bsiggel/motia-iii
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

feat: Enhance developer guide with updated platform evolution and workflow details

Browse Source
This commit is contained in:
bsiggel
2026-03-19 20:56:32 +00:00
parent 46085bd8dd
commit 8ce843415e
1 changed files with 447 additions and 14 deletions
Download Patch File Download Diff File Expand all files Collapse all files

461
docs/INDEX.md
View File

@@ -3,6 +3,7 @@
> **For AI Assistants**: This document contains all critical patterns, conventions, and best practices. Read this first to understand the codebase structure and ensure consistency.
**Quick Navigation:**
- [iii Platform & Development Workflow](#iii-platform--development-workflow) - Platform evolution and CLI tools
- [Core Concepts](#core-concepts) - System architecture and patterns
- [Design Principles](#design-principles) - Event Storm & Bidirectional References
- [Step Development](#step-development-best-practices) - How to create new steps
@@ -23,6 +24,244 @@
---
## iii Platform & Development Workflow
### Platform Evolution (v0.8 → v0.9+)
**Status:** March 2026 - iii v0.9+ production-ready
iii has evolved from an all-in-one development tool to a **modular, production-grade event engine** with clear separation between development and deployment workflows.
#### Structural Changes Overview
| Component | Before (v0.2-v0.7) | Now (v0.9+) | Impact |
|-----------|-------------------|-------------|--------|
| **Console/Dashboard** | Integrated in engine process (port 3111) | Separate process (`iii-cli console` or `dev`) | More flexibility, less resource overhead, better scaling |
| **CLI Tool** | Minimal or non-existent | `iii-cli` is the central dev tool | Terminal-based dev workflow, scriptable, faster iteration |
| **Project Structure** | Steps anywhere in project | **Recommended:** `src/` + `src/steps/` | Cleaner structure, reliable hot-reload |
| **Hot-Reload/Watcher** | Integrated in engine | Separate `shell::ExecModule` with `watch` paths | Only Python/TS files watched (configurable) |
| **Start & Services** | Single `iii` process | Engine (`iii` or `iii-cli start`) + Console separate | Better for production (engine) vs dev (console) |
| **Config Handling** | YAML + ENV | YAML + ENV + CLI flags prioritized | More control via CLI flags |
| **Observability** | Basic | Enhanced (OTel, Rollups, Alerts, Traces) | Production-ready telemetry |
| **Streams & State** | KV-Store (file/memory) | More adapters + file_based default | Better persistence handling |
**Key Takeaway:** iii is now a **modular, production-ready engine** where development (CLI + separate console) is clearly separated from production deployment.
---
### Development Workflow with iii-cli
**`iii-cli` is your primary tool for local development, debugging, and testing.**
#### Essential Commands
| Command | Purpose | When to Use | Example |
|---------|---------|------------|---------|
| `iii-cli dev` | Start dev server with hot-reload + integrated console | Local development, immediate feedback on code changes | `iii-cli dev` |
| `iii-cli console` | Start dashboard only (separate port) | When you only need the console (no dev reload) | `iii-cli console --host 0.0.0.0 --port 3113` |
| `iii-cli start` | Start engine standalone (like `motia.service`) | Testing engine in isolation | `iii-cli start -c iii-config.yaml` |
| `iii-cli logs` | Live logs of all flows/workers/triggers | Debugging, error investigation | `iii-cli logs --level debug` |
| `iii-cli trace <id>` | Show detailed trace information (OTel) | Debug specific request/flow | `iii-cli trace abc123` |
| `iii-cli state ls` | List states (KV storage) | Verify state persistence | `iii-cli state ls` |
| `iii-cli state get` | Get specific state value | Inspect state content | `iii-cli state get key` |
| `iii-cli stream ls` | List all streams + groups | Inspect stream/websocket connections | `iii-cli stream ls` |
| `iii-cli flow list` | Show all registered flows/triggers | Overview of active steps & endpoints | `iii-cli flow list` |
| `iii-cli worker logs` | Worker logs (Python/TS execution) | Debug issues in step handlers | `iii-cli worker logs` |
#### Typical Development Workflow
```bash
# 1. Navigate to project
cd /opt/motia-iii/bitbylaw
# 2. Start dev mode (hot-reload + console on port 3113)
iii-cli dev --host 0.0.0.0 --port 3113 --engine-port 3111
# Alternative: Separate engine + console
# Terminal 1:
iii-cli start -c iii-config.yaml
# Terminal 2:
iii-cli console --host 0.0.0.0 --port 3113 \
--engine-host 192.168.1.62 --engine-port 3111
# 3. Watch logs live (separate terminal)
iii-cli logs -f
# 4. Debug specific trace
iii-cli trace <trace-id-from-logs>
# 5. Inspect state
iii-cli state ls
iii-cli state get document:sync:status
# 6. Verify flows registered
iii-cli flow list
```
#### Development vs. Production
**Development:**
- Use `iii-cli dev` for hot-reload
- Console accessible on localhost:3113
- Logs visible in terminal
- Immediate feedback on code changes
**Production:**
- `systemd` service runs `iii-cli start`
- Console runs separately (if needed)
- Logs via `journalctl -u motia.service -f`
- No hot-reload (restart service for changes)
**Example Production Service:**
```ini
[Unit]
Description=Motia III Engine
After=network.target redis.service
[Service]
Type=simple
User=motia
WorkingDirectory=/opt/motia-iii/bitbylaw
ExecStart=/usr/local/bin/iii-cli start -c /opt/motia-iii/bitbylaw/iii-config.yaml
Restart=always
RestartSec=10
Environment="PATH=/usr/local/bin:/usr/bin"
[Install]
WantedBy=multi-user.target
```
#### Project Structure Best Practices
**Recommended Structure (v0.9+):**
```
bitbylaw/
├── iii-config.yaml # Main configuration
├── src/ # Source code root
│ └── steps/ # All steps here (hot-reload reliable)
│ ├── __init__.py
│ ├── vmh/
│ │ ├── __init__.py
│ │ ├── document_sync_event_step.py
│ │ └── webhook/
│ │ ├── __init__.py
│ │ └── document_create_api_step.py
│ └── advoware_proxy/
│ └── ...
├── services/ # Shared business logic
│ ├── __init__.py
│ ├── xai_service.py
│ ├── espocrm.py
│ └── ...
└── tests/ # Test files
```
**Why `src/steps/` is recommended:**
- **Hot-reload works reliably** - Watcher detects changes correctly
- **Cleaner project** - Source code isolated from config/docs
- **IDE support** - Better navigation and refactoring
- **Deployment** - Easier to package
**Note:** Old structure (steps in root) still works, but hot-reload may be less reliable.
#### Hot-Reload Configuration
**Hot-reload is configured via `shell::ExecModule` in `iii-config.yaml`:**
```yaml
modules:
- type: shell::ExecModule
config:
watch:
- "src/**/*.py" # Watch Python files in src/
- "services/**/*.py" # Watch service files
# Add more patterns as needed
ignore:
- "**/__pycache__/**"
- "**/*.pyc"
- "**/tests/**"
```
**Behavior:**
- Only files matching `watch` patterns trigger reload
- Changes in `ignore` patterns are ignored
- Reload is automatic in `iii-cli dev` mode
- Production mode (`iii-cli start`) does NOT watch files
---
### Observability & Debugging
#### OpenTelemetry Integration
**iii v0.9+ has built-in OpenTelemetry support:**
```python
# Traces are automatically created for:
# - HTTP requests
# - Queue processing
# - Cron execution
# - Service calls (if instrumented)
# Access trace ID in handler:
async def handler(request: ApiRequest, ctx: FlowContext) -> ApiResponse:
trace_id = ctx.trace_id # Use for debugging
ctx.logger.info(f"Trace ID: {trace_id}")
```
**View traces:**
```bash
# Get trace details
iii-cli trace <trace-id>
# Filter logs by trace
iii-cli logs --trace <trace-id>
```
#### Debugging Workflow
**1. Live Logs:**
```bash
# All logs
iii-cli logs -f
# Specific level
iii-cli logs --level error
# With grep
iii-cli logs -f | grep "document_sync"
```
**2. State Inspection:**
```bash
# List all state keys
iii-cli state ls
# Get specific state
iii-cli state get sync:document:last_run
```
**3. Flow Verification:**
```bash
# List all registered flows
iii-cli flow list
# Verify endpoint exists
iii-cli flow list | grep "/vmh/webhook"
```
**4. Worker Issues:**
```bash
# Worker-specific logs
iii-cli worker logs
# Check worker health
iii-cli worker status
```
---
## Core Concepts
### System Overview
@@ -1271,24 +1510,41 @@ sudo systemctl enable motia.service
sudo systemctl enable iii-console.service
```
**Manual (Development):**
**Development (iii-cli):**
```bash
# Start iii Engine
# Option 1: Dev mode with integrated console and hot-reload
cd /opt/motia-iii/bitbylaw
/opt/bin/iii -c iii-config.yaml
iii-cli dev --host 0.0.0.0 --port 3113 --engine-port 3111
# Start iii Console (Web UI)
/opt/bin/iii-console --enable-flow --host 0.0.0.0 --port 3113 \
--engine-host 192.168.67.233 --engine-port 3111 --ws-port 3114
# Option 2: Separate engine and console
# Terminal 1: Start engine
iii-cli start -c iii-config.yaml
# Terminal 2: Start console
iii-cli console --host 0.0.0.0 --port 3113 \
--engine-host 192.168.1.62 --engine-port 3111
# Option 3: Manual (legacy)
/opt/bin/iii -c iii-config.yaml
```
### Check Registered Steps
**Using iii-cli (recommended):**
```bash
# List all flows and triggers
iii-cli flow list
# Filter for specific step
iii-cli flow list | grep document_sync
```
**Using curl (legacy):**
```bash
curl http://localhost:3111/_console/functions | python3 -m json.tool
```
### Test HTTP Endpoint
### Test HTTP Endpoints
```bash
# Test document webhook
@@ -1298,6 +1554,11 @@ curl -X POST "http://localhost:3111/vmh/webhook/document/create" \
# Test advoware proxy
curl "http://localhost:3111/advoware/proxy?endpoint=employees"
# Test beteiligte sync
curl -X POST "http://localhost:3111/vmh/webhook/beteiligte/create" \
-H "Content-Type: application/json" \
-d '{"entity_type": "CBeteiligte", "entity_id": "abc123", "action": "create"}'
```
### Manually Trigger Cron
@@ -1308,36 +1569,208 @@ curl -X POST "http://localhost:3111/_console/cron/trigger" \
-d '{"function_id": "steps::VMH Beteiligte Sync Cron::trigger::0"}'
```
### View Logs
### View and Debug Logs
**Using iii-cli (recommended):**
```bash
# Live logs via journalctl
journalctl -u motia-iii -f
# Live logs (all)
iii-cli logs -f
# Live logs with specific level
iii-cli logs -f --level error
iii-cli logs -f --level debug
# Filter by component
iii-cli logs -f | grep "document_sync"
# Worker-specific logs
iii-cli worker logs
# Get specific trace
iii-cli trace <trace-id>
# Filter logs by trace ID
iii-cli logs --trace <trace-id>
```
**Using journalctl (production):**
```bash
# Live logs
journalctl -u motia.service -f
# Search for specific step
journalctl --since "today" | grep -i "document sync"
journalctl -u motia.service --since "today" | grep -i "document sync"
# Show errors only
journalctl -u motia.service -p err -f
# Last 100 lines
journalctl -u motia.service -n 100
# Specific time range
journalctl -u motia.service --since "2026-03-19 10:00" --until "2026-03-19 11:00"
```
**Using log files (legacy):**
```bash
# Check for errors
tail -100 /opt/motia-iii/bitbylaw/iii_new.log | grep -i error
# Follow log file
tail -f /opt/motia-iii/bitbylaw/iii_new.log
```
### Inspect State and Streams
**State Management:**
```bash
# List all state keys
iii-cli state ls
# Get specific state value
iii-cli state get document:sync:last_run
# Set state (if needed for testing)
iii-cli state set test:key "test value"
# Delete state
iii-cli state delete test:key
```
**Stream Management:**
```bash
# List all active streams
iii-cli stream ls
# Inspect specific stream
iii-cli stream info <stream-id>
# List consumer groups
iii-cli stream groups <stream-name>
```
### Debugging Workflow
**1. Identify the Issue:**
```bash
# Check if step is registered
iii-cli flow list | grep my_step
# View recent errors
iii-cli logs --level error -n 50
# Check service status
sudo systemctl status motia.service
```
**2. Get Detailed Information:**
```bash
# Live tail logs for specific step
iii-cli logs -f | grep "document_sync"
# Check worker processes
iii-cli worker logs
# Inspect state
iii-cli state ls
```
**3. Test Specific Functionality:**
```bash
# Trigger webhook manually
curl -X POST http://localhost:3111/vmh/webhook/...
# Check response and logs
iii-cli logs -f | grep "webhook"
# Verify state changed
iii-cli state get entity:sync:status
```
**4. Trace Specific Request:**
```bash
# Make request, note trace ID from logs
curl -X POST http://localhost:3111/vmh/webhook/document/create ...
# Get full trace
iii-cli trace <trace-id>
# View all logs for this trace
iii-cli logs --trace <trace-id>
```
### Performance Monitoring
**Check System Resources:**
```bash
# CPU and memory usage
htop
# Process-specific
ps aux | grep iii
# Redis memory
redis-cli info memory
# File descriptors
lsof -p $(pgrep -f "iii-cli start")
```
**Check Processing Metrics:**
```bash
# Queue lengths (if using Redis streams)
redis-cli XINFO STREAM vmh:document:sync
# Pending messages
redis-cli XPENDING vmh:document:sync group1
# Lock status
redis-cli KEYS "lock:*"
```
### Common Issues
**Step not showing up:**
1. Check file naming: Must end with `_step.py`
2. Check for import errors: `grep -i "importerror\|traceback" iii.log`
3. Verify `config` dict is present
4. Restart iii engine
2. Check for syntax errors: `iii-cli logs --level error`
3. Check for import errors: `iii-cli logs | grep -i "importerror\|traceback"`
4. Verify `config` dict is present
5. Restart: `sudo systemctl restart motia.service` or restart `iii-cli dev`
6. Verify hot-reload working: Check terminal output in `iii-cli dev`
**Redis connection failed:**
- Check `REDIS_HOST` and `REDIS_PORT` environment variables
- Verify Redis is running: `redis-cli ping`
- Check Redis logs: `journalctl -u redis -f`
- Service will work without Redis but with warnings
**Hot-reload not working:**
- Verify using `iii-cli dev` (not `iii-cli start`)
- Check `watch` patterns in `iii-config.yaml`
- Ensure files are in watched directories (`src/**/*.py`)
- Look for watcher errors: `iii-cli logs | grep -i "watch"`
**Handler not triggered:**
- Verify endpoint registered: `iii-cli flow list`
- Check HTTP method matches (GET, POST, etc.)
- Test with curl to isolate issue
- Check trigger configuration in step's `config` dict
**AttributeError '_log' not found:**
- Ensure service inherits from `BaseSyncUtils` OR
- Implement `_log()` method manually
**Trace not found:**
- Ensure OpenTelemetry enabled in config
- Check if trace ID is valid format
- Use `iii-cli logs` with filters instead
**Console not accessible:**
- Check if console service running: `systemctl status iii-console.service`
- Verify port not blocked by firewall: `sudo ufw status`
- Check console logs: `journalctl -u iii-console.service -f`
- Try accessing via `localhost:3113` instead of public IP
---
## Key Patterns Summary