diff --git a/docs/INDEX.md b/docs/INDEX.md index 80d85ca..813d2a5 100644 --- a/docs/INDEX.md +++ b/docs/INDEX.md @@ -3,6 +3,7 @@ > **For AI Assistants**: This document contains all critical patterns, conventions, and best practices. Read this first to understand the codebase structure and ensure consistency. **Quick Navigation:** +- [iii Platform & Development Workflow](#iii-platform--development-workflow) - Platform evolution and CLI tools - [Core Concepts](#core-concepts) - System architecture and patterns - [Design Principles](#design-principles) - Event Storm & Bidirectional References - [Step Development](#step-development-best-practices) - How to create new steps @@ -23,6 +24,244 @@ --- +## iii Platform & Development Workflow + +### Platform Evolution (v0.8 → v0.9+) + +**Status:** March 2026 - iii v0.9+ production-ready + +iii has evolved from an all-in-one development tool to a **modular, production-grade event engine** with clear separation between development and deployment workflows. + +#### Structural Changes Overview + +| Component | Before (v0.2-v0.7) | Now (v0.9+) | Impact | +|-----------|-------------------|-------------|--------| +| **Console/Dashboard** | Integrated in engine process (port 3111) | Separate process (`iii-cli console` or `dev`) | More flexibility, less resource overhead, better scaling | +| **CLI Tool** | Minimal or non-existent | `iii-cli` is the central dev tool | Terminal-based dev workflow, scriptable, faster iteration | +| **Project Structure** | Steps anywhere in project | **Recommended:** `src/` + `src/steps/` | Cleaner structure, reliable hot-reload | +| **Hot-Reload/Watcher** | Integrated in engine | Separate `shell::ExecModule` with `watch` paths | Only Python/TS files watched (configurable) | +| **Start & Services** | Single `iii` process | Engine (`iii` or `iii-cli start`) + Console separate | Better for production (engine) vs dev (console) | +| **Config Handling** | YAML + ENV | YAML + ENV + CLI flags prioritized | More control via CLI flags | +| **Observability** | Basic | Enhanced (OTel, Rollups, Alerts, Traces) | Production-ready telemetry | +| **Streams & State** | KV-Store (file/memory) | More adapters + file_based default | Better persistence handling | + +**Key Takeaway:** iii is now a **modular, production-ready engine** where development (CLI + separate console) is clearly separated from production deployment. + +--- + +### Development Workflow with iii-cli + +**`iii-cli` is your primary tool for local development, debugging, and testing.** + +#### Essential Commands + +| Command | Purpose | When to Use | Example | +|---------|---------|------------|---------| +| `iii-cli dev` | Start dev server with hot-reload + integrated console | Local development, immediate feedback on code changes | `iii-cli dev` | +| `iii-cli console` | Start dashboard only (separate port) | When you only need the console (no dev reload) | `iii-cli console --host 0.0.0.0 --port 3113` | +| `iii-cli start` | Start engine standalone (like `motia.service`) | Testing engine in isolation | `iii-cli start -c iii-config.yaml` | +| `iii-cli logs` | Live logs of all flows/workers/triggers | Debugging, error investigation | `iii-cli logs --level debug` | +| `iii-cli trace ` | Show detailed trace information (OTel) | Debug specific request/flow | `iii-cli trace abc123` | +| `iii-cli state ls` | List states (KV storage) | Verify state persistence | `iii-cli state ls` | +| `iii-cli state get` | Get specific state value | Inspect state content | `iii-cli state get key` | +| `iii-cli stream ls` | List all streams + groups | Inspect stream/websocket connections | `iii-cli stream ls` | +| `iii-cli flow list` | Show all registered flows/triggers | Overview of active steps & endpoints | `iii-cli flow list` | +| `iii-cli worker logs` | Worker logs (Python/TS execution) | Debug issues in step handlers | `iii-cli worker logs` | + +#### Typical Development Workflow + +```bash +# 1. Navigate to project +cd /opt/motia-iii/bitbylaw + +# 2. Start dev mode (hot-reload + console on port 3113) +iii-cli dev --host 0.0.0.0 --port 3113 --engine-port 3111 + +# Alternative: Separate engine + console +# Terminal 1: +iii-cli start -c iii-config.yaml + +# Terminal 2: +iii-cli console --host 0.0.0.0 --port 3113 \ + --engine-host 192.168.1.62 --engine-port 3111 + +# 3. Watch logs live (separate terminal) +iii-cli logs -f + +# 4. Debug specific trace +iii-cli trace + +# 5. Inspect state +iii-cli state ls +iii-cli state get document:sync:status + +# 6. Verify flows registered +iii-cli flow list +``` + +#### Development vs. Production + +**Development:** +- Use `iii-cli dev` for hot-reload +- Console accessible on localhost:3113 +- Logs visible in terminal +- Immediate feedback on code changes + +**Production:** +- `systemd` service runs `iii-cli start` +- Console runs separately (if needed) +- Logs via `journalctl -u motia.service -f` +- No hot-reload (restart service for changes) + +**Example Production Service:** +```ini +[Unit] +Description=Motia III Engine +After=network.target redis.service + +[Service] +Type=simple +User=motia +WorkingDirectory=/opt/motia-iii/bitbylaw +ExecStart=/usr/local/bin/iii-cli start -c /opt/motia-iii/bitbylaw/iii-config.yaml +Restart=always +RestartSec=10 +Environment="PATH=/usr/local/bin:/usr/bin" + +[Install] +WantedBy=multi-user.target +``` + +#### Project Structure Best Practices + +**Recommended Structure (v0.9+):** +``` +bitbylaw/ +├── iii-config.yaml # Main configuration +├── src/ # Source code root +│ └── steps/ # All steps here (hot-reload reliable) +│ ├── __init__.py +│ ├── vmh/ +│ │ ├── __init__.py +│ │ ├── document_sync_event_step.py +│ │ └── webhook/ +│ │ ├── __init__.py +│ │ └── document_create_api_step.py +│ └── advoware_proxy/ +│ └── ... +├── services/ # Shared business logic +│ ├── __init__.py +│ ├── xai_service.py +│ ├── espocrm.py +│ └── ... +└── tests/ # Test files +``` + +**Why `src/steps/` is recommended:** +- **Hot-reload works reliably** - Watcher detects changes correctly +- **Cleaner project** - Source code isolated from config/docs +- **IDE support** - Better navigation and refactoring +- **Deployment** - Easier to package + +**Note:** Old structure (steps in root) still works, but hot-reload may be less reliable. + +#### Hot-Reload Configuration + +**Hot-reload is configured via `shell::ExecModule` in `iii-config.yaml`:** + +```yaml +modules: + - type: shell::ExecModule + config: + watch: + - "src/**/*.py" # Watch Python files in src/ + - "services/**/*.py" # Watch service files + # Add more patterns as needed + ignore: + - "**/__pycache__/**" + - "**/*.pyc" + - "**/tests/**" +``` + +**Behavior:** +- Only files matching `watch` patterns trigger reload +- Changes in `ignore` patterns are ignored +- Reload is automatic in `iii-cli dev` mode +- Production mode (`iii-cli start`) does NOT watch files + +--- + +### Observability & Debugging + +#### OpenTelemetry Integration + +**iii v0.9+ has built-in OpenTelemetry support:** + +```python +# Traces are automatically created for: +# - HTTP requests +# - Queue processing +# - Cron execution +# - Service calls (if instrumented) + +# Access trace ID in handler: +async def handler(request: ApiRequest, ctx: FlowContext) -> ApiResponse: + trace_id = ctx.trace_id # Use for debugging + ctx.logger.info(f"Trace ID: {trace_id}") +``` + +**View traces:** +```bash +# Get trace details +iii-cli trace + +# Filter logs by trace +iii-cli logs --trace +``` + +#### Debugging Workflow + +**1. Live Logs:** +```bash +# All logs +iii-cli logs -f + +# Specific level +iii-cli logs --level error + +# With grep +iii-cli logs -f | grep "document_sync" +``` + +**2. State Inspection:** +```bash +# List all state keys +iii-cli state ls + +# Get specific state +iii-cli state get sync:document:last_run +``` + +**3. Flow Verification:** +```bash +# List all registered flows +iii-cli flow list + +# Verify endpoint exists +iii-cli flow list | grep "/vmh/webhook" +``` + +**4. Worker Issues:** +```bash +# Worker-specific logs +iii-cli worker logs + +# Check worker health +iii-cli worker status +``` + +--- + ## Core Concepts ### System Overview @@ -1271,24 +1510,41 @@ sudo systemctl enable motia.service sudo systemctl enable iii-console.service ``` -**Manual (Development):** +**Development (iii-cli):** ```bash -# Start iii Engine +# Option 1: Dev mode with integrated console and hot-reload cd /opt/motia-iii/bitbylaw -/opt/bin/iii -c iii-config.yaml +iii-cli dev --host 0.0.0.0 --port 3113 --engine-port 3111 -# Start iii Console (Web UI) -/opt/bin/iii-console --enable-flow --host 0.0.0.0 --port 3113 \ - --engine-host 192.168.67.233 --engine-port 3111 --ws-port 3114 +# Option 2: Separate engine and console +# Terminal 1: Start engine +iii-cli start -c iii-config.yaml + +# Terminal 2: Start console +iii-cli console --host 0.0.0.0 --port 3113 \ + --engine-host 192.168.1.62 --engine-port 3111 + +# Option 3: Manual (legacy) +/opt/bin/iii -c iii-config.yaml ``` ### Check Registered Steps +**Using iii-cli (recommended):** +```bash +# List all flows and triggers +iii-cli flow list + +# Filter for specific step +iii-cli flow list | grep document_sync +``` + +**Using curl (legacy):** ```bash curl http://localhost:3111/_console/functions | python3 -m json.tool ``` -### Test HTTP Endpoint +### Test HTTP Endpoints ```bash # Test document webhook @@ -1298,6 +1554,11 @@ curl -X POST "http://localhost:3111/vmh/webhook/document/create" \ # Test advoware proxy curl "http://localhost:3111/advoware/proxy?endpoint=employees" + +# Test beteiligte sync +curl -X POST "http://localhost:3111/vmh/webhook/beteiligte/create" \ + -H "Content-Type: application/json" \ + -d '{"entity_type": "CBeteiligte", "entity_id": "abc123", "action": "create"}' ``` ### Manually Trigger Cron @@ -1308,36 +1569,208 @@ curl -X POST "http://localhost:3111/_console/cron/trigger" \ -d '{"function_id": "steps::VMH Beteiligte Sync Cron::trigger::0"}' ``` -### View Logs +### View and Debug Logs +**Using iii-cli (recommended):** ```bash -# Live logs via journalctl -journalctl -u motia-iii -f +# Live logs (all) +iii-cli logs -f + +# Live logs with specific level +iii-cli logs -f --level error +iii-cli logs -f --level debug + +# Filter by component +iii-cli logs -f | grep "document_sync" + +# Worker-specific logs +iii-cli worker logs + +# Get specific trace +iii-cli trace + +# Filter logs by trace ID +iii-cli logs --trace +``` + +**Using journalctl (production):** +```bash +# Live logs +journalctl -u motia.service -f # Search for specific step -journalctl --since "today" | grep -i "document sync" +journalctl -u motia.service --since "today" | grep -i "document sync" +# Show errors only +journalctl -u motia.service -p err -f + +# Last 100 lines +journalctl -u motia.service -n 100 + +# Specific time range +journalctl -u motia.service --since "2026-03-19 10:00" --until "2026-03-19 11:00" +``` + +**Using log files (legacy):** +```bash # Check for errors tail -100 /opt/motia-iii/bitbylaw/iii_new.log | grep -i error + +# Follow log file +tail -f /opt/motia-iii/bitbylaw/iii_new.log +``` + +### Inspect State and Streams + +**State Management:** +```bash +# List all state keys +iii-cli state ls + +# Get specific state value +iii-cli state get document:sync:last_run + +# Set state (if needed for testing) +iii-cli state set test:key "test value" + +# Delete state +iii-cli state delete test:key +``` + +**Stream Management:** +```bash +# List all active streams +iii-cli stream ls + +# Inspect specific stream +iii-cli stream info + +# List consumer groups +iii-cli stream groups +``` + +### Debugging Workflow + +**1. Identify the Issue:** +```bash +# Check if step is registered +iii-cli flow list | grep my_step + +# View recent errors +iii-cli logs --level error -n 50 + +# Check service status +sudo systemctl status motia.service +``` + +**2. Get Detailed Information:** +```bash +# Live tail logs for specific step +iii-cli logs -f | grep "document_sync" + +# Check worker processes +iii-cli worker logs + +# Inspect state +iii-cli state ls +``` + +**3. Test Specific Functionality:** +```bash +# Trigger webhook manually +curl -X POST http://localhost:3111/vmh/webhook/... + +# Check response and logs +iii-cli logs -f | grep "webhook" + +# Verify state changed +iii-cli state get entity:sync:status +``` + +**4. Trace Specific Request:** +```bash +# Make request, note trace ID from logs +curl -X POST http://localhost:3111/vmh/webhook/document/create ... + +# Get full trace +iii-cli trace + +# View all logs for this trace +iii-cli logs --trace +``` + +### Performance Monitoring + +**Check System Resources:** +```bash +# CPU and memory usage +htop + +# Process-specific +ps aux | grep iii + +# Redis memory +redis-cli info memory + +# File descriptors +lsof -p $(pgrep -f "iii-cli start") +``` + +**Check Processing Metrics:** +```bash +# Queue lengths (if using Redis streams) +redis-cli XINFO STREAM vmh:document:sync + +# Pending messages +redis-cli XPENDING vmh:document:sync group1 + +# Lock status +redis-cli KEYS "lock:*" ``` ### Common Issues **Step not showing up:** 1. Check file naming: Must end with `_step.py` -2. Check for import errors: `grep -i "importerror\|traceback" iii.log` -3. Verify `config` dict is present -4. Restart iii engine +2. Check for syntax errors: `iii-cli logs --level error` +3. Check for import errors: `iii-cli logs | grep -i "importerror\|traceback"` +4. Verify `config` dict is present +5. Restart: `sudo systemctl restart motia.service` or restart `iii-cli dev` +6. Verify hot-reload working: Check terminal output in `iii-cli dev` **Redis connection failed:** - Check `REDIS_HOST` and `REDIS_PORT` environment variables - Verify Redis is running: `redis-cli ping` +- Check Redis logs: `journalctl -u redis -f` - Service will work without Redis but with warnings +**Hot-reload not working:** +- Verify using `iii-cli dev` (not `iii-cli start`) +- Check `watch` patterns in `iii-config.yaml` +- Ensure files are in watched directories (`src/**/*.py`) +- Look for watcher errors: `iii-cli logs | grep -i "watch"` + +**Handler not triggered:** +- Verify endpoint registered: `iii-cli flow list` +- Check HTTP method matches (GET, POST, etc.) +- Test with curl to isolate issue +- Check trigger configuration in step's `config` dict + **AttributeError '_log' not found:** - Ensure service inherits from `BaseSyncUtils` OR - Implement `_log()` method manually +**Trace not found:** +- Ensure OpenTelemetry enabled in config +- Check if trace ID is valid format +- Use `iii-cli logs` with filters instead + +**Console not accessible:** +- Check if console service running: `systemctl status iii-console.service` +- Verify port not blocked by firewall: `sudo ufw status` +- Check console logs: `journalctl -u iii-console.service -f` +- Try accessing via `localhost:3113` instead of public IP + --- ## Key Patterns Summary