diff --git a/README.md b/README.md index ac1b660..92aa023 100644 --- a/README.md +++ b/README.md @@ -1,34 +1,207 @@ -# Motia III - BitByLaw Backend +# bitbylaw - Motia III Integration Platform + +✅ **Migration Complete: 21/21 Steps (100%)** + +Event-driven Integration zwischen Advoware, EspoCRM und Google Calendar mit **Motia III v1.0-RC** (Pure Python). + +## Quick Start + +```bash +cd /opt/motia-iii/bitbylaw + +# Start iii Engine +/opt/bin/iii -c iii-config.yaml + +# Start iii Console (Web UI) - separate terminal +/opt/bin/iii-console --enable-flow --host 0.0.0.0 --port 3113 \ + --engine-host localhost --engine-port 3111 --ws-port 3114 +``` + +Open browser: `http://localhost:3113/` + +## Migration Status + +This project has been **fully migrated** from old Motia v0.17 (Node.js + Python) to **Motia III v1.0-RC** (Pure Python). + +See: +- [MIGRATION_STATUS.md](MIGRATION_STATUS.md) - Progress overview +- [MIGRATION_COMPLETE_ANALYSIS.md](MIGRATION_COMPLETE_ANALYSIS.md) - Complete analysis +- [docs/INDEX.md](docs/INDEX.md) - Complete documentation index + +## Komponenten + +1. **Advoware API Proxy** (4 Steps) - REST-API-Proxy mit HMAC-512 Auth + - GET, POST, PUT, DELETE proxies + - [Details](steps/advoware_proxy/README.md) + +2. **Calendar Sync** (4 Steps) - Bidirektionale Synchronisation Advoware ↔ Google + - Cron-triggered (every 15 min) + - Manual API trigger + - Per-employee sync + - [Details](steps/advoware_cal_sync/README.md) + +3. **VMH Integration** (9 Steps) - EspoCRM Webhook-Receiver & Sync + - Beteiligte sync (bidirectional) + - Bankverbindungen sync + - Webhook handlers (create/update/delete) + - [Details](steps/vmh/README.md) + +**Total: 17 Steps registered** + +## Architektur + +``` +┌─────────────┐ ┌──────────┐ ┌────────────┐ +│ EspoCRM │────▶│ Webhooks │────▶│ Redis │ +│ (VMH) │ │ (HTTP) │ │ Locking │ +└─────────────┘ └──────────┘ └────────────┘ + │ +┌─────────────┐ ┌──────────┐ │ +│ Clients │────▶│ Proxy │────▶ │ +│ │ │ API API │ │ +└─────────────┘ └──────────┘ ▼ + ┌────────────┐ + Cron (15min) │ Motia │ + │ │ Steps │ + └──────────────────────────▶│ (Python) │ + └────────────┘ + │ + ▼ + ┌────────────┐ + │ Advoware │ + │ Google │ + │ Calendar │ + └────────────┘ +``` + +## API Endpoints + +**Advoware Proxy:** +- `GET/POST/PUT/DELETE /advoware/proxy?endpoint=...` + +**Calendar Sync:** +- `POST /advoware/calendar/sync` - Manual trigger + ```bash + # Sync single employee + curl -X POST "http://localhost:3111/advoware/calendar/sync" \ + -H "Content-Type: application/json" \ + -d '{"kuerzel": "PB"}' + + # Sync all employees + curl -X POST "http://localhost:3111/advoware/calendar/sync" \ + -H "Content-Type: application/json" \ + -d '{"kuerzel": "ALL"}' + ``` + +**VMH Webhooks:** +- `POST /vmh/webhook/beteiligte/create` +- `POST /vmh/webhook/beteiligte/update` +- `POST /vmh/webhook/beteiligte/delete` +- `POST /vmh/webhook/bankverbindungen/create` +- `POST /vmh/webhook/bankverbindungen/update` +- `POST /vmh/webhook/bankverbindungen/delete` + + +## Configuration + +Environment variables loaded from systemd service or `.env` file: + +```bash +# Advoware API +ADVOWARE_API_BASE_URL=https://www2.advo-net.net:90/ +ADVOWARE_PRODUCT_ID=64 +ADVOWARE_APP_ID=your_app_id +ADVOWARE_API_KEY=your_base64_hmac_key +ADVOWARE_KANZLEI=your_kanzlei +ADVOWARE_DATABASE=your_database +ADVOWARE_USER=api_user +ADVOWARE_ROLE=2 +ADVOWARE_PASSWORD=your_password +ADVOWARE_TOKEN_LIFETIME_MINUTES=55 +ADVOWARE_API_TIMEOUT_SECONDS=30 + +# Redis +REDIS_HOST=localhost +REDIS_PORT=6379 +REDIS_DB_CALENDAR_SYNC=1 +REDIS_TIMEOUT_SECONDS=5 + +# Google Calendar +GOOGLE_CALENDAR_SERVICE_ACCOUNT_PATH=/path/to/service-account.json + +# PostgreSQL +POSTGRES_HOST=localhost +POSTGRES_PORT=5432 +POSTGRES_USER=bitbylaw +POSTGRES_PASSWORD=your_password +POSTGRES_DATABASE=bitbylaw +``` + +See [docs/INDEX.md](docs/INDEX.md) for complete configuration guide. -Motia Backend-Anwendung powered by the iii engine. ## Project Structure ``` bitbylaw/ -├── steps/ # Motia Steps (Business Logic) -├── iii-config.yaml # iii Engine Configuration -├── pyproject.toml # Python Dependencies -└── .venv/ # Python Virtual Environment -``` +├── docs/ # Documentation -## Services +## Services & Ports -- **Motia API**: Port 3111 (https://api-motia.bitbylaw.com) -- **iii Console**: Port 3113 (https://motia.bitbylaw.com) +- **iii Engine (API)**: Port 3111 (https://api-motia.bitbylaw.com) +- **iii Console (Web UI)**: Port 3113 (https://motia.bitbylaw.com) - **Streams/WebSocket**: Port 3114 +- **Redis**: Port 6379 (localhost) +- **PostgreSQL**: Port 5432 (localhost)eps) +│ ├── advoware_cal_sync/ # Calendar Sync (4 steps) +│ └── vmh/ # VMH Integration (9 steps) +├── services/ # Shared Services +│ ├── advoware_service.py # Advoware API Client +│ ├── espocrm.py # EspoCRM API Client +│ └── ... # Other services +├── iii-config.yaml # iii Engine Configuration +├── pyproject.toml # Python Dependencies (uv) +├── MIGRATION_STATUS.md # Migration progress +├── MIGRATION_COMPLETE_ANALYSIS.md # Migration analysis +└── .venv/ # Python Virtual Environment +``` ## Systemd Services -- `motia.service` - Backend Engine -- `iii-console.service` - Observability Dashboard +## Systemd Services + +- `motia.service` - iii Engine (Backend) +- `iii-console.service` - iii Console (Observability Dashboard) + +### Service Management + +```bash +# Status +systemctl status motia.service +systemctl status iii-console.service + +# Restart ## Development ```bash # Install dependencies +cd /opt/motia-iii/bitbylaw uv sync +# Start iii Engine (development) +iii -c iii-config.yaml + +# Start iii Console (separate terminal) +iii-console --enable-flow --host 0.0.0.0 --port 3113 \ + --engine-host localhost --engine-port 3111 --ws-port 3114 + +# Test step import +uv run python -c "from steps.advoware_proxy import advoware_api_proxy_get_step" + +# Check registered steps +curl http://localhost:3111/_console/functions + # Start locally iii -c iii-config.yaml ``` @@ -37,10 +210,70 @@ iii -c iii-config.yaml Services run automatically via systemd on boot. -```bash -# Restart services -systemctl restart motia.service iii-console.service +## Technology Stack + +- **Framework**: Motia III v1.0-RC (Pure Python, iii Engine) +- **Language**: Python 3.13 +- **Package Manager**: uv +- **Data Store**: Redis (Caching, Locking), PostgreSQL (Sync State) +- **External APIs**: + - Advoware REST API (HMAC-512 auth) + - Google Calendar API (Service Account) + - EspoCRM API (X-Api-Key auth) + +## Documentation + +### Getting Started +- [Documentation Index](docs/INDEX.md) - Complete index +- [Migration Status](MIGRATION_STATUS.md) - 100% complete +- [Migration Analysis](MIGRATION_COMPLETE_ANALYSIS.md) - Complete analysis + +### Technical Details +- [Architecture](docs/ARCHITECTURE.md) - System design (Motia III) +- [Advoware Proxy](steps/advoware_proxy/README.md) - API Proxy details +- [Calendar Sync](steps/advoware_cal_sync/README.md) - Sync logic +- [VMH Integration](steps/vmh/README.md) - Webhook handlers + +### Migration +- [Migration Guide](../MIGRATION_GUIDE.md) - Old Motia → Motia III patterns + +## Testing + +```bash +# Test HTTP endpoints +curl http://localhost:3111/advoware/proxy?endpoint=employees + +# Trigger calendar sync manually +curl -X POST "http://localhost:3111/advoware/calendar/sync" \ + -H "Content-Type: application/json" \ + -d '{"kuerzel": "ALL"}' + +# Check registered functions +curl http://localhost:3111/_console/functions | grep "Calendar" + +# View logs in Console +open http://localhost:3113/ +``` + +## Production + +Services run automatically via systemd on boot. + +**Deployed on**: motia.bitbylaw.com +**Deployment Date**: März 2026 + +```bash +# Restart production services +sudo systemctl restart motia.service iii-console.service + +# View production logs +journalctl -u motia.service -f +journalctl -u iii-console.service -f + +# Check service status +systemctl status motia.service iii-console.service +``` # View logs journalctl -u motia.service -f journalctl -u iii-console.service -f diff --git a/docs/INDEX.md b/docs/INDEX.md new file mode 100644 index 0000000..d0f70ac --- /dev/null +++ b/docs/INDEX.md @@ -0,0 +1,246 @@ +# Documentation Index - Motia III + +## Getting Started + +**New to the project?** Start here: + +1. [README.md](../README.md) - Project Overview & Quick Start +2. [MIGRATION_STATUS.md](../MIGRATION_STATUS.md) - Migration Progress (100% Complete!) +3. [MIGRATION_COMPLETE_ANALYSIS.md](../MIGRATION_COMPLETE_ANALYSIS.md) - Complete Migration Analysis + +## Migration to Motia III + +**Status: ✅ 100% Complete (21/21 Steps migrated)** + +- **[MIGRATION_GUIDE.md](../MIGRATION_GUIDE.md)** - Complete migration patterns + - Old Motia v0.17 → Motia III v1.0-RC + - TypeScript + Python Hybrid → Pure Python + - Configuration changes, trigger patterns, API differences +- **[MIGRATION_STATUS.md](../MIGRATION_STATUS.md)** - Current migration status +- **[MIGRATION_COMPLETE_ANALYSIS.md](../MIGRATION_COMPLETE_ANALYSIS.md)** - Complete analysis + +## Core Documentation + +### For Developers +- **[ARCHITECTURE.md](ARCHITECTURE.md)** - System design and architecture + - Components, Data Flow, Event-Driven Design + - Updated for Motia III patterns + +### For Operations +- **[DEPLOYMENT.md](DEPLOYMENT.md)** - Production deployment (if available) + - Installation, systemd, nginx, Monitoring +- **[CONFIGURATION.md](CONFIGURATION.md)** - Environment configuration (if available) + - All environment variables, secrets management + +## Component Documentation + +### Steps (Business Logic) + +**Advoware Proxy** ([Module README](../steps/advoware_proxy/README.md)): +Universal HTTP proxy for Advoware API with automatic authentication. +- GET, POST, PUT, DELETE proxies +- HMAC-512 authentication +- Redis token caching + +**Calendar Sync** ([Module README](../steps/advoware_cal_sync/README.md)): +Bidirectional sync between Advoware appointments and Google Calendar. +- `calendar_sync_cron_step.py` - Auto trigger (every 15 min) +- `calendar_sync_api_step.py` - Manual trigger endpoint +- `calendar_sync_all_step.py` - Employee cascade handler +- `calendar_sync_event_step.py` - Per-employee sync logic (1053 lines) + +**VMH Integration** ([Module README](../steps/vmh/README.md)): +Webhooks and bidirectional sync between EspoCRM and Advoware. +- **Beteiligte Sync** (Bidirectional EspoCRM ↔ Advoware) + - Cron job (every 15 min) + - Event handlers for create/update/delete +- **Webhooks** (6 endpoints) + - Beteiligte: create, update, delete + - Bankverbindungen: create, update, delete + +### Services + +Service modules providing API clients and business logic: +- `advoware_service.py` - Advoware API client (HMAC-512 auth, token caching) +- `espocrm.py` - EspoCRM API client +- `advoware.py` - Legacy Advoware service (deprecated) +- Sync utilities and mappers + +## Motia III Patterns + +### Step Configuration + +**Old (Motia v0.17):** +```python +config = { + 'type': 'api', # or 'event', 'cron' + 'name': 'MyStep', + 'method': 'POST', + 'path': '/my-step', + 'emits': ['my-event'], + 'subscribes': ['other-event'] +} +``` + +**New (Motia III):** +```python +from motia import http, queue, cron + +config = { + 'name': 'MyStep', + 'flows': ['my-flow'], + 'triggers': [ + http('POST', '/my-step') # or queue('topic') or cron('0 */15 * * * *') + ], + 'enqueues': ['my-event'] +} +``` + +### Handler Signatures + +**HTTP Trigger:** +```python +from motia import ApiRequest, ApiResponse, FlowContext + +async def handler(request: ApiRequest, ctx: FlowContext) -> ApiResponse: + # Access: request.body, request.query_params, request.path_params + # Enqueue: await ctx.enqueue(topic='...', data={...}) + # Log: ctx.logger.info('...') + return ApiResponse(status=200, body={...}) +``` + +**Queue Trigger:** +```python +async def handler(input_data: dict, ctx: FlowContext) -> None: + # Process queue data + await ctx.enqueue(topic='next-step', data={...}) +``` + +**Cron Trigger:** +```python +async def handler(input_data: None, ctx: FlowContext) -> None: + # Cron jobs receive no input + ctx.logger.info('Cron triggered') +``` + +### Key Differences from Old Motia + +| Old Motia v0.17 | Motia III v1.0-RC | +|-----------------|-------------------| +| `type: 'api'` | `triggers: [http()]` | +| `type: 'event'` | `triggers: [queue()]` | +| `type: 'cron'` | `triggers: [cron()]` | +| `context.emit()` | `ctx.enqueue()` | +| `emits: [...]` | `enqueues: [...]` | +| `subscribes: [...]` | Moved to trigger: `queue('topic')` | +| 5-field cron | 6-field cron (prepend seconds) | +| `context.logger` | `ctx.logger` | +| Motia Workbench | iii Console | +| Node.js + Python | Pure Python | + +## Documentation Structure + +``` +docs/ +├── INDEX.md # This file +├── ARCHITECTURE.md # System design (Motia III) +└── advoware/ + └── (optional API specs) + +steps/{module}/ +├── README.md # Module overview +└── {step_name}_step.py # Step implementation + +services/ +└── {service_name}.py # Service implementations + +MIGRATION_GUIDE.md # Complete migration guide +MIGRATION_STATUS.md # Migration progress +MIGRATION_COMPLETE_ANALYSIS.md # Final analysis +``` + +## Quick Reference + +### Common Tasks + +| Task | Documentation | +|------|---------------| +| Understand migration | [MIGRATION_GUIDE.md](../MIGRATION_GUIDE.md) | +| Check migration status | [MIGRATION_STATUS.md](../MIGRATION_STATUS.md) | +| Understand architecture | [ARCHITECTURE.md](ARCHITECTURE.md) | +| Calendar sync overview | [steps/advoware_cal_sync/README.md](../steps/advoware_cal_sync/README.md) | +| Proxy API usage | [steps/advoware_proxy/README.md](../steps/advoware_proxy/README.md) | +| VMH sync details | [steps/vmh/README.md](../steps/vmh/README.md) | + +### Code Locations + +| Component | Location | Documentation | +|-----------|----------|---------------| +| API Proxy Steps | `steps/advoware_proxy/` | [README](../steps/advoware_proxy/README.md) | +| Calendar Sync Steps | `steps/advoware_cal_sync/` | [README](../steps/advoware_cal_sync/README.md) | +| VMH Steps | `steps/vmh/` | [README](../steps/vmh/README.md) | +| Advoware Service | `services/advoware_service.py` | (in-code docs) | +| Configuration | `iii-config.yaml` | System config | +| Environment | `.env` or systemd | Environment variables | + +## Running the System + +### Start iii Engine +```bash +cd /opt/motia-iii/bitbylaw +/opt/bin/iii -c iii-config.yaml +``` + +### Start iii Console (Web UI) +```bash +/opt/bin/iii-console --enable-flow --host 0.0.0.0 --port 3113 \ + --engine-host 192.168.67.233 --engine-port 3111 --ws-port 3114 +``` + +### Access Web Console +Open browser: `http://localhost:3113/` + +### Check Registered Steps +```bash +curl http://localhost:3111/_console/functions | python3 -m json.tool +``` + +## Testing + +### Test HTTP Step +```bash +# Calendar sync API +curl -X POST "http://localhost:3111/advoware/calendar/sync" \ + -H "Content-Type: application/json" \ + -d '{"kuerzel": "PB"}' + +# Advoware proxy +curl "http://localhost:3111/advoware/proxy?endpoint=employees" +``` + +### Trigger Cron Manually +```bash +curl -X POST "http://localhost:3111/_console/cron/trigger" \ + -H "Content-Type: application/json" \ + -d '{"function_id": "steps::Calendar Sync Cron Job::trigger::0"}' +``` + +### Check Logs +View logs in iii Console or via API: +```bash +curl "http://localhost:3111/_console/logs" +``` + +## External Resources + +- [Motia III Documentation](https://iii.dev) +- [Python SDK](https://pypi.org/project/motia/) +- [Google Calendar API](https://developers.google.com/calendar) +- [Redis Documentation](https://redis.io/documentation) + +## Support + +- **Migration Questions**: Check [MIGRATION_GUIDE.md](../MIGRATION_GUIDE.md) +- **Runtime Issues**: Check iii Console logs +- **Step Not Showing**: Verify import errors in logs +- **Redis Issues**: Check Redis connection in `services/` diff --git a/steps/advoware_cal_sync/README.md b/steps/advoware_cal_sync/README.md new file mode 100644 index 0000000..b242058 --- /dev/null +++ b/steps/advoware_cal_sync/README.md @@ -0,0 +1,268 @@ +# Advoware Calendar Sync - Event-Driven Design + +Dieser Abschnitt implementiert die bidirektionale Synchronisation zwischen Advoware-Terminen und Google Calendar. Das System nutzt einen event-driven Ansatz mit **Motia III v1.0-RC**, der auf direkten API-Calls basiert, mit Redis für Locking und Deduplikation. Es stellt sicher, dass Termine konsistent gehalten werden, mit Fokus auf Robustheit, Fehlerbehandlung und korrekte Handhabung von mehrtägigen Terminen. + +## Übersicht + +Das System synchronisiert Termine zwischen: +- **Advoware**: Zentrale Terminverwaltung mit detaillierten Informationen. +- **Google Calendar**: Benutzerfreundliche Kalenderansicht für jeden Mitarbeiter. + +## Architektur + +### Event-Driven Design +- **Direkte API-Synchronisation**: Kein zentraler Hub; Sync läuft direkt zwischen APIs. +- **Redis Locking**: Per-Employee Locking verhindert Race-Conditions. +- **Event Emission**: Cron → All-Step → Employee-Step für skalierbare Verarbeitung. +- **Fehlerresistenz**: Einzelne Fehler stoppen nicht den gesamten Sync. +- **Logging**: Alle Logs erscheinen in der iii Console via `ctx.logger`. + +### Sync-Phasen +1. **Cron-Step**: Automatische Auslösung alle 15 Minuten. +2. **All-Step**: Fetcht alle Mitarbeiter und emittiert Events pro Employee. +3. **Employee-Step**: Synchronisiert Termine für einen einzelnen Mitarbeiter. + +### Datenmapping und Standardisierung +Beide Systeme werden auf gemeinsames Format normalisiert (Berlin TZ): +```python +{ + 'start': datetime, # Berlin TZ + 'end': datetime, + 'text': str, + 'notiz': str, + 'ort': str, + 'dauertermin': int, # 0/1 + 'turnus': int, # 0/1 + 'turnusArt': int, + 'recurrence': str # RRULE oder None +} +``` + +#### Advoware → Standard +- Start: `datum` + `uhrzeitVon` (Fallback 09:00), oder `datum` als datetime. +- End: `datumBis` + `uhrzeitBis` (Fallback 10:00), oder `datum` + 1h. +- All-Day: `dauertermin=1` oder Dauer >1 Tag. +- Recurring: `turnus`/`turnusArt` (vereinfacht, keine RRULE). + +#### Google → Standard +- Start/End: `dateTime` oder `date` (All-Day). +- All-Day: `dauertermin=1` wenn All-Day oder Dauer >1 Tag. +- Recurring: RRULE aus `recurrence`. + +#### Standard → Advoware +- POST/PUT: `datum`/`uhrzeitBis`/`datumBis` aus start/end. +- Defaults: `vorbereitungsDauer='00:00:00'`, `sb`/`anwalt`=employee_kuerzel. + +#### Standard → Google +- All-Day: `date` statt `dateTime`, end +1 Tag. +- Recurring: RRULE aus `recurrence`. + +## Funktionalität + +### Automatische Kalender-Erstellung +- Für jeden Advoware-Mitarbeiter wird ein Google Calendar mit dem Namen `AW-{Kuerzel}` erstellt. +- Beispiel: Mitarbeiter mit Kürzel "SB" → Calendar "AW-SB". +- Kalender wird mit dem Haupt-Google-Account (`lehmannundpartner@gmail.com`) als Owner geteilt. + +### Sync-Details + +#### Cron-Step (calendar_sync_cron_step.py) +- Läuft alle 15 Minuten und emittiert "calendar_sync_all". +- **Trigger**: `cron("0 */15 * * * *")` (6-field: Sekunde Minute Stunde Tag Monat Wochentag) + +#### All-Step (calendar_sync_all_step.py) +- Fetcht alle Mitarbeiter aus Advoware. +- Filtert Debug-Liste (falls konfiguriert). +- Setzt Redis-Lock pro Employee. +- Emittiert "calendar_sync" Event pro Employee. +- **Trigger**: `queue('calendar_sync_all')` + +#### Employee-Step (calendar_sync_event_step.py) +- Fetcht Advoware-Termine für den Employee. +- Fetcht Google-Events für den Employee. +- Synchronisiert: Neue erstellen, Updates anwenden, Deletes handhaben. +- Verwendet Locking, um parallele Syncs zu verhindern. +- **Trigger**: `queue('calendar_sync')` + +#### API-Step (calendar_sync_api_step.py) +- Manueller Trigger für einzelnen Employee oder "ALL". +- Bei "ALL": Emittiert "calendar_sync_all". +- Bei Employee: Setzt Lock und emittiert "calendar_sync". +- **Trigger**: `http('POST', '/advoware/calendar/sync')` + +## API-Schwächen und Fixes + +### Advoware API +- **Mehrtägige Termine**: `datumBis` wird korrekt für Enddatum verwendet; '00:00:00' als '23:59:59' interpretiert. +- **Zeitformate**: Robuste Parsing mit Fallbacks. +- **Keine 24h-Limit**: Termine können länger als 24h sein; Google Calendar unterstützt das. + +### Google Calendar API +- **Zeitbereiche**: Akzeptiert Events >24h ohne Probleme. +- **Rate Limits**: Backoff-Retry implementiert. + +## Step-Konfiguration (Motia III) + +### calendar_sync_cron_step.py +```python +config = { + 'name': 'Calendar Sync Cron Job', + 'flows': ['advoware'], + 'triggers': [ + cron("0 */15 * * * *") # Alle 15 Minuten (6-field format) + ], + 'enqueues': ['calendar_sync_all'] +} +``` + +### calendar_sync_all_step.py +```python +config = { + 'name': 'Calendar Sync All Step', + 'flows': ['advoware'], + 'triggers': [ + queue('calendar_sync_all') + ], + 'enqueues': ['calendar_sync'] +} +``` + +### calendar_sync_event_step.py +```python +config = { + 'name': 'Calendar Sync Event Step', + 'flows': ['advoware'], + 'triggers': [ + queue('calendar_sync') + ], + 'enqueues': [] +} +``` + +### calendar_sync_api_step.py +```python +config = { + 'name': 'Calendar Sync API Trigger', + 'flows': ['advoware'], + 'triggers': [ + http('POST', '/advoware/calendar/sync') + ], + 'enqueues': ['calendar_sync', 'calendar_sync_all'] +} +``` + +## Setup + +### Umgebungsvariablen +```env +# Google Calendar +GOOGLE_CALENDAR_SERVICE_ACCOUNT_PATH=service-account.json + +# Advoware API +ADVOWARE_API_BASE_URL=https://www2.advo-net.net:90/ +ADVOWARE_PRODUCT_ID=64 +ADVOWARE_APP_ID=your_app_id +ADVOWARE_API_KEY=your_api_key +ADVOWARE_KANZLEI=your_kanzlei +ADVOWARE_DATABASE=your_database +ADVOWARE_USER=your_user +ADVOWARE_ROLE=2 +ADVOWARE_PASSWORD=your_password +ADVOWARE_TOKEN_LIFETIME_MINUTES=55 +ADVOWARE_API_TIMEOUT_SECONDS=30 + +# Redis +REDIS_HOST=localhost +REDIS_PORT=6379 +REDIS_DB_CALENDAR_SYNC=1 +REDIS_TIMEOUT_SECONDS=5 + +# Debug +CALENDAR_SYNC_DEBUG_EMPLOYEES=PB,AI # Optional, filter employees +``` + +## Verwendung + +### Manueller Sync +```bash +# Sync für einen bestimmten Mitarbeiter +curl -X POST "http://localhost:3111/advoware/calendar/sync" \ + -H "Content-Type: application/json" \ + -d '{"kuerzel": "PB"}' + +# Sync für alle Mitarbeiter +curl -X POST "http://localhost:3111/advoware/calendar/sync" \ + -H "Content-Type: application/json" \ + -d '{"kuerzel": "ALL"}' +``` + +### Automatischer Sync +Cron-Step läuft automatisch alle 15 Minuten. + +## Fehlerbehandlung und Logging + +- **Locking**: Redis NX/EX verhindert parallele Syncs. +- **Logging**: `ctx.logger` für iii Console-Sichtbarkeit. +- **API-Fehler**: Retry mit Backoff. +- **Parsing-Fehler**: Robuste Fallbacks. + +## Sicherheit + +- Service Account für Google Calendar API. +- HMAC-512 Authentifizierung für Advoware API. +- Redis für Concurrency-Control. + +## Bekannte Probleme + +- **Recurring-Events**: Begrenzte Unterstützung für komplexe Wiederholungen. +- **Performance**: Bei vielen Terminen Paginierung beachten. +- **Timezone-Handling**: Alle Operationen in Europe/Berlin TZ. + +## Datenfluss + +``` +Cron (alle 15min) + → calendar_sync_cron_step + → ctx.enqueue(topic: "calendar_sync_all") + → calendar_sync_all_step + → Fetch Employees from Advoware + → For each Employee: + → Set Redis Lock (key: calendar_sync:employee:{kuerzel}) + → ctx.enqueue(topic: "calendar_sync", data: {kuerzel, ...}) + → calendar_sync_event_step + → Fetch Advoware Termine (frNr, datum, text, etc.) + → Fetch Google Calendar Events + → 4-Phase Sync: + 1. New from Advoware → Google + 2. New from Google → Advoware + 3. Process Deletes + 4. Process Updates + → Clear Redis Lock +``` + +## Weitere Dokumentation + +- **Individual Step Docs**: Siehe `docs/` Ordner in diesem Verzeichnis +- **Architecture Overview**: [../../docs/ARCHITECTURE.md](../../docs/ARCHITECTURE.md) +- **Google Setup Guide**: [../../docs/GOOGLE_SETUP.md](../../docs/GOOGLE_SETUP.md) +- **Troubleshooting**: [../../docs/TROUBLESHOOTING.md](../../docs/TROUBLESHOOTING.md) + +## Migration Notes + +Dieses System wurde von **Motia v0.17** nach **Motia III v1.0-RC** migriert: + +### Wichtige Änderungen: +- ✅ `type: 'event'` → `triggers: [queue('topic')]` +- ✅ `type: 'cron'` → `triggers: [cron('expression')]` (6-field format) +- ✅ `type: 'api'` → `triggers: [http('METHOD', 'path')]` +- ✅ `context.emit()` → `ctx.enqueue()` +- ✅ `emits: [...]` → `enqueues: [...]` +- ✅ Relative Imports → Absolute Imports mit `sys.path.insert()` +- ✅ Motia Workbench → iii Console + +### Kompatibilität: +- ✅ Alle 4 Steps vollständig migriert +- ✅ Google Calendar API Integration unverändert +- ✅ Advoware API Integration unverändert +- ✅ Redis Locking-Mechanismus unverändert +- ✅ Datenbank-Schema kompatibel diff --git a/steps/advoware_proxy/README.md b/steps/advoware_proxy/README.md new file mode 100644 index 0000000..c848e62 --- /dev/null +++ b/steps/advoware_proxy/README.md @@ -0,0 +1,314 @@ +# Advoware API Proxy Steps + +Dieser Ordner enthält die API-Proxy-Steps für die Advoware-Integration. Jeder Step implementiert eine HTTP-Methode als universellen Proxy zur Advoware-API mit **Motia III v1.0-RC**. + +## Übersicht + +Die Proxy-Steps fungieren als transparente Schnittstelle zwischen Clients und der Advoware-API. Sie handhaben Authentifizierung, Fehlerbehandlung und Logging automatisch. + +## Steps + +### 1. GET Proxy (`advoware_api_proxy_get_step.py`) + +**Zweck:** Universeller Proxy für GET-Requests an die Advoware-API. + +**Konfiguration:** +```python +config = { + 'name': 'Advoware Proxy GET', + 'flows': ['advoware'], + 'triggers': [ + http('GET', '/advoware/proxy') + ], + 'enqueues': [] +} +``` + +**Funktionalität:** +- Extrahiert den Ziel-Endpoint aus Query-Parametern (`endpoint`) +- Übergibt alle anderen Query-Parameter als API-Parameter +- Gibt das Ergebnis als JSON zurück + +**Beispiel Request:** +```bash +GET /advoware/proxy?endpoint=employees&limit=10&offset=0 +``` + +**Response:** +```json +{ + "result": { + "data": [...], + "total": 100 + } +} +``` + +### 2. POST Proxy (`advoware_api_proxy_post_step.py`) + +**Zweck:** Universeller Proxy für POST-Requests an die Advoware-API. + +**Konfiguration:** +```python +config = { + 'name': 'Advoware Proxy POST', + 'flows': ['advoware'], + 'triggers': [ + http('POST', '/advoware/proxy') + ], + 'enqueues': [] +} +``` + +**Funktionalität:** +- Extrahiert den Ziel-Endpoint aus Query-Parametern (`endpoint`) +- Verwendet den Request-Body als JSON-Daten für die API +- Erstellt neue Ressourcen in Advoware + +**Beispiel Request:** +```bash +POST /advoware/proxy?endpoint=employees +Content-Type: application/json + +{ + "name": "John Doe", + "email": "john@example.com" +} +``` + +### 3. PUT Proxy (`advoware_api_proxy_put_step.py`) + +**Zweck:** Universeller Proxy für PUT-Requests an die Advoware-API. + +**Konfiguration:** +```python +config = { + 'name': 'Advoware Proxy PUT', + 'flows': ['advoware'], + 'triggers': [ + http('PUT', '/advoware/proxy') + ], + 'enqueues': [] +} +``` + +**Funktionalität:** +- Extrahiert den Ziel-Endpoint aus Query-Parametern (`endpoint`) +- Verwendet den Request-Body als JSON-Daten für Updates +- Aktualisiert bestehende Ressourcen in Advoware + +**Beispiel Request:** +```bash +PUT /advoware/proxy?endpoint=employees/123 +Content-Type: application/json + +{ + "name": "John Smith", + "email": "johnsmith@example.com" +} +``` + +### 4. DELETE Proxy (`advoware_api_proxy_delete_step.py`) + +**Zweck:** Universeller Proxy für DELETE-Requests an die Advoware-API. + +**Konfiguration:** +```python +config = { + 'name': 'Advoware Proxy DELETE', + 'flows': ['advoware'], + 'triggers': [ + http('DELETE', '/advoware/proxy') + ], + 'enqueues': [] +} +``` + +**Funktionalität:** +- Extrahiert den Ziel-Endpoint aus Query-Parametern (`endpoint`) +- Löscht Ressourcen in Advoware + +**Beispiel Request:** +```bash +DELETE /advoware/proxy?endpoint=employees/123 +``` + +## Gemeinsame Features + +### Authentifizierung +Alle Steps verwenden den `AdvowareService` für automatische Token-Verwaltung und Authentifizierung: +- HMAC-512 basierte Signatur +- Token-Caching in Redis (55 Minuten Lifetime) +- Automatischer Token-Refresh bei 401-Errors + +### Fehlerbehandling +- **400 Bad Request:** Fehlender `endpoint` Parameter +- **500 Internal Server Error:** API-Fehler oder Exceptions +- **401 Unauthorized:** Automatischer Token-Refresh und Retry + +### Logging +Detaillierte Logs via `ctx.logger` für: +- Eingehende Requests +- API-Calls an Advoware +- Fehler und Exceptions +- Token-Management + +Alle Logs sind in der **iii Console** sichtbar. + +### Sicherheit +- Keine direkte Weitergabe sensibler Daten +- Authentifizierung über Service-Layer +- Input-Validation für erforderliche Parameter +- HMAC-512 Signatur für alle API-Requests + +## Handler-Struktur (Motia III) + +Alle Steps folgen dem gleichen Pattern: + +```python +from motia import http, ApiRequest, ApiResponse, FlowContext +from services.advoware_service import AdvowareService + +config = { + 'name': 'Advoware Proxy {METHOD}', + 'flows': ['advoware'], + 'triggers': [ + http('{METHOD}', '/advoware/proxy') + ], + 'enqueues': [] +} + +async def handler(request: ApiRequest, ctx: FlowContext) -> ApiResponse: + # Extract endpoint from query params + endpoint = request.query_params.get('endpoint') + + if not endpoint: + return ApiResponse( + status=400, + body={'error': 'Missing required query parameter: endpoint'} + ) + + # Call Advoware API + advoware_service = AdvowareService() + result = await advoware_service.{method}(endpoint, **params) + + return ApiResponse(status=200, body={'result': result}) +``` + +## Testing + +### Unit Tests +```bash +# Test GET Proxy +curl -X GET "http://localhost:3111/advoware/proxy?endpoint=employees" + +# Test POST Proxy +curl -X POST "http://localhost:3111/advoware/proxy?endpoint=employees" \ + -H "Content-Type: application/json" \ + -d '{"name": "Test Employee"}' + +# Test PUT Proxy +curl -X PUT "http://localhost:3111/advoware/proxy?endpoint=employees/1" \ + -H "Content-Type: application/json" \ + -d '{"name": "Updated Employee"}' + +# Test DELETE Proxy +curl -X DELETE "http://localhost:3111/advoware/proxy?endpoint=employees/1" +``` + +### Integration Tests +Überprüfen Sie die Logs in der iii Console: +```bash +# Check logs +curl http://localhost:3111/_console/logs +``` + +## Konfiguration + +### Umgebungsvariablen +Stellen Sie sicher, dass folgende Variablen gesetzt sind: +```env +ADVOWARE_API_BASE_URL=https://www2.advo-net.net:90/ +ADVOWARE_PRODUCT_ID=64 +ADVOWARE_APP_ID=your_app_id +ADVOWARE_API_KEY=your_api_key +ADVOWARE_KANZLEI=your_kanzlei +ADVOWARE_DATABASE=your_database +ADVOWARE_USER=your_user +ADVOWARE_ROLE=2 +ADVOWARE_PASSWORD=your_password +ADVOWARE_TOKEN_LIFETIME_MINUTES=55 +ADVOWARE_API_TIMEOUT_SECONDS=30 + +# Redis (für Token-Caching) +REDIS_HOST=localhost +REDIS_PORT=6379 +REDIS_DB=0 +``` + +### Dependencies +- `services/advoware_service.py` - Advoware API Client mit HMAC Auth +- `config.py` - Konfigurationsmanagement +- `motia` - Motia III Python SDK +- `asyncpg` - PostgreSQL Client +- `redis` - Redis Client für Token-Caching + +## Erweiterungen + +### Geplante Features +- Request/Response Caching für häufige Queries +- Rate Limiting pro Client +- Request Validation Schemas mit Pydantic +- Batch-Operations Support + +### Custom Endpoints +Für spezifische Endpoints können zusätzliche Steps erstellt werden, die direkt auf bestimmte Ressourcen zugreifen und erweiterte Validierung/Transformation bieten. + +## Architektur + +``` +Client Request + │ + ▼ +HTTP Trigger (http('METHOD', '/advoware/proxy')) + │ + ▼ +Handler (ApiRequest → ApiResponse) + │ + ├─► Extract 'endpoint' from query params + ├─► Extract other params/body + │ + ▼ +AdvowareService + │ + ├─► Check Redis for valid token + ├─► If expired: Get new token (HMAC-512 auth) + ├─► Build HTTP request + │ + ▼ +Advoware API + │ + ▼ +Response → Transform → Return ApiResponse +``` + +## Migration Notes + +Dieses System wurde von **Motia v0.17** nach **Motia III v1.0-RC** migriert: + +### Wichtige Änderungen: +- ✅ `type: 'api'` → `triggers: [http('METHOD', 'path')]` +- ✅ `ApiRouteConfig` → `StepConfig` mit `as const satisfies` +- ✅ `Handlers['StepName']` → `Handlers` +- ✅ `context` → `ctx` +- ✅ `req` dict → `ApiRequest` typed object +- ✅ Return dict → `ApiResponse` typed object +- ✅ `method`, `path` moved into trigger +- ✅ Motia Workbench → iii Console + +### Kompatibilität: +- ✅ Alle 4 Proxy Steps vollständig migriert +- ✅ AdvowareService kompatibel (keine Änderungen) +- ✅ Redis Token-Caching unverändert +- ✅ HMAC-512 Auth unverändert +- ✅ API-Endpoints identisch