Files
motia/bitbylaw/docs/ARCHITECTURE.md
2026-02-07 09:23:49 +00:00

643 lines
19 KiB
Markdown

# Architektur
## Systemübersicht
Das bitbylaw-System ist eine event-driven Integration zwischen Advoware, EspoCRM, Google Calendar, Vermieterhelden und 3CX Telefonie, basierend auf dem Motia Framework. Die Architektur folgt einem modularen, mikroservice-orientierten Ansatz mit klarer Separation of Concerns.
### Kernkomponenten
```
┌─────────────────────────────┐
│ KONG API Gateway │
│ api.bitbylaw.com │
│ (Auth, Rate Limiting) │
└──────────────┬──────────────┘
┌──────────────────────────┼──────────────────────────┐
│ │ │
┌────▼────────┐ ┌──────▼─────────┐ ┌─────▼──────┐
│ Vermieter- │ │ Motia │ │ 3CX │
│ helden.de │────────▶│ Framework │◀────────│ Telefonie │
│ (WordPress) │ │ (Middleware) │ │ (ralup) │
└─────────────┘ └────────┬───────┘ └────────────┘
Leads Input │ Call Handling
┌───────────────────────────┼───────────────────────────┐
│ │ │
┌────▼────┐ ┌──────▼──────┐ ┌──────▼─────┐
│Advoware │ │ VMH │ │ Calendar │
│ Proxy │ │ Webhooks │ │ Sync │
└────┬────┘ └─────┬───────┘ └─────┬──────┘
│ │ │
│ │ │
┌────▼─────────────────────────▼──────────────────────────▼────┐
│ Redis (3 DBs) │
│ DB 1: Caching & Locks │
│ DB 2: Calendar Sync State │
└───────────────────────────────────────────────────────────────┘
┌────▼────────────────────────────┐
│ External Services │
├─────────────────────────────────┤
│ • Advoware REST API │
│ • EspoCRM (VMH) │
│ • Google Calendar API │
│ • 3CX API (ralup.my3cx.de) │
│ • Vermieterhelden WordPress │
└─────────────────────────────────┘
```
## Komponenten-Details
### 0. KONG API Gateway
**Zweck**: Zentraler API-Gateway für alle öffentlichen APIs mit Authentifizierung und Rate Limiting.
**Domain**: `api.bitbylaw.com`
**Funktionen**:
- **Authentication**: API-Key-basiert, JWT, OAuth2
- **Rate Limiting**: Pro Consumer/API-Key
- **Request Routing**: Zu Backend-Services (Motia, etc.)
- **SSL/TLS Termination**: HTTPS-Handling
- **Logging & Monitoring**: Request-Logs, Metrics
- **CORS Handling**: Cross-Origin Requests
**Upstream Services**:
- Motia Framework (Advoware Proxy, Calendar Sync, VMH Webhooks)
- Zukünftig: Weitere Microservices
**Konfiguration**:
```yaml
# KONG Service Configuration
services:
- name: motia-backend
url: http://localhost:3000
routes:
- name: advoware-proxy
paths: [/advoware/*]
- name: calendar-sync
paths: [/calendar/*]
- name: vmh-webhooks
paths: [/vmh/*]
plugins:
- name: key-auth
- name: rate-limiting
config:
minute: 600
```
**Flow**:
```
Client → KONG (api.bitbylaw.com) → Auth Check → Rate Limit → Motia Backend
```
### 1. Advoware Proxy Layer
**Zweck**: Transparente REST-API-Proxy für Advoware mit Authentifizierung und Caching.
**Module**: `steps/advoware_proxy/`
- `advoware_api_proxy_get_step.py` - GET-Requests
- `advoware_api_proxy_post_step.py` - POST-Requests (Create)
- `advoware_api_proxy_put_step.py` - PUT-Requests (Update)
- `advoware_api_proxy_delete_step.py` - DELETE-Requests
**Services**: `services/advoware.py`
- Token-Management (HMAC-512 Authentifizierung)
- Redis-basiertes Token-Caching (55min Lifetime)
- Automatischer Token-Refresh bei 401-Errors
- Async API-Client mit aiohttp
**Datenfluss**:
```
Client → API-Step → AdvowareAPI Service → Redis (Token Cache) → Advoware API
```
### 2. Calendar Sync System
**Zweck**: Bidirektionale Synchronisation zwischen Advoware-Terminen und Google Calendar.
**Architecture Pattern**: Event-Driven Cascade
**Integration**: EspoCRM sendet Webhooks an KONG → Motia
**Datenfluss**:
```
EspoCRM (Vermieterhelden CRM) → KONG → Motia VMH Webhooks → Redis Dedup → Events
```
```
Cron (täglich)
→ calendar_sync_cron_step
→ Emit: "calendar_sync_all"
→ calendar_sync_all_step
→ Fetch Employees
→ For each Employee:
→ Set Redis Lock
→ Emit: "calendar_sync_employee"
→ calendar_sync_event_step
→ Fetch Advoware Events
→ Fetch Google Events
→ Sync (Create/Update/Delete)
→ Clear Redis Lock
```
**Module**: `steps/advoware_cal_sync/`
- `calendar_sync_cron_step.py` - Täglicher Trigger
- `calendar_sync_all_step.py` - Employee-List-Handler
- `calendar_sync_event_step.py` - Per-Employee Sync-Logic
- `calendar_sync_api_step.py` - Manueller Trigger-Endpoint
- `calendar_sync_utils.py` - Shared Utilities
- `audit_calendar_sync.py` - Audit & Diagnostics
**Key Features**:
- **Redis Locking**: Verhindert parallele Syncs für denselben Employee
- **Rate Limiting**: Token-Bucket-Algorithm (7 tokens, Redis-based)
- **Normalisierung**: Common format (Berlin TZ) für beide APIs
- **Error Isolation**: Employee-Fehler stoppen nicht Gesamt-Sync
**Datenmapping**:
```
Advoware Format → Standard Format → Google Calendar Format
↓ ↓ ↓
datum/uhrzeitVon start (datetime) dateTime
datumBis end (datetime) dateTime
dauertermin all_day (bool) date
turnus/turnusArt recurrence RRULE
```
### 3. VMH Webhook System
**Zweck**: Empfang und Verarbeitung von EspoCRM Webhooks für Beteiligte-Entitäten.
**Architecture Pattern**: Webhook → Deduplication → Event Emission
**Module**: `steps/vmh/`
- `webhook/beteiligte_create_api_step.py` - Create Webhook
- `webhook/beteiligte_update_api_step.py` - Update Webhook
- `webhook/beteiligte_delete_api_step.py` - Delete Webhook
- `beteiligte_sync_event_step.py` - Sync Event Handler (Placeholder)
**Webhook-Flow**:
```
EspoCRM → POST /vmh/webhook/beteiligte/create
Webhook Step
Extract Entity IDs
Redis Deduplication (SET: vmh:beteiligte:create_pending)
Emit Event: "vmh.beteiligte.create"
Sync Event Step (subscribes)
[TODO: Implementierung]
### 4. Vermieterhelden Integration
**Zweck**: Lead-Eingang von Vermieterhelden.de WordPress-Frontend.
**URL**: `https://vermieterhelden.de`
**Technologie**: WordPress-basiertes Frontend
**Funktionen**:
- **Lead-Formulare**: Mieter, Vermieter, Anfragen
- **Lead-Routing**: Zu EspoCRM (VMH) → Motia
- **Webhook-basiert**: POST zu KONG/Motia bei neuem Lead
**Datenfluss**:
```
Vermieterhelden.de → Lead erstellt → Webhook → KONG → Motia → EspoCRM/Advoware
```
**Lead-Typen**:
- Mieter-Anfragen
- Vermieter-Anfragen
- Kontaktformulare
- Newsletter-Anmeldungen
**Integration mit Motia**:
- Eigener Webhook-Endpoint: `/api/leads/vermieterhelden`
- Lead-Validierung und -Enrichment
- Weiterleitung an CRM-Systeme
### 5. 3CX Telefonie-Integration
**Zweck**: Telefonie-System-Integration für Call-Handling und Lead-Qualifizierung.
**URL**: `https://ralup.my3cx.de`
**Technologie**: 3CX Cloud PBX
**Funktionen**:
- **Outbound Calls**: Lead-Anrufe (automatisch oder manuell)
- **Inbound Calls**: Stammdatenabfrage (CTI - Computer Telephony Integration)
- **Call Logging**: Anrufprotokolle zu CRM
- **Call Recording**: Aufzeichnungen speichern und abrufen
- **Screen Pops**: Kundeninfo bei eingehendem Anruf
**API-Integrationen**:
**A) Outbound: Motia → 3CX**
```
Motia → KONG → 3CX API
- Initiate Call to Lead
- Get Call Status
```
**B) Inbound: 3CX → Motia**
```
3CX Webhook → KONG → Motia
- Call Started → Fetch Customer Data
- Call Ended → Log Call Record
```
**Datenfluss**:
**Call Initiation**:
```
Lead in CRM → Trigger Call → Motia → 3CX API → Dial Number
```
**Inbound Call**:
```
3CX detects call → Webhook to Motia → Lookup in Advoware/EspoCRM → Return data → 3CX Screen Pop
```
**Call Recording**:
```
Call ends → 3CX Webhook → Motia → Store metadata → Link to CRM entity
```
**Use Cases**:
- Lead-Qualifizierung nach Eingang
- Stammdatenabfrage bei Anruf
- Anrufprotokoll in EspoCRM/Advoware
- Automatische Follow-up-Tasks
```
**Deduplikation-Mechanismus**:
- Redis SET für pending IDs pro Action-Type (create/update/delete)
- Neue IDs werden zu SET hinzugefügt
- Events nur für neue (nicht-duplizierte) IDs emittiert
- SET-TTL verhindert Memory-Leaks
## Event-Driven Design
### Event-Topics
| Topic | Emitter | Subscriber | Payload |
|-------|---------|------------|---------|
| `calendar_sync_all` | cron_step | all_step | `{}` |
| `calendar_sync_employee` | all_step, api_step | event_step | `{kuerzel, full_content}` |
| `vmh.beteiligte.create` | create webhook | sync_event_step | `{entity_id, action, source, timestamp}` |
| `vmh.beteiligte.update` | update webhook | sync_event_step | `{entity_id, action, source, timestamp}` |
| `vmh.beteiligte.delete` | delete webhook | sync_event_step | `{entity_id, action, source, timestamp}` |
### Event-Flow Patterns
**1. Cascade Pattern** (Calendar Sync):
```
Trigger → Fetch List → Emit per Item → Process Item
```
**2. Webhook Pattern** (VMH):
```
External Event → Dedup → Internal Event → Processing
```
## Redis Architecture
### Database Layout
**DB 0**: Default (Motia internal)
**DB 1**: Advoware Cache & Locks
- `advoware_access_token` - Bearer Token (TTL: 53min)
- `advoware_token_timestamp` - Token Creation Time
- `calendar_sync:lock:{kuerzel}` - Per-Employee Lock (TTL: 5min)
- `vmh:beteiligte:create_pending` - Create Dedup SET
- `vmh:beteiligte:update_pending` - Update Dedup SET
- `vmh:beteiligte:delete_pending` - Delete Dedup SET
**DB 2**: Calendar Sync Rate Limiting
- `google_calendar_api_tokens` - Token Bucket for Rate Limiting
---
## External APIs
### Advoware REST API
**Base URL**: `https://advoware-api.example.com/api/v1/`
**Auth**: HMAC-512 (siehe `services/advoware.py`)
**Rate Limits**: Unknown (keine Limits bekannt)
**Documentation**: [Advoware API Swagger](../docs/advoware/advoware_api_swagger.json)
**Wichtige Endpoints**:
- `POST /auth/login` - Token generieren
- `GET /employees` - Employee-Liste
- `GET /events` - Termine abrufen
- `POST /events` - Termin erstellen
- `PUT /events/{id}` - Termin aktualisieren
### Redis Usage Patterns
**Token Caching**:
```python
# Set with expiration
redis.set('advoware_access_token', token, ex=3180) # 53min
# Get with fallback
token = redis.get('advoware_access_token')
if not token:
token = fetch_new_token()
```
### EspoCRM (VMH)
**Integration**: Webhook Sender (Outbound), API Consumer
**Endpoints**: Configured in EspoCRM, routed via KONG
**Format**: JSON POST with entity data
**Note**: Dient als CRM für Vermieterhelden-Leads
### 3CX Telefonie API
**Base URL**: `https://ralup.my3cx.de/api/v1/`
**Auth**: API Key oder Basic Auth
**Rate Limits**: Unknown (typisch 60 req/min)
**Key Endpoints**:
- `POST /calls/initiate` - Anruf starten
- `GET /calls/{id}/status` - Call-Status
- `GET /calls/{id}/recording` - Aufzeichnung abrufen
- `POST /webhook` - Webhook-Konfiguration (eingehend)
**Webhooks** (Inbound von 3CX):
- `call.started` - Anruf beginnt
- `call.ended` - Anruf beendet
- `call.transferred` - Anruf weitergeleitet
### Vermieterhelden
**Integration**: Webhook Sender (Lead-Eingang)
**Base**: WordPress mit Custom Plugins
**Format**: JSON POST zu Motia
**Webhook-Events**:
- `lead.created` - Neuer Lead
- `contact.submitted` - Kontaktformular
lock_key = f'calendar_sync:lock:{kuerzel}'
if not redis.set(lock_key, '1', nx=True, ex=300):
raise LockError("Already locked")
# Always release
redis.delete(lock_key)
```
**Deduplication**:
```python
# Check & Add atomically
existing = redis.smembers('vmh:beteiligte:create_pending')
new_ids = input_ids - existing
if new_ids:
redis.sadd('vmh:beteiligte:create_pending', *new_ids)
```
## Service Layer
### AdvowareAPI Service
**Location**: `services/advoware.py`
**Responsibilities**:
- HMAC-512 Authentication
- Token Management
- HTTP Client (aiohttp)
- Error Handling & Retries
**Key Methods**:
```python
get_access_token(force_refresh=False) -> str
api_call(endpoint, method, params, json_data) -> Any
```
**Authentication Flow**:
```
1. Generate HMAC-512 signature
- Message: "{product_id}:{app_id}:{nonce}:{timestamp}"
- Key: Base64-decoded API Key
- Hash: SHA512
2. POST to security.advo-net.net/api/v1/Token
- Body: {AppID, User, Password, HMAC512Signature, ...}
3. Extract access_token from response
4. Cache in Redis (53min TTL)
5. Use as Bearer Token: "Authorization: Bearer {token}"
```
## External API Integration
### Advoware API
**Base URL**: `https://www2.advo-net.net:90/`
**Auth**: HMAC-512 + Bearer Token
**Rate Limits**: Unknown (robust error handling)
**Key Endpoints**:
- `/employees` - Mitarbeiter-Liste
- `/appointments` - Termine
### Google Calendar API
**Auth**: Service Account (JSON Key)
**Rate Limits**: 600 requests/minute (enforced via Redis)
**Scopes**: `https://www.googleapis.com/auth/calendar`
**Key Operations**:
- `calendars().get()` - Calendar abrufen
- `calendars().insert()` - Calendar erstellen
- `events().list()` - Events abrufen
- `events().insert()` - Event erstellen
- KONG Gateway**: API-Key oder JWT-based Auth für externe Clients
**Advoware**: User-based Auth (ADVOWARE_USER + PASSWORD)
**Google**: Service Account (domain-wide delegation)
**3CX**: API Key oder Basic Auth
**Redis**: Localhost only (no password)
**Vermieterhelden**: Webhook-Secret für Validation
### EspoCRM
**Integration**: Webhook Sender (Outbound)
**Endpoints**: Configured in EspoCRM
**Format**: JSON POST with entity data
## Security
### Secrets Management
**Environment Variables**:
```bash
ADVOWARE_API_KEY # Base64-encoded HMAC Key
ADVOWARE_PASSWORD # User Password
GOOGLE_CALENDAR_SERVICE_ACCOUNT_PATH # Path to JSON Key
ESPOCRM_MARVIN_API_KEY # Webhook Validation (optional)
```
**Storage**:
- Environment variables in systemd service
- Service Account JSON: `/opt/motia-app/service-account.json` (chmod 600)
- No secrets in code or Git
### Access Control
**Advoware**: User-based Auth (ADVOWARE_USER + PASSWORD)
**Google**: Service Account (domain-wide delegation)
**Redis**: Localhost only (no password)
## Performance Characteristics
### Throughput
**Calendar Sync**:
- ~10 employees: 2-3 minutes
- Rate-limited by Google API (600 req/min)
- Per-employee parallelization: Nein (sequential via events)
**Webhooks**:
- Instant processing (<100ms)
- Batch support (multiple entities per request)
- Redis dedup overhead: <10ms
### Memory Usage
**Current**: 169MB (Peak: 276MB)
**Breakdown**:
- Node.js process: ~150MB
- Python dependencies: Lazy-loaded per step
- Redis memory: <10MB
### Scalability
**Horizontal**: Nicht ohne weiteres möglich (Redis Locks, Shared State)
**Vertical**: CPU-bound bei vielen parallel Employees
**Bottleneck**: Google Calendar API Rate Limits
## Monitoring & Observability
### Logging
**Framework**: Motia Workbench (structured logging)
**Levels**: DEBUG, INFO, ERROR
**Output**: journalctl (systemd) + Motia Workbench UI
**Key Log Points**:
- API-Requests (Method, URL, Status)
- Event Emission (Topic, Payload)
- Redis Operations (Keys, Success/Failure)
- Errors (Stack traces, Context)
### Metrics
**Available** (via Logs):
- Webhook receive count
- Calendar sync duration per employee
- API call count & latency
- Redis hit/miss ratio (implicit)
**Missing** (Future):
- Prometheus metrics
- Grafana dashboards
- Alerting
## Deployment
### systemd Service
**Unit**: `motia.service`
**User**: `www-data`
**WorkingDirectory**: `/opt/motia-app/bitbylaw`
**Restart**: `always` (10s delay)
**Environment**:
```bash
NODE_ENV=production
NODE_OPTIONS=--max-old-space-size=8192 --inspect
HOST=0.0.0.0
MOTIA_LOG_LEVEL=debug
```
### Dependencies
**Runtime**:
- Node.js 18+
- Python 3.13+
- Redis Server
- systemd
**Build**:
- npm (Node packages)
- pip (Python packages)
- Motia CLI
## Disaster Recovery
### Backup Strategy
**Redis**:
- RDB snapshots (automatisch)
- AOF persistence (optional)
**Configuration**:
- Git-versioniert
- Environment Variables in systemd
**Service Account**:
- Manual backup: `/opt/motia-app/service-account.json`
### Recovery Procedures
**Service Restart**:
```bash
systemctl restart motia.service
```
**Clear Redis Cache**:
```bash
redis-cli -n 1 FLUSHDB # Advoware Cache
redis-cli -n 2 FLUSHDB # Calendar Sync
```
**Clear Employee Lock**:
```bash
python /opt/motia-app/bitbylaw/delete_employee_locks.py
```
## Future Enhancements
### P3CX Full Integration**: Complete call handling, CTI features
3. **Vermieterhelden Lead Processing**: Automated lead routing and enrichment
4. **Horizontal Scaling**: Distributed locking (Redis Cluster)
5. **Metrics & Monitoring**: Prometheus exporters
6. **Health Checks**: `/health` endpoint via KONG
### Considered
1. **PostgreSQL Hub**: Persistent sync state (currently Redis-only)
2. **Webhook Signatures**: Validation von Vermieterhelden/3CX requests
3. **Multi-Tenant**: Support für mehrere Kanzleien
4. **KONG Plugins**: Custom plugins für business logic
1. **PostgreSQL Hub**: Persistent sync state (currently Redis-only)
2. **Webhook Signatures**: Validation von EspoCRM requests
3. **Multi-Tenant**: Support für mehrere Kanzleien
## Related Documentation
- [Development Guide](DEVELOPMENT.md)
- [API Reference](API.md)
- [Configuration](CONFIGURATION.md)
- [Troubleshooting](TROUBLESHOOTING.md)
- [Deployment Guide](DEPLOYMENT.md)