motia/bitbylaw/docs/ARCHITECTURE.md

Architektur

Systemübersicht

Das bitbylaw-System ist eine event-driven Integration zwischen Advoware, EspoCRM, Google Calendar, Vermieterhelden und 3CX Telefonie, basierend auf dem Motia Framework. Die Architektur folgt einem modularen, mikroservice-orientierten Ansatz mit klarer Separation of Concerns.

Kernkomponenten

                    ┌─────────────────────────────┐
                    │   KONG API Gateway          │
                    │   api.bitbylaw.com          │
                    │   (Auth, Rate Limiting)     │
                    └──────────────┬──────────────┘
                                   │
        ┌──────────────────────────┼──────────────────────────┐
        │                          │                          │
   ┌────▼────────┐         ┌──────▼─────────┐         ┌─────▼──────┐
   │ Vermieter-  │         │     Motia      │         │    3CX     │
   │  helden.de  │────────▶│   Framework    │◀────────│  Telefonie │
   │ (WordPress) │         │   (Middleware) │         │  (ralup)   │
   └─────────────┘         └────────┬───────┘         └────────────┘
    Leads Input                     │                   Call Handling
                                    │
        ┌───────────────────────────┼───────────────────────────┐
        │                           │                           │
   ┌────▼────┐              ┌──────▼──────┐            ┌──────▼─────┐
   │Advoware │              │     VMH     │            │  Calendar  │
   │  Proxy  │              │  Webhooks   │            │    Sync    │
   └────┬────┘              └─────┬───────┘            └─────┬──────┘
        │                         │                          │
        │                         │                          │
   ┌────▼─────────────────────────▼──────────────────────────▼────┐
   │                     Redis (3 DBs)                             │
   │  DB 1: Caching & Locks                                        │
   │  DB 2: Calendar Sync State                                    │
   └───────────────────────────────────────────────────────────────┘
        │
   ┌────▼────────────────────────────┐
   │      External Services          │
   ├─────────────────────────────────┤
   │ • Advoware REST API             │
   │ • EspoCRM (VMH)                 │
   │ • Google Calendar API           │
   │ • 3CX API (ralup.my3cx.de)      │
   │ • Vermieterhelden WordPress     │
   └─────────────────────────────────┘

Komponenten-Details

0. KONG API Gateway

Zweck: Zentraler API-Gateway für alle öffentlichen APIs mit Authentifizierung und Rate Limiting.

Domain: api.bitbylaw.com

Funktionen:

• Authentication: API-Key-basiert, JWT, OAuth2
• Rate Limiting: Pro Consumer/API-Key
• Request Routing: Zu Backend-Services (Motia, etc.)
• SSL/TLS Termination: HTTPS-Handling
• Logging & Monitoring: Request-Logs, Metrics
• CORS Handling: Cross-Origin Requests

Upstream Services:

• Motia Framework (Advoware Proxy, Calendar Sync, VMH Webhooks)
• Zukünftig: Weitere Microservices

Konfiguration:

# KONG Service Configuration
services:
  - name: motia-backend
    url: http://localhost:3000
    routes:
      - name: advoware-proxy
        paths: [/advoware/*]
      - name: calendar-sync
        paths: [/calendar/*]
      - name: vmh-webhooks
        paths: [/vmh/*]
    plugins:
      - name: key-auth
      - name: rate-limiting
        config:
          minute: 600

Flow:

Client → KONG (api.bitbylaw.com) → Auth Check → Rate Limit → Motia Backend

1. Advoware Proxy Layer

Zweck: Transparente REST-API-Proxy für Advoware mit Authentifizierung und Caching.

Module: steps/advoware_proxy/

• advoware_api_proxy_get_step.py - GET-Requests
• advoware_api_proxy_post_step.py - POST-Requests (Create)
• advoware_api_proxy_put_step.py - PUT-Requests (Update)
• advoware_api_proxy_delete_step.py - DELETE-Requests

Services: services/advoware.py

• Token-Management (HMAC-512 Authentifizierung)
• Redis-basiertes Token-Caching (55min Lifetime)
• Automatischer Token-Refresh bei 401-Errors
• Async API-Client mit aiohttp

Datenfluss:

Client → API-Step → AdvowareAPI Service → Redis (Token Cache) → Advoware API

2. Calendar Sync System

Zweck: Bidirektionale Synchronisation zwischen Advoware-Terminen und Google Calendar.

Architecture Pattern: Event-Driven Cascade

Integration: EspoCRM sendet Webhooks an KONG → Motia

Datenfluss:

EspoCRM (Vermieterhelden CRM) → KONG → Motia VMH Webhooks → Redis Dedup → Events

Cron (täglich)
  → calendar_sync_cron_step
      → Emit: "calendar_sync_all"
          → calendar_sync_all_step
              → Fetch Employees
              → For each Employee:
                  → Set Redis Lock
                  → Emit: "calendar_sync_employee"
                      → calendar_sync_event_step
                          → Fetch Advoware Events
                          → Fetch Google Events
                          → Sync (Create/Update/Delete)
                          → Clear Redis Lock

Module: steps/advoware_cal_sync/

• calendar_sync_cron_step.py - Täglicher Trigger
• calendar_sync_all_step.py - Employee-List-Handler
• calendar_sync_event_step.py - Per-Employee Sync-Logic
• calendar_sync_api_step.py - Manueller Trigger-Endpoint
• calendar_sync_utils.py - Shared Utilities
• audit_calendar_sync.py - Audit & Diagnostics

Key Features:

• Redis Locking: Verhindert parallele Syncs für denselben Employee
• Rate Limiting: Token-Bucket-Algorithm (7 tokens, Redis-based)
• Normalisierung: Common format (Berlin TZ) für beide APIs
• Error Isolation: Employee-Fehler stoppen nicht Gesamt-Sync

Datenmapping:

Advoware Format → Standard Format → Google Calendar Format
     ↓                 ↓                    ↓
datum/uhrzeitVon   start (datetime)    dateTime
datumBis           end (datetime)      dateTime
dauertermin        all_day (bool)      date
turnus/turnusArt   recurrence          RRULE

3. VMH Webhook System

Zweck: Empfang und Verarbeitung von EspoCRM Webhooks für Beteiligte-Entitäten.

Architecture Pattern: Webhook → Deduplication → Event Emission

Module: steps/vmh/

• webhook/beteiligte_create_api_step.py - Create Webhook
• webhook/beteiligte_update_api_step.py - Update Webhook
• webhook/beteiligte_delete_api_step.py - Delete Webhook
• beteiligte_sync_event_step.py - Sync Event Handler (Placeholder)

Webhook-Flow:

EspoCRM → POST /vmh/webhook/beteiligte/create
           ↓
       Webhook Step
           ↓
     Extract Entity IDs
           ↓
    Redis Deduplication (SET: vmh:beteiligte:create_pending)
           ↓
    Emit Event: "vmh.beteiligte.create"
           ↓
    Sync Event Step (subscribes)
           ↓
    [TODO: Implementierung]

### 4. Vermieterhelden Integration

**Zweck**: Lead-Eingang von Vermieterhelden.de WordPress-Frontend.

**URL**: `https://vermieterhelden.de`

**Technologie**: WordPress-basiertes Frontend

**Funktionen**:
- **Lead-Formulare**: Mieter, Vermieter, Anfragen
- **Lead-Routing**: Zu EspoCRM (VMH) → Motia
- **Webhook-basiert**: POST zu KONG/Motia bei neuem Lead

**Datenfluss**:

Vermieterhelden.de → Lead erstellt → Webhook → KONG → Motia → EspoCRM/Advoware

**Lead-Typen**:
- Mieter-Anfragen
- Vermieter-Anfragen
- Kontaktformulare
- Newsletter-Anmeldungen

**Integration mit Motia**:
- Eigener Webhook-Endpoint: `/api/leads/vermieterhelden`
- Lead-Validierung und -Enrichment
- Weiterleitung an CRM-Systeme

### 5. 3CX Telefonie-Integration

**Zweck**: Telefonie-System-Integration für Call-Handling und Lead-Qualifizierung.

**URL**: `https://ralup.my3cx.de`

**Technologie**: 3CX Cloud PBX

**Funktionen**:
- **Outbound Calls**: Lead-Anrufe (automatisch oder manuell)
- **Inbound Calls**: Stammdatenabfrage (CTI - Computer Telephony Integration)
- **Call Logging**: Anrufprotokolle zu CRM
- **Call Recording**: Aufzeichnungen speichern und abrufen
- **Screen Pops**: Kundeninfo bei eingehendem Anruf

**API-Integrationen**:

**A) Outbound: Motia → 3CX**

Motia → KONG → 3CX API

• Initiate Call to Lead
• Get Call Status

**B) Inbound: 3CX → Motia**

3CX Webhook → KONG → Motia

• Call Started → Fetch Customer Data
• Call Ended → Log Call Record

**Datenfluss**:

**Call Initiation**:

Lead in CRM → Trigger Call → Motia → 3CX API → Dial Number

**Inbound Call**:

3CX detects call → Webhook to Motia → Lookup in Advoware/EspoCRM → Return data → 3CX Screen Pop

**Call Recording**:

Call ends → 3CX Webhook → Motia → Store metadata → Link to CRM entity

**Use Cases**:
- Lead-Qualifizierung nach Eingang
- Stammdatenabfrage bei Anruf
- Anrufprotokoll in EspoCRM/Advoware
- Automatische Follow-up-Tasks

Deduplikation-Mechanismus:

• Redis SET für pending IDs pro Action-Type (create/update/delete)
• Neue IDs werden zu SET hinzugefügt
• Events nur für neue (nicht-duplizierte) IDs emittiert
• SET-TTL verhindert Memory-Leaks

Event-Driven Design

Event-Topics

Topic	Emitter	Subscriber	Payload
calendar_sync_all	cron_step	all_step	{}
calendar_sync_employee	all_step, api_step	event_step	{kuerzel, full_content}
vmh.beteiligte.create	create webhook	sync_event_step	{entity_id, action, source, timestamp}
vmh.beteiligte.update	update webhook	sync_event_step	{entity_id, action, source, timestamp}
vmh.beteiligte.delete	delete webhook	sync_event_step	{entity_id, action, source, timestamp}

Event-Flow Patterns

1. Cascade Pattern (Calendar Sync):

Trigger → Fetch List → Emit per Item → Process Item

2. Webhook Pattern (VMH):

External Event → Dedup → Internal Event → Processing

Redis Architecture

Database Layout

DB 0: Default (Motia internal) DB 1: Advoware Cache & Locks

• advoware_access_token - Bearer Token (TTL: 53min)
• advoware_token_timestamp - Token Creation Time
• calendar_sync:lock:{kuerzel} - Per-Employee Lock (TTL: 5min)
• vmh:beteiligte:create_pending - Create Dedup SET
• vmh:beteiligte:update_pending - Update Dedup SET
• vmh:beteiligte:delete_pending - Delete Dedup SET

DB 2: Calendar Sync Rate Limiting

• google_calendar_api_tokens - Token Bucket for Rate Limiting

---

External APIs

Advoware REST API

Base URL: https://advoware-api.example.com/api/v1/ Auth: HMAC-512 (siehe services/advoware.py) Rate Limits: Unknown (keine Limits bekannt) Documentation: Advoware API Swagger

Wichtige Endpoints:

• POST /auth/login - Token generieren
• GET /employees - Employee-Liste
• GET /events - Termine abrufen
• POST /events - Termin erstellen
• PUT /events/{id} - Termin aktualisieren

Redis Usage Patterns

Token Caching:

# Set with expiration
redis.set('advoware_access_token', token, ex=3180)  # 53min

# Get with fallback
token = redis.get('advoware_access_token')
if not token:
    token = fetch_new_token()

EspoCRM (VMH)

Integration: Webhook Sender (Outbound), API Consumer Endpoints: Configured in EspoCRM, routed via KONG Format: JSON POST with entity data Note: Dient als CRM für Vermieterhelden-Leads

3CX Telefonie API

Base URL: https://ralup.my3cx.de/api/v1/ Auth: API Key oder Basic Auth Rate Limits: Unknown (typisch 60 req/min)

Key Endpoints:

• POST /calls/initiate - Anruf starten
• GET /calls/{id}/status - Call-Status
• GET /calls/{id}/recording - Aufzeichnung abrufen
• POST /webhook - Webhook-Konfiguration (eingehend)

Webhooks (Inbound von 3CX):

• call.started - Anruf beginnt
• call.ended - Anruf beendet
• call.transferred - Anruf weitergeleitet

Vermieterhelden

Integration: Webhook Sender (Lead-Eingang) Base: WordPress mit Custom Plugins Format: JSON POST zu Motia

Webhook-Events:

• lead.created - Neuer Lead
• contact.submitted - Kontaktformular lock_key = f'calendar_sync🔒{kuerzel}' if not redis.set(lock_key, '1', nx=True, ex=300): raise LockError("Already locked")

Always release

redis.delete(lock_key)

**Deduplication**:
```python
# Check & Add atomically
existing = redis.smembers('vmh:beteiligte:create_pending')
new_ids = input_ids - existing
if new_ids:
    redis.sadd('vmh:beteiligte:create_pending', *new_ids)

Service Layer

AdvowareAPI Service

Location: services/advoware.py

Responsibilities:

• HMAC-512 Authentication
• Token Management
• HTTP Client (aiohttp)
• Error Handling & Retries

Key Methods:

get_access_token(force_refresh=False) -> str
api_call(endpoint, method, params, json_data) -> Any

Authentication Flow:

1. Generate HMAC-512 signature
   - Message: "{product_id}:{app_id}:{nonce}:{timestamp}"
   - Key: Base64-decoded API Key
   - Hash: SHA512

2. POST to security.advo-net.net/api/v1/Token
   - Body: {AppID, User, Password, HMAC512Signature, ...}
   
3. Extract access_token from response

4. Cache in Redis (53min TTL)

5. Use as Bearer Token: "Authorization: Bearer {token}"

External API Integration

Advoware API

Base URL: https://www2.advo-net.net:90/ Auth: HMAC-512 + Bearer Token Rate Limits: Unknown (robust error handling)

Key Endpoints:

• /employees - Mitarbeiter-Liste
• /appointments - Termine

Google Calendar API

Auth: Service Account (JSON Key) Rate Limits: 600 requests/minute (enforced via Redis) Scopes: https://www.googleapis.com/auth/calendar

Key Operations:

• calendars().get() - Calendar abrufen
• calendars().insert() - Calendar erstellen
• events().list() - Events abrufen
• events().insert() - Event erstellen
• KONG Gateway**: API-Key oder JWT-based Auth für externe Clients Advoware: User-based Auth (ADVOWARE_USER + PASSWORD) Google: Service Account (domain-wide delegation) 3CX: API Key oder Basic Auth Redis: Localhost only (no password) Vermieterhelden: Webhook-Secret für Validation

EspoCRM

Integration: Webhook Sender (Outbound) Endpoints: Configured in EspoCRM Format: JSON POST with entity data

Security

Secrets Management

Environment Variables:

ADVOWARE_API_KEY          # Base64-encoded HMAC Key
ADVOWARE_PASSWORD         # User Password
GOOGLE_CALENDAR_SERVICE_ACCOUNT_PATH  # Path to JSON Key
ESPOCRM_MARVIN_API_KEY   # Webhook Validation (optional)

Storage:

• Environment variables in systemd service
• Service Account JSON: /opt/motia-app/service-account.json (chmod 600)
• No secrets in code or Git

Access Control

Advoware: User-based Auth (ADVOWARE_USER + PASSWORD) Google: Service Account (domain-wide delegation) Redis: Localhost only (no password)

Performance Characteristics

Throughput

Calendar Sync:

• ~10 employees: 2-3 minutes
• Rate-limited by Google API (600 req/min)
• Per-employee parallelization: Nein (sequential via events)

Webhooks:

• Instant processing (<100ms)
• Batch support (multiple entities per request)
• Redis dedup overhead: <10ms

Memory Usage

Current: 169MB (Peak: 276MB) Breakdown:

• Node.js process: ~150MB
• Python dependencies: Lazy-loaded per step
• Redis memory: <10MB

Scalability

Horizontal: Nicht ohne weiteres möglich (Redis Locks, Shared State) Vertical: CPU-bound bei vielen parallel Employees Bottleneck: Google Calendar API Rate Limits

Monitoring & Observability

Logging

Framework: Motia Workbench (structured logging) Levels: DEBUG, INFO, ERROR Output: journalctl (systemd) + Motia Workbench UI

Key Log Points:

• API-Requests (Method, URL, Status)
• Event Emission (Topic, Payload)
• Redis Operations (Keys, Success/Failure)
• Errors (Stack traces, Context)

Metrics

Available (via Logs):

• Webhook receive count
• Calendar sync duration per employee
• API call count & latency
• Redis hit/miss ratio (implicit)

Missing (Future):

• Prometheus metrics
• Grafana dashboards
• Alerting

Deployment

systemd Service

Unit: motia.service User: www-data WorkingDirectory: /opt/motia-app/bitbylaw Restart: always (10s delay)

Environment:

NODE_ENV=production
NODE_OPTIONS=--max-old-space-size=8192 --inspect
HOST=0.0.0.0
MOTIA_LOG_LEVEL=debug

Dependencies

Runtime:

• Node.js 18+
• Python 3.13+
• Redis Server
• systemd

Build:

• npm (Node packages)
• pip (Python packages)
• Motia CLI

Disaster Recovery

Backup Strategy

Redis:

• RDB snapshots (automatisch)
• AOF persistence (optional)

Configuration:

• Git-versioniert
• Environment Variables in systemd

Service Account:

• Manual backup: /opt/motia-app/service-account.json

Recovery Procedures

Service Restart:

systemctl restart motia.service

Clear Redis Cache:

redis-cli -n 1 FLUSHDB  # Advoware Cache
redis-cli -n 2 FLUSHDB  # Calendar Sync

Clear Employee Lock:

python /opt/motia-app/bitbylaw/delete_employee_locks.py

Future Enhancements

P3CX Full Integration**: Complete call handling, CTI features

3. Vermieterhelden Lead Processing: Automated lead routing and enrichment
4. Horizontal Scaling: Distributed locking (Redis Cluster)
5. Metrics & Monitoring: Prometheus exporters
6. Health Checks: /health endpoint via KONG

Considered

1. PostgreSQL Hub: Persistent sync state (currently Redis-only)
2. Webhook Signatures: Validation von Vermieterhelden/3CX requests
3. Multi-Tenant: Support für mehrere Kanzleien
4. KONG Plugins: Custom plugins für business logic
5. PostgreSQL Hub: Persistent sync state (currently Redis-only)
6. Webhook Signatures: Validation von EspoCRM requests
7. Multi-Tenant: Support für mehrere Kanzleien

Related Documentation

• Development Guide
• API Reference
• Configuration
• Troubleshooting
• Deployment Guide