Add comprehensive test scripts for thumbnail generation and xAI collections API

- Implemented `test_thumbnail_generation.py` to validate the complete flow of document thumbnail generation in EspoCRM, including document creation, file upload, webhook triggering, and preview verification.
- Created `test_xai_collections_api.py` to test critical operations of the xAI Collections API, covering file uploads, collection CRUD operations, document management, and response validation.
- Both scripts include detailed logging for success and error states, ensuring robust testing and easier debugging.

MIGRATION_COMPLETE_ANALYSIS.md

@@ -1,320 +0,0 @@
-# Vollständige Migrations-Analyse
-## Motia v0.17 → Motia III v1.0-RC
-
-**Datum:** 1. März 2026  
-**Status:** 🎉 **100% KOMPLETT - ALLE PHASEN ABGESCHLOSSEN!** 🎉
-
----
-
-## ✅ MIGRIERT - Production-Ready
-
-### 1. Steps (21 von 21 Steps - 100% Complete!)
-
-#### Phase 1: Advoware Proxy (4 Steps)
-- ✅ [`advoware_api_proxy_get_step.py`](steps/advoware_proxy/advoware_api_proxy_get_step.py) - GET Proxy
-- ✅ [`advoware_api_proxy_post_step.py`](steps/advoware_proxy/advoware_api_proxy_post_step.py) - POST Proxy
-- ✅ [`advoware_api_proxy_put_step.py`](steps/advoware_proxy/advoware_api_proxy_put_step.py) - PUT Proxy
-- ✅ [`advoware_api_proxy_delete_step.py`](steps/advoware_proxy/advoware_api_proxy_delete_step.py) - DELETE Proxy
-
-#### Phase 2: VMH Webhooks (6 Steps)
-- ✅ [`beteiligte_create_api_step.py`](steps/vmh/webhook/beteiligte_create_api_step.py) - POST /vmh/webhook/beteiligte/create
-- ✅ [`beteiligte_update_api_step.py`](steps/vmh/webhook/beteiligte_update_api_step.py) - POST /vmh/webhook/beteiligte/update
-- ✅ [`beteiligte_delete_api_step.py`](steps/vmh/webhook/beteiligte_delete_api_step.py) - POST /vmh/webhook/beteiligte/delete
-- ✅ [`bankverbindungen_create_api_step.py`](steps/vmh/webhook/bankverbindungen_create_api_step.py) - POST /vmh/webhook/bankverbindungen/create
-- ✅ [`bankverbindungen_update_api_step.py`](steps/vmh/webhook/bankverbindungen_update_api_step.py) - POST /vmh/webhook/bankverbindungen/update
-- ✅ [`bankverbindungen_delete_api_step.py`](steps/vmh/webhook/bankverbindungen_delete_api_step.py) - POST /vmh/webhook/bankverbindungen/delete
-
-#### Phase 3: VMH Sync Handlers (3 Steps)
-- ✅ [`beteiligte_sync_event_step.py`](steps/vmh/beteiligte_sync_event_step.py) - Subscriber für Queue-Events (mit Kommunikation-Integration!)
-- ✅ [`bankverbindungen_sync_event_step.py`](steps/vmh/bankverbindungen_sync_event_step.py) - Subscriber für Queue-Events
-- ✅ [`beteiligte_sync_cron_step.py`](steps/vmh/beteiligte_sync_cron_step.py) - Cron-Job alle 15 Min.
-
----
-
-### 2. Services (11 Module, 100% komplett)
-
-#### Core APIs
-- ✅ [`advoware.py`](services/advoware.py) (310 Zeilen) - Advoware API Client mit Token-Auth
-- ✅ [`advoware_service.py`](services/advoware_service.py) (179 Zeilen) - High-Level Advoware Service
-- ✅ [`espocrm.py`](services/espocrm.py) (293 Zeilen) - EspoCRM API Client
-
-#### Mapper & Sync Utils  
-- ✅ [`espocrm_mapper.py`](services/espocrm_mapper.py) (663 Zeilen) - Beteiligte Mapping
-- ✅ [`bankverbindungen_mapper.py`](services/bankverbindungen_mapper.py) (141 Zeilen) - Bankverbindungen Mapping
-- ✅ [`beteiligte_sync_utils.py`](services/beteiligte_sync_utils.py) (663 Zeilen) - Distributed Locking, Retry Logic
-- ✅ [`notification_utils.py`](services/notification_utils.py) (200 Zeilen) - In-App Notifications
-
-#### Phase 4: Kommunikation Sync
-- ✅ [`kommunikation_mapper.py`](services/kommunikation_mapper.py) (334 Zeilen) - Email/Phone Mapping mit Base64 Marker
-- ✅ [`kommunikation_sync_utils.py`](services/kommunikation_sync_utils.py) (999 Zeilen) - Bidirektionaler Sync mit 3-Way Diffing
-
-#### Phase 5: Adressen Sync (2 Module - Phase 5)
-- ✅ [`adressen_mapper.py`](services/adressen_mapper.py) (267 Zeilen) - Adressen Mapping
-- ✅ [`adressen_sync.py`](services/adressen_sync.py) (697 Zeilen) - Adressen Sync mit READ-ONLY Detection
-
-#### Phase 6: Google Calendar Sync (4 Steps + Utils)
-- ✅ [`calendar_sync_cron_step.py`](steps/advoware_cal_sync/calendar_sync_cron_step.py) - Cron-Trigger alle 15 Min.
-- ✅ [`calendar_sync_all_step.py`](steps/advoware_cal_sync/calendar_sync_all_step.py) - Bulk-Sync mit Redis-Priorisierung
-- ✅ [`calendar_sync_event_step.py`](steps/advoware_cal_sync/calendar_sync_event_step.py) - **1053 Zeilen!** Main Sync Handler
-- ✅ [`calendar_sync_a9 Topics - 100% Complete!)
-
-#### VMH Beteiligte
-- ✅ `vmh.beteiligte.create` - Webhook → Sync Handler
-- ✅ `vmh.beteiligte.update` - Webhook → Sync Handler
-- ✅ `vmh.beteiligte.delete` - Webhook → Sync Handler
-- ✅ `vmh.beteiligte.sync_check` - Cron → Sync Handler
-
-#### VMH Bankverbindungen  
-- ✅ `vmh.bankverbindungen.create` - Webhook → Sync Handler
-- ✅ `vmh.bankverbindungen.update` - Webhook → Sync Handler
-- ✅ `vmh.bankverbindungen.delete` - Webhook → Sync Handler
-
-#### Calendar Sync
-- ✅ `calendar_sync_all` - Cron/API → All Step → Employee Events
-- ✅ `calendar_sync_employee` - All/API → Event Step (Main Sync Logic)
-
----
-
-### 4. HTTP Endpoints (14 Endpoints - 100% Complete!
-- ✅ `vmh.bankverbindungen.create` - Webhook → Sync Handler
-- ✅ `vmh.bankverbindungen.update` - Webhook → Sync Handler
-- ✅ `vmh.bankverbindungen.delete` - Webhook → Sync Handler
-
----
-
-### 4. HTTP Endpoints (13 Endpoints, 100% komplett)
-
-#### Advoware Proxy (4 Endpoints)
-- ✅ `GET /advoware/proxy?path=...` - Advoware API Proxy
-- ✅ `POST /advoware/proxy?path=...` - Advoware API Proxy
-- ✅ `PUT /advoware/proxy?path=...` - Advoware API Proxy
-- ✅ `DELETE /advoware/proxy?path=...` - Advoware API Proxy
-
-#### VMH Webhooks - Beteiligte (3 Endpoints)
-- ✅ `POST /vmh/webhook/beteiligte/create` - EspoCRM Webhook Handler
-- ✅ `POST /vmh/webhook/beteiligte/update` - EspoCRM Webhook Handler
-- ✅ `POST /vmh/webhook/beteiligte/delete` - EspoCRM Webhook Handler
-
-#### VMH Webhooks - Bankverbindungen (3 Endpoints)
-- ✅ `Calendar Sync (1 Endpoint)
-- ✅ `POST /advoware/calendar/sync` - Manual Calendar Sync Trigger (kuerzel or "ALL")
-
-#### POST /vmh/webhook/bankverbindungen/create` - EspoCRM Webhook Handler
-- ✅ `POST /vmh/webhook/bankverbindungen/update` - EspoCRM Webhook Handler
-- ✅ `POST /vmh/webhook/bankverbindungen/delete` - EspoCRM Webhook Handler
-
-#### Example Ticketing (6 Endpoints - Demo)
-- ✅ `POST /tickets` - Create Ticket
-- ✅ `GET /tickets` - List Tickets
-- ✅ `POST /tickets/{id}/triage` - Triage
-- ✅ `POST /tickets/{id}/escalate` - Escalate
-- ✅ `POST /tickets/{id}/notify` - Notify Customer
-- ✅ Cron: SLA Monitor
-2 Jobs - 100% Complete!)
-
-- ✅ **VMH Beteiligte Sync Cron** (alle 15 Min.)
-  - Findet Entities mit Status: `pending_sync`, `dirty`, `failed`
-  - Auto-Reset für `permanently_failed` nach 24h
-  - Findet `clean` Entities > 24h nicht gesynct
-  - Emittiert `vmh.beteiligte.sync_check` Events
-
-- ✅ **Calendar Sync Cron** (alle 15 Min.)
-  - Emittiert `calendar_sync_all` Events
-  - Triggered Bulk-Sync für alle oder priorisierte Mitarbeiter
-  - Redis-basierte Priorisierung (älteste zuerst)
-
----
-
-### 6. Dependencies (pyproject.toml - 100% Complete!
----
-
-### 6. Dependencies (pyproject.toml aktualisiert)
-
-```toml
-dependencies = [
-    "asyncpg>=0.29.0",  # ✅ NEU für Calendar Sync (PostgreSQL)
-    "google-api-python-client>=2.100.0",  # ✅ NEU für Calendar Sync
-    "google-auth>=2.23.0",  # ✅ NEU für Calendar Sync
-    "backoff>=2.2.1",  # ✅ NEU für Calendar Sync (Retry Logic)
-]
-```
-
----
-
-## ❌ NICHT MIGRIERT → ALLE MIGRIERT! 🎉
-
-~~### Phase 6: Google Calendar Sync (4 Steps)~~
-
-**Status:** ✅ **VOLLSTÄNDIG MIGRIERT!** (1. März 2026)
-
-- ✅ `calendar_sync_cron_step.py` - Cron-Trigger (alle 15 Min.)
-- ✅ `calendar_sync_all_step.py` - Bulk-Sync Handler  
-- ✅ `calendar_sync_event_step.py` - Queue-Event Handler (**1053 Zeilen!**)
-- ✅ `calendar_sync_api_step.py` - HTTP API für manuellen Trigger
-- ✅ `calendar_sync_utils.py` - Hilfs-Funktionen
-
-**Dependencies (ALLE installiert):**
-- ✅ `google-api-python-client` - Google Calendar API
-- ✅ `google-auth` - Google OAuth2
-- ✅ `asyncpg` - PostgreSQL Connection
-- ✅ `backoff` - Retry/Backoff Logic
-
-**Migration abgeschlossen in:** ~4 Stunden (statt geschätzt 3-5 Tage
-
-**Dependencies (nicht benötigt):**
-- ❌ `google-api-python-client` - Google Calendar API
-- ❌ `google-auth` - Google OAuth2
-- ❌ PostgreSQL Connection - Für Termine-Datenbank
-
-**Geschätzte Migration:** 3-5 Tage (komplex wegen Google API + PostgreSQL)  
-**Priorität:** MEDIUM (funktioniert aktuell im old-motia)
-
----
-
-### Root-Level Steps (Test/Specialized Logic)
-
-**Status:** Bewusst NICHT migriert (nicht Teil der Core-Funktionalität)
-
-- ❌ `/opt/motia-iii/old-motia/steps/crm-bbl-vmh-reset-nextcall_step.py` (96 Zeilen)
-  - **Zweck:** CVmhErstgespraech Status-Check Cron-Job
-  - **Grund:** Spezialisierte Business-Logik, nicht Teil der Core-Sync-Infrastruktur
-  - **Status:** Kann bei Bedarf später migriert werden
-
-- ❌ `/opt/motia-iii/old-motia/steps/event_step.py` (Test/Demo)
-- ❌ `/opt/motia-iii/old-motia/steps/hello_step.py` (Test/Demo)
-
----
-
-## 📊 Migrations-Statistik
-
-| Kategorie | Migriert | 21 | 0 | 21 | **100%** ✅ |
-| **Service Module** | 11 | 0 | 11 | **100%** ✅ |
-| **Queue Events** | 9 | 0 | 9 | **100%** ✅ |
-| **HTTP Endpoints** | 14 | 0 | 14 | **100%** ✅ |
-| **Cron Jobs** | 2 | 0 | 2 | **100%** ✅ |
-| **Code (Zeilen)** | ~9.000 | 0 | ~9.000 | **100%** ✅ |
-
----
-
-## 🎯 Funktionalitäts-Matrix
-
-| Feature | Old-Motia | Motia III | Status |
-|---------|-----------|-----------|--------|
-| **Advoware Proxy API** | ✅ | ✅ | ✅ KOMPLETT |
-| **VMH Beteiligte Sync** | ✅ | ✅ | ✅ KOMPLETT |
-| **VMH Bankverbindungen Sync** | ✅ | ✅ | ✅ KOMPLETT |
-| **Kommunikation Sync (Email/Phone)** | ✅ | ✅ | ✅ KOMPLETT |
-| **Adressen Sync** | ✅ | ✅ | ✅ KOMPLETT |
-| **EspoCRM Webhooks** | ✅ | ✅ | ✅ KOMPLETT |
-| **Distributed Locking** | ✅ | ✅ | ✅ KOMPLETT |
-| **Retry Logic & Backoff** | ✅ | ✅ | ✅ KOMPLETT |
-| **Notifications** | ✅ | ✅ | ✅ KOMPLETT |
-| **Sync Validation** | ✅ | ✅ | ✅ KOMPLETT |
-| **Cron-basierter Auto-Retry** | ✅ | ✅ | ✅ KOMPLETT |
-| **Google Calendar Sync** | ✅ | ✅ | ✅ **KOMPLETT** |
-
----
-
-## 🏆 Migration erfolgreich abgeschlossen!
-
-**Alle 21 Production Steps, 11 Service Module, 9 Queue Events, 14 HTTP Endpoints und 2 Cron Jobs wurden erfolgreich migriert!**
-| **Cron-basierter Auto-Retry** | ✅ | ✅ | ✅ KOMPLETT |
-| **Google Calendar Sync** | ✅ | ❌ | ⏳ PHASE 6 |
-| **CVmhErstgespraech Logic** | ✅ | ❌ | ⏳ Optional |
-
----
-
-## 🔄 Sync-Architektur Übersicht
-
-```
-┌─────────────────┐
-│   EspoCRM API   │
-└────────┬────────┘
-         │ Webhooks
-         ▼
-┌─────────────────────────────────────┐
-│  VMH Webhook Steps (6 Endpoints)    │
-│  • Batch & Single Entity Support    │
-│  • Deduplication                     │
-└────────┬────────────────────────────┘
-         │ Emits Queue Events
-         ▼
-┌─────────────────────────────────────┐
-│  Queue System (Redis/Builtin)       │
-│  • vmh.beteiligte.*                 │
-│  • vmh.bankverbindungen.*           │
-└────────┬────────────────────────────┘
-         │
-         ▼
-┌─────────────────────────────────────┐
-│  Sync Event Handlers (3 Steps)      │
-│  • Distributed Locking (Redis)      │
-│  • Retry Logic & Backoff            │
-│  • Conflict Resolution              │
-└────────┬────────────────────────────┘
-         │
-         ├──► Stammdaten Sync
-         │    (espocrm_mapper.py)
-         │
-         ├──► Kommunikation Sync ✅ NEW!
-         │    (kommunikation_sync_utils.py)
-         │    • 3-Way Diffing
-         │    • Bidirectional
-         │    • Slot-Management
-         │
-         └──► Adressen Sync ✅ NEW!
-              (adressen_sync.py)
-              • CREATE/UPDATE/DELETE
-              • READ-ONLY Detection
-         
-         ▼
-┌─────────────────────────────────────┐
-│  Advoware API (advoware.py)         │
-│  • Token-based Auth                 │
-│  • HMAC Signing                     │
-└─────────────────────────────────────┘
-
-         ┌──────────────────┐
-         │  Cron Job (15min)│
-         └────────┬─────────┘
-                  │
-                  ▼ Emits sync_check Events
-         ┌─────────────────────────┐
-         │  Auto-Retry & Cleanup   │
-         │  • pending_sync         │
-         │  • dirty                │
-         │  • failed → retry       │
-         │  • permanently_failed   │
-         │    → auto-reset (24h)   │
-         └─────────────────────────┘
-```
-
----
-
-## ✅ FAZIT
-
-**Die gesamte Core-Funktionalität (außer Google Calendar) wurde erfolgreich migriert!**
-
-### Production-Ready Features:
-1. ✅ Vollständige Advoware ↔ EspoCRM Synchronisation
-2. ✅ Bidirektionale Kommunikationsdaten (Email/Phone)
-3. ✅ Bidirektionale Adressen
-4. ✅ Webhook-basierte Event-Verarbeitung
-5. ✅ Automatisches Retry-System
-6. ✅ Distributed Locking
-7. ✅ Konflikt-Erkennung & Resolution
-
-### Code-Qualität:
-- ✅ Keine Compile-Errors
-- ✅ Motia III API korrekt verwendet
-- ✅ Alle Dependencies vorhanden
-- ✅ Type-Hints (Pydantic Models)
-- ✅ Error-Handling & Logging
-
-### Deployment:
-- ✅ Alle Steps registriert
-- ✅ Queue-System konfiguriert
-- ✅ Cron-Jobs aktiv
-- ✅ Redis-Integration
-
-**Das System ist bereit für Production! 🚀**

MIGRATION_STATUS.md

@@ -1,276 +0,0 @@
-# Motia Migration Status
-
-**🎉 MIGRATION 100% KOMPLETT**
-
-> 📋 Detaillierte Analyse: [MIGRATION_COMPLETE_ANALYSIS.md](MIGRATION_COMPLETE_ANALYSIS.md)
-
-## Quick Stats
-
-- ✅ **21 von 21 Steps** migriert (100%)
-- ✅ **11 von 11 Service-Module** migriert (100%)
-- ✅ **~9.000 Zeilen Code** migriert (100%)
-- ✅ **14 HTTP Endpoints** aktiv
-- ✅ **9 Queue Events** konfiguriert
-- ✅ **2 Cron Jobs** (VMH: alle 15 Min., Calendar: alle 15 Min.)
-
----
-
-## Overview
-Migrating from **old-motia v0.17** (Node.js + Python hybrid) to **Motia III v1.0-RC** (pure Python).
-
-## Old System Analysis
-
-### Location
-- Old system: `/opt/motia-iii/old-motia/`
-- Old project dir: `/opt/motia-iii/old-motia/bitbylaw/`
-
-### Steps Found in Old System
-
-#### Root Steps (`/opt/motia-iii/old-motia/steps/`)
-1. `crm-bbl-vmh-reset-nextcall_step.py`
-2. `event_step.py`
-3. `hello_step.py`
-
-#### BitByLaw Steps (`/opt/motia-iii/old-motia/bitbylaw/steps/`)
-
-**Advoware Calendar Sync** (`advoware_cal_sync/`):
-- `calendar_sync_all_step.py`
-- `calendar_sync_api_step.py`
-- `calendar_sync_cron_step.py`
-- `calendar_sync_event_step.py`
-- `audit_calendar_sync.py`
-- `calendar_sync_utils.py` (utility module)
-
-**Advoware Proxy** (`advoware_proxy/`):
-- `advoware_api_proxy_get_step.py`
-- `advoware_api_proxy_post_step.py`
-- `advoware_api_proxy_put_step.py`
-- `advoware_api_proxy_delete_step.py`
-
-**VMH Integration** (`vmh/`):
-- `beteiligte_sync_cron_step.py`
-- `beteiligte_sync_event_step.py`
-- `bankverbindungen_sync_event_step.py`
-- `webhook/bankverbindungen_create_api_step.py`
-- `webhook/bankverbindungen_update_api_step.py`
-- `webhook/bankverbindungen_delete_api_step.py`
-- `webhook/beteiligte_create_api_step.py`
-- `webhook/beteiligte_update_api_step.py`
-- `webhook/beteiligte_delete_api_step.py`
-
-### Supporting Services/Modules
-
-From `/opt/motia-iii/old-motia/bitbylaw/`:
-- `services/advoware.py` - Advoware API wrapper
-- `config.py` - Configuration module
-- Dependencies: PostgreSQL, Redis, Google Calendar API
-
-## Migration Changes Required
-
-### Key Structural Changes
-
-#### 1. Config Format
-```python
-# OLD
-config = {
-    "type": "api",  # or "event", "cron"
-    "name": "StepName",
-    "path": "/endpoint",
-    "method": "GET",
-    "cron": "0 5 * * *",
-    "subscribes": ["topic"],
-    "emits": ["other-topic"]
-}
-
-# NEW
-from motia import http, queue, cron
-
-config = {
-    "name": "StepName",
-    "flows": ["flow-name"],
-    "triggers": [
-        http("GET", "/endpoint")
-        # or queue("topic", input=schema)
-        # or cron("0 0 5 * * *")  # 6-field!
-    ],
-    "enqueues": ["other-topic"]
-}
-```
-
-#### 2. Handler Signature
-```python
-# OLD - API
-async def handler(req, context):
-    body = req.get('body', {})
-    await context.emit({"topic": "x", "data": {...}})
-    return {"status": 200, "body": {...}}
-
-# NEW - API
-from motia import ApiRequest, ApiResponse, FlowContext
-
-async def handler(request: ApiRequest, ctx: FlowContext) -> ApiResponse:
-    body = request.body
-    await ctx.enqueue({"topic": "x", "data": {...}})
-    return ApiResponse(status=200, body={...})
-
-# OLD - Event/Queue
-async def handler(data, context):
-    context.logger.info(data['field'])
-
-# NEW - Queue
-async def handler(input_data: dict, ctx: FlowContext):
-    ctx.logger.info(input_data['field'])
-
-# OLD - Cron
-async def handler(context):
-    context.logger.info("Running")
-
-# NEW - Cron
-async def handler(input_data: dict, ctx: FlowContext):
-    ctx.logger.info("Running")
-```
-
-#### 3. Method Changes
-- `context.emit()` → `ctx.enqueue()`
-- `req.get('body')` → `request.body`
-- `req.get('queryParams')` → `request.query_params`
-- `req.get('pathParams')` → `request.path_params`
-- `req.get('headers')` → `request.headers`
-- Return dict → `ApiResponse` object
-
-#### 4. Cron Format
-- OLD: 5-field `"0 5 * * *"` (minute hour day month weekday)
-- NEW: 6-field `"0 0 5 * * *"` (second minute hour day month weekday)
-
-## Migration Strategy
-
-### Phase 1: Simple Steps (Priority)
-Start with simple API proxy steps as they're straightforward:
-1. ✅ Example ticketing steps (already in new system)
-2. ⏳ Advoware proxy steps (GET, POST, PUT, DELETE)
-3. ⏳ Simple webhook handlers
-
-### Phase 2: Complex Integration Steps
-Steps with external dependencies:
-4. ⏳ VMH sync steps (beteiligte, bankverbindungen)
-5. ⏳ Calendar sync steps (most complex - Google Calendar + Redis + PostgreSQL)
-
-### Phase 3: Supporting Infrastructure
-- Migrate `services/` modules (advoware.py wrapper)
-- Migrate `config.py` to use environment variables properly
-- Update dependencies in `pyproject.toml`
-
-### Dependencies to Review
-From old `requirements.txt` and code analysis:
-- `asyncpg` - PostgreSQL async driver
-- `redis` - Redis client
-- `google-api-python-client` - Google Calendar API
-- `google-auth` - Google OAuth2
-- `backoff` - Retry/backoff decorator
-- `pytz` - Timezone handling
-- `pydantic` - Already in new system
-- `requests` / `aiohttp` - HTTP clients for Advoware API
-
-## Migration Roadmap
-
-### ✅ COMPLETED
-
-| Phase | Module | Lines | Status |
-|-------|--------|-------|--------|
-| **1** | Advoware Proxy (GET, POST, PUT, DELETE) | ~400 | ✅ Complete |
-| **1** | `advoware.py`, `advoware_service.py` | ~800 | ✅ Complete |
-| **2** | VMH Webhook Steps (6 endpoints) | ~900 | ✅ Complete |
-| **2** | `espocrm.py`, `espocrm_mapper.py` | ~900 | ✅ Complete |
-| **2** | `bankverbindungen_mapper.py`, `beteiligte_sync_utils.py`, `notification_utils.py` | ~1200 | ✅ Complete |
-| **3** | VMH Sync Event Steps (2 handlers + 1 cron) | ~1000 | ✅ Complete |
-| **4** | Kommunikation Sync (`kommunikation_mapper.py`, `kommunikation_sync_utils.py`) | ~1333 | ✅ Complete |
-| **5** | Adressen Sync (`adressen_mapper.py`, `adressen_sync.py`) | ~964 | ✅ Complete |
-| **6** | **Google Calendar Sync** (`calendar_sync_*.py`, `calendar_sync_utils.py`) | ~1500 | ✅ **Complete** |
-
-**Total migrated: ~9.000 lines of production code**
-
-### ✅ Phase 6 COMPLETED: Google Calendar Sync
-
-**Advoware Calendar Sync** - Google Calendar ↔ Advoware Sync:
-- ✅ `calendar_sync_cron_step.py` - Cron-Trigger (alle 15 Min.)
-- ✅ `calendar_sync_all_step.py` - Bulk-Sync Handler mit Redis-basierter Priorisierung
-- ✅ `calendar_sync_event_step.py` - Queue-Event Handler (**1053 Zeilen komplexe Sync-Logik!**)
-- ✅ `calendar_sync_api_step.py` - HTTP API für manuellen Trigger
-- ✅ `calendar_sync_utils.py` - Hilfs-Funktionen (DB, Google Service, Redis, Logging)
-
-**Dependencies:**
-- ✅ `google-api-python-client` - Google Calendar API
-- ✅ `google-auth` - Google OAuth2
-- ✅ `asyncpg` - PostgreSQL async driver
-- ✅ `backoff` - Retry/backoff decorator
-
-**Features:**
-- ✅ Bidirektionale Synchronisation (Google ↔ Advoware)
-- ✅ 4-Phase Sync-Algorithmus (New Adv→Google, New Google→Adv, Deletes, Updates)
-- ✅ PostgreSQL als Sync-State Hub (calendar_sync Tabelle)
-- ✅ Redis-basiertes Rate Limiting (Token Bucket für Google API)
-- ✅ Distributed Locking per Employee
-- ✅ Automatische Calendar-Creation mit ACL
-- ✅ Recurring Events Support (RRULE)
-- ✅ Timezone-Handling (Europe/Berlin)
-- ✅ Backoff-Retry für API-Fehler
-- ✅ Write-Protection für Advoware
-- ✅ Source-System-Wins & Last-Change-Wins Strategien
-
-### ⏳ REMAINING
-
-**Keine! Die Migration ist zu 100% abgeschlossen.**
-
-### Completed
-- ✅ Analysis of old system structure
-- ✅ MIGRATION_GUIDE.md reviewed
-- ✅ Migration patterns documented
-- ✅ New system has example ticketing steps
-- ✅ **Phase 1: Advoware Proxy Steps migrated** (GET, POST, PUT, DELETE)
-- ✅ **Advoware API service module migrated** (services/advoware.py)
-- ✅ **Phase 2: VMH Integration - Webhook Steps migrated** (6 endpoints)
-- ✅ **EspoCRM API service module migrated** (services/espocrm.py)
-- ✅ All endpoints registered and running:
-  - **Advoware Proxy:**
-    - `GET /advoware/proxy6 Complete ✅
-
-**🎉 ALLE PHASEN ABGESCHLOSSEN! 100% MIGRATION ERFOLGREICH!**
-
-**Phase 6** - Google Calendar Sync:
-- ✅ `calendar_sync_cron_step.py` (Cron-Trigger alle 15 Min.)
-- ✅ `calendar_sync_all_step.py` (Bulk-Handler mit Redis-Priorisierung)
-- ✅ `calendar_sync_event_step.py` (1053 Zeilen - 4-Phase Sync-Algorithmus)
-- ✅ `calendar_sync_api_step.py` (HTTP API für manuelle Triggers)
-- ✅ `calendar_sync_utils.py` (DB, Google Service, Redis Client)
-
-**Sync-Architektur komplett:**
-
-1. **Advoware Proxy** (Phase 1) → HTTP API für Advoware-Zugriff
-2. **Webhooks** (Phase 2) → Emittieren Queue-Events
-3. **Event Handler** (Phase 3) → Verarbeiten Events mit Stammdaten-Sync
-4. **Kommunikation Sync** (Phase 4) → Bidirektionale Email/Phone-Synchronisation
-5. **Adressen Sync** (Phase 5) → Bidirektionale Adressen-Synchronisation
-6. **Calendar Sync** (Phase 6) → Google Calendar ↔ Advoware Bidirektional
-7. **Cron Jobs** (Phase 3 & 6) → Regelmäßige Sync-Checks & Auto-Retries
-
-Die vollständige Synchronisations- und Integrations-Pipeline ist nun zu 100%
-**Phase 5** - Adressen Sync:
-- ✅ `adressen_mapper.py` (267 Zeilen - CAdressen ↔ Advoware Adressen)
-- ✅ `adressen_sync.py` (697 Zeilen - CREATE/UPDATE mit READ-ONLY Detection)
-
-### Sync-Architektur komplett:
-
-1. **Webhooks** (Phase 2) → Emittieren Queue-Events
-2. **Event Handler** (Phase 3) → Verarbeiten Events mit Stammdaten-Sync
-3. **Kommunikation Sync** (Phase 4) → Bidirektionale Email/Phone-Synchronisation
-4. **Adressen Sync** (Phase 5) → Bidirektionale Adressen-Synchronisation
-5. **Cron Job** (Phase 3) → Regelmäßige Sync-Checks & Auto-Retries
-
-Die vollständige Synchronisations-Pipeline ist nun einsatzbereit!
-
-## Notes
-- Old system was Node.js + Python hybrid (Python steps as child processes)
-- New system is pure Python (standalone SDK)
-- No need for Node.js/npm anymore
-- iii engine handles all infrastructure (queues, state, HTTP, cron)
-- Console replaced Workbench

services/document_sync_utils.py

@@ -11,14 +11,11 @@ Hilfsfunktionen für Document-Synchronisation mit xAI:
 from typing import Dict, Any, Optional, List, Tuple
 from datetime import datetime, timedelta
 import logging
-import redis
-import os
+
+from services.sync_utils_base import BaseSyncUtils
 
 logger = logging.getLogger(__name__)
 
-# Lock TTL in seconds (prevents deadlocks)
-LOCK_TTL_SECONDS = 900  # 15 minutes
-
 # Max retry before permanent failure
 MAX_SYNC_RETRIES = 5
 
@@ -26,40 +23,12 @@ MAX_SYNC_RETRIES = 5
 RETRY_BACKOFF_MINUTES = [1, 5, 15, 60, 240]  # 1min, 5min, 15min, 1h, 4h
 
 
-class DocumentSync:
+class DocumentSync(BaseSyncUtils):
     """Utility-Klasse für Document-Synchronisation mit xAI"""
     
-    def __init__(self, espocrm_api, redis_client: redis.Redis = None, context=None):
-        self.espocrm = espocrm_api
-        self.context = context
-        self.logger = context.logger if context else logger
-        self.redis = redis_client or self._init_redis()
-    
-    def _init_redis(self) -> redis.Redis:
-        """Initialize Redis client for distributed locking"""
-        try:
-            redis_host = os.getenv('REDIS_HOST', 'localhost')
-            redis_port = int(os.getenv('REDIS_PORT', '6379'))
-            redis_db = int(os.getenv('REDIS_DB_ADVOWARE_CACHE', '1'))
-            
-            client = redis.Redis(
-                host=redis_host,
-                port=redis_port,
-                db=redis_db,
-                decode_responses=True
-            )
-            client.ping()
-            return client
-        except Exception as e:
-            self._log(f"Redis connection failed: {e}", level='error')
-            return None
-    
-    def _log(self, message: str, level: str = 'info'):
-        """Logging mit Context-Support"""
-        if self.context and hasattr(self.context, 'logger'):
-            getattr(self.context.logger, level)(message)
-        else:
-            getattr(logger, level)(message)
+    def _get_lock_key(self, entity_id: str) -> str:
+        """Redis Lock-Key für Documents"""
+        return f"sync_lock:document:{entity_id}"
     
     async def acquire_sync_lock(self, entity_id: str, entity_type: str = 'CDokumente') -> bool:
         """
@@ -74,13 +43,10 @@ class DocumentSync:
         """
         try:
             # STEP 1: Atomic Redis lock (prevents race conditions)
-            if self.redis:
-                lock_key = f"sync_lock:document:{entity_id}"
-                acquired = self.redis.set(lock_key, "locked", nx=True, ex=LOCK_TTL_SECONDS)
-                
-                if not acquired:
-                    self._log(f"Redis lock bereits aktiv für {entity_type} {entity_id}", level='warn')
-                    return False
+            lock_key = self._get_lock_key(entity_id)
+            if not self._acquire_redis_lock(lock_key):
+                self._log(f"Redis lock bereits aktiv für {entity_type} {entity_id}", level='warn')
+                return False
             
             # STEP 2: Update syncStatus (für UI visibility) - nur bei Document Entity
             # CDokumente hat dieses Feld nicht - überspringen
@@ -98,12 +64,8 @@ class DocumentSync:
         except Exception as e:
             self._log(f"Fehler beim Acquire Lock: {e}", level='error')
             # Clean up Redis lock on error
-            if self.redis:
-                try:
-                    lock_key = f"sync_lock:document:{entity_id}"
-                    self.redis.delete(lock_key)
-                except:
-                    pass
+            lock_key = self._get_lock_key(entity_id)
+            self._release_redis_lock(lock_key)
             return False
     
     async def release_sync_lock(
@@ -149,19 +111,14 @@ class DocumentSync:
             self._log(f"Sync-Lock released: {entity_type} {entity_id} → {'success' if success else 'failed'}")
             
             # Release Redis lock
-            if self.redis:
-                lock_key = f"sync_lock:document:{entity_id}"
-                self.redis.delete(lock_key)
+            lock_key = self._get_lock_key(entity_id)
+            self._release_redis_lock(lock_key)
             
         except Exception as e:
             self._log(f"Fehler beim Release Lock: {e}", level='error')
             # Ensure Redis lock is released even on error
-            if self.redis:
-                try:
-                    lock_key = f"sync_lock:document:{entity_id}"
-                    self.redis.delete(lock_key)
-                except:
-                    pass
+            lock_key = self._get_lock_key(entity_id)
+            self._release_redis_lock(lock_key)
     
     async def should_sync_to_xai(self, document: Dict[str, Any]) -> Tuple[bool, List[str], str]:
         """

services/sync_utils_base.py

@@ -0,0 +1,150 @@
+"""
+Base Sync Utilities
+
+Gemeinsame Funktionalität für alle Sync-Operationen:
+- Redis Distributed Locking
+- Context-aware Logging
+- EspoCRM API Helpers
+"""
+
+from typing import Dict, Any, Optional
+from datetime import datetime
+import logging
+import redis
+import os
+import pytz
+
+logger = logging.getLogger(__name__)
+
+# Lock TTL in seconds (prevents deadlocks)
+LOCK_TTL_SECONDS = 900  # 15 minutes
+
+
+class BaseSyncUtils:
+    """Base-Klasse mit gemeinsamer Sync-Funktionalität"""
+    
+    def __init__(self, espocrm_api, redis_client: redis.Redis = None, context=None):
+        """
+        Args:
+            espocrm_api: EspoCRM API client instance
+            redis_client: Optional Redis client (wird sonst initialisiert)
+            context: Optional Motia FlowContext für Logging
+        """
+        self.espocrm = espocrm_api
+        self.context = context
+        self.logger = context.logger if context else logger
+        self.redis = redis_client or self._init_redis()
+    
+    def _init_redis(self) -> Optional[redis.Redis]:
+        """Initialize Redis client for distributed locking"""
+        try:
+            redis_host = os.getenv('REDIS_HOST', 'localhost')
+            redis_port = int(os.getenv('REDIS_PORT', '6379'))
+            redis_db = int(os.getenv('REDIS_DB_ADVOWARE_CACHE', '1'))
+            
+            client = redis.Redis(
+                host=redis_host,
+                port=redis_port,
+                db=redis_db,
+                decode_responses=True
+            )
+            client.ping()
+            return client
+        except Exception as e:
+            self._log(f"Redis connection failed: {e}", level='error')
+            return None
+    
+    def _log(self, message: str, level: str = 'info'):
+        """
+        Context-aware logging
+        
+        Falls ein FlowContext vorhanden ist, wird dessen Logger verwendet.
+        Sonst fallback auf Standard-Logger.
+        """
+        if self.context and hasattr(self.context, 'logger'):
+            getattr(self.context.logger, level)(message)
+        else:
+            getattr(logger, level)(message)
+    
+    def _get_lock_key(self, entity_id: str) -> str:
+        """
+        Erzeugt Redis Lock-Key für eine Entity
+        
+        Muss in Subklassen überschrieben werden, um entity-spezifische Prefixes zu nutzen.
+        z.B. 'sync_lock:cbeteiligte:{entity_id}' oder 'sync_lock:document:{entity_id}'
+        """
+        raise NotImplementedError("Subclass must implement _get_lock_key()")
+    
+    def _acquire_redis_lock(self, lock_key: str) -> bool:
+        """
+        Atomic Redis lock acquisition
+        
+        Args:
+            lock_key: Redis key für den Lock
+            
+        Returns:
+            True wenn Lock erfolgreich, False wenn bereits locked
+        """
+        if not self.redis:
+            self._log("Redis nicht verfügbar, Lock-Mechanismus deaktiviert", level='warn')
+            return True  # Fallback: Wenn kein Redis, immer lock erlauben
+        
+        try:
+            acquired = self.redis.set(lock_key, "locked", nx=True, ex=LOCK_TTL_SECONDS)
+            return bool(acquired)
+        except Exception as e:
+            self._log(f"Redis lock error: {e}", level='error')
+            return True  # Bei Fehler: Lock erlauben, um Deadlocks zu vermeiden
+    
+    def _release_redis_lock(self, lock_key: str) -> None:
+        """
+        Redis lock freigeben
+        
+        Args:
+            lock_key: Redis key für den Lock
+        """
+        if not self.redis:
+            return
+        
+        try:
+            self.redis.delete(lock_key)
+        except Exception as e:
+            self._log(f"Redis unlock error: {e}", level='error')
+    
+    def _get_espocrm_datetime(self, dt: Optional[datetime] = None) -> str:
+        """
+        Formatiert datetime für EspoCRM (ohne Timezone!)
+        
+        Args:
+            dt: Optional datetime object (default: now UTC)
+            
+        Returns:
+            String im Format 'YYYY-MM-DD HH:MM:SS'
+        """
+        if dt is None:
+            dt = datetime.now(pytz.UTC)
+        elif dt.tzinfo is None:
+            dt = pytz.UTC.localize(dt)
+        
+        return dt.strftime('%Y-%m-%d %H:%M:%S')
+    
+    async def acquire_sync_lock(self, entity_id: str, **kwargs) -> bool:
+        """
+        Erwirbt Sync-Lock für eine Entity
+        
+        Muss in Subklassen implementiert werden, um entity-spezifische
+        Status-Updates durchzuführen.
+        
+        Returns:
+            True wenn Lock erfolgreich, False wenn bereits locked
+        """
+        raise NotImplementedError("Subclass must implement acquire_sync_lock()")
+    
+    async def release_sync_lock(self, entity_id: str, **kwargs) -> None:
+        """
+        Gibt Sync-Lock frei und setzt finalen Status
+        
+        Muss in Subklassen implementiert werden, um entity-spezifische
+        Status-Updates durchzuführen.
+        """
+        raise NotImplementedError("Subclass must implement release_sync_lock()")

steps/espocrm_webhooks/__init__.py

@@ -1,6 +0,0 @@
-"""
-EspoCRM Generic Webhooks
-
-Empfängt Webhooks von EspoCRM für verschiedene Entities.
-Zentrale Anlaufstelle für alle EspoCRM-Events außerhalb VMH-Kontext.
-"""

steps/espocrm_webhooks/document_create_webhook_api_step.py

@@ -1,208 +0,0 @@
-"""EspoCRM Webhook - Document Create
-
-Empfängt Create-Webhooks von EspoCRM für Documents.
-Loggt detailliert alle Payload-Informationen für Analyse.
-"""
-import json
-import datetime
-from typing import Any
-from motia import FlowContext, http, ApiRequest, ApiResponse
-
-
-config = {
-    "name": "VMH Webhook Document Create",
-    "description": "Empfängt Create-Webhooks von EspoCRM für Document Entities",
-    "flows": ["vmh-documents"],
-    "triggers": [
-        http("POST", "/vmh/webhook/document/create")
-    ],
-    "enqueues": ["vmh.document.create"],
-}
-
-
-async def handler(request: ApiRequest, ctx: FlowContext[Any]) -> ApiResponse:
-    """
-    Webhook handler for Document creation in EspoCRM.
-    
-    Receives notifications when documents are created and emits queue events
-    for processing (xAI sync, etc.).
-    
-    Payload Analysis Mode: Logs comprehensive details about webhook structure.
-    """
-    try:
-        payload = request.body or []
-        
-        # ═══════════════════════════════════════════════════════════════
-        # DETAILLIERTES LOGGING FÜR ANALYSE
-        # ═══════════════════════════════════════════════════════════════
-        
-        ctx.logger.info("=" * 80)
-        ctx.logger.info("📥 EspoCRM DOCUMENT CREATE WEBHOOK EMPFANGEN")
-        ctx.logger.info("=" * 80)
-        
-        # Log Request Headers
-        ctx.logger.info("\n🔍 REQUEST HEADERS:")
-        if hasattr(request, 'headers'):
-            for key, value in request.headers.items():
-                ctx.logger.info(f"   {key}: {value}")
-        else:
-            ctx.logger.info("   (keine Headers verfügbar)")
-        
-        # Log Payload Type & Structure
-        ctx.logger.info(f"\n📦 PAYLOAD TYPE: {type(payload).__name__}")
-        ctx.logger.info(f"📦 PAYLOAD LENGTH: {len(payload) if isinstance(payload, (list, dict)) else 'N/A'}")
-        
-        # Log Full Payload (pretty-printed)
-        ctx.logger.info("\n📄 FULL PAYLOAD:")
-        ctx.logger.info(json.dumps(payload, indent=2, ensure_ascii=False))
-        
-        # ═══════════════════════════════════════════════════════════════
-        # PAYLOAD ANALYSE & ID EXTRAKTION
-        # ═══════════════════════════════════════════════════════════════
-        
-        entity_ids = set()
-        payload_details = []
-        
-        if isinstance(payload, list):
-            ctx.logger.info(f"\n✅ Payload ist LIST mit {len(payload)} Einträgen")
-            for idx, entity in enumerate(payload):
-                if isinstance(entity, dict):
-                    entity_id = entity.get('id')
-                    if entity_id:
-                        entity_ids.add(entity_id)
-                        
-                        # Sammle Details für Logging
-                        detail = {
-                            'index': idx,
-                            'id': entity_id,
-                            'name': entity.get('name', 'N/A'),
-                            'type': entity.get('type', 'N/A'),
-                            'size': entity.get('size', 'N/A'),
-                            'all_fields': list(entity.keys())
-                        }
-                        payload_details.append(detail)
-                        
-                        ctx.logger.info(f"\n   📄 Document #{idx + 1}:")
-                        ctx.logger.info(f"      ID: {entity_id}")
-                        ctx.logger.info(f"      Name: {entity.get('name', 'N/A')}")
-                        ctx.logger.info(f"      Type: {entity.get('type', 'N/A')}")
-                        ctx.logger.info(f"      Size: {entity.get('size', 'N/A')} bytes")
-                        ctx.logger.info(f"      Verfügbare Felder: {', '.join(entity.keys())}")
-                        
-                        # xAI-relevante Felder (falls vorhanden)
-                        xai_fields = {k: v for k, v in entity.items() 
-                                     if 'xai' in k.lower() or 'collection' in k.lower()}
-                        if xai_fields:
-                            ctx.logger.info(f"      🤖 xAI-Felder: {json.dumps(xai_fields, ensure_ascii=False)}")
-                        
-                        # Parent/Relationship Felder
-                        rel_fields = {k: v for k, v in entity.items() 
-                                     if 'parent' in k.lower() or 'related' in k.lower() or 
-                                        'link' in k.lower() or k.endswith('Id') or k.endswith('Ids')}
-                        if rel_fields:
-                            ctx.logger.info(f"      🔗 Relationship-Felder: {json.dumps(rel_fields, ensure_ascii=False)}")
-                
-        elif isinstance(payload, dict):
-            ctx.logger.info("\n✅ Payload ist SINGLE DICT")
-            entity_id = payload.get('id')
-            if entity_id:
-                entity_ids.add(entity_id)
-                
-                ctx.logger.info(f"\n   📄 Document:")
-                ctx.logger.info(f"      ID: {entity_id}")
-                ctx.logger.info(f"      Name: {payload.get('name', 'N/A')}")
-                ctx.logger.info(f"      Type: {payload.get('type', 'N/A')}")
-                ctx.logger.info(f"      Size: {payload.get('size', 'N/A')} bytes")
-                ctx.logger.info(f"      Verfügbare Felder: {', '.join(payload.keys())}")
-                
-                # xAI-relevante Felder
-                xai_fields = {k: v for k, v in payload.items() 
-                             if 'xai' in k.lower() or 'collection' in k.lower()}
-                if xai_fields:
-                    ctx.logger.info(f"      🤖 xAI-Felder: {json.dumps(xai_fields, ensure_ascii=False)}")
-                
-                # Relationship Felder
-                rel_fields = {k: v for k, v in payload.items() 
-                             if 'parent' in k.lower() or 'related' in k.lower() or 
-                                'link' in k.lower() or k.endswith('Id') or k.endswith('Ids')}
-                if rel_fields:
-                    ctx.logger.info(f"      🔗 Relationship-Felder: {json.dumps(rel_fields, ensure_ascii=False)}")
-        else:
-            ctx.logger.warning(f"⚠️  Unerwarteter Payload-Typ: {type(payload)}")
-        
-        # ═══════════════════════════════════════════════════════════════
-        # QUEUE EVENTS EMITTIEREN
-        # ═══════════════════════════════════════════════════════════════
-        
-        ctx.logger.info("\n" + "=" * 80)
-        ctx.logger.info(f"📊 ZUSAMMENFASSUNG: {len(entity_ids)} Document(s) gefunden")
-        ctx.logger.info("=" * 80)
-        
-        if not entity_ids:
-            ctx.logger.warning("⚠️  Keine Document-IDs im Payload gefunden!")
-            return ApiResponse(
-                status=200,
-                body={
-                    'status': 'received',
-                    'action': 'create',
-                    'ids_count': 0,
-                    'warning': 'No document IDs found in payload'
-                }
-            )
-        
-        # Emit events für Queue-Processing (Deduplizierung erfolgt im Event-Handler via Lock)
-        # Versuche Entity-Type zu ermitteln
-        entity_type = 'CDokumente'  # Default für VMH
-        if isinstance(payload, list) and payload:
-            entity_type = payload[0].get('entityType') or payload[0].get('_scope') or 'CDokumente'
-        elif isinstance(payload, dict):
-            entity_type = payload.get('entityType') or payload.get('_scope') or 'CDokumente'
-        
-        ctx.logger.info(f"📝 Entity-Type: {entity_type}")
-        
-        for entity_id in entity_ids:
-            await ctx.enqueue({
-                'topic': 'vmh.document.create',
-                'data': {
-                    'entity_id': entity_id,
-                    'entity_type': entity_type,
-                    'action': 'create',
-                    'source': 'webhook',
-                    'timestamp': datetime.datetime.now().isoformat()
-                }
-            })
-            ctx.logger.info(f"✅ Event emittiert: vmh.document.create für ID {entity_id} (Type: {entity_type})")
-        
-        ctx.logger.info("\n" + "=" * 80)
-        ctx.logger.info(f"✅ WEBHOOK VERARBEITUNG ABGESCHLOSSEN")
-        ctx.logger.info("=" * 80)
-        
-        return ApiResponse(
-            status=200,
-            body={
-                'status': 'received',
-                'action': 'create',
-                'ids_count': len(entity_ids),
-                'document_ids': list(entity_ids)
-            }
-        )
-    
-    except Exception as e:
-        ctx.logger.error("=" * 80)
-        ctx.logger.error(f"❌ FEHLER beim Verarbeiten des Document Create Webhooks")
-        ctx.logger.error("=" * 80)
-        ctx.logger.error(f"Error Type: {type(e).__name__}")
-        ctx.logger.error(f"Error Message: {str(e)}")
-        
-        # Log Stack Trace
-        import traceback
-        ctx.logger.error(f"Stack Trace:\n{traceback.format_exc()}")
-        
-        return ApiResponse(
-            status=500,
-            body={
-                'error': 'Internal server error',
-                'error_type': type(e).__name__,
-                'details': str(e)
-            }
-        )

steps/espocrm_webhooks/document_delete_webhook_api_step.py

@@ -1,184 +0,0 @@
-"""EspoCRM Webhook - Document Delete
-
-Empfängt Delete-Webhooks von EspoCRM für Documents.
-Loggt detailliert alle Payload-Informationen für Analyse.
-"""
-import json
-import datetime
-from typing import Any
-from motia import FlowContext, http, ApiRequest, ApiResponse
-
-
-config = {
-    "name": "VMH Webhook Document Delete",
-    "description": "Empfängt Delete-Webhooks von EspoCRM für Document Entities",
-    "flows": ["vmh-documents"],
-    "triggers": [
-        http("POST", "/vmh/webhook/document/delete")
-    ],
-    "enqueues": ["vmh.document.delete"],
-}
-
-
-async def handler(request: ApiRequest, ctx: FlowContext[Any]) -> ApiResponse:
-    """
-    Webhook handler for Document deletion in EspoCRM.
-    
-    Receives notifications when documents are deleted.
-    Note: Bei Deletion haben wir ggf. nur die ID, keine vollständigen Entity-Daten.
-    """
-    try:
-        payload = request.body or []
-        
-        # ═══════════════════════════════════════════════════════════════
-        # DETAILLIERTES LOGGING FÜR ANALYSE
-        # ═══════════════════════════════════════════════════════════════
-        
-        ctx.logger.info("=" * 80)
-        ctx.logger.info("📥 EspoCRM DOCUMENT DELETE WEBHOOK EMPFANGEN")
-        ctx.logger.info("=" * 80)
-        
-        # Log Request Headers
-        ctx.logger.info("\n🔍 REQUEST HEADERS:")
-        if hasattr(request, 'headers'):
-            for key, value in request.headers.items():
-                ctx.logger.info(f"   {key}: {value}")
-        else:
-            ctx.logger.info("   (keine Headers verfügbar)")
-        
-        # Log Payload Type & Structure
-        ctx.logger.info(f"\n📦 PAYLOAD TYPE: {type(payload).__name__}")
-        ctx.logger.info(f"📦 PAYLOAD LENGTH: {len(payload) if isinstance(payload, (list, dict)) else 'N/A'}")
-        
-        # Log Full Payload (pretty-printed)
-        ctx.logger.info("\n📄 FULL PAYLOAD:")
-        ctx.logger.info(json.dumps(payload, indent=2, ensure_ascii=False))
-        
-        # ═══════════════════════════════════════════════════════════════
-        # PAYLOAD ANALYSE & ID EXTRAKTION
-        # ═══════════════════════════════════════════════════════════════
-        
-        entity_ids = set()
-        
-        if isinstance(payload, list):
-            ctx.logger.info(f"\n✅ Payload ist LIST mit {len(payload)} Einträgen")
-            for idx, entity in enumerate(payload):
-                if isinstance(entity, dict):
-                    entity_id = entity.get('id')
-                    if entity_id:
-                        entity_ids.add(entity_id)
-                        
-                        ctx.logger.info(f"\n   🗑️  Document #{idx + 1}:")
-                        ctx.logger.info(f"      ID: {entity_id}")
-                        ctx.logger.info(f"      Verfügbare Felder: {', '.join(entity.keys())}")
-                        
-                        # Bei Delete haben wir oft nur minimale Daten
-                        if 'name' in entity:
-                            ctx.logger.info(f"      Name: {entity.get('name')}")
-                        if 'deletedAt' in entity or 'deleted' in entity:
-                            ctx.logger.info(f"      Deleted At: {entity.get('deletedAt', entity.get('deleted', 'N/A'))}")
-                        
-                        # xAI-relevante Felder (falls vorhanden)
-                        xai_fields = {k: v for k, v in entity.items() 
-                                     if 'xai' in k.lower() or 'collection' in k.lower()}
-                        if xai_fields:
-                            ctx.logger.info(f"      🤖 xAI-Felder: {json.dumps(xai_fields, ensure_ascii=False)}")
-                
-        elif isinstance(payload, dict):
-            ctx.logger.info("\n✅ Payload ist SINGLE DICT")
-            entity_id = payload.get('id')
-            if entity_id:
-                entity_ids.add(entity_id)
-                
-                ctx.logger.info(f"\n   🗑️  Document:")
-                ctx.logger.info(f"      ID: {entity_id}")
-                ctx.logger.info(f"      Verfügbare Felder: {', '.join(payload.keys())}")
-                
-                if 'name' in payload:
-                    ctx.logger.info(f"      Name: {payload.get('name')}")
-                if 'deletedAt' in payload or 'deleted' in payload:
-                    ctx.logger.info(f"      Deleted At: {payload.get('deletedAt', payload.get('deleted', 'N/A'))}")
-                
-                # xAI-relevante Felder
-                xai_fields = {k: v for k, v in payload.items() 
-                             if 'xai' in k.lower() or 'collection' in k.lower()}
-                if xai_fields:
-                    ctx.logger.info(f"      🤖 xAI-Felder: {json.dumps(xai_fields, ensure_ascii=False)}")
-        else:
-            ctx.logger.warning(f"⚠️  Unerwarteter Payload-Typ: {type(payload)}")
-        
-        # ═══════════════════════════════════════════════════════════════
-        # QUEUE EVENTS EMITTIEREN
-        # ═══════════════════════════════════════════════════════════════
-        
-        ctx.logger.info("\n" + "=" * 80)
-        ctx.logger.info(f"📊 ZUSAMMENFASSUNG: {len(entity_ids)} Document(s) gefunden")
-        ctx.logger.info("=" * 80)
-        
-        if not entity_ids:
-            ctx.logger.warning("⚠️  Keine Document-IDs im Payload gefunden!")
-            return ApiResponse(
-                status=200,
-                body={
-                    'status': 'received',
-                    'action': 'delete',
-                    'ids_count': 0,
-                    'warning': 'No document IDs found in payload'
-                }
-            )
-        
-        # Emit events für Queue-Processing
-        # Versuche Entity-Type zu ermitteln
-        entity_type = 'CDokumente'  # Default für VMH
-        if isinstance(payload, list) and payload:
-            entity_type = payload[0].get('entityType') or payload[0].get('_scope') or 'CDokumente'
-        elif isinstance(payload, dict):
-            entity_type = payload.get('entityType') or payload.get('_scope') or 'CDokumente'
-        
-        ctx.logger.info(f"📝 Entity-Type: {entity_type}")
-        
-        for entity_id in entity_ids:
-            await ctx.enqueue({
-                'topic': 'vmh.document.delete',
-                'data': {
-                    'entity_id': entity_id,
-                    'entity_type': entity_type,
-                    'action': 'delete',
-                    'source': 'webhook',
-                    'timestamp': datetime.datetime.now().isoformat()
-                }
-            })
-            ctx.logger.info(f"✅ Event emittiert: vmh.document.delete für ID {entity_id} (Type: {entity_type})")
-        
-        ctx.logger.info("\n" + "=" * 80)
-        ctx.logger.info(f"✅ WEBHOOK VERARBEITUNG ABGESCHLOSSEN")
-        ctx.logger.info("=" * 80)
-        
-        return ApiResponse(
-            status=200,
-            body={
-                'status': 'received',
-                'action': 'delete',
-                'ids_count': len(entity_ids),
-                'document_ids': list(entity_ids)
-            }
-        )
-    
-    except Exception as e:
-        ctx.logger.error("=" * 80)
-        ctx.logger.error(f"❌ FEHLER beim Verarbeiten des Document Delete Webhooks")
-        ctx.logger.error("=" * 80)
-        ctx.logger.error(f"Error Type: {type(e).__name__}")
-        ctx.logger.error(f"Error Message: {str(e)}")
-        
-        import traceback
-        ctx.logger.error(f"Stack Trace:\n{traceback.format_exc()}")
-        
-        return ApiResponse(
-            status=500,
-            body={
-                'error': 'Internal server error',
-                'error_type': type(e).__name__,
-                'details': str(e)
-            }
-        )

steps/espocrm_webhooks/document_update_webhook_api_step.py

@@ -1,206 +0,0 @@
-"""EspoCRM Webhook - Document Update
-
-Empfängt Update-Webhooks von EspoCRM für Documents.
-Loggt detailliert alle Payload-Informationen für Analyse.
-"""
-import json
-import datetime
-from typing import Any
-from motia import FlowContext, http, ApiRequest, ApiResponse
-
-
-config = {
-    "name": "VMH Webhook Document Update",
-    "description": "Empfängt Update-Webhooks von EspoCRM für Document Entities",
-    "flows": ["vmh-documents"],
-    "triggers": [
-        http("POST", "/vmh/webhook/document/update")
-    ],
-    "enqueues": ["vmh.document.update"],
-}
-
-
-async def handler(request: ApiRequest, ctx: FlowContext[Any]) -> ApiResponse:
-    """
-    Webhook handler for Document updates in EspoCRM.
-    
-    Receives notifications when documents are updated and emits queue events
-    for processing (xAI sync, etc.).
-    
-    Note: Loop-Prevention sollte auf EspoCRM-Seite implementiert werden.
-    xAI-Feld-Updates sollten keine neuen Webhooks triggern.
-    """
-    try:
-        payload = request.body or []
-        
-        # ═══════════════════════════════════════════════════════════════
-        # DETAILLIERTES LOGGING FÜR ANALYSE
-        # ═══════════════════════════════════════════════════════════════
-        
-        ctx.logger.info("=" * 80)
-        ctx.logger.info("📥 EspoCRM DOCUMENT UPDATE WEBHOOK EMPFANGEN")
-        ctx.logger.info("=" * 80)
-        
-        # Log Request Headers
-        ctx.logger.info("\n🔍 REQUEST HEADERS:")
-        if hasattr(request, 'headers'):
-            for key, value in request.headers.items():
-                ctx.logger.info(f"   {key}: {value}")
-        else:
-            ctx.logger.info("   (keine Headers verfügbar)")
-        
-        # Log Payload Type & Structure
-        ctx.logger.info(f"\n📦 PAYLOAD TYPE: {type(payload).__name__}")
-        ctx.logger.info(f"📦 PAYLOAD LENGTH: {len(payload) if isinstance(payload, (list, dict)) else 'N/A'}")
-        
-        # Log Full Payload (pretty-printed)
-        ctx.logger.info("\n📄 FULL PAYLOAD:")
-        ctx.logger.info(json.dumps(payload, indent=2, ensure_ascii=False))
-        
-        # ═══════════════════════════════════════════════════════════════
-        # PAYLOAD ANALYSE & ID EXTRAKTION
-        # ═══════════════════════════════════════════════════════════════
-        
-        entity_ids = set()
-        
-        if isinstance(payload, list):
-            ctx.logger.info(f"\n✅ Payload ist LIST mit {len(payload)} Einträgen")
-            for idx, entity in enumerate(payload):
-                if isinstance(entity, dict):
-                    entity_id = entity.get('id')
-                    if entity_id:
-                        entity_ids.add(entity_id)
-                        
-                        ctx.logger.info(f"\n   📄 Document #{idx + 1}:")
-                        ctx.logger.info(f"      ID: {entity_id}")
-                        ctx.logger.info(f"      Name: {entity.get('name', 'N/A')}")
-                        ctx.logger.info(f"      Modified At: {entity.get('modifiedAt', 'N/A')}")
-                        ctx.logger.info(f"      Modified By: {entity.get('modifiedById', 'N/A')}")
-                        ctx.logger.info(f"      Verfügbare Felder: {', '.join(entity.keys())}")
-                        
-                        # Prüfe ob CHANGED fields mitgeliefert werden
-                        changed_fields = entity.get('changedFields') or entity.get('changed') or entity.get('modifiedFields')
-                        if changed_fields:
-                            ctx.logger.info(f"      🔄 Geänderte Felder: {json.dumps(changed_fields, ensure_ascii=False)}")
-                        
-                        # xAI-relevante Felder
-                        xai_fields = {k: v for k, v in entity.items() 
-                                     if 'xai' in k.lower() or 'collection' in k.lower()}
-                        if xai_fields:
-                            ctx.logger.info(f"      🤖 xAI-Felder: {json.dumps(xai_fields, ensure_ascii=False)}")
-                        
-                        # Relationship Felder
-                        rel_fields = {k: v for k, v in entity.items() 
-                                     if 'parent' in k.lower() or 'related' in k.lower() or 
-                                        'link' in k.lower() or k.endswith('Id') or k.endswith('Ids')}
-                        if rel_fields:
-                            ctx.logger.info(f"      🔗 Relationship-Felder: {json.dumps(rel_fields, ensure_ascii=False)}")
-                
-        elif isinstance(payload, dict):
-            ctx.logger.info("\n✅ Payload ist SINGLE DICT")
-            entity_id = payload.get('id')
-            if entity_id:
-                entity_ids.add(entity_id)
-                
-                ctx.logger.info(f"\n   📄 Document:")
-                ctx.logger.info(f"      ID: {entity_id}")
-                ctx.logger.info(f"      Name: {payload.get('name', 'N/A')}")
-                ctx.logger.info(f"      Modified At: {payload.get('modifiedAt', 'N/A')}")
-                ctx.logger.info(f"      Modified By: {payload.get('modifiedById', 'N/A')}")
-                ctx.logger.info(f"      Verfügbare Felder: {', '.join(payload.keys())}")
-                
-                # Geänderte Felder
-                changed_fields = payload.get('changedFields') or payload.get('changed') or payload.get('modifiedFields')
-                if changed_fields:
-                    ctx.logger.info(f"      🔄 Geänderte Felder: {json.dumps(changed_fields, ensure_ascii=False)}")
-                
-                # xAI-relevante Felder
-                xai_fields = {k: v for k, v in payload.items() 
-                             if 'xai' in k.lower() or 'collection' in k.lower()}
-                if xai_fields:
-                    ctx.logger.info(f"      🤖 xAI-Felder: {json.dumps(xai_fields, ensure_ascii=False)}")
-                
-                # Relationship Felder
-                rel_fields = {k: v for k, v in payload.items() 
-                             if 'parent' in k.lower() or 'related' in k.lower() or 
-                                'link' in k.lower() or k.endswith('Id') or k.endswith('Ids')}
-                if rel_fields:
-                    ctx.logger.info(f"      🔗 Relationship-Felder: {json.dumps(rel_fields, ensure_ascii=False)}")
-        else:
-            ctx.logger.warning(f"⚠️  Unerwarteter Payload-Typ: {type(payload)}")
-        
-        # ═══════════════════════════════════════════════════════════════
-        # QUEUE EVENTS EMITTIEREN
-        # ═══════════════════════════════════════════════════════════════
-        
-        ctx.logger.info("\n" + "=" * 80)
-        ctx.logger.info(f"📊 ZUSAMMENFASSUNG: {len(entity_ids)} Document(s) gefunden")
-        ctx.logger.info("=" * 80)
-        
-        if not entity_ids:
-            ctx.logger.warning("⚠️  Keine Document-IDs im Payload gefunden!")
-            return ApiResponse(
-                status=200,
-                body={
-                    'status': 'received',
-                    'action': 'update',
-                    'ids_count': 0,
-                    'warning': 'No document IDs found in payload'
-                }
-            )
-        
-        # Emit events für Queue-Processing
-        # Versuche Entity-Type zu ermitteln
-        entity_type = 'CDokumente'  # Default für VMH
-        if isinstance(payload, list) and payload:
-            entity_type = payload[0].get('entityType') or payload[0].get('_scope') or 'CDokumente'
-        elif isinstance(payload, dict):
-            entity_type = payload.get('entityType') or payload.get('_scope') or 'CDokumente'
-        
-        ctx.logger.info(f"📝 Entity-Type: {entity_type}")
-        
-        for entity_id in entity_ids:
-            await ctx.enqueue({
-                'topic': 'vmh.document.update',
-                'data': {
-                    'entity_id': entity_id,
-                    'entity_type': entity_type,
-                    'action': 'update',
-                    'source': 'webhook',
-                    'timestamp': datetime.datetime.now().isoformat()
-                }
-            })
-            ctx.logger.info(f"✅ Event emittiert: vmh.document.update für ID {entity_id} (Type: {entity_type})")
-        
-        ctx.logger.info("\n" + "=" * 80)
-        ctx.logger.info(f"✅ WEBHOOK VERARBEITUNG ABGESCHLOSSEN")
-        ctx.logger.info("=" * 80)
-        
-        return ApiResponse(
-            status=200,
-            body={
-                'status': 'received',
-                'action': 'update',
-                'ids_count': len(entity_ids),
-                'document_ids': list(entity_ids)
-            }
-        )
-    
-    except Exception as e:
-        ctx.logger.error("=" * 80)
-        ctx.logger.error(f"❌ FEHLER beim Verarbeiten des Document Update Webhooks")
-        ctx.logger.error("=" * 80)
-        ctx.logger.error(f"Error Type: {type(e).__name__}")
-        ctx.logger.error(f"Error Message: {str(e)}")
-        
-        import traceback
-        ctx.logger.error(f"Stack Trace:\n{traceback.format_exc()}")
-        
-        return ApiResponse(
-            status=500,
-            body={
-                'error': 'Internal server error',
-                'error_type': type(e).__name__,
-                'details': str(e)
-            }
-        )

steps/vmh/webhook/document_create_api_step.py

@@ -0,0 +1,77 @@
+"""VMH Webhook - Document Create"""
+import json
+from typing import Any
+from motia import FlowContext, http, ApiRequest, ApiResponse
+
+
+config = {
+    "name": "VMH Webhook Document Create",
+    "description": "Empfängt Create-Webhooks von EspoCRM für Documents",
+    "flows": ["vmh-documents"],
+    "triggers": [
+        http("POST", "/vmh/webhook/document/create")
+    ],
+    "enqueues": ["vmh.document.create"],
+}
+
+
+async def handler(request: ApiRequest, ctx: FlowContext[Any]) -> ApiResponse:
+    """
+    Webhook handler for Document creation in EspoCRM.
+    
+    Receives batch or single entity notifications and emits queue events
+    for each entity ID to be synced to xAI.
+    """
+    try:
+        payload = request.body or []
+
+        ctx.logger.info("VMH Webhook Document Create empfangen")
+        ctx.logger.debug(f"Payload: {json.dumps(payload, indent=2, ensure_ascii=False)}")
+
+        # Sammle alle IDs aus dem Batch
+        entity_ids = set()
+
+        if isinstance(payload, list):
+            for entity in payload:
+                if isinstance(entity, dict) and 'id' in entity:
+                    entity_ids.add(entity['id'])
+                    # Extrahiere entityType falls vorhanden
+                    entity_type = entity.get('entityType', 'CDokumente')
+        elif isinstance(payload, dict) and 'id' in payload:
+            entity_ids.add(payload['id'])
+            entity_type = payload.get('entityType', 'CDokumente')
+
+        ctx.logger.info(f"{len(entity_ids)} Document IDs zum Create-Sync gefunden")
+
+        # Emit events für Queue-Processing (Deduplizierung erfolgt im Event-Handler via Lock)
+        for entity_id in entity_ids:
+            await ctx.enqueue({
+                'topic': 'vmh.document.create',
+                'data': {
+                    'entity_id': entity_id,
+                    'entity_type': entity_type if 'entity_type' in locals() else 'CDokumente',
+                    'action': 'create',
+                    'timestamp': payload[0].get('modifiedAt') if isinstance(payload, list) and payload else None
+                }
+            })
+
+        return ApiResponse(
+            status_code=200,
+            body={
+                'success': True,
+                'message': f'{len(entity_ids)} Document(s) zum Sync enqueued',
+                'entity_ids': list(entity_ids)
+            }
+        )
+
+    except Exception as e:
+        ctx.logger.error(f"Fehler im Document Create Webhook: {e}")
+        ctx.logger.error(f"Payload: {request.body}")
+        
+        return ApiResponse(
+            status_code=500,
+            body={
+                'success': False,
+                'error': str(e)
+            }
+        )

steps/vmh/webhook/document_delete_api_step.py

@@ -0,0 +1,76 @@
+"""VMH Webhook - Document Delete"""
+import json
+from typing import Any
+from motia import FlowContext, http, ApiRequest, ApiResponse
+
+
+config = {
+    "name": "VMH Webhook Document Delete",
+    "description": "Empfängt Delete-Webhooks von EspoCRM für Documents",
+    "flows": ["vmh-documents"],
+    "triggers": [
+        http("POST", "/vmh/webhook/document/delete")
+    ],
+    "enqueues": ["vmh.document.delete"],
+}
+
+
+async def handler(request: ApiRequest, ctx: FlowContext[Any]) -> ApiResponse:
+    """
+    Webhook handler for Document deletion in EspoCRM.
+    
+    Receives batch or single entity notifications and emits queue events
+    for each entity ID to be removed from xAI.
+    """
+    try:
+        payload = request.body or []
+
+        ctx.logger.info("VMH Webhook Document Delete empfangen")
+        ctx.logger.debug(f"Payload: {json.dumps(payload, indent=2, ensure_ascii=False)}")
+
+        # Sammle alle IDs aus dem Batch
+        entity_ids = set()
+
+        if isinstance(payload, list):
+            for entity in payload:
+                if isinstance(entity, dict) and 'id' in entity:
+                    entity_ids.add(entity['id'])
+                    entity_type = entity.get('entityType', 'CDokumente')
+        elif isinstance(payload, dict) and 'id' in payload:
+            entity_ids.add(payload['id'])
+            entity_type = payload.get('entityType', 'CDokumente')
+
+        ctx.logger.info(f"{len(entity_ids)} Document IDs zum Delete-Sync gefunden")
+
+        # Emit events für Queue-Processing
+        for entity_id in entity_ids:
+            await ctx.enqueue({
+                'topic': 'vmh.document.delete',
+                'data': {
+                    'entity_id': entity_id,
+                    'entity_type': entity_type if 'entity_type' in locals() else 'CDokumente',
+                    'action': 'delete',
+                    'timestamp': payload[0].get('deletedAt') if isinstance(payload, list) and payload else None
+                }
+            })
+
+        return ApiResponse(
+            status_code=200,
+            body={
+                'success': True,
+                'message': f'{len(entity_ids)} Document(s) zum Delete enqueued',
+                'entity_ids': list(entity_ids)
+            }
+        )
+
+    except Exception as e:
+        ctx.logger.error(f"Fehler im Document Delete Webhook: {e}")
+        ctx.logger.error(f"Payload: {request.body}")
+        
+        return ApiResponse(
+            status_code=500,
+            body={
+                'success': False,
+                'error': str(e)
+            }
+        )

steps/vmh/webhook/document_update_api_step.py

@@ -0,0 +1,76 @@
+"""VMH Webhook - Document Update"""
+import json
+from typing import Any
+from motia import FlowContext, http, ApiRequest, ApiResponse
+
+
+config = {
+    "name": "VMH Webhook Document Update",
+    "description": "Empfängt Update-Webhooks von EspoCRM für Documents",
+    "flows": ["vmh-documents"],
+    "triggers": [
+        http("POST", "/vmh/webhook/document/update")
+    ],
+    "enqueues": ["vmh.document.update"],
+}
+
+
+async def handler(request: ApiRequest, ctx: FlowContext[Any]) -> ApiResponse:
+    """
+    Webhook handler for Document updates in EspoCRM.
+    
+    Receives batch or single entity notifications and emits queue events
+    for each entity ID to be synced to xAI.
+    """
+    try:
+        payload = request.body or []
+
+        ctx.logger.info("VMH Webhook Document Update empfangen")
+        ctx.logger.debug(f"Payload: {json.dumps(payload, indent=2, ensure_ascii=False)}")
+
+        # Sammle alle IDs aus dem Batch
+        entity_ids = set()
+
+        if isinstance(payload, list):
+            for entity in payload:
+                if isinstance(entity, dict) and 'id' in entity:
+                    entity_ids.add(entity['id'])
+                    entity_type = entity.get('entityType', 'CDokumente')
+        elif isinstance(payload, dict) and 'id' in payload:
+            entity_ids.add(payload['id'])
+            entity_type = payload.get('entityType', 'CDokumente')
+
+        ctx.logger.info(f"{len(entity_ids)} Document IDs zum Update-Sync gefunden")
+
+        # Emit events für Queue-Processing
+        for entity_id in entity_ids:
+            await ctx.enqueue({
+                'topic': 'vmh.document.update',
+                'data': {
+                    'entity_id': entity_id,
+                    'entity_type': entity_type if 'entity_type' in locals() else 'CDokumente',
+                    'action': 'update',
+                    'timestamp': payload[0].get('modifiedAt') if isinstance(payload, list) and payload else None
+                }
+            })
+
+        return ApiResponse(
+            status_code=200,
+            body={
+                'success': True,
+                'message': f'{len(entity_ids)} Document(s) zum Sync enqueued',
+                'entity_ids': list(entity_ids)
+            }
+        )
+
+    except Exception as e:
+        ctx.logger.error(f"Fehler im Document Update Webhook: {e}")
+        ctx.logger.error(f"Payload: {request.body}")
+        
+        return ApiResponse(
+            status_code=500,
+            body={
+                'success': False,
+                'error': str(e)
+            }
+        )

tests/README.md

@@ -0,0 +1,110 @@
+# Test Scripts
+
+This directory contains test scripts for the Motia III xAI Collections integration.
+
+## Test Files
+
+### `test_xai_collections_api.py`
+Tests xAI Collections API authentication and basic operations.
+
+**Usage:**
+```bash
+cd /opt/motia-iii/bitbylaw
+python tests/test_xai_collections_api.py
+```
+
+**Required Environment Variables:**
+- `XAI_MANAGEMENT_API_KEY` - xAI Management API key for collection operations
+- `XAI_API_KEY` - xAI Regular API key for file operations
+
+**Tests:**
+- ✅ Management API authentication
+- ✅ Regular API authentication
+- ✅ Collection listing
+- ✅ Collection creation
+- ✅ File upload
+- ✅ Collection deletion
+- ✅ Error handling
+
+### `test_preview_upload.py`
+Tests preview/thumbnail upload to EspoCRM CDokumente entity.
+
+**Usage:**
+```bash
+cd /opt/motia-iii/bitbylaw
+python tests/test_preview_upload.py
+```
+
+**Required Environment Variables:**
+- `ESPOCRM_URL` - EspoCRM instance URL (default: https://crm.bitbylaw.com)
+- `ESPOCRM_API_KEY` - EspoCRM API key
+
+**Tests:**
+- ✅ Preview image generation (WebP format, 600x800px)
+- ✅ Base64 Data URI encoding
+- ✅ Attachment upload via JSON POST
+- ✅ Entity update with previewId/previewName
+
+**Status:** ✅ Successfully tested - Attachment ID `69a71194c7c6baebf` created
+
+### `test_thumbnail_generation.py`
+Tests thumbnail generation for various document types.
+
+**Usage:**
+```bash
+cd /opt/motia-iii/bitbylaw
+python tests/test_thumbnail_generation.py
+```
+
+**Supported Formats:**
+- PDF → WebP (first page)
+- DOCX/DOC → PDF → WebP
+- Images (JPEG, PNG, etc.) → WebP resize
+
+**Dependencies:**
+- `python3-pil` - PIL/Pillow for image processing
+- `poppler-utils` - PDF rendering
+- `libreoffice` - DOCX to PDF conversion
+- `pdf2image` - PDF to image conversion
+
+## Running Tests
+
+### All Tests
+```bash
+cd /opt/motia-iii/bitbylaw
+python -m pytest tests/ -v
+```
+
+### Individual Tests
+```bash
+cd /opt/motia-iii/bitbylaw
+python tests/test_xai_collections_api.py
+python tests/test_preview_upload.py
+python tests/test_thumbnail_generation.py
+```
+
+## Environment Setup
+
+Create `.env` file in `/opt/motia-iii/bitbylaw/`:
+```bash
+# xAI Collections API
+XAI_MANAGEMENT_API_KEY=xai-token-xxx...
+XAI_API_KEY=xai-xxx...
+
+# EspoCRM API
+ESPOCRM_URL=https://crm.bitbylaw.com
+ESPOCRM_API_KEY=xxx...
+
+# Redis (for locking)
+REDIS_HOST=localhost
+REDIS_PORT=6379
+REDIS_DB_ADVOWARE_CACHE=1
+```
+
+## Test Results
+
+Last test run: Successfully validated preview upload functionality
+- Preview upload works with base64 Data URI format
+- Attachment created with ID: `69a71194c7c6baebf`
+- CDokumente entity updated with previewId/previewName
+- WebP format at 600x800px confirmed working