Compare commits
62 Commits
5c204ba16c
...
main
| Author | SHA1 | Date | |
|---|---|---|---|
|
|
71f583481a | ||
|
|
48d440a860 | ||
|
|
c02a5d8823 | ||
|
|
edae5f6081 | ||
|
|
8ce843415e | ||
|
|
46085bd8dd | ||
|
|
2ac83df1e0 | ||
|
|
7fffdb2660 | ||
|
|
69f0c6a44d | ||
|
|
949a5fd69c | ||
|
|
8e53fd6345 | ||
|
|
59fdd7d9ec | ||
|
|
eaab14ae57 | ||
|
|
331d43390a | ||
|
|
18f2ff775e | ||
|
|
c032e24d7a | ||
|
|
4a5065aea4 | ||
|
|
bb13d59ddb | ||
|
|
b0fceef4e2 | ||
|
|
e727582584 | ||
|
|
2292fd4762 | ||
|
|
9ada48d8c8 | ||
|
|
9a3e01d447 | ||
|
|
e945333c1a | ||
|
|
6f7f847939 | ||
|
|
46c0bbf381 | ||
|
|
8f1533337c | ||
|
|
6bf2343a12 | ||
|
|
8ed7cca432 | ||
|
|
9bbfa61b3b | ||
|
|
a5a122b688 | ||
|
|
6c3cf3ca91 | ||
|
|
1c765d1eec | ||
|
|
a0cf845877 | ||
|
|
f392ec0f06 | ||
|
|
2532bd89ee | ||
|
|
2e449d2928 | ||
|
|
fd0196ec31 | ||
|
|
d71b5665b6 | ||
|
|
d69801ed97 | ||
|
|
6e2303c5eb | ||
|
|
93d4d89531 | ||
|
|
4ed752b19e | ||
|
|
ba657ecd3b | ||
|
|
9e7e163933 | ||
|
|
82b48eee8e | ||
|
|
7fd6eed86d | ||
|
|
91ae2947e5 | ||
|
|
6f7d62293e | ||
|
|
d7b2b5543f | ||
|
|
a53051ea8e | ||
|
|
69a48f5f9a | ||
|
|
bcb6454b2a | ||
|
|
c45bfb7233 | ||
|
|
0e521f22f8 | ||
|
|
70265c9adf | ||
|
|
ee9aab049f | ||
|
|
cb0e170ee9 | ||
|
|
bc917bd885 | ||
|
|
0282149613 | ||
|
|
0740952063 | ||
|
|
f1ac5ffc7e |
@@ -1,320 +0,0 @@
|
||||
# Vollständige Migrations-Analyse
|
||||
## Motia v0.17 → Motia III v1.0-RC
|
||||
|
||||
**Datum:** 1. März 2026
|
||||
**Status:** 🎉 **100% KOMPLETT - ALLE PHASEN ABGESCHLOSSEN!** 🎉
|
||||
|
||||
---
|
||||
|
||||
## ✅ MIGRIERT - Production-Ready
|
||||
|
||||
### 1. Steps (21 von 21 Steps - 100% Complete!)
|
||||
|
||||
#### Phase 1: Advoware Proxy (4 Steps)
|
||||
- ✅ [`advoware_api_proxy_get_step.py`](steps/advoware_proxy/advoware_api_proxy_get_step.py) - GET Proxy
|
||||
- ✅ [`advoware_api_proxy_post_step.py`](steps/advoware_proxy/advoware_api_proxy_post_step.py) - POST Proxy
|
||||
- ✅ [`advoware_api_proxy_put_step.py`](steps/advoware_proxy/advoware_api_proxy_put_step.py) - PUT Proxy
|
||||
- ✅ [`advoware_api_proxy_delete_step.py`](steps/advoware_proxy/advoware_api_proxy_delete_step.py) - DELETE Proxy
|
||||
|
||||
#### Phase 2: VMH Webhooks (6 Steps)
|
||||
- ✅ [`beteiligte_create_api_step.py`](steps/vmh/webhook/beteiligte_create_api_step.py) - POST /vmh/webhook/beteiligte/create
|
||||
- ✅ [`beteiligte_update_api_step.py`](steps/vmh/webhook/beteiligte_update_api_step.py) - POST /vmh/webhook/beteiligte/update
|
||||
- ✅ [`beteiligte_delete_api_step.py`](steps/vmh/webhook/beteiligte_delete_api_step.py) - POST /vmh/webhook/beteiligte/delete
|
||||
- ✅ [`bankverbindungen_create_api_step.py`](steps/vmh/webhook/bankverbindungen_create_api_step.py) - POST /vmh/webhook/bankverbindungen/create
|
||||
- ✅ [`bankverbindungen_update_api_step.py`](steps/vmh/webhook/bankverbindungen_update_api_step.py) - POST /vmh/webhook/bankverbindungen/update
|
||||
- ✅ [`bankverbindungen_delete_api_step.py`](steps/vmh/webhook/bankverbindungen_delete_api_step.py) - POST /vmh/webhook/bankverbindungen/delete
|
||||
|
||||
#### Phase 3: VMH Sync Handlers (3 Steps)
|
||||
- ✅ [`beteiligte_sync_event_step.py`](steps/vmh/beteiligte_sync_event_step.py) - Subscriber für Queue-Events (mit Kommunikation-Integration!)
|
||||
- ✅ [`bankverbindungen_sync_event_step.py`](steps/vmh/bankverbindungen_sync_event_step.py) - Subscriber für Queue-Events
|
||||
- ✅ [`beteiligte_sync_cron_step.py`](steps/vmh/beteiligte_sync_cron_step.py) - Cron-Job alle 15 Min.
|
||||
|
||||
---
|
||||
|
||||
### 2. Services (11 Module, 100% komplett)
|
||||
|
||||
#### Core APIs
|
||||
- ✅ [`advoware.py`](services/advoware.py) (310 Zeilen) - Advoware API Client mit Token-Auth
|
||||
- ✅ [`advoware_service.py`](services/advoware_service.py) (179 Zeilen) - High-Level Advoware Service
|
||||
- ✅ [`espocrm.py`](services/espocrm.py) (293 Zeilen) - EspoCRM API Client
|
||||
|
||||
#### Mapper & Sync Utils
|
||||
- ✅ [`espocrm_mapper.py`](services/espocrm_mapper.py) (663 Zeilen) - Beteiligte Mapping
|
||||
- ✅ [`bankverbindungen_mapper.py`](services/bankverbindungen_mapper.py) (141 Zeilen) - Bankverbindungen Mapping
|
||||
- ✅ [`beteiligte_sync_utils.py`](services/beteiligte_sync_utils.py) (663 Zeilen) - Distributed Locking, Retry Logic
|
||||
- ✅ [`notification_utils.py`](services/notification_utils.py) (200 Zeilen) - In-App Notifications
|
||||
|
||||
#### Phase 4: Kommunikation Sync
|
||||
- ✅ [`kommunikation_mapper.py`](services/kommunikation_mapper.py) (334 Zeilen) - Email/Phone Mapping mit Base64 Marker
|
||||
- ✅ [`kommunikation_sync_utils.py`](services/kommunikation_sync_utils.py) (999 Zeilen) - Bidirektionaler Sync mit 3-Way Diffing
|
||||
|
||||
#### Phase 5: Adressen Sync (2 Module - Phase 5)
|
||||
- ✅ [`adressen_mapper.py`](services/adressen_mapper.py) (267 Zeilen) - Adressen Mapping
|
||||
- ✅ [`adressen_sync.py`](services/adressen_sync.py) (697 Zeilen) - Adressen Sync mit READ-ONLY Detection
|
||||
|
||||
#### Phase 6: Google Calendar Sync (4 Steps + Utils)
|
||||
- ✅ [`calendar_sync_cron_step.py`](steps/advoware_cal_sync/calendar_sync_cron_step.py) - Cron-Trigger alle 15 Min.
|
||||
- ✅ [`calendar_sync_all_step.py`](steps/advoware_cal_sync/calendar_sync_all_step.py) - Bulk-Sync mit Redis-Priorisierung
|
||||
- ✅ [`calendar_sync_event_step.py`](steps/advoware_cal_sync/calendar_sync_event_step.py) - **1053 Zeilen!** Main Sync Handler
|
||||
- ✅ [`calendar_sync_a9 Topics - 100% Complete!)
|
||||
|
||||
#### VMH Beteiligte
|
||||
- ✅ `vmh.beteiligte.create` - Webhook → Sync Handler
|
||||
- ✅ `vmh.beteiligte.update` - Webhook → Sync Handler
|
||||
- ✅ `vmh.beteiligte.delete` - Webhook → Sync Handler
|
||||
- ✅ `vmh.beteiligte.sync_check` - Cron → Sync Handler
|
||||
|
||||
#### VMH Bankverbindungen
|
||||
- ✅ `vmh.bankverbindungen.create` - Webhook → Sync Handler
|
||||
- ✅ `vmh.bankverbindungen.update` - Webhook → Sync Handler
|
||||
- ✅ `vmh.bankverbindungen.delete` - Webhook → Sync Handler
|
||||
|
||||
#### Calendar Sync
|
||||
- ✅ `calendar_sync_all` - Cron/API → All Step → Employee Events
|
||||
- ✅ `calendar_sync_employee` - All/API → Event Step (Main Sync Logic)
|
||||
|
||||
---
|
||||
|
||||
### 4. HTTP Endpoints (14 Endpoints - 100% Complete!
|
||||
- ✅ `vmh.bankverbindungen.create` - Webhook → Sync Handler
|
||||
- ✅ `vmh.bankverbindungen.update` - Webhook → Sync Handler
|
||||
- ✅ `vmh.bankverbindungen.delete` - Webhook → Sync Handler
|
||||
|
||||
---
|
||||
|
||||
### 4. HTTP Endpoints (13 Endpoints, 100% komplett)
|
||||
|
||||
#### Advoware Proxy (4 Endpoints)
|
||||
- ✅ `GET /advoware/proxy?path=...` - Advoware API Proxy
|
||||
- ✅ `POST /advoware/proxy?path=...` - Advoware API Proxy
|
||||
- ✅ `PUT /advoware/proxy?path=...` - Advoware API Proxy
|
||||
- ✅ `DELETE /advoware/proxy?path=...` - Advoware API Proxy
|
||||
|
||||
#### VMH Webhooks - Beteiligte (3 Endpoints)
|
||||
- ✅ `POST /vmh/webhook/beteiligte/create` - EspoCRM Webhook Handler
|
||||
- ✅ `POST /vmh/webhook/beteiligte/update` - EspoCRM Webhook Handler
|
||||
- ✅ `POST /vmh/webhook/beteiligte/delete` - EspoCRM Webhook Handler
|
||||
|
||||
#### VMH Webhooks - Bankverbindungen (3 Endpoints)
|
||||
- ✅ `Calendar Sync (1 Endpoint)
|
||||
- ✅ `POST /advoware/calendar/sync` - Manual Calendar Sync Trigger (kuerzel or "ALL")
|
||||
|
||||
#### POST /vmh/webhook/bankverbindungen/create` - EspoCRM Webhook Handler
|
||||
- ✅ `POST /vmh/webhook/bankverbindungen/update` - EspoCRM Webhook Handler
|
||||
- ✅ `POST /vmh/webhook/bankverbindungen/delete` - EspoCRM Webhook Handler
|
||||
|
||||
#### Example Ticketing (6 Endpoints - Demo)
|
||||
- ✅ `POST /tickets` - Create Ticket
|
||||
- ✅ `GET /tickets` - List Tickets
|
||||
- ✅ `POST /tickets/{id}/triage` - Triage
|
||||
- ✅ `POST /tickets/{id}/escalate` - Escalate
|
||||
- ✅ `POST /tickets/{id}/notify` - Notify Customer
|
||||
- ✅ Cron: SLA Monitor
|
||||
2 Jobs - 100% Complete!)
|
||||
|
||||
- ✅ **VMH Beteiligte Sync Cron** (alle 15 Min.)
|
||||
- Findet Entities mit Status: `pending_sync`, `dirty`, `failed`
|
||||
- Auto-Reset für `permanently_failed` nach 24h
|
||||
- Findet `clean` Entities > 24h nicht gesynct
|
||||
- Emittiert `vmh.beteiligte.sync_check` Events
|
||||
|
||||
- ✅ **Calendar Sync Cron** (alle 15 Min.)
|
||||
- Emittiert `calendar_sync_all` Events
|
||||
- Triggered Bulk-Sync für alle oder priorisierte Mitarbeiter
|
||||
- Redis-basierte Priorisierung (älteste zuerst)
|
||||
|
||||
---
|
||||
|
||||
### 6. Dependencies (pyproject.toml - 100% Complete!
|
||||
---
|
||||
|
||||
### 6. Dependencies (pyproject.toml aktualisiert)
|
||||
|
||||
```toml
|
||||
dependencies = [
|
||||
"asyncpg>=0.29.0", # ✅ NEU für Calendar Sync (PostgreSQL)
|
||||
"google-api-python-client>=2.100.0", # ✅ NEU für Calendar Sync
|
||||
"google-auth>=2.23.0", # ✅ NEU für Calendar Sync
|
||||
"backoff>=2.2.1", # ✅ NEU für Calendar Sync (Retry Logic)
|
||||
]
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## ❌ NICHT MIGRIERT → ALLE MIGRIERT! 🎉
|
||||
|
||||
~~### Phase 6: Google Calendar Sync (4 Steps)~~
|
||||
|
||||
**Status:** ✅ **VOLLSTÄNDIG MIGRIERT!** (1. März 2026)
|
||||
|
||||
- ✅ `calendar_sync_cron_step.py` - Cron-Trigger (alle 15 Min.)
|
||||
- ✅ `calendar_sync_all_step.py` - Bulk-Sync Handler
|
||||
- ✅ `calendar_sync_event_step.py` - Queue-Event Handler (**1053 Zeilen!**)
|
||||
- ✅ `calendar_sync_api_step.py` - HTTP API für manuellen Trigger
|
||||
- ✅ `calendar_sync_utils.py` - Hilfs-Funktionen
|
||||
|
||||
**Dependencies (ALLE installiert):**
|
||||
- ✅ `google-api-python-client` - Google Calendar API
|
||||
- ✅ `google-auth` - Google OAuth2
|
||||
- ✅ `asyncpg` - PostgreSQL Connection
|
||||
- ✅ `backoff` - Retry/Backoff Logic
|
||||
|
||||
**Migration abgeschlossen in:** ~4 Stunden (statt geschätzt 3-5 Tage
|
||||
|
||||
**Dependencies (nicht benötigt):**
|
||||
- ❌ `google-api-python-client` - Google Calendar API
|
||||
- ❌ `google-auth` - Google OAuth2
|
||||
- ❌ PostgreSQL Connection - Für Termine-Datenbank
|
||||
|
||||
**Geschätzte Migration:** 3-5 Tage (komplex wegen Google API + PostgreSQL)
|
||||
**Priorität:** MEDIUM (funktioniert aktuell im old-motia)
|
||||
|
||||
---
|
||||
|
||||
### Root-Level Steps (Test/Specialized Logic)
|
||||
|
||||
**Status:** Bewusst NICHT migriert (nicht Teil der Core-Funktionalität)
|
||||
|
||||
- ❌ `/opt/motia-iii/old-motia/steps/crm-bbl-vmh-reset-nextcall_step.py` (96 Zeilen)
|
||||
- **Zweck:** CVmhErstgespraech Status-Check Cron-Job
|
||||
- **Grund:** Spezialisierte Business-Logik, nicht Teil der Core-Sync-Infrastruktur
|
||||
- **Status:** Kann bei Bedarf später migriert werden
|
||||
|
||||
- ❌ `/opt/motia-iii/old-motia/steps/event_step.py` (Test/Demo)
|
||||
- ❌ `/opt/motia-iii/old-motia/steps/hello_step.py` (Test/Demo)
|
||||
|
||||
---
|
||||
|
||||
## 📊 Migrations-Statistik
|
||||
|
||||
| Kategorie | Migriert | 21 | 0 | 21 | **100%** ✅ |
|
||||
| **Service Module** | 11 | 0 | 11 | **100%** ✅ |
|
||||
| **Queue Events** | 9 | 0 | 9 | **100%** ✅ |
|
||||
| **HTTP Endpoints** | 14 | 0 | 14 | **100%** ✅ |
|
||||
| **Cron Jobs** | 2 | 0 | 2 | **100%** ✅ |
|
||||
| **Code (Zeilen)** | ~9.000 | 0 | ~9.000 | **100%** ✅ |
|
||||
|
||||
---
|
||||
|
||||
## 🎯 Funktionalitäts-Matrix
|
||||
|
||||
| Feature | Old-Motia | Motia III | Status |
|
||||
|---------|-----------|-----------|--------|
|
||||
| **Advoware Proxy API** | ✅ | ✅ | ✅ KOMPLETT |
|
||||
| **VMH Beteiligte Sync** | ✅ | ✅ | ✅ KOMPLETT |
|
||||
| **VMH Bankverbindungen Sync** | ✅ | ✅ | ✅ KOMPLETT |
|
||||
| **Kommunikation Sync (Email/Phone)** | ✅ | ✅ | ✅ KOMPLETT |
|
||||
| **Adressen Sync** | ✅ | ✅ | ✅ KOMPLETT |
|
||||
| **EspoCRM Webhooks** | ✅ | ✅ | ✅ KOMPLETT |
|
||||
| **Distributed Locking** | ✅ | ✅ | ✅ KOMPLETT |
|
||||
| **Retry Logic & Backoff** | ✅ | ✅ | ✅ KOMPLETT |
|
||||
| **Notifications** | ✅ | ✅ | ✅ KOMPLETT |
|
||||
| **Sync Validation** | ✅ | ✅ | ✅ KOMPLETT |
|
||||
| **Cron-basierter Auto-Retry** | ✅ | ✅ | ✅ KOMPLETT |
|
||||
| **Google Calendar Sync** | ✅ | ✅ | ✅ **KOMPLETT** |
|
||||
|
||||
---
|
||||
|
||||
## 🏆 Migration erfolgreich abgeschlossen!
|
||||
|
||||
**Alle 21 Production Steps, 11 Service Module, 9 Queue Events, 14 HTTP Endpoints und 2 Cron Jobs wurden erfolgreich migriert!**
|
||||
| **Cron-basierter Auto-Retry** | ✅ | ✅ | ✅ KOMPLETT |
|
||||
| **Google Calendar Sync** | ✅ | ❌ | ⏳ PHASE 6 |
|
||||
| **CVmhErstgespraech Logic** | ✅ | ❌ | ⏳ Optional |
|
||||
|
||||
---
|
||||
|
||||
## 🔄 Sync-Architektur Übersicht
|
||||
|
||||
```
|
||||
┌─────────────────┐
|
||||
│ EspoCRM API │
|
||||
└────────┬────────┘
|
||||
│ Webhooks
|
||||
▼
|
||||
┌─────────────────────────────────────┐
|
||||
│ VMH Webhook Steps (6 Endpoints) │
|
||||
│ • Batch & Single Entity Support │
|
||||
│ • Deduplication │
|
||||
└────────┬────────────────────────────┘
|
||||
│ Emits Queue Events
|
||||
▼
|
||||
┌─────────────────────────────────────┐
|
||||
│ Queue System (Redis/Builtin) │
|
||||
│ • vmh.beteiligte.* │
|
||||
│ • vmh.bankverbindungen.* │
|
||||
└────────┬────────────────────────────┘
|
||||
│
|
||||
▼
|
||||
┌─────────────────────────────────────┐
|
||||
│ Sync Event Handlers (3 Steps) │
|
||||
│ • Distributed Locking (Redis) │
|
||||
│ • Retry Logic & Backoff │
|
||||
│ • Conflict Resolution │
|
||||
└────────┬────────────────────────────┘
|
||||
│
|
||||
├──► Stammdaten Sync
|
||||
│ (espocrm_mapper.py)
|
||||
│
|
||||
├──► Kommunikation Sync ✅ NEW!
|
||||
│ (kommunikation_sync_utils.py)
|
||||
│ • 3-Way Diffing
|
||||
│ • Bidirectional
|
||||
│ • Slot-Management
|
||||
│
|
||||
└──► Adressen Sync ✅ NEW!
|
||||
(adressen_sync.py)
|
||||
• CREATE/UPDATE/DELETE
|
||||
• READ-ONLY Detection
|
||||
|
||||
▼
|
||||
┌─────────────────────────────────────┐
|
||||
│ Advoware API (advoware.py) │
|
||||
│ • Token-based Auth │
|
||||
│ • HMAC Signing │
|
||||
└─────────────────────────────────────┘
|
||||
|
||||
┌──────────────────┐
|
||||
│ Cron Job (15min)│
|
||||
└────────┬─────────┘
|
||||
│
|
||||
▼ Emits sync_check Events
|
||||
┌─────────────────────────┐
|
||||
│ Auto-Retry & Cleanup │
|
||||
│ • pending_sync │
|
||||
│ • dirty │
|
||||
│ • failed → retry │
|
||||
│ • permanently_failed │
|
||||
│ → auto-reset (24h) │
|
||||
└─────────────────────────┘
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## ✅ FAZIT
|
||||
|
||||
**Die gesamte Core-Funktionalität (außer Google Calendar) wurde erfolgreich migriert!**
|
||||
|
||||
### Production-Ready Features:
|
||||
1. ✅ Vollständige Advoware ↔ EspoCRM Synchronisation
|
||||
2. ✅ Bidirektionale Kommunikationsdaten (Email/Phone)
|
||||
3. ✅ Bidirektionale Adressen
|
||||
4. ✅ Webhook-basierte Event-Verarbeitung
|
||||
5. ✅ Automatisches Retry-System
|
||||
6. ✅ Distributed Locking
|
||||
7. ✅ Konflikt-Erkennung & Resolution
|
||||
|
||||
### Code-Qualität:
|
||||
- ✅ Keine Compile-Errors
|
||||
- ✅ Motia III API korrekt verwendet
|
||||
- ✅ Alle Dependencies vorhanden
|
||||
- ✅ Type-Hints (Pydantic Models)
|
||||
- ✅ Error-Handling & Logging
|
||||
|
||||
### Deployment:
|
||||
- ✅ Alle Steps registriert
|
||||
- ✅ Queue-System konfiguriert
|
||||
- ✅ Cron-Jobs aktiv
|
||||
- ✅ Redis-Integration
|
||||
|
||||
**Das System ist bereit für Production! 🚀**
|
||||
@@ -1,276 +0,0 @@
|
||||
# Motia Migration Status
|
||||
|
||||
**🎉 MIGRATION 100% KOMPLETT**
|
||||
|
||||
> 📋 Detaillierte Analyse: [MIGRATION_COMPLETE_ANALYSIS.md](MIGRATION_COMPLETE_ANALYSIS.md)
|
||||
|
||||
## Quick Stats
|
||||
|
||||
- ✅ **21 von 21 Steps** migriert (100%)
|
||||
- ✅ **11 von 11 Service-Module** migriert (100%)
|
||||
- ✅ **~9.000 Zeilen Code** migriert (100%)
|
||||
- ✅ **14 HTTP Endpoints** aktiv
|
||||
- ✅ **9 Queue Events** konfiguriert
|
||||
- ✅ **2 Cron Jobs** (VMH: alle 15 Min., Calendar: alle 15 Min.)
|
||||
|
||||
---
|
||||
|
||||
## Overview
|
||||
Migrating from **old-motia v0.17** (Node.js + Python hybrid) to **Motia III v1.0-RC** (pure Python).
|
||||
|
||||
## Old System Analysis
|
||||
|
||||
### Location
|
||||
- Old system: `/opt/motia-iii/old-motia/`
|
||||
- Old project dir: `/opt/motia-iii/old-motia/bitbylaw/`
|
||||
|
||||
### Steps Found in Old System
|
||||
|
||||
#### Root Steps (`/opt/motia-iii/old-motia/steps/`)
|
||||
1. `crm-bbl-vmh-reset-nextcall_step.py`
|
||||
2. `event_step.py`
|
||||
3. `hello_step.py`
|
||||
|
||||
#### BitByLaw Steps (`/opt/motia-iii/old-motia/bitbylaw/steps/`)
|
||||
|
||||
**Advoware Calendar Sync** (`advoware_cal_sync/`):
|
||||
- `calendar_sync_all_step.py`
|
||||
- `calendar_sync_api_step.py`
|
||||
- `calendar_sync_cron_step.py`
|
||||
- `calendar_sync_event_step.py`
|
||||
- `audit_calendar_sync.py`
|
||||
- `calendar_sync_utils.py` (utility module)
|
||||
|
||||
**Advoware Proxy** (`advoware_proxy/`):
|
||||
- `advoware_api_proxy_get_step.py`
|
||||
- `advoware_api_proxy_post_step.py`
|
||||
- `advoware_api_proxy_put_step.py`
|
||||
- `advoware_api_proxy_delete_step.py`
|
||||
|
||||
**VMH Integration** (`vmh/`):
|
||||
- `beteiligte_sync_cron_step.py`
|
||||
- `beteiligte_sync_event_step.py`
|
||||
- `bankverbindungen_sync_event_step.py`
|
||||
- `webhook/bankverbindungen_create_api_step.py`
|
||||
- `webhook/bankverbindungen_update_api_step.py`
|
||||
- `webhook/bankverbindungen_delete_api_step.py`
|
||||
- `webhook/beteiligte_create_api_step.py`
|
||||
- `webhook/beteiligte_update_api_step.py`
|
||||
- `webhook/beteiligte_delete_api_step.py`
|
||||
|
||||
### Supporting Services/Modules
|
||||
|
||||
From `/opt/motia-iii/old-motia/bitbylaw/`:
|
||||
- `services/advoware.py` - Advoware API wrapper
|
||||
- `config.py` - Configuration module
|
||||
- Dependencies: PostgreSQL, Redis, Google Calendar API
|
||||
|
||||
## Migration Changes Required
|
||||
|
||||
### Key Structural Changes
|
||||
|
||||
#### 1. Config Format
|
||||
```python
|
||||
# OLD
|
||||
config = {
|
||||
"type": "api", # or "event", "cron"
|
||||
"name": "StepName",
|
||||
"path": "/endpoint",
|
||||
"method": "GET",
|
||||
"cron": "0 5 * * *",
|
||||
"subscribes": ["topic"],
|
||||
"emits": ["other-topic"]
|
||||
}
|
||||
|
||||
# NEW
|
||||
from motia import http, queue, cron
|
||||
|
||||
config = {
|
||||
"name": "StepName",
|
||||
"flows": ["flow-name"],
|
||||
"triggers": [
|
||||
http("GET", "/endpoint")
|
||||
# or queue("topic", input=schema)
|
||||
# or cron("0 0 5 * * *") # 6-field!
|
||||
],
|
||||
"enqueues": ["other-topic"]
|
||||
}
|
||||
```
|
||||
|
||||
#### 2. Handler Signature
|
||||
```python
|
||||
# OLD - API
|
||||
async def handler(req, context):
|
||||
body = req.get('body', {})
|
||||
await context.emit({"topic": "x", "data": {...}})
|
||||
return {"status": 200, "body": {...}}
|
||||
|
||||
# NEW - API
|
||||
from motia import ApiRequest, ApiResponse, FlowContext
|
||||
|
||||
async def handler(request: ApiRequest, ctx: FlowContext) -> ApiResponse:
|
||||
body = request.body
|
||||
await ctx.enqueue({"topic": "x", "data": {...}})
|
||||
return ApiResponse(status=200, body={...})
|
||||
|
||||
# OLD - Event/Queue
|
||||
async def handler(data, context):
|
||||
context.logger.info(data['field'])
|
||||
|
||||
# NEW - Queue
|
||||
async def handler(input_data: dict, ctx: FlowContext):
|
||||
ctx.logger.info(input_data['field'])
|
||||
|
||||
# OLD - Cron
|
||||
async def handler(context):
|
||||
context.logger.info("Running")
|
||||
|
||||
# NEW - Cron
|
||||
async def handler(input_data: dict, ctx: FlowContext):
|
||||
ctx.logger.info("Running")
|
||||
```
|
||||
|
||||
#### 3. Method Changes
|
||||
- `context.emit()` → `ctx.enqueue()`
|
||||
- `req.get('body')` → `request.body`
|
||||
- `req.get('queryParams')` → `request.query_params`
|
||||
- `req.get('pathParams')` → `request.path_params`
|
||||
- `req.get('headers')` → `request.headers`
|
||||
- Return dict → `ApiResponse` object
|
||||
|
||||
#### 4. Cron Format
|
||||
- OLD: 5-field `"0 5 * * *"` (minute hour day month weekday)
|
||||
- NEW: 6-field `"0 0 5 * * *"` (second minute hour day month weekday)
|
||||
|
||||
## Migration Strategy
|
||||
|
||||
### Phase 1: Simple Steps (Priority)
|
||||
Start with simple API proxy steps as they're straightforward:
|
||||
1. ✅ Example ticketing steps (already in new system)
|
||||
2. ⏳ Advoware proxy steps (GET, POST, PUT, DELETE)
|
||||
3. ⏳ Simple webhook handlers
|
||||
|
||||
### Phase 2: Complex Integration Steps
|
||||
Steps with external dependencies:
|
||||
4. ⏳ VMH sync steps (beteiligte, bankverbindungen)
|
||||
5. ⏳ Calendar sync steps (most complex - Google Calendar + Redis + PostgreSQL)
|
||||
|
||||
### Phase 3: Supporting Infrastructure
|
||||
- Migrate `services/` modules (advoware.py wrapper)
|
||||
- Migrate `config.py` to use environment variables properly
|
||||
- Update dependencies in `pyproject.toml`
|
||||
|
||||
### Dependencies to Review
|
||||
From old `requirements.txt` and code analysis:
|
||||
- `asyncpg` - PostgreSQL async driver
|
||||
- `redis` - Redis client
|
||||
- `google-api-python-client` - Google Calendar API
|
||||
- `google-auth` - Google OAuth2
|
||||
- `backoff` - Retry/backoff decorator
|
||||
- `pytz` - Timezone handling
|
||||
- `pydantic` - Already in new system
|
||||
- `requests` / `aiohttp` - HTTP clients for Advoware API
|
||||
|
||||
## Migration Roadmap
|
||||
|
||||
### ✅ COMPLETED
|
||||
|
||||
| Phase | Module | Lines | Status |
|
||||
|-------|--------|-------|--------|
|
||||
| **1** | Advoware Proxy (GET, POST, PUT, DELETE) | ~400 | ✅ Complete |
|
||||
| **1** | `advoware.py`, `advoware_service.py` | ~800 | ✅ Complete |
|
||||
| **2** | VMH Webhook Steps (6 endpoints) | ~900 | ✅ Complete |
|
||||
| **2** | `espocrm.py`, `espocrm_mapper.py` | ~900 | ✅ Complete |
|
||||
| **2** | `bankverbindungen_mapper.py`, `beteiligte_sync_utils.py`, `notification_utils.py` | ~1200 | ✅ Complete |
|
||||
| **3** | VMH Sync Event Steps (2 handlers + 1 cron) | ~1000 | ✅ Complete |
|
||||
| **4** | Kommunikation Sync (`kommunikation_mapper.py`, `kommunikation_sync_utils.py`) | ~1333 | ✅ Complete |
|
||||
| **5** | Adressen Sync (`adressen_mapper.py`, `adressen_sync.py`) | ~964 | ✅ Complete |
|
||||
| **6** | **Google Calendar Sync** (`calendar_sync_*.py`, `calendar_sync_utils.py`) | ~1500 | ✅ **Complete** |
|
||||
|
||||
**Total migrated: ~9.000 lines of production code**
|
||||
|
||||
### ✅ Phase 6 COMPLETED: Google Calendar Sync
|
||||
|
||||
**Advoware Calendar Sync** - Google Calendar ↔ Advoware Sync:
|
||||
- ✅ `calendar_sync_cron_step.py` - Cron-Trigger (alle 15 Min.)
|
||||
- ✅ `calendar_sync_all_step.py` - Bulk-Sync Handler mit Redis-basierter Priorisierung
|
||||
- ✅ `calendar_sync_event_step.py` - Queue-Event Handler (**1053 Zeilen komplexe Sync-Logik!**)
|
||||
- ✅ `calendar_sync_api_step.py` - HTTP API für manuellen Trigger
|
||||
- ✅ `calendar_sync_utils.py` - Hilfs-Funktionen (DB, Google Service, Redis, Logging)
|
||||
|
||||
**Dependencies:**
|
||||
- ✅ `google-api-python-client` - Google Calendar API
|
||||
- ✅ `google-auth` - Google OAuth2
|
||||
- ✅ `asyncpg` - PostgreSQL async driver
|
||||
- ✅ `backoff` - Retry/backoff decorator
|
||||
|
||||
**Features:**
|
||||
- ✅ Bidirektionale Synchronisation (Google ↔ Advoware)
|
||||
- ✅ 4-Phase Sync-Algorithmus (New Adv→Google, New Google→Adv, Deletes, Updates)
|
||||
- ✅ PostgreSQL als Sync-State Hub (calendar_sync Tabelle)
|
||||
- ✅ Redis-basiertes Rate Limiting (Token Bucket für Google API)
|
||||
- ✅ Distributed Locking per Employee
|
||||
- ✅ Automatische Calendar-Creation mit ACL
|
||||
- ✅ Recurring Events Support (RRULE)
|
||||
- ✅ Timezone-Handling (Europe/Berlin)
|
||||
- ✅ Backoff-Retry für API-Fehler
|
||||
- ✅ Write-Protection für Advoware
|
||||
- ✅ Source-System-Wins & Last-Change-Wins Strategien
|
||||
|
||||
### ⏳ REMAINING
|
||||
|
||||
**Keine! Die Migration ist zu 100% abgeschlossen.**
|
||||
|
||||
### Completed
|
||||
- ✅ Analysis of old system structure
|
||||
- ✅ MIGRATION_GUIDE.md reviewed
|
||||
- ✅ Migration patterns documented
|
||||
- ✅ New system has example ticketing steps
|
||||
- ✅ **Phase 1: Advoware Proxy Steps migrated** (GET, POST, PUT, DELETE)
|
||||
- ✅ **Advoware API service module migrated** (services/advoware.py)
|
||||
- ✅ **Phase 2: VMH Integration - Webhook Steps migrated** (6 endpoints)
|
||||
- ✅ **EspoCRM API service module migrated** (services/espocrm.py)
|
||||
- ✅ All endpoints registered and running:
|
||||
- **Advoware Proxy:**
|
||||
- `GET /advoware/proxy6 Complete ✅
|
||||
|
||||
**🎉 ALLE PHASEN ABGESCHLOSSEN! 100% MIGRATION ERFOLGREICH!**
|
||||
|
||||
**Phase 6** - Google Calendar Sync:
|
||||
- ✅ `calendar_sync_cron_step.py` (Cron-Trigger alle 15 Min.)
|
||||
- ✅ `calendar_sync_all_step.py` (Bulk-Handler mit Redis-Priorisierung)
|
||||
- ✅ `calendar_sync_event_step.py` (1053 Zeilen - 4-Phase Sync-Algorithmus)
|
||||
- ✅ `calendar_sync_api_step.py` (HTTP API für manuelle Triggers)
|
||||
- ✅ `calendar_sync_utils.py` (DB, Google Service, Redis Client)
|
||||
|
||||
**Sync-Architektur komplett:**
|
||||
|
||||
1. **Advoware Proxy** (Phase 1) → HTTP API für Advoware-Zugriff
|
||||
2. **Webhooks** (Phase 2) → Emittieren Queue-Events
|
||||
3. **Event Handler** (Phase 3) → Verarbeiten Events mit Stammdaten-Sync
|
||||
4. **Kommunikation Sync** (Phase 4) → Bidirektionale Email/Phone-Synchronisation
|
||||
5. **Adressen Sync** (Phase 5) → Bidirektionale Adressen-Synchronisation
|
||||
6. **Calendar Sync** (Phase 6) → Google Calendar ↔ Advoware Bidirektional
|
||||
7. **Cron Jobs** (Phase 3 & 6) → Regelmäßige Sync-Checks & Auto-Retries
|
||||
|
||||
Die vollständige Synchronisations- und Integrations-Pipeline ist nun zu 100%
|
||||
**Phase 5** - Adressen Sync:
|
||||
- ✅ `adressen_mapper.py` (267 Zeilen - CAdressen ↔ Advoware Adressen)
|
||||
- ✅ `adressen_sync.py` (697 Zeilen - CREATE/UPDATE mit READ-ONLY Detection)
|
||||
|
||||
### Sync-Architektur komplett:
|
||||
|
||||
1. **Webhooks** (Phase 2) → Emittieren Queue-Events
|
||||
2. **Event Handler** (Phase 3) → Verarbeiten Events mit Stammdaten-Sync
|
||||
3. **Kommunikation Sync** (Phase 4) → Bidirektionale Email/Phone-Synchronisation
|
||||
4. **Adressen Sync** (Phase 5) → Bidirektionale Adressen-Synchronisation
|
||||
5. **Cron Job** (Phase 3) → Regelmäßige Sync-Checks & Auto-Retries
|
||||
|
||||
Die vollständige Synchronisations-Pipeline ist nun einsatzbereit!
|
||||
|
||||
## Notes
|
||||
- Old system was Node.js + Python hybrid (Python steps as child processes)
|
||||
- New system is pure Python (standalone SDK)
|
||||
- No need for Node.js/npm anymore
|
||||
- iii engine handles all infrastructure (queues, state, HTTP, cron)
|
||||
- Console replaced Workbench
|
||||
382
REFACTORING_SUMMARY.md
Normal file
382
REFACTORING_SUMMARY.md
Normal file
@@ -0,0 +1,382 @@
|
||||
# Code Refactoring - Verbesserungen Übersicht
|
||||
|
||||
Datum: 3. März 2026
|
||||
|
||||
## Zusammenfassung
|
||||
|
||||
Umfassendes Refactoring zur Verbesserung von Robustheit, Eleganz und Effizienz des BitByLaw Integration Codes.
|
||||
|
||||
## Implementierte Verbesserungen
|
||||
|
||||
### 1. ✅ Custom Exception Classes ([services/exceptions.py](services/exceptions.py))
|
||||
|
||||
**Problem:** Zu generisches Exception Handling mit `except Exception`
|
||||
|
||||
**Lösung:** Hierarchische Exception-Struktur:
|
||||
|
||||
```python
|
||||
from services.exceptions import (
|
||||
AdvowareAPIError,
|
||||
AdvowareAuthError,
|
||||
AdvowareTimeoutError,
|
||||
EspoCRMAPIError,
|
||||
EspoCRMAuthError,
|
||||
RetryableError,
|
||||
NonRetryableError,
|
||||
LockAcquisitionError,
|
||||
ValidationError
|
||||
)
|
||||
|
||||
# Verwendung:
|
||||
try:
|
||||
result = await advoware.api_call(...)
|
||||
except AdvowareTimeoutError:
|
||||
# Spezifisch für Timeouts
|
||||
raise RetryableError()
|
||||
except AdvowareAuthError:
|
||||
# Auth-Fehler nicht retryable
|
||||
raise
|
||||
except AdvowareAPIError as e:
|
||||
# Andere API-Fehler
|
||||
if is_retryable(e):
|
||||
# Retry logic
|
||||
```
|
||||
|
||||
**Vorteile:**
|
||||
- Präzise Fehlerbehandlung
|
||||
- Besseres Error Tracking
|
||||
- Automatische Retry-Klassifizierung mit `is_retryable()`
|
||||
|
||||
---
|
||||
|
||||
### 2. ✅ Redis Client Factory ([services/redis_client.py](services/redis_client.py))
|
||||
|
||||
**Problem:** Duplizierte Redis-Initialisierung in 4+ Dateien
|
||||
|
||||
**Lösung:** Zentralisierte Redis Client Factory mit Singleton Pattern:
|
||||
|
||||
```python
|
||||
from services.redis_client import get_redis_client, is_redis_available
|
||||
|
||||
# Strict mode: Exception bei Fehler
|
||||
redis_client = get_redis_client(strict=True)
|
||||
|
||||
# Optional mode: None bei Fehler (für optionale Features)
|
||||
redis_client = get_redis_client(strict=False)
|
||||
|
||||
# Health Check
|
||||
if is_redis_available():
|
||||
# Redis verfügbar
|
||||
```
|
||||
|
||||
**Vorteile:**
|
||||
- DRY (Don't Repeat Yourself)
|
||||
- Connection Pooling
|
||||
- Zentrale Konfiguration
|
||||
- Health Checks
|
||||
|
||||
---
|
||||
|
||||
### 3. ✅ Pydantic Models für Validation ([services/models.py](services/models.py))
|
||||
|
||||
**Problem:** Keine Datenvalidierung, unsichere Typen
|
||||
|
||||
**Lösung:** Pydantic Models mit automatischer Validierung:
|
||||
|
||||
```python
|
||||
from services.models import (
|
||||
AdvowareBeteiligteCreate,
|
||||
EspoCRMBeteiligteCreate,
|
||||
validate_beteiligte_advoware
|
||||
)
|
||||
|
||||
# Automatische Validierung:
|
||||
try:
|
||||
validated = AdvowareBeteiligteCreate.model_validate(data)
|
||||
except ValidationError as e:
|
||||
# Handle validation errors
|
||||
|
||||
# Helper:
|
||||
validated = validate_beteiligte_advoware(data)
|
||||
```
|
||||
|
||||
**Features:**
|
||||
- Type Safety
|
||||
- Automatische Validierung (Geburtsdatum, Name, etc.)
|
||||
- Enums für Status/Rechtsformen
|
||||
- Field Validators
|
||||
|
||||
---
|
||||
|
||||
### 4. ✅ Zentrale Konfiguration ([services/config.py](services/config.py))
|
||||
|
||||
**Problem:** Magic Numbers und Strings überall im Code
|
||||
|
||||
**Lösung:** Zentrale Config mit Dataclasses:
|
||||
|
||||
```python
|
||||
from services.config import (
|
||||
SYNC_CONFIG,
|
||||
API_CONFIG,
|
||||
ADVOWARE_CONFIG,
|
||||
ESPOCRM_CONFIG,
|
||||
FEATURE_FLAGS,
|
||||
get_retry_delay_seconds,
|
||||
get_lock_key
|
||||
)
|
||||
|
||||
# Verwendung:
|
||||
max_retries = SYNC_CONFIG.max_retries # 5
|
||||
lock_ttl = SYNC_CONFIG.lock_ttl_seconds # 900
|
||||
backoff = SYNC_CONFIG.retry_backoff_minutes # [1, 5, 15, 60, 240]
|
||||
|
||||
# Helper Functions:
|
||||
lock_key = get_lock_key('cbeteiligte', entity_id)
|
||||
retry_delay = get_retry_delay_seconds(attempt=2) # 15 * 60 seconds
|
||||
```
|
||||
|
||||
**Konfigurationsbereiche:**
|
||||
- `SYNC_CONFIG` - Retry, Locking, Change Detection
|
||||
- `API_CONFIG` - Timeouts, Rate Limiting
|
||||
- `ADVOWARE_CONFIG` - Token, Auth, Read-only Fields
|
||||
- `ESPOCRM_CONFIG` - Pagination, Notifications
|
||||
- `FEATURE_FLAGS` - Feature Toggles
|
||||
|
||||
---
|
||||
|
||||
### 5. ✅ Konsistentes Logging ([services/logging_utils.py](services/logging_utils.py))
|
||||
|
||||
**Problem:** Inkonsistentes Logging (3 verschiedene Patterns)
|
||||
|
||||
**Lösung:** Unified Logger mit Context-Support:
|
||||
|
||||
```python
|
||||
from services.logging_utils import get_logger, get_service_logger
|
||||
|
||||
# Service Logger:
|
||||
logger = get_service_logger('advoware', context)
|
||||
logger.info("Message", entity_id="123")
|
||||
|
||||
# Mit Context Manager für Timing:
|
||||
with logger.operation('sync_entity', entity_id='123'):
|
||||
# Do work
|
||||
pass # Automatisches Timing und Error Logging
|
||||
|
||||
# API Call Tracking:
|
||||
with logger.api_call('/api/v1/Beteiligte', method='POST'):
|
||||
result = await api.post(...)
|
||||
```
|
||||
|
||||
**Features:**
|
||||
- Motia FlowContext Support
|
||||
- Structured Logging
|
||||
- Automatisches Performance Tracking
|
||||
- Context Fields
|
||||
|
||||
---
|
||||
|
||||
### 6. ✅ Spezifische Exceptions in Services
|
||||
|
||||
**Aktualisierte Services:**
|
||||
- [advoware.py](services/advoware.py) - AdvowareAPIError, AdvowareAuthError, AdvowareTimeoutError
|
||||
- [espocrm.py](services/espocrm.py) - EspoCRMAPIError, EspoCRMAuthError, EspoCRMTimeoutError
|
||||
- [sync_utils_base.py](services/sync_utils_base.py) - LockAcquisitionError
|
||||
- [beteiligte_sync_utils.py](services/beteiligte_sync_utils.py) - SyncError
|
||||
|
||||
**Beispiel:**
|
||||
```python
|
||||
# Vorher:
|
||||
except Exception as e:
|
||||
logger.error(f"Error: {e}")
|
||||
|
||||
# Nachher:
|
||||
except AdvowareTimeoutError:
|
||||
raise RetryableError("Request timed out")
|
||||
except AdvowareAuthError:
|
||||
raise # Nicht retryable
|
||||
except AdvowareAPIError as e:
|
||||
if is_retryable(e):
|
||||
# Retry
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
### 7. ✅ Type Hints ergänzt
|
||||
|
||||
**Verbesserte Type Hints in:**
|
||||
- Service-Methoden (advoware.py, espocrm.py)
|
||||
- Mapper-Funktionen (espocrm_mapper.py)
|
||||
- Utility-Klassen (sync_utils_base.py, beteiligte_sync_utils.py)
|
||||
- Step Handler
|
||||
|
||||
**Beispiel:**
|
||||
```python
|
||||
# Vorher:
|
||||
async def handler(event_data, ctx):
|
||||
...
|
||||
|
||||
# Nachher:
|
||||
async def handler(
|
||||
event_data: Dict[str, Any],
|
||||
ctx: FlowContext[Any]
|
||||
) -> Optional[Dict[str, Any]]:
|
||||
...
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Migration Guide
|
||||
|
||||
### Für bestehenden Code
|
||||
|
||||
1. **Exception Handling aktualisieren:**
|
||||
```python
|
||||
# Alt:
|
||||
try:
|
||||
result = await api.call()
|
||||
except Exception as e:
|
||||
logger.error(f"Error: {e}")
|
||||
|
||||
# Neu:
|
||||
try:
|
||||
result = await api.call()
|
||||
except AdvowareTimeoutError:
|
||||
# Spezifisch behandeln
|
||||
raise RetryableError()
|
||||
except AdvowareAPIError as e:
|
||||
logger.error(f"API Error: {e}")
|
||||
if is_retryable(e):
|
||||
# Retry
|
||||
```
|
||||
|
||||
2. **Redis initialisieren:**
|
||||
```python
|
||||
# Alt:
|
||||
redis_client = redis.Redis(host=..., port=...)
|
||||
|
||||
# Neu:
|
||||
from services.redis_client import get_redis_client
|
||||
redis_client = get_redis_client(strict=False)
|
||||
```
|
||||
|
||||
3. **Konstanten verwenden:**
|
||||
```python
|
||||
# Alt:
|
||||
MAX_RETRIES = 5
|
||||
LOCK_TTL = 900
|
||||
|
||||
# Neu:
|
||||
from services.config import SYNC_CONFIG
|
||||
max_retries = SYNC_CONFIG.max_retries
|
||||
lock_ttl = SYNC_CONFIG.lock_ttl_seconds
|
||||
```
|
||||
|
||||
4. **Logging standardisieren:**
|
||||
```python
|
||||
# Alt:
|
||||
logger = logging.getLogger(__name__)
|
||||
logger.info("Message")
|
||||
|
||||
# Neu:
|
||||
from services.logging_utils import get_service_logger
|
||||
logger = get_service_logger('my_service', context)
|
||||
logger.info("Message", entity_id="123")
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Performance-Verbesserungen
|
||||
|
||||
- ✅ Redis Connection Pooling (max 50 Connections)
|
||||
- ✅ Token Caching optimiert
|
||||
- ✅ Bessere Error Classification (weniger unnötige Retries)
|
||||
- ⚠️ Noch TODO: Batch Operations für parallele Syncs
|
||||
|
||||
---
|
||||
|
||||
## Feature Flags
|
||||
|
||||
Neue Features können über `FEATURE_FLAGS` gesteuert werden:
|
||||
|
||||
```python
|
||||
from services.config import FEATURE_FLAGS
|
||||
|
||||
# Aktivieren/Deaktivieren:
|
||||
FEATURE_FLAGS.strict_validation = True # Pydantic Validation
|
||||
FEATURE_FLAGS.kommunikation_sync_enabled = False # Noch in Entwicklung
|
||||
FEATURE_FLAGS.parallel_sync_enabled = False # Experimentell
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Testing
|
||||
|
||||
**Unit Tests sollten nun leichter sein:**
|
||||
|
||||
```python
|
||||
# Mock Redis:
|
||||
from services.redis_client import RedisClientFactory
|
||||
RedisClientFactory._instance = mock_redis
|
||||
|
||||
# Mock Exceptions:
|
||||
from services.exceptions import AdvowareAPIError
|
||||
raise AdvowareAPIError("Test error", status_code=500)
|
||||
|
||||
# Validate Models:
|
||||
from services.models import validate_beteiligte_advoware
|
||||
with pytest.raises(ValidationError):
|
||||
validate_beteiligte_advoware(invalid_data)
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Nächste Schritte
|
||||
|
||||
1. **Unit Tests schreiben** (min. 60% Coverage)
|
||||
- Exception Handling Tests
|
||||
- Mapper Tests mit Pydantic
|
||||
- Redis Factory Tests
|
||||
|
||||
2. **Batch Operations** implementieren
|
||||
- Parallele API-Calls
|
||||
- Bulk Updates
|
||||
|
||||
3. **Monitoring** verbessern
|
||||
- Performance Metrics aus Logger nutzen
|
||||
- Redis Health Checks
|
||||
|
||||
4. **Dokumentation** erweitern
|
||||
- API-Docs generieren (Sphinx)
|
||||
- Error Handling Guide
|
||||
|
||||
---
|
||||
|
||||
## Breakfree Changes
|
||||
|
||||
⚠️ **Minimale Breaking Changes:**
|
||||
|
||||
1. Import-Pfade haben sich geändert:
|
||||
- `AdvowareTokenError` → `AdvowareAuthError`
|
||||
- `EspoCRMError` → `EspoCRMAPIError`
|
||||
|
||||
2. Redis wird jetzt über Factory bezogen:
|
||||
- Statt direktem `redis.Redis()` → `get_redis_client()`
|
||||
|
||||
**Migration ist einfach:** Imports aktualisieren, Code läuft sonst identisch.
|
||||
|
||||
---
|
||||
|
||||
## Autoren
|
||||
|
||||
- Code Refactoring: GitHub Copilot
|
||||
- Review: BitByLaw Team
|
||||
- Datum: 3. März 2026
|
||||
|
||||
---
|
||||
|
||||
## Fragen?
|
||||
|
||||
Bei Fragen zum Refactoring siehe:
|
||||
- [services/README.md](services/README.md) - Service-Layer Dokumentation
|
||||
- [exceptions.py](services/exceptions.py) - Exception Hierarchie
|
||||
- [config.py](services/config.py) - Alle Konfigurationsoptionen
|
||||
599
docs/AI_KNOWLEDGE_SYNC.md
Normal file
599
docs/AI_KNOWLEDGE_SYNC.md
Normal file
@@ -0,0 +1,599 @@
|
||||
# AI Knowledge Collection Sync - Dokumentation
|
||||
|
||||
**Version**: 1.0
|
||||
**Datum**: 11. März 2026
|
||||
**Status**: ✅ Implementiert
|
||||
|
||||
---
|
||||
|
||||
## Überblick
|
||||
|
||||
Synchronisiert EspoCRM `CAIKnowledge` Entities mit XAI Collections für semantische Dokumentensuche. Unterstützt vollständigen Collection-Lifecycle, BLAKE3-basierte Integritätsprüfung und robustes Hash-basiertes Change Detection.
|
||||
|
||||
## Features
|
||||
|
||||
✅ **Collection Lifecycle Management**
|
||||
- NEW → Collection erstellen in XAI
|
||||
- ACTIVE → Automatischer Sync der Dokumente
|
||||
- PAUSED → Sync pausiert, Collection bleibt
|
||||
- DEACTIVATED → Collection aus XAI löschen
|
||||
|
||||
✅ **Dual-Hash Change Detection**
|
||||
- EspoCRM Hash (MD5/SHA256) für lokale Änderungserkennung
|
||||
- XAI BLAKE3 Hash für Remote-Integritätsverifikation
|
||||
- Metadata-Hash für Beschreibungs-Änderungen
|
||||
|
||||
✅ **Robustheit**
|
||||
- BLAKE3 Verification nach jedem Upload
|
||||
- Metadata-Only Updates via PATCH
|
||||
- Orphan Detection & Cleanup
|
||||
- Distributed Locking (Redis)
|
||||
- Daily Full Sync (02:00 Uhr nachts)
|
||||
|
||||
✅ **Fehlerbehandlung**
|
||||
- Unsupported MIME Types → Status "unsupported"
|
||||
- Transient Errors → Retry mit Exponential Backoff
|
||||
- Partial Failures toleriert
|
||||
|
||||
---
|
||||
|
||||
## Architektur
|
||||
|
||||
```
|
||||
┌─────────────────────────────────────────────────────────────────┐
|
||||
│ EspoCRM CAIKnowledge │
|
||||
│ ├─ activationStatus: new/active/paused/deactivated │
|
||||
│ ├─ syncStatus: unclean/pending_sync/synced/failed │
|
||||
│ └─ datenbankId: XAI Collection ID │
|
||||
└─────────────────────────────────────────────────────────────────┘
|
||||
↓ Webhook
|
||||
┌─────────────────────────────────────────────────────────────────┐
|
||||
│ Motia Webhook Handler │
|
||||
│ → POST /vmh/webhook/aiknowledge/update │
|
||||
└─────────────────────────────────────────────────────────────────┘
|
||||
↓ Emit Event
|
||||
┌─────────────────────────────────────────────────────────────────┐
|
||||
│ Queue: aiknowledge.sync │
|
||||
└─────────────────────────────────────────────────────────────────┘
|
||||
↓ Lock: aiknowledge:{id}
|
||||
┌─────────────────────────────────────────────────────────────────┐
|
||||
│ Sync Handler │
|
||||
│ ├─ Check activationStatus │
|
||||
│ ├─ Manage Collection Lifecycle │
|
||||
│ ├─ Sync Documents (with BLAKE3 verification) │
|
||||
│ └─ Update Statuses │
|
||||
└─────────────────────────────────────────────────────────────────┘
|
||||
↓
|
||||
┌─────────────────────────────────────────────────────────────────┐
|
||||
│ XAI Collections API │
|
||||
│ └─ Collections with embedded documents │
|
||||
└─────────────────────────────────────────────────────────────────┘
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## EspoCRM Konfiguration
|
||||
|
||||
### 1. Entity: CAIKnowledge
|
||||
|
||||
**Felder:**
|
||||
|
||||
| Feld | Typ | Beschreibung | Werte |
|
||||
|------|-----|--------------|-------|
|
||||
| `name` | varchar(255) | Name der Knowledge Base | - |
|
||||
| `datenbankId` | varchar(255) | XAI Collection ID | Automatisch gefüllt |
|
||||
| `activationStatus` | enum | Lifecycle-Status | new, active, paused, deactivated |
|
||||
| `syncStatus` | enum | Sync-Status | unclean, pending_sync, synced, failed |
|
||||
| `lastSync` | datetime | Letzter erfolgreicher Sync | ISO 8601 |
|
||||
| `syncError` | text | Fehlermeldung bei Failure | Max 2000 Zeichen |
|
||||
|
||||
**Enum-Definitionen:**
|
||||
|
||||
```json
|
||||
{
|
||||
"activationStatus": {
|
||||
"type": "enum",
|
||||
"options": ["new", "active", "paused", "deactivated"],
|
||||
"default": "new"
|
||||
},
|
||||
"syncStatus": {
|
||||
"type": "enum",
|
||||
"options": ["unclean", "pending_sync", "synced", "failed"],
|
||||
"default": "unclean"
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
### 2. Junction: CAIKnowledgeCDokumente
|
||||
|
||||
**additionalColumns:**
|
||||
|
||||
| Feld | Typ | Beschreibung |
|
||||
|------|-----|--------------|
|
||||
| `aiDocumentId` | varchar(255) | XAI file_id |
|
||||
| `syncstatus` | enum | Per-Document Sync-Status |
|
||||
| `syncedHash` | varchar(64) | MD5/SHA256 von EspoCRM |
|
||||
| `xaiBlake3Hash` | varchar(128) | BLAKE3 Hash von XAI |
|
||||
| `syncedMetadataHash` | varchar(64) | Hash der Metadaten |
|
||||
| `lastSync` | datetime | Letzter Sync dieses Dokuments |
|
||||
|
||||
**Enum-Definition:**
|
||||
|
||||
```json
|
||||
{
|
||||
"syncstatus": {
|
||||
"type": "enum",
|
||||
"options": ["new", "unclean", "synced", "failed", "unsupported"]
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
### 3. Webhooks
|
||||
|
||||
**Webhook 1: CREATE**
|
||||
```json
|
||||
{
|
||||
"event": "CAIKnowledge.afterSave",
|
||||
"url": "https://your-motia-domain.com/vmh/webhook/aiknowledge/update",
|
||||
"method": "POST",
|
||||
"payload": "{\"entity_id\": \"{$id}\", \"entity_type\": \"CAIKnowledge\", \"action\": \"create\"}",
|
||||
"condition": "entity.isNew()"
|
||||
}
|
||||
```
|
||||
|
||||
**Webhook 2: UPDATE**
|
||||
```json
|
||||
{
|
||||
"event": "CAIKnowledge.afterSave",
|
||||
"url": "https://your-motia-domain.com/vmh/webhook/aiknowledge/update",
|
||||
"method": "POST",
|
||||
"payload": "{\"entity_id\": \"{$id}\", \"entity_type\": \"CAIKnowledge\", \"action\": \"update\"}",
|
||||
"condition": "!entity.isNew()"
|
||||
}
|
||||
```
|
||||
|
||||
**Webhook 3: DELETE (Optional)**
|
||||
```json
|
||||
{
|
||||
"event": "CAIKnowledge.afterRemove",
|
||||
"url": "https://your-motia-domain.com/vmh/webhook/aiknowledge/delete",
|
||||
"method": "POST",
|
||||
"payload": "{\"entity_id\": \"{$id}\", \"entity_type\": \"CAIKnowledge\", \"action\": \"delete\"}"
|
||||
}
|
||||
```
|
||||
|
||||
**Empfehlung**: Nur CREATE + UPDATE verwenden. DELETE über `activationStatus="deactivated"` steuern.
|
||||
|
||||
### 4. Hooks (EspoCRM Backend)
|
||||
|
||||
**Hook 1: Document Link → syncStatus auf "unclean"**
|
||||
|
||||
```php
|
||||
// Hooks/Custom/CAIKnowledge/AfterRelateLinkMultiple.php
|
||||
namespace Espo\Custom\Hooks\CAIKnowledge;
|
||||
|
||||
class AfterRelateLinkMultiple extends \Espo\Core\Hooks\Base
|
||||
{
|
||||
public function afterRelateLinkMultiple($entity, $options, $data)
|
||||
{
|
||||
if ($data['link'] === 'dokumentes') {
|
||||
// Mark as unclean when documents linked
|
||||
$entity->set('syncStatus', 'unclean');
|
||||
$this->getEntityManager()->saveEntity($entity);
|
||||
}
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
**Hook 2: Document Change → Junction auf "unclean"**
|
||||
|
||||
```php
|
||||
// Hooks/Custom/CDokumente/AfterSave.php
|
||||
namespace Espo\Custom\Hooks\CDokumente;
|
||||
|
||||
class AfterSave extends \Espo\Core\Hooks\Base
|
||||
{
|
||||
public function afterSave($entity, $options)
|
||||
{
|
||||
if ($entity->isAttributeChanged('description') ||
|
||||
$entity->isAttributeChanged('md5') ||
|
||||
$entity->isAttributeChanged('sha256')) {
|
||||
|
||||
// Mark all junction entries as unclean
|
||||
$this->updateJunctionStatuses($entity->id, 'unclean');
|
||||
|
||||
// Mark all related CAIKnowledge as unclean
|
||||
$this->markRelatedKnowledgeUnclean($entity->id);
|
||||
}
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Environment Variables
|
||||
|
||||
```bash
|
||||
# XAI API Keys (erforderlich)
|
||||
XAI_API_KEY=your_xai_api_key_here
|
||||
XAI_MANAGEMENT_KEY=your_xai_management_key_here
|
||||
|
||||
# Redis (für Locking)
|
||||
REDIS_HOST=localhost
|
||||
REDIS_PORT=6379
|
||||
|
||||
# EspoCRM
|
||||
ESPOCRM_API_BASE_URL=https://crm.bitbylaw.com/api/v1
|
||||
ESPOCRM_API_KEY=your_espocrm_api_key
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Workflows
|
||||
|
||||
### Workflow 1: Neue Knowledge Base erstellen
|
||||
|
||||
```
|
||||
1. User erstellt CAIKnowledge in EspoCRM
|
||||
└─ activationStatus: "new" (default)
|
||||
|
||||
2. Webhook CREATE gefeuert
|
||||
└─ Event: aiknowledge.sync
|
||||
|
||||
3. Sync Handler:
|
||||
└─ activationStatus="new" → Collection erstellen in XAI
|
||||
└─ Update EspoCRM:
|
||||
├─ datenbankId = collection_id
|
||||
├─ activationStatus = "active"
|
||||
└─ syncStatus = "unclean"
|
||||
|
||||
4. Nächster Webhook (UPDATE):
|
||||
└─ activationStatus="active" → Dokumente syncen
|
||||
```
|
||||
|
||||
### Workflow 2: Dokumente hinzufügen
|
||||
|
||||
```
|
||||
1. User verknüpft Dokumente mit CAIKnowledge
|
||||
└─ EspoCRM Hook setzt syncStatus = "unclean"
|
||||
|
||||
2. Webhook UPDATE gefeuert
|
||||
└─ Event: aiknowledge.sync
|
||||
|
||||
3. Sync Handler:
|
||||
└─ Für jedes Junction-Entry:
|
||||
├─ Check: MIME Type supported?
|
||||
├─ Check: Hash changed?
|
||||
├─ Download von EspoCRM
|
||||
├─ Upload zu XAI mit Metadata
|
||||
├─ Verify Upload (BLAKE3)
|
||||
└─ Update Junction: syncstatus="synced"
|
||||
|
||||
4. Update CAIKnowledge:
|
||||
└─ syncStatus = "synced"
|
||||
└─ lastSync = now()
|
||||
```
|
||||
|
||||
### Workflow 3: Metadata-Änderung
|
||||
|
||||
```
|
||||
1. User ändert Document.description in EspoCRM
|
||||
└─ EspoCRM Hook setzt Junction syncstatus = "unclean"
|
||||
└─ EspoCRM Hook setzt CAIKnowledge syncStatus = "unclean"
|
||||
|
||||
2. Webhook UPDATE gefeuert
|
||||
|
||||
3. Sync Handler:
|
||||
└─ Berechne Metadata-Hash
|
||||
└─ Hash unterschiedlich? → PATCH zu XAI
|
||||
└─ Falls PATCH fehlschlägt → Fallback: Re-upload
|
||||
└─ Update Junction: syncedMetadataHash
|
||||
```
|
||||
|
||||
### Workflow 4: Knowledge Base deaktivieren
|
||||
|
||||
```
|
||||
1. User setzt activationStatus = "deactivated"
|
||||
|
||||
2. Webhook UPDATE gefeuert
|
||||
|
||||
3. Sync Handler:
|
||||
└─ Collection aus XAI löschen
|
||||
└─ Alle Junction Entries zurücksetzen:
|
||||
├─ syncstatus = "new"
|
||||
└─ aiDocumentId = NULL
|
||||
└─ CAIKnowledge bleibt in EspoCRM (mit datenbankId)
|
||||
```
|
||||
|
||||
### Workflow 5: Daily Full Sync
|
||||
|
||||
```
|
||||
Cron: Täglich um 02:00 Uhr
|
||||
|
||||
1. Lade alle CAIKnowledge mit:
|
||||
└─ activationStatus = "active"
|
||||
└─ syncStatus IN ("unclean", "failed")
|
||||
|
||||
2. Für jedes:
|
||||
└─ Emit: aiknowledge.sync Event
|
||||
|
||||
3. Queue verarbeitet alle sequenziell
|
||||
└─ Fängt verpasste Webhooks ab
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Monitoring & Troubleshooting
|
||||
|
||||
### Logs prüfen
|
||||
|
||||
```bash
|
||||
# Motia Service Logs
|
||||
sudo journalctl -u motia-iii -f | grep -i "ai knowledge"
|
||||
|
||||
# Letzte 100 Sync-Events
|
||||
sudo journalctl -u motia-iii -n 100 | grep "AI KNOWLEDGE SYNC"
|
||||
|
||||
# Fehler der letzten 24 Stunden
|
||||
sudo journalctl -u motia-iii --since "24 hours ago" | grep "❌"
|
||||
```
|
||||
|
||||
### EspoCRM Status prüfen
|
||||
|
||||
```sql
|
||||
-- Alle Knowledge Bases mit Status
|
||||
SELECT
|
||||
id,
|
||||
name,
|
||||
activation_status,
|
||||
sync_status,
|
||||
last_sync,
|
||||
sync_error
|
||||
FROM c_ai_knowledge
|
||||
WHERE activation_status = 'active';
|
||||
|
||||
-- Junction Entries mit Sync-Problemen
|
||||
SELECT
|
||||
j.id,
|
||||
k.name AS knowledge_name,
|
||||
d.name AS document_name,
|
||||
j.syncstatus,
|
||||
j.last_sync
|
||||
FROM c_ai_knowledge_c_dokumente j
|
||||
JOIN c_ai_knowledge k ON j.c_ai_knowledge_id = k.id
|
||||
JOIN c_dokumente d ON j.c_dokumente_id = d.id
|
||||
WHERE j.syncstatus IN ('failed', 'unsupported');
|
||||
```
|
||||
|
||||
### Häufige Probleme
|
||||
|
||||
#### Problem: "Lock busy for aiknowledge:xyz"
|
||||
|
||||
**Ursache**: Vorheriger Sync noch aktiv oder abgestürzt
|
||||
|
||||
**Lösung**:
|
||||
```bash
|
||||
# Redis lock manuell freigeben
|
||||
redis-cli
|
||||
> DEL sync_lock:aiknowledge:xyz
|
||||
```
|
||||
|
||||
#### Problem: "Unsupported MIME type"
|
||||
|
||||
**Ursache**: Document hat MIME Type, den XAI nicht unterstützt
|
||||
|
||||
**Lösung**:
|
||||
- Dokument konvertieren (z.B. RTF → PDF)
|
||||
- Oder: Akzeptieren (bleibt mit Status "unsupported")
|
||||
|
||||
#### Problem: "Upload verification failed"
|
||||
|
||||
**Ursache**: XAI liefert kein BLAKE3 Hash oder Hash-Mismatch
|
||||
|
||||
**Lösung**:
|
||||
1. Prüfe XAI API Dokumentation (Hash-Format geändert?)
|
||||
2. Falls temporär: Retry läuft automatisch
|
||||
3. Falls persistent: XAI Support kontaktieren
|
||||
|
||||
#### Problem: "Collection not found"
|
||||
|
||||
**Ursache**: Collection wurde manuell in XAI gelöscht
|
||||
|
||||
**Lösung**: Automatisch gelöst - Sync erstellt neue Collection
|
||||
|
||||
---
|
||||
|
||||
## API Endpoints
|
||||
|
||||
### Webhook Endpoint
|
||||
|
||||
```http
|
||||
POST /vmh/webhook/aiknowledge/update
|
||||
Content-Type: application/json
|
||||
|
||||
{
|
||||
"entity_id": "kb-123",
|
||||
"entity_type": "CAIKnowledge",
|
||||
"action": "update"
|
||||
}
|
||||
```
|
||||
|
||||
**Response:**
|
||||
```json
|
||||
{
|
||||
"success": true,
|
||||
"knowledge_id": "kb-123"
|
||||
}
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Performance
|
||||
|
||||
### Typische Sync-Zeiten
|
||||
|
||||
| Szenario | Zeit | Notizen |
|
||||
|----------|------|---------|
|
||||
| Collection erstellen | < 1s | Nur API Call |
|
||||
| 1 Dokument (1 MB) | 2-4s | Upload + Verify |
|
||||
| 10 Dokumente (10 MB) | 20-40s | Sequenziell |
|
||||
| 100 Dokumente (100 MB) | 3-6 min | Lock TTL: 30 min |
|
||||
| Metadata-only Update | < 1s | Nur PATCH |
|
||||
| Orphan Cleanup | 1-3s | Pro 10 Dokumente |
|
||||
|
||||
### Lock TTLs
|
||||
|
||||
- **AIKnowledge Sync**: 30 Minuten (1800 Sekunden)
|
||||
- **Redis Lock**: Same as above
|
||||
- **Auto-Release**: Bei Timeout (TTL expired)
|
||||
|
||||
### Rate Limits
|
||||
|
||||
**XAI API:**
|
||||
- Files Upload: ~100 requests/minute
|
||||
- Management API: ~1000 requests/minute
|
||||
|
||||
**Strategie bei Rate Limit (429)**:
|
||||
- Exponential Backoff: 2s, 4s, 8s, 16s, 32s
|
||||
- Respect `Retry-After` Header
|
||||
- Max 5 Retries
|
||||
|
||||
---
|
||||
|
||||
## XAI Collections Metadata
|
||||
|
||||
### Document Metadata Fields
|
||||
|
||||
Werden für jedes Dokument in XAI gespeichert:
|
||||
|
||||
```json
|
||||
{
|
||||
"fields": {
|
||||
"document_name": "Vertrag.pdf",
|
||||
"description": "Mietvertrag Mustermann",
|
||||
"created_at": "2024-01-01T00:00:00Z",
|
||||
"modified_at": "2026-03-10T15:30:00Z",
|
||||
"espocrm_id": "dok-123"
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
**inject_into_chunk**: `true` für `document_name` und `description`
|
||||
→ Verbessert semantische Suche
|
||||
|
||||
### Collection Metadata
|
||||
|
||||
```json
|
||||
{
|
||||
"metadata": {
|
||||
"espocrm_entity_type": "CAIKnowledge",
|
||||
"espocrm_entity_id": "kb-123",
|
||||
"created_at": "2026-03-11T10:00:00Z"
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Testing
|
||||
|
||||
### Manueller Test
|
||||
|
||||
```bash
|
||||
# 1. Erstelle CAIKnowledge in EspoCRM
|
||||
# 2. Prüfe Logs
|
||||
sudo journalctl -u motia-iii -f
|
||||
|
||||
# 3. Prüfe Redis Lock
|
||||
redis-cli
|
||||
> KEYS sync_lock:aiknowledge:*
|
||||
|
||||
# 4. Prüfe XAI Collection
|
||||
curl -H "Authorization: Bearer $XAI_MANAGEMENT_KEY" \
|
||||
https://management-api.x.ai/v1/collections
|
||||
```
|
||||
|
||||
### Integration Test
|
||||
|
||||
```python
|
||||
# tests/test_aiknowledge_sync.py
|
||||
|
||||
async def test_full_sync_workflow():
|
||||
"""Test complete sync workflow"""
|
||||
|
||||
# 1. Create CAIKnowledge with status "new"
|
||||
knowledge = await espocrm.create_entity('CAIKnowledge', {
|
||||
'name': 'Test KB',
|
||||
'activationStatus': 'new'
|
||||
})
|
||||
|
||||
# 2. Trigger webhook
|
||||
await trigger_webhook(knowledge['id'])
|
||||
|
||||
# 3. Wait for sync
|
||||
await asyncio.sleep(5)
|
||||
|
||||
# 4. Check collection created
|
||||
knowledge = await espocrm.get_entity('CAIKnowledge', knowledge['id'])
|
||||
assert knowledge['datenbankId'] is not None
|
||||
assert knowledge['activationStatus'] == 'active'
|
||||
|
||||
# 5. Link document
|
||||
await espocrm.link_entities('CAIKnowledge', knowledge['id'], 'CDokumente', doc_id)
|
||||
|
||||
# 6. Trigger webhook again
|
||||
await trigger_webhook(knowledge['id'])
|
||||
await asyncio.sleep(10)
|
||||
|
||||
# 7. Check junction synced
|
||||
junction = await espocrm.get_junction_entries(
|
||||
'CAIKnowledgeCDokumente',
|
||||
'cAIKnowledgeId',
|
||||
knowledge['id']
|
||||
)
|
||||
assert junction[0]['syncstatus'] == 'synced'
|
||||
assert junction[0]['xaiBlake3Hash'] is not None
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Maintenance
|
||||
|
||||
### Wöchentliche Checks
|
||||
|
||||
- [ ] Prüfe failed Syncs in EspoCRM
|
||||
- [ ] Prüfe Redis Memory Usage
|
||||
- [ ] Prüfe XAI Storage Usage
|
||||
- [ ] Review Logs für Patterns
|
||||
|
||||
### Monatliche Tasks
|
||||
|
||||
- [ ] Cleanup alte syncError Messages
|
||||
- [ ] Verify XAI Collection Integrity
|
||||
- [ ] Review Performance Metrics
|
||||
- [ ] Update MIME Type Support List
|
||||
|
||||
---
|
||||
|
||||
## Support
|
||||
|
||||
**Bei Problemen:**
|
||||
|
||||
1. **Logs prüfen**: `journalctl -u motia-iii -f`
|
||||
2. **EspoCRM Status prüfen**: SQL Queries (siehe oben)
|
||||
3. **Redis Locks prüfen**: `redis-cli KEYS sync_lock:*`
|
||||
4. **XAI API Status**: https://status.x.ai
|
||||
|
||||
**Kontakt:**
|
||||
- Team: BitByLaw Development
|
||||
- Motia Docs: `/opt/motia-iii/bitbylaw/docs/INDEX.md`
|
||||
|
||||
---
|
||||
|
||||
**Version History:**
|
||||
|
||||
- **1.0** (11.03.2026) - Initial Release
|
||||
- Collection Lifecycle Management
|
||||
- BLAKE3 Hash Verification
|
||||
- Daily Full Sync
|
||||
- Metadata Change Detection
|
||||
160
docs/DOCUMENT_SYNC_XAI_STATUS.md
Normal file
160
docs/DOCUMENT_SYNC_XAI_STATUS.md
Normal file
@@ -0,0 +1,160 @@
|
||||
# Document Sync mit xAI Collections - Implementierungs-Status
|
||||
|
||||
## ✅ Implementiert
|
||||
|
||||
### 1. Webhook Endpunkte
|
||||
- **POST** `/vmh/webhook/document/create`
|
||||
- **POST** `/vmh/webhook/document/update`
|
||||
- **POST** `/vmh/webhook/document/delete`
|
||||
|
||||
### 2. Event Handler (`document_sync_event_step.py`)
|
||||
- Queue Topics: `vmh.document.{create|update|delete}`
|
||||
- Redis Distributed Locking
|
||||
- Vollständiges Document Loading von EspoCRM
|
||||
|
||||
### 3. Sync Utilities (`document_sync_utils.py`)
|
||||
- **✅ Datei-Status Prüfung**: "Neu", "Geändert" → xAI-Sync erforderlich
|
||||
- **✅ Hash-basierte Change Detection**: MD5/SHA Vergleich für Updates
|
||||
- **✅ Related Entities Discovery**: Many-to-Many Attachments durchsuchen
|
||||
- **✅ Collection Requirements**: Automatische Ermittlung welche Collections nötig sind
|
||||
|
||||
## ⏳ In Arbeit
|
||||
|
||||
### 4. Preview-Generierung (`generate_thumbnail()`)
|
||||
|
||||
**✅ Implementiert** - Bereit zum Installieren der Dependencies
|
||||
|
||||
**Konfiguration:**
|
||||
- **Feld in EspoCRM**: `preview` (Attachment)
|
||||
- **Format**: **WebP** (bessere Kompression als PNG/JPEG)
|
||||
- **Größe**: **600x800px** (behält Aspect Ratio)
|
||||
- **Qualität**: 85% (guter Kompromiss zwischen Qualität und Dateigröße)
|
||||
|
||||
**Unterstützte Formate:**
|
||||
- ✅ PDF: Erste Seite als Preview
|
||||
- ✅ DOCX/DOC: Konvertierung zu PDF, dann erste Seite
|
||||
- ✅ Images (JPG, PNG, etc.): Resize auf Preview-Größe
|
||||
- ❌ Andere: Kein Preview (TODO: Generic File-Icons)
|
||||
|
||||
**Benötigte Dependencies:**
|
||||
```bash
|
||||
# Python Packages
|
||||
pip install pdf2image Pillow docx2pdf
|
||||
|
||||
# System Dependencies (Ubuntu/Debian)
|
||||
apt-get install poppler-utils libreoffice
|
||||
```
|
||||
|
||||
**Installation:**
|
||||
```bash
|
||||
cd /opt/motia-iii/bitbylaw
|
||||
/opt/bin/uv pip install pdf2image Pillow docx2pdf
|
||||
|
||||
# System packages
|
||||
sudo apt-get update
|
||||
sudo apt-get install -y poppler-utils libreoffice
|
||||
```
|
||||
|
||||
## ❌ Noch nicht implementiert
|
||||
|
||||
### 5. xAI Service (`xai_service.py`)
|
||||
|
||||
**Anforderungen:**
|
||||
- File Upload zu xAI (basierend auf `test_xai_collections_api.py`)
|
||||
- Add File zu Collections
|
||||
- Remove File von Collections
|
||||
- File Download von EspoCRM
|
||||
|
||||
**Referenz-Code vorhanden:**
|
||||
- `/opt/motia-iii/bitbylaw/test_xai_collections_api.py` (630 Zeilen, alle xAI Operations getestet)
|
||||
|
||||
**Implementierungs-Plan:**
|
||||
|
||||
```python
|
||||
class XAIService:
|
||||
def __init__(self, context=None):
|
||||
self.management_key = os.getenv('XAI_MANAGEMENT_KEY')
|
||||
self.api_key = os.getenv('XAI_API_KEY')
|
||||
self.context = context
|
||||
|
||||
async def upload_file(self, file_content: bytes, filename: str) -> str:
|
||||
"""Upload File zu xAI → returns file_id"""
|
||||
# Multipart/form-data upload
|
||||
# POST https://api.x.ai/v1/files
|
||||
pass
|
||||
|
||||
async def add_to_collection(self, collection_id: str, file_id: str):
|
||||
"""Add File zu Collection"""
|
||||
# POST https://management-api.x.ai/v1/collections/{collection_id}/documents/{file_id}
|
||||
pass
|
||||
|
||||
async def remove_from_collection(self, collection_id: str, file_id: str):
|
||||
"""Remove File von Collection"""
|
||||
# DELETE https://management-api.x.ai/v1/collections/{collection_id}/documents/{file_id}
|
||||
pass
|
||||
|
||||
async def download_from_espocrm(self, attachment_id: str) -> bytes:
|
||||
"""Download File von EspoCRM Attachment"""
|
||||
# GET https://crm.bitbylaw.com/api/v1/Attachment/file/{attachment_id}
|
||||
pass
|
||||
```
|
||||
|
||||
## 📋 Integration Checklist
|
||||
|
||||
### Vollständiger Upload-Flow:
|
||||
|
||||
1. ✅ Webhook empfangen → Event emittieren
|
||||
2. ✅ Event Handler: Lock acquire
|
||||
3. ✅ Document laden von EspoCRM
|
||||
4. ✅ Entscheidung: Sync nötig? (Datei-Status, Hash-Check, Collections)
|
||||
5. ⏳ Download File von EspoCRM
|
||||
6. ⏳ Hash berechnen (MD5/SHA)
|
||||
7. ⏳ Thumbnail generieren
|
||||
8. ❌ Upload zu xAI (falls neu oder Hash changed)
|
||||
9. ❌ Add zu Collections
|
||||
10. ⏳ Update EspoCRM Metadaten (xaiFileId, xaiCollections, xaiSyncedHash, thumbnail)
|
||||
11. ✅ Lock release
|
||||
|
||||
### Datei-Stati in EspoCRM:
|
||||
|
||||
- **"Neu"**: Komplett neue Datei → xAI Upload + Collection Add
|
||||
- **"Geändert"**: File-Inhalt geändert → xAI Re-Upload + Collection Update
|
||||
- **"Gesynct"**: Erfolgreich gesynct, keine Änderungen
|
||||
- **"Fehler"**: Sync fehlgeschlagen (mit Error-Message)
|
||||
|
||||
### EspoCRM Custom Fields:
|
||||
|
||||
**Erforderlich für Document Entity:**
|
||||
- `dateiStatus` (Enum): "Neu", "Geändert", "Gesynct", "Fehler"
|
||||
- `md5` (String): MD5 Hash des Files
|
||||
- `sha` (String): SHA Hash des Files
|
||||
- `xaiFileId` (String): xAI File ID
|
||||
- `xaiCollections` (Array): JSON Array von Collection IDs
|
||||
- `xaiSyncedHash` (String): Hash beim letzten erfolgreichen Sync
|
||||
- `xaiSyncStatus` (Enum): "syncing", "synced", "failed"
|
||||
- `xaiSyncError` (Text): Fehlermeldung bei Sync-Fehler
|
||||
- **`preview` (Attachment)**: Vorschaubild im WebP-Format (600x800px)
|
||||
|
||||
## 🚀 Nächste Schritte
|
||||
|
||||
**Priorität 1: xAI Service**
|
||||
- Code aus `test_xai_collections_api.py` extrahieren
|
||||
- In `services/xai_service.py` übertragen
|
||||
- EspoCRM Download-Funktion implementieren
|
||||
|
||||
**Priorität 2: Thumbnail-Generator**
|
||||
- Dependencies installieren
|
||||
- PDF-Thumbnail implementieren
|
||||
- EspoCRM Upload-Methode erweitern
|
||||
|
||||
**Priorität 3: Integration testen**
|
||||
- Document in EspoCRM anlegen
|
||||
- Datei-Status auf "Neu" setzen
|
||||
- Webhook triggern
|
||||
- Logs analysieren
|
||||
|
||||
## 📚 Referenzen
|
||||
|
||||
- **xAI API Tests**: `/opt/motia-iii/bitbylaw/test_xai_collections_api.py`
|
||||
- **EspoCRM API**: `services/espocrm.py`
|
||||
- **Beteiligte Sync** (Referenz-Implementierung): `steps/vmh/beteiligte_sync_event_step.py`
|
||||
2069
docs/INDEX.md
2069
docs/INDEX.md
File diff suppressed because it is too large
Load Diff
@@ -78,6 +78,6 @@ modules:
|
||||
- class: modules::shell::ExecModule
|
||||
config:
|
||||
watch:
|
||||
- steps/**/*.py
|
||||
- src/steps/**/*.py
|
||||
exec:
|
||||
- /opt/bin/uv run python -m motia.cli run --dir steps
|
||||
- /usr/local/bin/uv run python -m motia.cli run --dir src/steps
|
||||
|
||||
@@ -18,5 +18,8 @@ dependencies = [
|
||||
"google-api-python-client>=2.100.0", # Google Calendar API
|
||||
"google-auth>=2.23.0", # Google OAuth2
|
||||
"backoff>=2.2.1", # Retry/backoff decorator
|
||||
"langchain>=0.3.0", # LangChain framework
|
||||
"langchain-xai>=0.2.0", # xAI integration for LangChain
|
||||
"langchain-core>=0.3.0", # LangChain core
|
||||
]
|
||||
|
||||
|
||||
@@ -7,9 +7,6 @@ Basierend auf ADRESSEN_SYNC_ANALYSE.md Abschnitt 12.
|
||||
|
||||
from typing import Dict, Any, Optional
|
||||
from datetime import datetime
|
||||
import logging
|
||||
|
||||
logger = logging.getLogger(__name__)
|
||||
|
||||
|
||||
class AdressenMapper:
|
||||
|
||||
@@ -26,8 +26,6 @@ from services.espocrm import EspoCRMAPI
|
||||
from services.adressen_mapper import AdressenMapper
|
||||
from services.notification_utils import NotificationManager
|
||||
|
||||
logger = logging.getLogger(__name__)
|
||||
|
||||
|
||||
class AdressenSync:
|
||||
"""Sync-Klasse für Adressen zwischen EspoCRM und Advoware"""
|
||||
|
||||
@@ -8,16 +8,17 @@ import hashlib
|
||||
import base64
|
||||
import os
|
||||
import datetime
|
||||
import redis
|
||||
import logging
|
||||
from typing import Optional, Dict, Any
|
||||
|
||||
logger = logging.getLogger(__name__)
|
||||
|
||||
|
||||
class AdvowareTokenError(Exception):
|
||||
"""Raised when token acquisition fails"""
|
||||
pass
|
||||
from services.exceptions import (
|
||||
AdvowareAPIError,
|
||||
AdvowareAuthError,
|
||||
AdvowareTimeoutError,
|
||||
RetryableError
|
||||
)
|
||||
from services.redis_client import get_redis_client
|
||||
from services.config import ADVOWARE_CONFIG, API_CONFIG
|
||||
from services.logging_utils import get_service_logger
|
||||
|
||||
|
||||
class AdvowareAPI:
|
||||
@@ -34,15 +35,8 @@ class AdvowareAPI:
|
||||
- ADVOWARE_USER
|
||||
- ADVOWARE_ROLE
|
||||
- ADVOWARE_PASSWORD
|
||||
- REDIS_HOST (optional, default: localhost)
|
||||
- REDIS_PORT (optional, default: 6379)
|
||||
- REDIS_DB_ADVOWARE_CACHE (optional, default: 1)
|
||||
"""
|
||||
|
||||
AUTH_URL = "https://security.advo-net.net/api/v1/Token"
|
||||
TOKEN_CACHE_KEY = 'advoware_access_token'
|
||||
TOKEN_TIMESTAMP_CACHE_KEY = 'advoware_token_timestamp'
|
||||
|
||||
def __init__(self, context=None):
|
||||
"""
|
||||
Initialize Advoware API client.
|
||||
@@ -51,7 +45,8 @@ class AdvowareAPI:
|
||||
context: Motia FlowContext for logging (optional)
|
||||
"""
|
||||
self.context = context
|
||||
self._log("AdvowareAPI initializing", level='debug')
|
||||
self.logger = get_service_logger('advoware', context)
|
||||
self.logger.debug("AdvowareAPI initializing")
|
||||
|
||||
# Load configuration from environment
|
||||
self.API_BASE_URL = os.getenv('ADVOWARE_API_BASE_URL', 'https://www2.advo-net.net:90/')
|
||||
@@ -63,30 +58,33 @@ class AdvowareAPI:
|
||||
self.user = os.getenv('ADVOWARE_USER', '')
|
||||
self.role = int(os.getenv('ADVOWARE_ROLE', '2'))
|
||||
self.password = os.getenv('ADVOWARE_PASSWORD', '')
|
||||
self.token_lifetime_minutes = int(os.getenv('ADVOWARE_TOKEN_LIFETIME_MINUTES', '55'))
|
||||
self.api_timeout_seconds = int(os.getenv('ADVOWARE_API_TIMEOUT_SECONDS', '30'))
|
||||
self.token_lifetime_minutes = ADVOWARE_CONFIG.token_lifetime_minutes
|
||||
self.api_timeout_seconds = API_CONFIG.default_timeout_seconds
|
||||
|
||||
# Initialize Redis for token caching
|
||||
try:
|
||||
redis_host = os.getenv('REDIS_HOST', 'localhost')
|
||||
redis_port = int(os.getenv('REDIS_PORT', '6379'))
|
||||
redis_db = int(os.getenv('REDIS_DB_ADVOWARE_CACHE', '1'))
|
||||
redis_timeout = int(os.getenv('REDIS_TIMEOUT_SECONDS', '5'))
|
||||
# Initialize Redis for token caching (centralized)
|
||||
self.redis_client = get_redis_client(strict=False)
|
||||
if self.redis_client:
|
||||
self.logger.info("Connected to Redis for token caching")
|
||||
else:
|
||||
self.logger.warning("⚠️ Redis unavailable - token caching disabled!")
|
||||
|
||||
self.redis_client = redis.Redis(
|
||||
host=redis_host,
|
||||
port=redis_port,
|
||||
db=redis_db,
|
||||
socket_timeout=redis_timeout,
|
||||
socket_connect_timeout=redis_timeout
|
||||
)
|
||||
self.redis_client.ping()
|
||||
self._log("Connected to Redis for token caching")
|
||||
except (redis.exceptions.ConnectionError, Exception) as e:
|
||||
self._log(f"Could not connect to Redis: {e}. Token caching disabled.", level='warning')
|
||||
self.redis_client = None
|
||||
self.logger.info("AdvowareAPI initialized")
|
||||
|
||||
self._log("AdvowareAPI initialized")
|
||||
self._session: Optional[aiohttp.ClientSession] = None
|
||||
|
||||
def _log(self, message: str, level: str = 'info') -> None:
|
||||
"""Internal logging helper"""
|
||||
log_func = getattr(self.logger, level, self.logger.info)
|
||||
log_func(message)
|
||||
|
||||
async def _get_session(self) -> aiohttp.ClientSession:
|
||||
if self._session is None or self._session.closed:
|
||||
self._session = aiohttp.ClientSession()
|
||||
return self._session
|
||||
|
||||
async def close(self) -> None:
|
||||
if self._session and not self._session.closed:
|
||||
await self._session.close()
|
||||
|
||||
def _generate_hmac(self, request_time_stamp: str, nonce: Optional[str] = None) -> str:
|
||||
"""Generate HMAC-SHA512 signature for authentication"""
|
||||
@@ -97,7 +95,7 @@ class AdvowareAPI:
|
||||
|
||||
try:
|
||||
api_key_bytes = base64.b64decode(self.api_key)
|
||||
logger.debug("API Key decoded from base64")
|
||||
self.logger.debug("API Key decoded from base64")
|
||||
except Exception as e:
|
||||
self._log(f"API Key not base64-encoded, using as-is: {e}", level='debug')
|
||||
api_key_bytes = self.api_key.encode('utf-8') if isinstance(self.api_key, str) else self.api_key
|
||||
@@ -105,9 +103,9 @@ class AdvowareAPI:
|
||||
signature = hmac.new(api_key_bytes, message, hashlib.sha512)
|
||||
return base64.b64encode(signature.digest()).decode('utf-8')
|
||||
|
||||
def _fetch_new_access_token(self) -> str:
|
||||
"""Fetch new access token from Advoware Auth API"""
|
||||
self._log("Fetching new access token from Advoware")
|
||||
async def _fetch_new_access_token(self) -> str:
|
||||
"""Fetch new access token from Advoware Auth API (async)"""
|
||||
self.logger.info("Fetching new access token from Advoware")
|
||||
|
||||
nonce = str(uuid.uuid4())
|
||||
request_time_stamp = datetime.datetime.utcnow().replace(microsecond=0).isoformat() + "Z"
|
||||
@@ -127,39 +125,61 @@ class AdvowareAPI:
|
||||
"RequestTimeStamp": request_time_stamp
|
||||
}
|
||||
|
||||
self._log(f"Token request: AppID={self.app_id}, User={self.user}", level='debug')
|
||||
self.logger.debug(f"Token request: AppID={self.app_id}, User={self.user}")
|
||||
|
||||
# Using synchronous requests for token fetch (called from sync context)
|
||||
import requests
|
||||
response = requests.post(
|
||||
self.AUTH_URL,
|
||||
json=data,
|
||||
headers=headers,
|
||||
timeout=self.api_timeout_seconds
|
||||
)
|
||||
# Async token fetch using aiohttp
|
||||
session = await self._get_session()
|
||||
|
||||
self._log(f"Token response status: {response.status_code}")
|
||||
response.raise_for_status()
|
||||
try:
|
||||
async with session.post(
|
||||
ADVOWARE_CONFIG.auth_url,
|
||||
json=data,
|
||||
headers=headers,
|
||||
timeout=aiohttp.ClientTimeout(total=self.api_timeout_seconds)
|
||||
) as response:
|
||||
self.logger.debug(f"Token response status: {response.status}")
|
||||
|
||||
if response.status == 401:
|
||||
raise AdvowareAuthError(
|
||||
"Authentication failed - check credentials",
|
||||
status_code=401
|
||||
)
|
||||
|
||||
if response.status >= 400:
|
||||
error_text = await response.text()
|
||||
raise AdvowareAPIError(
|
||||
f"Token request failed ({response.status}): {error_text}",
|
||||
status_code=response.status
|
||||
)
|
||||
|
||||
result = await response.json()
|
||||
|
||||
except asyncio.TimeoutError:
|
||||
raise AdvowareTimeoutError(
|
||||
"Token request timed out",
|
||||
status_code=408
|
||||
)
|
||||
except aiohttp.ClientError as e:
|
||||
raise AdvowareAPIError(f"Token request failed: {str(e)}")
|
||||
|
||||
result = response.json()
|
||||
access_token = result.get("access_token")
|
||||
|
||||
if not access_token:
|
||||
self._log("No access_token in response", level='error')
|
||||
raise AdvowareTokenError("No access_token received from Advoware")
|
||||
self.logger.error("No access_token in response")
|
||||
raise AdvowareAuthError("No access_token received from Advoware")
|
||||
|
||||
self._log("Access token fetched successfully")
|
||||
self.logger.info("Access token fetched successfully")
|
||||
|
||||
# Cache token in Redis
|
||||
if self.redis_client:
|
||||
effective_ttl = max(1, (self.token_lifetime_minutes - 2) * 60)
|
||||
self.redis_client.set(self.TOKEN_CACHE_KEY, access_token, ex=effective_ttl)
|
||||
self.redis_client.set(self.TOKEN_TIMESTAMP_CACHE_KEY, str(time.time()), ex=effective_ttl)
|
||||
self._log(f"Token cached in Redis with TTL {effective_ttl}s")
|
||||
self.redis_client.set(ADVOWARE_CONFIG.token_cache_key, access_token, ex=effective_ttl)
|
||||
self.redis_client.set(ADVOWARE_CONFIG.token_timestamp_key, str(time.time()), ex=effective_ttl)
|
||||
self.logger.debug(f"Token cached in Redis with TTL {effective_ttl}s")
|
||||
|
||||
return access_token
|
||||
|
||||
def get_access_token(self, force_refresh: bool = False) -> str:
|
||||
async def get_access_token(self, force_refresh: bool = False) -> str:
|
||||
"""
|
||||
Get valid access token (from cache or fetch new).
|
||||
|
||||
@@ -169,33 +189,34 @@ class AdvowareAPI:
|
||||
Returns:
|
||||
Valid access token
|
||||
"""
|
||||
self._log("Getting access token", level='debug')
|
||||
self.logger.debug("Getting access token")
|
||||
|
||||
if not self.redis_client:
|
||||
self._log("No Redis available, fetching new token")
|
||||
return self._fetch_new_access_token()
|
||||
self.logger.info("No Redis available, fetching new token")
|
||||
return await self._fetch_new_access_token()
|
||||
|
||||
if force_refresh:
|
||||
self._log("Force refresh requested, fetching new token")
|
||||
return self._fetch_new_access_token()
|
||||
self.logger.info("Force refresh requested, fetching new token")
|
||||
return await self._fetch_new_access_token()
|
||||
|
||||
# Check cache
|
||||
cached_token = self.redis_client.get(self.TOKEN_CACHE_KEY)
|
||||
token_timestamp = self.redis_client.get(self.TOKEN_TIMESTAMP_CACHE_KEY)
|
||||
cached_token = self.redis_client.get(ADVOWARE_CONFIG.token_cache_key)
|
||||
token_timestamp = self.redis_client.get(ADVOWARE_CONFIG.token_timestamp_key)
|
||||
|
||||
if cached_token and token_timestamp:
|
||||
try:
|
||||
timestamp = float(token_timestamp.decode('utf-8'))
|
||||
# Redis decode_responses=True returns strings
|
||||
timestamp = float(token_timestamp)
|
||||
age_seconds = time.time() - timestamp
|
||||
|
||||
if age_seconds < (self.token_lifetime_minutes - 1) * 60:
|
||||
self._log(f"Using cached token (age: {age_seconds:.0f}s)", level='debug')
|
||||
return cached_token.decode('utf-8')
|
||||
except (ValueError, AttributeError) as e:
|
||||
self._log(f"Error reading cached token: {e}", level='debug')
|
||||
self.logger.debug(f"Using cached token (age: {age_seconds:.0f}s)")
|
||||
return cached_token
|
||||
except (ValueError, AttributeError, TypeError) as e:
|
||||
self.logger.debug(f"Error reading cached token: {e}")
|
||||
|
||||
self._log("Cached token expired or invalid, fetching new")
|
||||
return self._fetch_new_access_token()
|
||||
self.logger.info("Cached token expired or invalid, fetching new")
|
||||
return await self._fetch_new_access_token()
|
||||
|
||||
async def api_call(
|
||||
self,
|
||||
@@ -223,6 +244,11 @@ class AdvowareAPI:
|
||||
|
||||
Returns:
|
||||
JSON response or None
|
||||
|
||||
Raises:
|
||||
AdvowareAuthError: Authentication failed
|
||||
AdvowareTimeoutError: Request timed out
|
||||
AdvowareAPIError: Other API errors
|
||||
"""
|
||||
# Clean endpoint
|
||||
endpoint = endpoint.lstrip('/')
|
||||
@@ -233,7 +259,12 @@ class AdvowareAPI:
|
||||
)
|
||||
|
||||
# Get auth token
|
||||
token = self.get_access_token()
|
||||
try:
|
||||
token = await self.get_access_token()
|
||||
except AdvowareAuthError:
|
||||
raise
|
||||
except Exception as e:
|
||||
raise AdvowareAPIError(f"Failed to get access token: {str(e)}")
|
||||
|
||||
# Prepare headers
|
||||
effective_headers = headers.copy() if headers else {}
|
||||
@@ -243,21 +274,21 @@ class AdvowareAPI:
|
||||
# Use 'data' parameter if provided, otherwise 'json_data'
|
||||
json_payload = data if data is not None else json_data
|
||||
|
||||
async with aiohttp.ClientSession(timeout=effective_timeout) as session:
|
||||
try:
|
||||
self._log(f"API call: {method} {url}", level='debug')
|
||||
|
||||
session = await self._get_session()
|
||||
try:
|
||||
with self.logger.api_call(endpoint, method):
|
||||
async with session.request(
|
||||
method,
|
||||
url,
|
||||
headers=effective_headers,
|
||||
params=params,
|
||||
json=json_payload
|
||||
json=json_payload,
|
||||
timeout=effective_timeout
|
||||
) as response:
|
||||
# Handle 401 - retry with fresh token
|
||||
if response.status == 401:
|
||||
self._log("401 Unauthorized, refreshing token")
|
||||
token = self.get_access_token(force_refresh=True)
|
||||
self.logger.warning("401 Unauthorized, refreshing token")
|
||||
token = await self.get_access_token(force_refresh=True)
|
||||
effective_headers['Authorization'] = f'Bearer {token}'
|
||||
|
||||
async with session.request(
|
||||
@@ -265,17 +296,57 @@ class AdvowareAPI:
|
||||
url,
|
||||
headers=effective_headers,
|
||||
params=params,
|
||||
json=json_payload
|
||||
json=json_payload,
|
||||
timeout=effective_timeout
|
||||
) as retry_response:
|
||||
if retry_response.status == 401:
|
||||
raise AdvowareAuthError(
|
||||
"Authentication failed even after token refresh",
|
||||
status_code=401
|
||||
)
|
||||
|
||||
if retry_response.status >= 500:
|
||||
error_text = await retry_response.text()
|
||||
raise RetryableError(
|
||||
f"Server error {retry_response.status}: {error_text}"
|
||||
)
|
||||
|
||||
retry_response.raise_for_status()
|
||||
return await self._parse_response(retry_response)
|
||||
|
||||
response.raise_for_status()
|
||||
# Handle other error codes
|
||||
if response.status == 404:
|
||||
error_text = await response.text()
|
||||
raise AdvowareAPIError(
|
||||
f"Resource not found: {endpoint}",
|
||||
status_code=404,
|
||||
response_body=error_text
|
||||
)
|
||||
|
||||
if response.status >= 500:
|
||||
error_text = await response.text()
|
||||
raise RetryableError(
|
||||
f"Server error {response.status}: {error_text}"
|
||||
)
|
||||
|
||||
if response.status >= 400:
|
||||
error_text = await response.text()
|
||||
raise AdvowareAPIError(
|
||||
f"API error {response.status}: {error_text}",
|
||||
status_code=response.status,
|
||||
response_body=error_text
|
||||
)
|
||||
|
||||
return await self._parse_response(response)
|
||||
|
||||
except aiohttp.ClientError as e:
|
||||
self._log(f"API call failed: {e}", level='error')
|
||||
raise
|
||||
except asyncio.TimeoutError:
|
||||
raise AdvowareTimeoutError(
|
||||
f"Request timed out after {effective_timeout.total}s",
|
||||
status_code=408
|
||||
)
|
||||
except aiohttp.ClientError as e:
|
||||
self.logger.error(f"API call failed: {e}")
|
||||
raise AdvowareAPIError(f"Request failed: {str(e)}")
|
||||
|
||||
async def _parse_response(self, response: aiohttp.ClientResponse) -> Any:
|
||||
"""Parse API response"""
|
||||
@@ -283,27 +354,6 @@ class AdvowareAPI:
|
||||
try:
|
||||
return await response.json()
|
||||
except Exception as e:
|
||||
self._log(f"JSON parse error: {e}", level='debug')
|
||||
self.logger.debug(f"JSON parse error: {e}")
|
||||
return None
|
||||
return None
|
||||
|
||||
def _log(self, message: str, level: str = 'info'):
|
||||
"""Log message via context or standard logger"""
|
||||
if self.context:
|
||||
if level == 'debug':
|
||||
self.context.logger.debug(message)
|
||||
elif level == 'warning':
|
||||
self.context.logger.warning(message)
|
||||
elif level == 'error':
|
||||
self.context.logger.error(message)
|
||||
else:
|
||||
self.context.logger.info(message)
|
||||
else:
|
||||
if level == 'debug':
|
||||
logger.debug(message)
|
||||
elif level == 'warning':
|
||||
logger.warning(message)
|
||||
elif level == 'error':
|
||||
logger.error(message)
|
||||
else:
|
||||
logger.info(message)
|
||||
|
||||
@@ -1,24 +1,29 @@
|
||||
"""
|
||||
Advoware Service Wrapper
|
||||
Erweitert AdvowareAPI mit höheren Operations
|
||||
|
||||
Extends AdvowareAPI with higher-level operations for business logic.
|
||||
"""
|
||||
|
||||
import logging
|
||||
from typing import Dict, Any, Optional
|
||||
from services.advoware import AdvowareAPI
|
||||
|
||||
logger = logging.getLogger(__name__)
|
||||
from services.logging_utils import get_service_logger
|
||||
|
||||
|
||||
class AdvowareService:
|
||||
"""
|
||||
Service-Layer für Advoware Operations
|
||||
Verwendet AdvowareAPI für API-Calls
|
||||
Service layer for Advoware operations.
|
||||
Uses AdvowareAPI for API calls.
|
||||
"""
|
||||
|
||||
def __init__(self, context=None):
|
||||
self.api = AdvowareAPI(context)
|
||||
self.context = context
|
||||
self.logger = get_service_logger('advoware_service', context)
|
||||
|
||||
def _log(self, message: str, level: str = 'info') -> None:
|
||||
"""Internal logging helper"""
|
||||
log_func = getattr(self.logger, level, self.logger.info)
|
||||
log_func(message)
|
||||
|
||||
async def api_call(self, *args, **kwargs):
|
||||
"""Delegate api_call to underlying AdvowareAPI"""
|
||||
@@ -26,29 +31,29 @@ class AdvowareService:
|
||||
|
||||
# ========== BETEILIGTE ==========
|
||||
|
||||
async def get_beteiligter(self, betnr: int) -> Optional[Dict]:
|
||||
async def get_beteiligter(self, betnr: int) -> Optional[Dict[str, Any]]:
|
||||
"""
|
||||
Lädt Beteiligten mit allen Daten
|
||||
Load Beteiligte with all data.
|
||||
|
||||
Returns:
|
||||
Beteiligte-Objekt
|
||||
Beteiligte object or None
|
||||
"""
|
||||
try:
|
||||
endpoint = f"api/v1/advonet/Beteiligte/{betnr}"
|
||||
result = await self.api.api_call(endpoint, method='GET')
|
||||
return result
|
||||
except Exception as e:
|
||||
logger.error(f"[ADVO] Fehler beim Laden von Beteiligte {betnr}: {e}", exc_info=True)
|
||||
self._log(f"[ADVO] Error loading Beteiligte {betnr}: {e}", level='error')
|
||||
return None
|
||||
|
||||
# ========== KOMMUNIKATION ==========
|
||||
|
||||
async def create_kommunikation(self, betnr: int, data: Dict[str, Any]) -> Optional[Dict]:
|
||||
async def create_kommunikation(self, betnr: int, data: Dict[str, Any]) -> Optional[Dict[str, Any]]:
|
||||
"""
|
||||
Erstellt neue Kommunikation
|
||||
Create new Kommunikation.
|
||||
|
||||
Args:
|
||||
betnr: Beteiligten-Nummer
|
||||
betnr: Beteiligte number
|
||||
data: {
|
||||
'tlf': str, # Required
|
||||
'bemerkung': str, # Optional
|
||||
@@ -57,68 +62,68 @@ class AdvowareService:
|
||||
}
|
||||
|
||||
Returns:
|
||||
Neue Kommunikation mit 'id'
|
||||
New Kommunikation with 'id' or None
|
||||
"""
|
||||
try:
|
||||
endpoint = f"api/v1/advonet/Beteiligte/{betnr}/Kommunikationen"
|
||||
result = await self.api.api_call(endpoint, method='POST', json_data=data)
|
||||
|
||||
if result:
|
||||
logger.info(f"[ADVO] ✅ Created Kommunikation: betnr={betnr}, kommKz={data.get('kommKz')}")
|
||||
self._log(f"[ADVO] ✅ Created Kommunikation: betnr={betnr}, kommKz={data.get('kommKz')}")
|
||||
|
||||
return result
|
||||
|
||||
except Exception as e:
|
||||
logger.error(f"[ADVO] Fehler beim Erstellen von Kommunikation: {e}", exc_info=True)
|
||||
self._log(f"[ADVO] Error creating Kommunikation: {e}", level='error')
|
||||
return None
|
||||
|
||||
async def update_kommunikation(self, betnr: int, komm_id: int, data: Dict[str, Any]) -> bool:
|
||||
"""
|
||||
Aktualisiert bestehende Kommunikation
|
||||
Update existing Kommunikation.
|
||||
|
||||
Args:
|
||||
betnr: Beteiligten-Nummer
|
||||
komm_id: Kommunikation-ID
|
||||
betnr: Beteiligte number
|
||||
komm_id: Kommunikation ID
|
||||
data: {
|
||||
'tlf': str, # Optional
|
||||
'bemerkung': str, # Optional
|
||||
'online': bool # Optional
|
||||
}
|
||||
|
||||
NOTE: kommKz ist READ-ONLY und kann nicht geändert werden
|
||||
NOTE: kommKz is READ-ONLY and cannot be changed
|
||||
|
||||
Returns:
|
||||
True wenn erfolgreich
|
||||
True if successful
|
||||
"""
|
||||
try:
|
||||
endpoint = f"api/v1/advonet/Beteiligte/{betnr}/Kommunikationen/{komm_id}"
|
||||
await self.api.api_call(endpoint, method='PUT', json_data=data)
|
||||
|
||||
logger.info(f"[ADVO] ✅ Updated Kommunikation: betnr={betnr}, komm_id={komm_id}")
|
||||
self._log(f"[ADVO] ✅ Updated Kommunikation: betnr={betnr}, komm_id={komm_id}")
|
||||
return True
|
||||
|
||||
except Exception as e:
|
||||
logger.error(f"[ADVO] Fehler beim Update von Kommunikation: {e}", exc_info=True)
|
||||
self._log(f"[ADVO] Error updating Kommunikation: {e}", level='error')
|
||||
return False
|
||||
|
||||
async def delete_kommunikation(self, betnr: int, komm_id: int) -> bool:
|
||||
"""
|
||||
Löscht Kommunikation (aktuell 403 Forbidden)
|
||||
Delete Kommunikation (currently returns 403 Forbidden).
|
||||
|
||||
NOTE: DELETE ist in Advoware API deaktiviert
|
||||
Verwende stattdessen: Leere Slots mit empty_slot_marker
|
||||
NOTE: DELETE is disabled in Advoware API.
|
||||
Use empty slots with empty_slot_marker instead.
|
||||
|
||||
Returns:
|
||||
True wenn erfolgreich
|
||||
True if successful
|
||||
"""
|
||||
try:
|
||||
endpoint = f"api/v1/advonet/Beteiligte/{betnr}/Kommunikationen/{komm_id}"
|
||||
await self.api.api_call(endpoint, method='DELETE')
|
||||
|
||||
logger.info(f"[ADVO] ✅ Deleted Kommunikation: betnr={betnr}, komm_id={komm_id}")
|
||||
self._log(f"[ADVO] ✅ Deleted Kommunikation: betnr={betnr}, komm_id={komm_id}")
|
||||
return True
|
||||
|
||||
except Exception as e:
|
||||
# Expected: 403 Forbidden
|
||||
logger.warning(f"[ADVO] DELETE not allowed (expected): {e}")
|
||||
self._log(f"[ADVO] DELETE not allowed (expected): {e}", level='warning')
|
||||
return False
|
||||
|
||||
545
services/aiknowledge_sync_utils.py
Normal file
545
services/aiknowledge_sync_utils.py
Normal file
@@ -0,0 +1,545 @@
|
||||
"""
|
||||
AI Knowledge Sync Utilities
|
||||
|
||||
Utility functions for synchronizing CAIKnowledge entities with XAI Collections:
|
||||
- Collection lifecycle management (create, delete)
|
||||
- Document synchronization with BLAKE3 hash verification
|
||||
- Metadata-only updates via PATCH
|
||||
- Orphan detection and cleanup
|
||||
"""
|
||||
|
||||
import hashlib
|
||||
import json
|
||||
from typing import Dict, Any, Optional, List, Tuple
|
||||
from datetime import datetime
|
||||
from urllib.parse import unquote
|
||||
|
||||
from services.sync_utils_base import BaseSyncUtils
|
||||
from services.models import (
|
||||
AIKnowledgeActivationStatus,
|
||||
AIKnowledgeSyncStatus,
|
||||
JunctionSyncStatus
|
||||
)
|
||||
|
||||
|
||||
class AIKnowledgeSync(BaseSyncUtils):
|
||||
"""Utility class for AI Knowledge ↔ XAI Collections synchronization"""
|
||||
|
||||
def _get_lock_key(self, entity_id: str) -> str:
|
||||
"""Redis lock key for AI Knowledge entities"""
|
||||
return f"sync_lock:aiknowledge:{entity_id}"
|
||||
|
||||
async def acquire_sync_lock(self, knowledge_id: str) -> bool:
|
||||
"""
|
||||
Acquire distributed lock via Redis + update EspoCRM syncStatus.
|
||||
|
||||
Args:
|
||||
knowledge_id: CAIKnowledge entity ID
|
||||
|
||||
Returns:
|
||||
True if lock acquired, False if already locked
|
||||
"""
|
||||
try:
|
||||
# STEP 1: Atomic Redis lock
|
||||
lock_key = self._get_lock_key(knowledge_id)
|
||||
if not self._acquire_redis_lock(lock_key):
|
||||
self._log(f"Redis lock already active for {knowledge_id}", level='warn')
|
||||
return False
|
||||
|
||||
# STEP 2: Update syncStatus to pending_sync
|
||||
try:
|
||||
await self.espocrm.update_entity('CAIKnowledge', knowledge_id, {
|
||||
'syncStatus': AIKnowledgeSyncStatus.PENDING_SYNC.value
|
||||
})
|
||||
except Exception as e:
|
||||
self._log(f"Could not set syncStatus: {e}", level='debug')
|
||||
|
||||
self._log(f"Sync lock acquired for {knowledge_id}")
|
||||
return True
|
||||
|
||||
except Exception as e:
|
||||
self._log(f"Error acquiring lock: {e}", level='error')
|
||||
# Clean up Redis lock on error
|
||||
lock_key = self._get_lock_key(knowledge_id)
|
||||
self._release_redis_lock(lock_key)
|
||||
return False
|
||||
|
||||
async def release_sync_lock(
|
||||
self,
|
||||
knowledge_id: str,
|
||||
success: bool = True,
|
||||
error_message: Optional[str] = None
|
||||
) -> None:
|
||||
"""
|
||||
Release sync lock and set final status.
|
||||
|
||||
Args:
|
||||
knowledge_id: CAIKnowledge entity ID
|
||||
success: Whether sync succeeded
|
||||
error_message: Optional error message
|
||||
"""
|
||||
try:
|
||||
update_data = {
|
||||
'syncStatus': AIKnowledgeSyncStatus.SYNCED.value if success else AIKnowledgeSyncStatus.FAILED.value
|
||||
}
|
||||
|
||||
if success:
|
||||
update_data['lastSync'] = datetime.now().isoformat()
|
||||
update_data['syncError'] = None
|
||||
elif error_message:
|
||||
update_data['syncError'] = error_message[:2000]
|
||||
|
||||
await self.espocrm.update_entity('CAIKnowledge', knowledge_id, update_data)
|
||||
|
||||
self._log(f"Sync lock released: {knowledge_id} → {'success' if success else 'failed'}")
|
||||
|
||||
# Release Redis lock
|
||||
lock_key = self._get_lock_key(knowledge_id)
|
||||
self._release_redis_lock(lock_key)
|
||||
|
||||
except Exception as e:
|
||||
self._log(f"Error releasing lock: {e}", level='error')
|
||||
# Ensure Redis lock is released
|
||||
lock_key = self._get_lock_key(knowledge_id)
|
||||
self._release_redis_lock(lock_key)
|
||||
|
||||
async def sync_knowledge_to_xai(self, knowledge_id: str, ctx) -> None:
|
||||
"""
|
||||
Main sync orchestrator with activation status handling.
|
||||
|
||||
Args:
|
||||
knowledge_id: CAIKnowledge entity ID
|
||||
ctx: Motia context for logging
|
||||
"""
|
||||
from services.espocrm import EspoCRMAPI
|
||||
from services.xai_service import XAIService
|
||||
|
||||
espocrm = EspoCRMAPI(ctx)
|
||||
xai = XAIService(ctx)
|
||||
|
||||
try:
|
||||
# 1. Load knowledge entity
|
||||
knowledge = await espocrm.get_entity('CAIKnowledge', knowledge_id)
|
||||
|
||||
activation_status = knowledge.get('aktivierungsstatus')
|
||||
collection_id = knowledge.get('datenbankId')
|
||||
|
||||
ctx.logger.info("=" * 80)
|
||||
ctx.logger.info(f"📋 Processing: {knowledge['name']}")
|
||||
ctx.logger.info(f" aktivierungsstatus: {activation_status}")
|
||||
ctx.logger.info(f" datenbankId: {collection_id or 'NONE'}")
|
||||
ctx.logger.info("=" * 80)
|
||||
|
||||
# ═══════════════════════════════════════════════════════════
|
||||
# CASE 1: NEW → Create Collection
|
||||
# ═══════════════════════════════════════════════════════════
|
||||
if activation_status == AIKnowledgeActivationStatus.NEW.value:
|
||||
ctx.logger.info("🆕 Status 'new' → Creating XAI Collection")
|
||||
|
||||
collection = await xai.create_collection(
|
||||
name=knowledge['name'],
|
||||
metadata={
|
||||
'espocrm_entity_type': 'CAIKnowledge',
|
||||
'espocrm_entity_id': knowledge_id,
|
||||
'created_at': datetime.now().isoformat()
|
||||
}
|
||||
)
|
||||
|
||||
# XAI API returns 'collection_id' not 'id'
|
||||
collection_id = collection.get('collection_id') or collection.get('id')
|
||||
|
||||
# Update EspoCRM: Set datenbankId + change status to 'active'
|
||||
await espocrm.update_entity('CAIKnowledge', knowledge_id, {
|
||||
'datenbankId': collection_id,
|
||||
'aktivierungsstatus': AIKnowledgeActivationStatus.ACTIVE.value,
|
||||
'syncStatus': AIKnowledgeSyncStatus.UNCLEAN.value
|
||||
})
|
||||
|
||||
ctx.logger.info(f"✅ Collection created: {collection_id}")
|
||||
ctx.logger.info(" Status changed to 'active', now syncing documents...")
|
||||
|
||||
# Continue to document sync immediately (don't return)
|
||||
# Fall through to sync logic below
|
||||
|
||||
# ═══════════════════════════════════════════════════════════
|
||||
# CASE 2: DEACTIVATED → Delete Collection from XAI
|
||||
# ═══════════════════════════════════════════════════════════
|
||||
elif activation_status == AIKnowledgeActivationStatus.DEACTIVATED.value:
|
||||
ctx.logger.info("🗑️ Status 'deactivated' → Deleting XAI Collection")
|
||||
|
||||
if collection_id:
|
||||
try:
|
||||
await xai.delete_collection(collection_id)
|
||||
ctx.logger.info(f"✅ Collection deleted from XAI: {collection_id}")
|
||||
except Exception as e:
|
||||
ctx.logger.error(f"❌ Failed to delete collection: {e}")
|
||||
else:
|
||||
ctx.logger.info("⏭️ No collection ID, nothing to delete")
|
||||
|
||||
# Reset junction entries
|
||||
documents = await espocrm.get_knowledge_documents_with_junction(knowledge_id)
|
||||
|
||||
for doc in documents:
|
||||
doc_id = doc['documentId']
|
||||
try:
|
||||
await espocrm.update_knowledge_document_junction(
|
||||
knowledge_id,
|
||||
doc_id,
|
||||
{
|
||||
'syncstatus': 'new',
|
||||
'aiDocumentId': None
|
||||
},
|
||||
update_last_sync=False
|
||||
)
|
||||
except Exception as e:
|
||||
ctx.logger.warn(f"⚠️ Failed to reset junction for {doc_id}: {e}")
|
||||
|
||||
ctx.logger.info(f"✅ Deactivation complete, {len(documents)} junction entries reset")
|
||||
return
|
||||
|
||||
# ═══════════════════════════════════════════════════════════
|
||||
# CASE 3: PAUSED → Skip Sync
|
||||
# ═══════════════════════════════════════════════════════════
|
||||
elif activation_status == AIKnowledgeActivationStatus.PAUSED.value:
|
||||
ctx.logger.info("⏸️ Status 'paused' → No sync performed")
|
||||
return
|
||||
|
||||
# ═══════════════════════════════════════════════════════════
|
||||
# CASE 4: ACTIVE → Normal Sync (or just created from NEW)
|
||||
# ═══════════════════════════════════════════════════════════
|
||||
if activation_status in (AIKnowledgeActivationStatus.ACTIVE.value, AIKnowledgeActivationStatus.NEW.value):
|
||||
if not collection_id:
|
||||
ctx.logger.error("❌ Status 'active' but no datenbankId!")
|
||||
raise RuntimeError("Active knowledge without collection ID")
|
||||
|
||||
if activation_status == AIKnowledgeActivationStatus.ACTIVE.value:
|
||||
ctx.logger.info(f"🔄 Status 'active' → Syncing documents to {collection_id}")
|
||||
|
||||
# Verify collection exists
|
||||
collection = await xai.get_collection(collection_id)
|
||||
if not collection:
|
||||
ctx.logger.warn(f"⚠️ Collection {collection_id} not found, recreating")
|
||||
collection = await xai.create_collection(
|
||||
name=knowledge['name'],
|
||||
metadata={
|
||||
'espocrm_entity_type': 'CAIKnowledge',
|
||||
'espocrm_entity_id': knowledge_id
|
||||
}
|
||||
)
|
||||
collection_id = collection['id']
|
||||
await espocrm.update_entity('CAIKnowledge', knowledge_id, {
|
||||
'datenbankId': collection_id
|
||||
})
|
||||
|
||||
# Sync documents (both for ACTIVE status and after NEW → ACTIVE transition)
|
||||
await self._sync_knowledge_documents(knowledge_id, collection_id, ctx)
|
||||
|
||||
elif activation_status not in (AIKnowledgeActivationStatus.DEACTIVATED.value, AIKnowledgeActivationStatus.PAUSED.value):
|
||||
ctx.logger.error(f"❌ Unknown aktivierungsstatus: {activation_status}")
|
||||
raise ValueError(f"Invalid aktivierungsstatus: {activation_status}")
|
||||
|
||||
finally:
|
||||
await xai.close()
|
||||
|
||||
async def _sync_knowledge_documents(
|
||||
self,
|
||||
knowledge_id: str,
|
||||
collection_id: str,
|
||||
ctx
|
||||
) -> None:
|
||||
"""
|
||||
Sync all documents of a knowledge base to XAI collection.
|
||||
|
||||
Uses efficient JunctionData endpoint to get all documents with junction data
|
||||
and blake3 hashes in a single API call. Hash comparison is always performed.
|
||||
|
||||
Args:
|
||||
knowledge_id: CAIKnowledge entity ID
|
||||
collection_id: XAI Collection ID
|
||||
ctx: Motia context
|
||||
"""
|
||||
from services.espocrm import EspoCRMAPI
|
||||
from services.xai_service import XAIService
|
||||
|
||||
espocrm = EspoCRMAPI(ctx)
|
||||
xai = XAIService(ctx)
|
||||
|
||||
# ═══════════════════════════════════════════════════════════════
|
||||
# STEP 1: Load all documents with junction data (single API call)
|
||||
# ═══════════════════════════════════════════════════════════════
|
||||
ctx.logger.info(f"📥 Loading documents with junction data for knowledge {knowledge_id}")
|
||||
|
||||
documents = await espocrm.get_knowledge_documents_with_junction(knowledge_id)
|
||||
|
||||
ctx.logger.info(f"📊 Found {len(documents)} document(s)")
|
||||
|
||||
if not documents:
|
||||
ctx.logger.info("✅ No documents to sync")
|
||||
return
|
||||
|
||||
# ═══════════════════════════════════════════════════════════════
|
||||
# STEP 2: Sync each document based on status/hash
|
||||
# ═══════════════════════════════════════════════════════════════
|
||||
successful = 0
|
||||
failed = 0
|
||||
skipped = 0
|
||||
# Track aiDocumentIds for orphan detection (collected during sync)
|
||||
synced_file_ids: set = set()
|
||||
for doc in documents:
|
||||
doc_id = doc['documentId']
|
||||
doc_name = doc.get('documentName', 'Unknown')
|
||||
junction_status = doc.get('syncstatus', 'new')
|
||||
ai_document_id = doc.get('aiDocumentId')
|
||||
blake3_hash = doc.get('blake3hash')
|
||||
|
||||
ctx.logger.info(f"\n📄 {doc_name} (ID: {doc_id})")
|
||||
ctx.logger.info(f" Status: {junction_status}")
|
||||
ctx.logger.info(f" aiDocumentId: {ai_document_id or 'N/A'}")
|
||||
ctx.logger.info(f" blake3hash: {blake3_hash[:16] if blake3_hash else 'N/A'}...")
|
||||
|
||||
try:
|
||||
# Decide if sync needed
|
||||
needs_sync = False
|
||||
reason = ""
|
||||
|
||||
if junction_status in ['new', 'unclean', 'failed']:
|
||||
needs_sync = True
|
||||
reason = f"status={junction_status}"
|
||||
elif junction_status == 'synced':
|
||||
# Synced status should have both blake3_hash and ai_document_id
|
||||
if not blake3_hash:
|
||||
needs_sync = True
|
||||
reason = "inconsistency: synced but no blake3 hash"
|
||||
ctx.logger.warn(f" ⚠️ Synced document missing blake3 hash!")
|
||||
elif not ai_document_id:
|
||||
needs_sync = True
|
||||
reason = "inconsistency: synced but no aiDocumentId"
|
||||
ctx.logger.warn(f" ⚠️ Synced document missing aiDocumentId!")
|
||||
else:
|
||||
# Verify Blake3 hash with XAI (always, since hash from JunctionData API is free)
|
||||
try:
|
||||
xai_doc_info = await xai.get_collection_document(collection_id, ai_document_id)
|
||||
if xai_doc_info:
|
||||
xai_blake3 = xai_doc_info.get('blake3_hash')
|
||||
|
||||
if xai_blake3 != blake3_hash:
|
||||
needs_sync = True
|
||||
reason = f"blake3 mismatch (XAI: {xai_blake3[:16] if xai_blake3 else 'N/A'}... vs EspoCRM: {blake3_hash[:16]}...)"
|
||||
ctx.logger.info(f" 🔄 Blake3 mismatch detected!")
|
||||
else:
|
||||
ctx.logger.info(f" ✅ Blake3 hash matches")
|
||||
else:
|
||||
needs_sync = True
|
||||
reason = "file not found in XAI collection"
|
||||
ctx.logger.warn(f" ⚠️ Document marked synced but not in XAI!")
|
||||
except Exception as e:
|
||||
needs_sync = True
|
||||
reason = f"verification failed: {e}"
|
||||
ctx.logger.warn(f" ⚠️ Failed to verify Blake3, will re-sync: {e}")
|
||||
|
||||
if not needs_sync:
|
||||
ctx.logger.info(f" ⏭️ Skipped (no sync needed)")
|
||||
# Document is already synced, track its aiDocumentId
|
||||
if ai_document_id:
|
||||
synced_file_ids.add(ai_document_id)
|
||||
skipped += 1
|
||||
continue
|
||||
|
||||
ctx.logger.info(f" 🔄 Syncing: {reason}")
|
||||
|
||||
# Get complete document entity with attachment info
|
||||
doc_entity = await espocrm.get_entity('CDokumente', doc_id)
|
||||
attachment_id = doc_entity.get('dokumentId')
|
||||
|
||||
if not attachment_id:
|
||||
ctx.logger.error(f" ❌ No attachment ID found for document {doc_id}")
|
||||
failed += 1
|
||||
continue
|
||||
|
||||
# Get attachment details for MIME type and original filename
|
||||
try:
|
||||
attachment = await espocrm.get_entity('Attachment', attachment_id)
|
||||
mime_type = attachment.get('type', 'application/octet-stream')
|
||||
file_size = attachment.get('size', 0)
|
||||
original_filename = attachment.get('name', doc_name) # Original filename with extension
|
||||
# URL-decode filename (fixes special chars like §, ä, ö, ü, etc.)
|
||||
original_filename = unquote(original_filename)
|
||||
except Exception as e:
|
||||
ctx.logger.warn(f" ⚠️ Failed to get attachment details: {e}, using defaults")
|
||||
mime_type = 'application/octet-stream'
|
||||
file_size = 0
|
||||
original_filename = unquote(doc_name) # Also decode fallback name
|
||||
|
||||
ctx.logger.info(f" 📎 Attachment: {attachment_id} ({mime_type}, {file_size} bytes)")
|
||||
ctx.logger.info(f" 📄 Original filename: {original_filename}")
|
||||
|
||||
# Download document
|
||||
file_content = await espocrm.download_attachment(attachment_id)
|
||||
ctx.logger.info(f" 📥 Downloaded {len(file_content)} bytes")
|
||||
|
||||
# Upload to XAI with original filename (includes extension)
|
||||
filename = original_filename
|
||||
|
||||
xai_file_id = await xai.upload_file(file_content, filename, mime_type)
|
||||
ctx.logger.info(f" 📤 Uploaded to XAI: {xai_file_id}")
|
||||
|
||||
# Add to collection
|
||||
await xai.add_to_collection(collection_id, xai_file_id)
|
||||
ctx.logger.info(f" ✅ Added to collection {collection_id}")
|
||||
|
||||
# Update junction
|
||||
await espocrm.update_knowledge_document_junction(
|
||||
knowledge_id,
|
||||
doc_id,
|
||||
{
|
||||
'aiDocumentId': xai_file_id,
|
||||
'syncstatus': 'synced'
|
||||
},
|
||||
update_last_sync=True
|
||||
)
|
||||
ctx.logger.info(f" ✅ Junction updated")
|
||||
|
||||
# Track the new aiDocumentId for orphan detection
|
||||
synced_file_ids.add(xai_file_id)
|
||||
|
||||
successful += 1
|
||||
|
||||
except Exception as e:
|
||||
failed += 1
|
||||
ctx.logger.error(f" ❌ Sync failed: {e}")
|
||||
|
||||
# Mark as failed in junction
|
||||
try:
|
||||
await espocrm.update_knowledge_document_junction(
|
||||
knowledge_id,
|
||||
doc_id,
|
||||
{'syncstatus': 'failed'},
|
||||
update_last_sync=False
|
||||
)
|
||||
except Exception as update_err:
|
||||
ctx.logger.error(f" ❌ Failed to update junction status: {update_err}")
|
||||
|
||||
# ═══════════════════════════════════════════════════════════════
|
||||
# STEP 3: Remove orphaned documents from XAI collection
|
||||
# ═══════════════════════════════════════════════════════════════
|
||||
try:
|
||||
ctx.logger.info(f"\n🧹 Checking for orphaned documents in XAI collection...")
|
||||
|
||||
# Get all files in XAI collection (normalized structure)
|
||||
xai_documents = await xai.list_collection_documents(collection_id)
|
||||
xai_file_ids = {doc.get('file_id') for doc in xai_documents if doc.get('file_id')}
|
||||
|
||||
# Use synced_file_ids (collected during this sync) for orphan detection
|
||||
# This includes both pre-existing synced docs and newly uploaded ones
|
||||
ctx.logger.info(f" XAI has {len(xai_file_ids)} files, we have {len(synced_file_ids)} synced")
|
||||
|
||||
# Find orphans (in XAI but not in our current sync)
|
||||
orphans = xai_file_ids - synced_file_ids
|
||||
|
||||
if orphans:
|
||||
ctx.logger.info(f" Found {len(orphans)} orphaned file(s)")
|
||||
for orphan_id in orphans:
|
||||
try:
|
||||
await xai.remove_from_collection(collection_id, orphan_id)
|
||||
ctx.logger.info(f" 🗑️ Removed {orphan_id}")
|
||||
except Exception as e:
|
||||
ctx.logger.warn(f" ⚠️ Failed to remove {orphan_id}: {e}")
|
||||
else:
|
||||
ctx.logger.info(f" ✅ No orphans found")
|
||||
|
||||
except Exception as e:
|
||||
ctx.logger.warn(f"⚠️ Failed to clean up orphans: {e}")
|
||||
|
||||
# ═══════════════════════════════════════════════════════════════
|
||||
# STEP 4: Summary
|
||||
# ═══════════════════════════════════════════════════════════════
|
||||
ctx.logger.info("")
|
||||
ctx.logger.info("=" * 80)
|
||||
ctx.logger.info(f"📊 Sync Statistics:")
|
||||
ctx.logger.info(f" ✅ Synced: {successful}")
|
||||
ctx.logger.info(f" ⏭️ Skipped: {skipped}")
|
||||
ctx.logger.info(f" ❌ Failed: {failed}")
|
||||
ctx.logger.info(f" Mode: Blake3 hash verification enabled")
|
||||
ctx.logger.info("=" * 80)
|
||||
|
||||
def _calculate_metadata_hash(self, document: Dict) -> str:
|
||||
"""
|
||||
Calculate hash of sync-relevant metadata.
|
||||
|
||||
Args:
|
||||
document: CDokumente entity
|
||||
|
||||
Returns:
|
||||
MD5 hash (32 chars)
|
||||
"""
|
||||
metadata = {
|
||||
'name': document.get('name', ''),
|
||||
'description': document.get('description', ''),
|
||||
}
|
||||
|
||||
metadata_str = json.dumps(metadata, sort_keys=True)
|
||||
return hashlib.md5(metadata_str.encode()).hexdigest()
|
||||
|
||||
def _build_xai_metadata(self, document: Dict) -> Dict[str, str]:
|
||||
"""
|
||||
Build XAI metadata from CDokumente entity.
|
||||
|
||||
Args:
|
||||
document: CDokumente entity
|
||||
|
||||
Returns:
|
||||
Metadata dict for XAI
|
||||
"""
|
||||
return {
|
||||
'document_name': document.get('name', ''),
|
||||
'description': document.get('description', ''),
|
||||
'created_at': document.get('createdAt', ''),
|
||||
'modified_at': document.get('modifiedAt', ''),
|
||||
'espocrm_id': document.get('id', '')
|
||||
}
|
||||
|
||||
async def _get_document_download_info(
|
||||
self,
|
||||
document: Dict,
|
||||
ctx
|
||||
) -> Optional[Dict[str, Any]]:
|
||||
"""
|
||||
Get download info for CDokumente entity.
|
||||
|
||||
Args:
|
||||
document: CDokumente entity
|
||||
ctx: Motia context
|
||||
|
||||
Returns:
|
||||
Dict with attachment_id, filename, mime_type
|
||||
"""
|
||||
from services.espocrm import EspoCRMAPI
|
||||
|
||||
espocrm = EspoCRMAPI(ctx)
|
||||
|
||||
# Check for dokumentId (CDokumente custom field)
|
||||
attachment_id = None
|
||||
filename = None
|
||||
|
||||
if document.get('dokumentId'):
|
||||
attachment_id = document.get('dokumentId')
|
||||
filename = document.get('dokumentName')
|
||||
elif document.get('fileId'):
|
||||
attachment_id = document.get('fileId')
|
||||
filename = document.get('fileName')
|
||||
|
||||
if not attachment_id:
|
||||
ctx.logger.error(f"❌ No attachment ID for document {document['id']}")
|
||||
return None
|
||||
|
||||
# Get attachment details
|
||||
try:
|
||||
attachment = await espocrm.get_entity('Attachment', attachment_id)
|
||||
return {
|
||||
'attachment_id': attachment_id,
|
||||
'filename': filename or attachment.get('name', 'unknown'),
|
||||
'mime_type': attachment.get('type', 'application/octet-stream')
|
||||
}
|
||||
except Exception as e:
|
||||
ctx.logger.error(f"❌ Failed to get attachment {attachment_id}: {e}")
|
||||
return None
|
||||
110
services/aktenzeichen_utils.py
Normal file
110
services/aktenzeichen_utils.py
Normal file
@@ -0,0 +1,110 @@
|
||||
"""Aktenzeichen-Erkennung und Validation
|
||||
|
||||
Utility functions für das Erkennen, Validieren und Normalisieren von
|
||||
Aktenzeichen im Format '1234/56' oder 'ABC/23'.
|
||||
"""
|
||||
import re
|
||||
from typing import Optional
|
||||
|
||||
|
||||
# Regex für Aktenzeichen: 1-4 Zeichen (alphanumerisch) + "/" + 2 Ziffern
|
||||
AKTENZEICHEN_REGEX = re.compile(r'^([A-Za-z0-9]{1,4}/\d{2})\s*', re.IGNORECASE)
|
||||
|
||||
|
||||
def extract_aktenzeichen(text: str) -> Optional[str]:
|
||||
"""
|
||||
Extrahiert Aktenzeichen vom Anfang des Textes.
|
||||
|
||||
Pattern: ^[A-Za-z0-9]{1,4}/\d{2}
|
||||
|
||||
Examples:
|
||||
>>> extract_aktenzeichen("1234/56 Was ist der Stand?")
|
||||
"1234/56"
|
||||
>>> extract_aktenzeichen("ABC/23 Frage zum Vertrag")
|
||||
"ABC/23"
|
||||
>>> extract_aktenzeichen("Kein Aktenzeichen hier")
|
||||
None
|
||||
|
||||
Args:
|
||||
text: Eingabetext (z.B. erste Message)
|
||||
|
||||
Returns:
|
||||
Aktenzeichen als String, oder None wenn nicht gefunden
|
||||
"""
|
||||
if not text or not isinstance(text, str):
|
||||
return None
|
||||
|
||||
match = AKTENZEICHEN_REGEX.match(text.strip())
|
||||
return match.group(1) if match else None
|
||||
|
||||
|
||||
def remove_aktenzeichen(text: str) -> str:
|
||||
"""
|
||||
Entfernt Aktenzeichen vom Anfang des Textes.
|
||||
|
||||
Examples:
|
||||
>>> remove_aktenzeichen("1234/56 Was ist der Stand?")
|
||||
"Was ist der Stand?"
|
||||
>>> remove_aktenzeichen("Kein Aktenzeichen")
|
||||
"Kein Aktenzeichen"
|
||||
|
||||
Args:
|
||||
text: Eingabetext mit Aktenzeichen
|
||||
|
||||
Returns:
|
||||
Text ohne Aktenzeichen (whitespace getrimmt)
|
||||
"""
|
||||
if not text or not isinstance(text, str):
|
||||
return text
|
||||
|
||||
return AKTENZEICHEN_REGEX.sub('', text, count=1).strip()
|
||||
|
||||
|
||||
def validate_aktenzeichen(az: str) -> bool:
|
||||
"""
|
||||
Validiert Aktenzeichen-Format.
|
||||
|
||||
Pattern: ^[A-Za-z0-9]{1,4}/\d{2}$
|
||||
|
||||
Examples:
|
||||
>>> validate_aktenzeichen("1234/56")
|
||||
True
|
||||
>>> validate_aktenzeichen("ABC/23")
|
||||
True
|
||||
>>> validate_aktenzeichen("12345/567") # Zu lang
|
||||
False
|
||||
>>> validate_aktenzeichen("1234-56") # Falsches Trennzeichen
|
||||
False
|
||||
|
||||
Args:
|
||||
az: Aktenzeichen zum Validieren
|
||||
|
||||
Returns:
|
||||
True wenn valide, False sonst
|
||||
"""
|
||||
if not az or not isinstance(az, str):
|
||||
return False
|
||||
|
||||
return bool(re.match(r'^[A-Za-z0-9]{1,4}/\d{2}$', az, re.IGNORECASE))
|
||||
|
||||
|
||||
def normalize_aktenzeichen(az: str) -> str:
|
||||
"""
|
||||
Normalisiert Aktenzeichen (uppercase, trim whitespace).
|
||||
|
||||
Examples:
|
||||
>>> normalize_aktenzeichen("abc/23")
|
||||
"ABC/23"
|
||||
>>> normalize_aktenzeichen(" 1234/56 ")
|
||||
"1234/56"
|
||||
|
||||
Args:
|
||||
az: Aktenzeichen zum Normalisieren
|
||||
|
||||
Returns:
|
||||
Normalisiertes Aktenzeichen (uppercase, getrimmt)
|
||||
"""
|
||||
if not az or not isinstance(az, str):
|
||||
return az
|
||||
|
||||
return az.strip().upper()
|
||||
@@ -6,9 +6,6 @@ Transformiert Bankverbindungen zwischen den beiden Systemen
|
||||
|
||||
from typing import Dict, Any, Optional, List
|
||||
from datetime import datetime
|
||||
import logging
|
||||
|
||||
logger = logging.getLogger(__name__)
|
||||
|
||||
|
||||
class BankverbindungenMapper:
|
||||
|
||||
@@ -13,63 +13,43 @@ Hilfsfunktionen für Sync-Operationen:
|
||||
from typing import Dict, Any, Optional, Tuple, Literal
|
||||
from datetime import datetime, timedelta
|
||||
import pytz
|
||||
import logging
|
||||
import redis
|
||||
import os
|
||||
|
||||
logger = logging.getLogger(__name__)
|
||||
from services.exceptions import LockAcquisitionError, SyncError, ValidationError
|
||||
from services.redis_client import get_redis_client
|
||||
from services.config import SYNC_CONFIG, get_lock_key, get_retry_delay_seconds
|
||||
from services.logging_utils import get_service_logger
|
||||
|
||||
import redis
|
||||
|
||||
# Timestamp-Vergleich Ergebnis-Typen
|
||||
TimestampResult = Literal["espocrm_newer", "advoware_newer", "conflict", "no_change"]
|
||||
|
||||
# Max retry before permanent failure
|
||||
MAX_SYNC_RETRIES = 5
|
||||
# Lock TTL in seconds (prevents deadlocks)
|
||||
LOCK_TTL_SECONDS = 900 # 15 minutes
|
||||
# Retry backoff: Wartezeit zwischen Retries (in Minuten)
|
||||
RETRY_BACKOFF_MINUTES = [1, 5, 15, 60, 240] # 1min, 5min, 15min, 1h, 4h
|
||||
# Auto-Reset nach 24h (für permanently_failed entities)
|
||||
AUTO_RESET_HOURS = 24
|
||||
|
||||
|
||||
class BeteiligteSync:
|
||||
"""Utility-Klasse für Beteiligte-Synchronisation"""
|
||||
|
||||
def __init__(self, espocrm_api, redis_client: redis.Redis = None, context=None):
|
||||
def __init__(self, espocrm_api, redis_client: Optional[redis.Redis] = None, context=None):
|
||||
self.espocrm = espocrm_api
|
||||
self.context = context
|
||||
self.logger = context.logger if context else logger
|
||||
self.redis = redis_client or self._init_redis()
|
||||
self.logger = get_service_logger('beteiligte_sync', context)
|
||||
|
||||
# Use provided Redis client or get from factory
|
||||
self.redis = redis_client or get_redis_client(strict=False)
|
||||
|
||||
if not self.redis:
|
||||
self.logger.error(
|
||||
"⚠️ KRITISCH: Redis nicht verfügbar! "
|
||||
"Distributed Locking deaktiviert - Race Conditions möglich!"
|
||||
)
|
||||
|
||||
# Import NotificationManager only when needed
|
||||
from services.notification_utils import NotificationManager
|
||||
self.notification_manager = NotificationManager(espocrm_api=self.espocrm, context=context)
|
||||
|
||||
def _init_redis(self) -> redis.Redis:
|
||||
"""Initialize Redis client for distributed locking"""
|
||||
try:
|
||||
redis_host = os.getenv('REDIS_HOST', 'localhost')
|
||||
redis_port = int(os.getenv('REDIS_PORT', '6379'))
|
||||
redis_db = int(os.getenv('REDIS_DB_ADVOWARE_CACHE', '1'))
|
||||
|
||||
client = redis.Redis(
|
||||
host=redis_host,
|
||||
port=redis_port,
|
||||
db=redis_db,
|
||||
decode_responses=True
|
||||
)
|
||||
client.ping()
|
||||
return client
|
||||
except Exception as e:
|
||||
self._log(f"Redis connection failed: {e}", level='error')
|
||||
return None
|
||||
|
||||
def _log(self, message: str, level: str = 'info'):
|
||||
"""Logging mit Context-Support"""
|
||||
if self.context and hasattr(self.context, 'logger'):
|
||||
getattr(self.context.logger, level)(message)
|
||||
else:
|
||||
getattr(logger, level)(message)
|
||||
def _log(self, message: str, level: str = 'info') -> None:
|
||||
"""Delegate logging to the logger with optional level"""
|
||||
log_func = getattr(self.logger, level, self.logger.info)
|
||||
log_func(message)
|
||||
|
||||
async def acquire_sync_lock(self, entity_id: str) -> bool:
|
||||
"""
|
||||
@@ -80,27 +60,39 @@ class BeteiligteSync:
|
||||
|
||||
Returns:
|
||||
True wenn Lock erfolgreich, False wenn bereits im Sync
|
||||
|
||||
Raises:
|
||||
SyncError: Bei kritischen Sync-Problemen
|
||||
"""
|
||||
try:
|
||||
# STEP 1: Atomic Redis lock (prevents race conditions)
|
||||
if self.redis:
|
||||
lock_key = f"sync_lock:cbeteiligte:{entity_id}"
|
||||
acquired = self.redis.set(lock_key, "locked", nx=True, ex=LOCK_TTL_SECONDS)
|
||||
lock_key = get_lock_key('cbeteiligte', entity_id)
|
||||
acquired = self.redis.set(
|
||||
lock_key,
|
||||
"locked",
|
||||
nx=True,
|
||||
ex=SYNC_CONFIG.lock_ttl_seconds
|
||||
)
|
||||
|
||||
if not acquired:
|
||||
self._log(f"Redis lock bereits aktiv für {entity_id}", level='warn')
|
||||
self.logger.warning(f"Redis lock bereits aktiv für {entity_id}")
|
||||
return False
|
||||
else:
|
||||
self.logger.error(
|
||||
f"⚠️ WARNUNG: Sync ohne Redis-Lock für {entity_id} - Race Condition möglich!"
|
||||
)
|
||||
|
||||
# STEP 2: Update syncStatus (für UI visibility)
|
||||
await self.espocrm.update_entity('CBeteiligte', entity_id, {
|
||||
'syncStatus': 'syncing'
|
||||
})
|
||||
|
||||
self._log(f"Sync-Lock für {entity_id} erworben")
|
||||
self.logger.info(f"Sync-Lock für {entity_id} erworben")
|
||||
return True
|
||||
|
||||
except Exception as e:
|
||||
self._log(f"Fehler beim Acquire Lock: {e}", level='error')
|
||||
self.logger.error(f"Fehler beim Acquire Lock: {e}")
|
||||
# Clean up Redis lock on error
|
||||
if self.redis:
|
||||
try:
|
||||
@@ -152,32 +144,42 @@ class BeteiligteSync:
|
||||
update_data['syncRetryCount'] = new_retry
|
||||
|
||||
# Exponential backoff - berechne nächsten Retry-Zeitpunkt
|
||||
if new_retry <= len(RETRY_BACKOFF_MINUTES):
|
||||
backoff_minutes = RETRY_BACKOFF_MINUTES[new_retry - 1]
|
||||
backoff_minutes = SYNC_CONFIG.retry_backoff_minutes
|
||||
if new_retry <= len(backoff_minutes):
|
||||
backoff_min = backoff_minutes[new_retry - 1]
|
||||
else:
|
||||
backoff_minutes = RETRY_BACKOFF_MINUTES[-1] # Letzte Backoff-Zeit
|
||||
backoff_min = backoff_minutes[-1] # Letzte Backoff-Zeit
|
||||
|
||||
next_retry = now_utc + timedelta(minutes=backoff_minutes)
|
||||
next_retry = now_utc + timedelta(minutes=backoff_min)
|
||||
update_data['syncNextRetry'] = next_retry.strftime('%Y-%m-%d %H:%M:%S')
|
||||
|
||||
self._log(f"Retry {new_retry}/{MAX_SYNC_RETRIES}, nächster Versuch in {backoff_minutes} Minuten")
|
||||
self.logger.info(
|
||||
f"Retry {new_retry}/{SYNC_CONFIG.max_retries}, "
|
||||
f"nächster Versuch in {backoff_min} Minuten"
|
||||
)
|
||||
|
||||
# Check max retries - mark as permanently failed
|
||||
if new_retry >= MAX_SYNC_RETRIES:
|
||||
if new_retry >= SYNC_CONFIG.max_retries:
|
||||
update_data['syncStatus'] = 'permanently_failed'
|
||||
|
||||
# Auto-Reset Timestamp für Wiederherstellung nach 24h
|
||||
auto_reset_time = now_utc + timedelta(hours=AUTO_RESET_HOURS)
|
||||
auto_reset_time = now_utc + timedelta(hours=SYNC_CONFIG.auto_reset_hours)
|
||||
update_data['syncAutoResetAt'] = auto_reset_time.strftime('%Y-%m-%d %H:%M:%S')
|
||||
|
||||
await self.send_notification(
|
||||
entity_id,
|
||||
'error',
|
||||
extra_data={
|
||||
'message': f"Sync fehlgeschlagen nach {MAX_SYNC_RETRIES} Versuchen. Auto-Reset in {AUTO_RESET_HOURS}h."
|
||||
'message': (
|
||||
f"Sync fehlgeschlagen nach {SYNC_CONFIG.max_retries} Versuchen. "
|
||||
f"Auto-Reset in {SYNC_CONFIG.auto_reset_hours}h."
|
||||
)
|
||||
}
|
||||
)
|
||||
self._log(f"Max retries ({MAX_SYNC_RETRIES}) erreicht für {entity_id}, Auto-Reset um {auto_reset_time}", level='error')
|
||||
self.logger.error(
|
||||
f"Max retries ({SYNC_CONFIG.max_retries}) erreicht für {entity_id}, "
|
||||
f"Auto-Reset um {auto_reset_time}"
|
||||
)
|
||||
else:
|
||||
update_data['syncRetryCount'] = 0
|
||||
update_data['syncNextRetry'] = None
|
||||
@@ -188,33 +190,32 @@ class BeteiligteSync:
|
||||
|
||||
await self.espocrm.update_entity('CBeteiligte', entity_id, update_data)
|
||||
|
||||
self._log(f"Sync-Lock released: {entity_id} → {new_status}")
|
||||
self.logger.info(f"Sync-Lock released: {entity_id} → {new_status}")
|
||||
|
||||
# Release Redis lock
|
||||
if self.redis:
|
||||
lock_key = f"sync_lock:cbeteiligte:{entity_id}"
|
||||
lock_key = get_lock_key('cbeteiligte', entity_id)
|
||||
self.redis.delete(lock_key)
|
||||
|
||||
except Exception as e:
|
||||
self._log(f"Fehler beim Release Lock: {e}", level='error')
|
||||
self.logger.error(f"Fehler beim Release Lock: {e}")
|
||||
# Ensure Redis lock is released even on error
|
||||
if self.redis:
|
||||
try:
|
||||
lock_key = f"sync_lock:cbeteiligte:{entity_id}"
|
||||
lock_key = get_lock_key('cbeteiligte', entity_id)
|
||||
self.redis.delete(lock_key)
|
||||
except:
|
||||
pass
|
||||
|
||||
@staticmethod
|
||||
def parse_timestamp(ts: Any) -> Optional[datetime]:
|
||||
def parse_timestamp(self, ts: Any) -> Optional[datetime]:
|
||||
"""
|
||||
Parse verschiedene Timestamp-Formate zu datetime
|
||||
Parse various timestamp formats to datetime.
|
||||
|
||||
Args:
|
||||
ts: String, datetime oder None
|
||||
ts: String, datetime or None
|
||||
|
||||
Returns:
|
||||
datetime-Objekt oder None
|
||||
datetime object or None
|
||||
"""
|
||||
if not ts:
|
||||
return None
|
||||
@@ -223,13 +224,13 @@ class BeteiligteSync:
|
||||
return ts
|
||||
|
||||
if isinstance(ts, str):
|
||||
# EspoCRM Format: "2026-02-07 14:30:00"
|
||||
# Advoware Format: "2026-02-07T14:30:00" oder "2026-02-07T14:30:00Z"
|
||||
# EspoCRM format: "2026-02-07 14:30:00"
|
||||
# Advoware format: "2026-02-07T14:30:00" or "2026-02-07T14:30:00Z"
|
||||
try:
|
||||
# Entferne trailing Z falls vorhanden
|
||||
# Remove trailing Z if present
|
||||
ts = ts.rstrip('Z')
|
||||
|
||||
# Versuche verschiedene Formate
|
||||
# Try various formats
|
||||
for fmt in [
|
||||
'%Y-%m-%d %H:%M:%S',
|
||||
'%Y-%m-%dT%H:%M:%S',
|
||||
@@ -240,11 +241,11 @@ class BeteiligteSync:
|
||||
except ValueError:
|
||||
continue
|
||||
|
||||
# Fallback: ISO-Format
|
||||
# Fallback: ISO format
|
||||
return datetime.fromisoformat(ts)
|
||||
|
||||
except Exception as e:
|
||||
logger.warning(f"Konnte Timestamp nicht parsen: {ts} - {e}")
|
||||
self._log(f"Could not parse timestamp: {ts} - {e}", level='warning')
|
||||
return None
|
||||
|
||||
return None
|
||||
|
||||
338
services/config.py
Normal file
338
services/config.py
Normal file
@@ -0,0 +1,338 @@
|
||||
"""
|
||||
Zentrale Konfiguration für BitByLaw Integration
|
||||
|
||||
Alle Magic Numbers und Strings sind hier zentralisiert.
|
||||
"""
|
||||
|
||||
from typing import List, Dict
|
||||
from dataclasses import dataclass
|
||||
import os
|
||||
|
||||
|
||||
# ========== Sync Configuration ==========
|
||||
|
||||
@dataclass
|
||||
class SyncConfig:
|
||||
"""Konfiguration für Sync-Operationen"""
|
||||
|
||||
# Retry-Konfiguration
|
||||
max_retries: int = 5
|
||||
"""Maximale Anzahl von Retry-Versuchen"""
|
||||
|
||||
retry_backoff_minutes: List[int] = None
|
||||
"""Exponential Backoff in Minuten: [1, 5, 15, 60, 240]"""
|
||||
|
||||
auto_reset_hours: int = 24
|
||||
"""Auto-Reset für permanently_failed Entities (in Stunden)"""
|
||||
|
||||
# Lock-Konfiguration
|
||||
lock_ttl_seconds: int = 900 # 15 Minuten
|
||||
"""TTL für distributed locks (verhindert Deadlocks)"""
|
||||
|
||||
lock_prefix: str = "sync_lock"
|
||||
"""Prefix für Redis Lock Keys"""
|
||||
|
||||
# Validation
|
||||
validate_before_sync: bool = True
|
||||
"""Validiere Entities vor dem Sync (empfohlen)"""
|
||||
|
||||
# Change Detection
|
||||
use_rowid_change_detection: bool = True
|
||||
"""Nutze rowId für Change Detection (Advoware)"""
|
||||
|
||||
def __post_init__(self):
|
||||
if self.retry_backoff_minutes is None:
|
||||
# Default exponential backoff: 1, 5, 15, 60, 240 Minuten
|
||||
self.retry_backoff_minutes = [1, 5, 15, 60, 240]
|
||||
|
||||
|
||||
# Singleton Instance
|
||||
SYNC_CONFIG = SyncConfig()
|
||||
|
||||
|
||||
# ========== API Configuration ==========
|
||||
|
||||
@dataclass
|
||||
class APIConfig:
|
||||
"""API-spezifische Konfiguration"""
|
||||
|
||||
# Timeouts
|
||||
default_timeout_seconds: int = 30
|
||||
"""Default Timeout für API-Calls"""
|
||||
|
||||
long_running_timeout_seconds: int = 120
|
||||
"""Timeout für lange Operations (z.B. Uploads)"""
|
||||
|
||||
# Retry
|
||||
max_api_retries: int = 3
|
||||
"""Anzahl Retries bei API-Fehlern"""
|
||||
|
||||
retry_status_codes: List[int] = None
|
||||
"""HTTP Status Codes die Retry auslösen"""
|
||||
|
||||
# Rate Limiting
|
||||
rate_limit_enabled: bool = True
|
||||
"""Aktiviere Rate Limiting"""
|
||||
|
||||
rate_limit_calls_per_minute: int = 60
|
||||
"""Max. API-Calls pro Minute"""
|
||||
|
||||
def __post_init__(self):
|
||||
if self.retry_status_codes is None:
|
||||
# Retry bei: 408 (Timeout), 429 (Rate Limit), 500, 502, 503, 504
|
||||
self.retry_status_codes = [408, 429, 500, 502, 503, 504]
|
||||
|
||||
|
||||
API_CONFIG = APIConfig()
|
||||
|
||||
|
||||
# ========== Advoware Configuration ==========
|
||||
|
||||
@dataclass
|
||||
class AdvowareConfig:
|
||||
"""Advoware-spezifische Konfiguration"""
|
||||
|
||||
# Token Management
|
||||
token_lifetime_minutes: int = 55
|
||||
"""Token-Lifetime (tatsächlich 60min, aber 5min Puffer)"""
|
||||
|
||||
token_cache_key: str = "advoware_access_token"
|
||||
"""Redis Key für Token Cache"""
|
||||
|
||||
token_timestamp_key: str = "advoware_token_timestamp"
|
||||
"""Redis Key für Token Timestamp"""
|
||||
|
||||
# Auth
|
||||
auth_url: str = "https://security.advo-net.net/api/v1/Token"
|
||||
"""Advoware Auth-Endpoint"""
|
||||
|
||||
product_id: int = 64
|
||||
"""Advoware Product ID"""
|
||||
|
||||
# Field Mapping
|
||||
readonly_fields: List[str] = None
|
||||
"""Felder die nicht via PUT geändert werden können"""
|
||||
|
||||
def __post_init__(self):
|
||||
if self.readonly_fields is None:
|
||||
# Diese Felder können nicht via PUT geändert werden
|
||||
self.readonly_fields = [
|
||||
'betNr', 'rowId', 'kommKz', # Kommunikation: kommKz ist read-only!
|
||||
'handelsRegisterNummer', 'registergericht' # Werden ignoriert von API
|
||||
]
|
||||
|
||||
|
||||
ADVOWARE_CONFIG = AdvowareConfig()
|
||||
|
||||
|
||||
# ========== EspoCRM Configuration ==========
|
||||
|
||||
@dataclass
|
||||
class EspoCRMConfig:
|
||||
"""EspoCRM-spezifische Konfiguration"""
|
||||
|
||||
# API
|
||||
default_page_size: int = 50
|
||||
"""Default Seitengröße für Listen-Abfragen"""
|
||||
|
||||
max_page_size: int = 200
|
||||
"""Maximale Seitengröße"""
|
||||
|
||||
# Sync Status Fields
|
||||
sync_status_field: str = "syncStatus"
|
||||
"""Feldname für Sync-Status"""
|
||||
|
||||
sync_error_field: str = "syncErrorMessage"
|
||||
"""Feldname für Sync-Fehler"""
|
||||
|
||||
sync_retry_field: str = "syncRetryCount"
|
||||
"""Feldname für Retry-Counter"""
|
||||
|
||||
# Notifications
|
||||
notification_enabled: bool = True
|
||||
"""In-App Notifications aktivieren"""
|
||||
|
||||
notification_user_id: str = "1"
|
||||
"""User-ID für Notifications (Marvin)"""
|
||||
|
||||
|
||||
ESPOCRM_CONFIG = EspoCRMConfig()
|
||||
|
||||
|
||||
# ========== Redis Configuration ==========
|
||||
|
||||
@dataclass
|
||||
class RedisConfig:
|
||||
"""Redis-spezifische Konfiguration"""
|
||||
|
||||
# Connection
|
||||
host: str = "localhost"
|
||||
port: int = 6379
|
||||
db: int = 1
|
||||
timeout_seconds: int = 5
|
||||
max_connections: int = 50
|
||||
|
||||
# Behavior
|
||||
decode_responses: bool = True
|
||||
"""Auto-decode bytes zu strings"""
|
||||
|
||||
health_check_interval: int = 30
|
||||
"""Health-Check Interval in Sekunden"""
|
||||
|
||||
# Keys
|
||||
key_prefix: str = "bitbylaw"
|
||||
"""Prefix für alle Redis Keys"""
|
||||
|
||||
def get_key(self, key: str) -> str:
|
||||
"""Gibt vollen Redis Key mit Prefix zurück"""
|
||||
return f"{self.key_prefix}:{key}"
|
||||
|
||||
@classmethod
|
||||
def from_env(cls) -> 'RedisConfig':
|
||||
"""Lädt Redis-Config aus Environment Variables"""
|
||||
return cls(
|
||||
host=os.getenv('REDIS_HOST', 'localhost'),
|
||||
port=int(os.getenv('REDIS_PORT', '6379')),
|
||||
db=int(os.getenv('REDIS_DB_ADVOWARE_CACHE', '1')),
|
||||
timeout_seconds=int(os.getenv('REDIS_TIMEOUT_SECONDS', '5')),
|
||||
max_connections=int(os.getenv('REDIS_MAX_CONNECTIONS', '50'))
|
||||
)
|
||||
|
||||
|
||||
REDIS_CONFIG = RedisConfig.from_env()
|
||||
|
||||
|
||||
# ========== Logging Configuration ==========
|
||||
|
||||
@dataclass
|
||||
class LoggingConfig:
|
||||
"""Logging-Konfiguration"""
|
||||
|
||||
# Levels
|
||||
default_level: str = "INFO"
|
||||
"""Default Log-Level"""
|
||||
|
||||
api_level: str = "INFO"
|
||||
"""Log-Level für API-Calls"""
|
||||
|
||||
sync_level: str = "INFO"
|
||||
"""Log-Level für Sync-Operations"""
|
||||
|
||||
# Format
|
||||
log_format: str = "[{timestamp}] {level} {logger}: {message}"
|
||||
"""Log-Format"""
|
||||
|
||||
include_context: bool = True
|
||||
"""Motia FlowContext in Logs einbinden"""
|
||||
|
||||
# Performance
|
||||
log_api_timings: bool = True
|
||||
"""API Call Timings loggen"""
|
||||
|
||||
log_sync_duration: bool = True
|
||||
"""Sync-Dauer loggen"""
|
||||
|
||||
|
||||
LOGGING_CONFIG = LoggingConfig()
|
||||
|
||||
|
||||
# ========== Calendar Sync Configuration ==========
|
||||
|
||||
@dataclass
|
||||
class CalendarSyncConfig:
|
||||
"""Konfiguration für Google Calendar Sync"""
|
||||
|
||||
# Sync Window
|
||||
sync_days_past: int = 7
|
||||
"""Tage in die Vergangenheit syncen"""
|
||||
|
||||
sync_days_future: int = 90
|
||||
"""Tage in die Zukunft syncen"""
|
||||
|
||||
# Cron
|
||||
cron_schedule: str = "0 */15 * * * *"
|
||||
"""Cron-Schedule (jede 15 Minuten)"""
|
||||
|
||||
# Batch Size
|
||||
batch_size: int = 10
|
||||
"""Anzahl Mitarbeiter pro Batch"""
|
||||
|
||||
|
||||
CALENDAR_SYNC_CONFIG = CalendarSyncConfig()
|
||||
|
||||
|
||||
# ========== Feature Flags ==========
|
||||
|
||||
@dataclass
|
||||
class FeatureFlags:
|
||||
"""Feature Flags für schrittweises Rollout"""
|
||||
|
||||
# Validation
|
||||
strict_validation: bool = True
|
||||
"""Strenge Validierung mit Pydantic"""
|
||||
|
||||
# Sync Features
|
||||
kommunikation_sync_enabled: bool = False
|
||||
"""Kommunikation-Sync aktivieren (noch in Entwicklung)"""
|
||||
|
||||
document_sync_enabled: bool = False
|
||||
"""Document-Sync aktivieren (noch in Entwicklung)"""
|
||||
|
||||
# Advanced Features
|
||||
parallel_sync_enabled: bool = False
|
||||
"""Parallele Sync-Operations (experimentell)"""
|
||||
|
||||
auto_conflict_resolution: bool = False
|
||||
"""Automatische Konfliktauflösung (experimentell)"""
|
||||
|
||||
# Debug
|
||||
debug_mode: bool = False
|
||||
"""Debug-Modus (mehr Logging, langsamer)"""
|
||||
|
||||
|
||||
FEATURE_FLAGS = FeatureFlags()
|
||||
|
||||
|
||||
# ========== Helper Functions ==========
|
||||
|
||||
def get_retry_delay_seconds(attempt: int) -> int:
|
||||
"""
|
||||
Gibt Retry-Delay in Sekunden für gegebenen Versuch zurück.
|
||||
|
||||
Args:
|
||||
attempt: Versuchs-Nummer (0-indexed)
|
||||
|
||||
Returns:
|
||||
Delay in Sekunden
|
||||
"""
|
||||
backoff_minutes = SYNC_CONFIG.retry_backoff_minutes
|
||||
if attempt < len(backoff_minutes):
|
||||
return backoff_minutes[attempt] * 60
|
||||
return backoff_minutes[-1] * 60
|
||||
|
||||
|
||||
def get_lock_key(entity_type: str, entity_id: str) -> str:
|
||||
"""
|
||||
Erzeugt Redis Lock-Key für Entity.
|
||||
|
||||
Args:
|
||||
entity_type: Entity-Typ (z.B. 'cbeteiligte')
|
||||
entity_id: Entity-ID
|
||||
|
||||
Returns:
|
||||
Redis Key
|
||||
"""
|
||||
return f"{SYNC_CONFIG.lock_prefix}:{entity_type.lower()}:{entity_id}"
|
||||
|
||||
|
||||
def is_retryable_status_code(status_code: int) -> bool:
|
||||
"""
|
||||
Prüft ob HTTP Status Code Retry auslösen soll.
|
||||
|
||||
Args:
|
||||
status_code: HTTP Status Code
|
||||
|
||||
Returns:
|
||||
True wenn retryable
|
||||
"""
|
||||
return status_code in API_CONFIG.retry_status_codes
|
||||
622
services/document_sync_utils.py
Normal file
622
services/document_sync_utils.py
Normal file
@@ -0,0 +1,622 @@
|
||||
"""
|
||||
Document Sync Utilities
|
||||
|
||||
Utility functions for document synchronization with xAI:
|
||||
- Distributed locking via Redis + syncStatus
|
||||
- Decision logic: When does a document need xAI sync?
|
||||
- Related entities determination (Many-to-Many attachments)
|
||||
- xAI Collection management
|
||||
"""
|
||||
|
||||
from typing import Dict, Any, Optional, List, Tuple
|
||||
from datetime import datetime, timedelta
|
||||
from urllib.parse import unquote
|
||||
|
||||
from services.sync_utils_base import BaseSyncUtils
|
||||
from services.models import FileStatus, XAISyncStatus
|
||||
|
||||
# Max retry before permanent failure
|
||||
MAX_SYNC_RETRIES = 5
|
||||
|
||||
# Retry backoff: Wartezeit zwischen Retries (in Minuten)
|
||||
RETRY_BACKOFF_MINUTES = [1, 5, 15, 60, 240] # 1min, 5min, 15min, 1h, 4h
|
||||
|
||||
# Legacy file status values (for backward compatibility)
|
||||
# These are old German and English status values that may still exist in the database
|
||||
LEGACY_NEW_STATUS_VALUES = {'neu', 'Neu', 'New'}
|
||||
LEGACY_CHANGED_STATUS_VALUES = {'geändert', 'Geändert', 'Changed'}
|
||||
LEGACY_SYNCED_STATUS_VALUES = {'synced', 'Synced', 'synchronized', 'Synchronized'}
|
||||
|
||||
|
||||
class DocumentSync(BaseSyncUtils):
|
||||
"""Utility class for document synchronization with xAI"""
|
||||
|
||||
def _get_lock_key(self, entity_id: str) -> str:
|
||||
"""Redis lock key for documents"""
|
||||
return f"sync_lock:document:{entity_id}"
|
||||
|
||||
async def acquire_sync_lock(self, entity_id: str, entity_type: str = 'CDokumente') -> bool:
|
||||
"""
|
||||
Atomic distributed lock via Redis + syncStatus update
|
||||
|
||||
Args:
|
||||
entity_id: EspoCRM Document ID
|
||||
entity_type: Entity-Type (CDokumente oder Document)
|
||||
|
||||
Returns:
|
||||
True wenn Lock erfolgreich, False wenn bereits im Sync
|
||||
"""
|
||||
try:
|
||||
# STEP 1: Atomic Redis lock (prevents race conditions)
|
||||
lock_key = self._get_lock_key(entity_id)
|
||||
if not self._acquire_redis_lock(lock_key):
|
||||
self._log(f"Redis lock bereits aktiv für {entity_type} {entity_id}", level='warn')
|
||||
return False
|
||||
|
||||
# STEP 2: Update xaiSyncStatus to pending_sync
|
||||
try:
|
||||
await self.espocrm.update_entity(entity_type, entity_id, {
|
||||
'xaiSyncStatus': XAISyncStatus.PENDING_SYNC.value
|
||||
})
|
||||
except Exception as e:
|
||||
self._log(f"Could not set xaiSyncStatus: {e}", level='debug')
|
||||
|
||||
self._log(f"Sync-Lock für {entity_type} {entity_id} erworben")
|
||||
return True
|
||||
|
||||
except Exception as e:
|
||||
self._log(f"Fehler beim Acquire Lock: {e}", level='error')
|
||||
# Clean up Redis lock on error
|
||||
lock_key = self._get_lock_key(entity_id)
|
||||
self._release_redis_lock(lock_key)
|
||||
return False
|
||||
|
||||
async def release_sync_lock(
|
||||
self,
|
||||
entity_id: str,
|
||||
success: bool = True,
|
||||
error_message: Optional[str] = None,
|
||||
extra_fields: Optional[Dict[str, Any]] = None,
|
||||
entity_type: str = 'CDokumente'
|
||||
) -> None:
|
||||
"""
|
||||
Gibt Sync-Lock frei und setzt finalen Status
|
||||
|
||||
Args:
|
||||
entity_id: EspoCRM Document ID
|
||||
success: Ob Sync erfolgreich war
|
||||
error_message: Optional: Fehlermeldung
|
||||
extra_fields: Optional: Zusätzliche Felder (z.B. xaiFileId, xaiCollections)
|
||||
entity_type: Entity-Type (CDokumente oder Document)
|
||||
"""
|
||||
try:
|
||||
update_data = {}
|
||||
|
||||
# Set xaiSyncStatus: clean on success, failed on error
|
||||
try:
|
||||
update_data['xaiSyncStatus'] = XAISyncStatus.CLEAN.value if success else XAISyncStatus.FAILED.value
|
||||
|
||||
if error_message:
|
||||
update_data['xaiSyncError'] = error_message[:2000]
|
||||
else:
|
||||
update_data['xaiSyncError'] = None
|
||||
except:
|
||||
pass # Fields may not exist
|
||||
|
||||
# Merge extra fields (z.B. xaiFileId, xaiCollections)
|
||||
if extra_fields:
|
||||
update_data.update(extra_fields)
|
||||
|
||||
if update_data:
|
||||
await self.espocrm.update_entity(entity_type, entity_id, update_data)
|
||||
|
||||
self._log(f"Sync-Lock released: {entity_type} {entity_id} → {'success' if success else 'failed'}")
|
||||
|
||||
# Release Redis lock
|
||||
lock_key = self._get_lock_key(entity_id)
|
||||
self._release_redis_lock(lock_key)
|
||||
|
||||
except Exception as e:
|
||||
self._log(f"Fehler beim Release Lock: {e}", level='error')
|
||||
# Ensure Redis lock is released even on error
|
||||
lock_key = self._get_lock_key(entity_id)
|
||||
self._release_redis_lock(lock_key)
|
||||
|
||||
async def should_sync_to_xai(
|
||||
self,
|
||||
document: Dict[str, Any],
|
||||
entity_type: str = 'CDokumente'
|
||||
) -> Tuple[bool, List[str], str]:
|
||||
"""
|
||||
Decide if a document needs to be synchronized to xAI.
|
||||
|
||||
Checks:
|
||||
1. File status field ("new", "changed")
|
||||
2. Hash values for change detection
|
||||
3. Related entities with xAI collections
|
||||
|
||||
Args:
|
||||
document: Complete document entity from EspoCRM
|
||||
|
||||
Returns:
|
||||
Tuple[bool, List[str], str]:
|
||||
- bool: Whether sync is needed
|
||||
- List[str]: List of collection IDs where the document should go
|
||||
- str: Reason/description of the decision
|
||||
"""
|
||||
doc_id = document.get('id')
|
||||
doc_name = document.get('name', 'Unbenannt')
|
||||
|
||||
# xAI-relevant fields
|
||||
xai_file_id = document.get('xaiFileId')
|
||||
xai_collections = document.get('xaiCollections') or []
|
||||
xai_sync_status = document.get('xaiSyncStatus')
|
||||
|
||||
# File status and hash fields
|
||||
datei_status = document.get('dateiStatus') or document.get('fileStatus')
|
||||
file_md5 = document.get('md5') or document.get('fileMd5')
|
||||
file_sha = document.get('sha') or document.get('fileSha')
|
||||
xai_synced_hash = document.get('xaiSyncedHash') # Hash at last xAI sync
|
||||
|
||||
self._log(f"📋 Document analysis: {doc_name} (ID: {doc_id})")
|
||||
self._log(f" xaiFileId: {xai_file_id or 'N/A'}")
|
||||
self._log(f" xaiCollections: {xai_collections}")
|
||||
self._log(f" xaiSyncStatus: {xai_sync_status or 'N/A'}")
|
||||
self._log(f" fileStatus: {datei_status or 'N/A'}")
|
||||
self._log(f" MD5: {file_md5[:16] if file_md5 else 'N/A'}...")
|
||||
self._log(f" SHA: {file_sha[:16] if file_sha else 'N/A'}...")
|
||||
self._log(f" xaiSyncedHash: {xai_synced_hash[:16] if xai_synced_hash else 'N/A'}...")
|
||||
|
||||
# Determine target collections from relations (CDokumente -> linked entities)
|
||||
target_collections = await self._get_required_collections_from_relations(
|
||||
doc_id,
|
||||
entity_type=entity_type
|
||||
)
|
||||
|
||||
# Check xaiSyncStatus="no_sync" -> no sync for this document
|
||||
if xai_sync_status == XAISyncStatus.NO_SYNC.value:
|
||||
self._log("⏭️ No xAI sync needed: xaiSyncStatus='no_sync'")
|
||||
return (False, [], "xaiSyncStatus is 'no_sync'")
|
||||
|
||||
if not target_collections:
|
||||
self._log("⏭️ No xAI sync needed: No related entities with xAI collections")
|
||||
return (False, [], "No linked entities with xAI collections")
|
||||
|
||||
# ═══════════════════════════════════════════════════════════════
|
||||
# PRIORITY CHECK 1: xaiSyncStatus="unclean" -> document was changed
|
||||
# ═══════════════════════════════════════════════════════════════
|
||||
if xai_sync_status == XAISyncStatus.UNCLEAN.value:
|
||||
self._log(f"🆕 xaiSyncStatus='unclean' → xAI sync REQUIRED")
|
||||
return (True, target_collections, "xaiSyncStatus='unclean'")
|
||||
|
||||
# ═══════════════════════════════════════════════════════════════
|
||||
# PRIORITY CHECK 2: fileStatus "new" or "changed"
|
||||
# ═══════════════════════════════════════════════════════════════
|
||||
# Check for standard enum values and legacy values
|
||||
is_new = (datei_status == FileStatus.NEW.value or datei_status in LEGACY_NEW_STATUS_VALUES)
|
||||
is_changed = (datei_status == FileStatus.CHANGED.value or datei_status in LEGACY_CHANGED_STATUS_VALUES)
|
||||
|
||||
if is_new or is_changed:
|
||||
self._log(f"🆕 fileStatus: '{datei_status}' → xAI sync REQUIRED")
|
||||
|
||||
if target_collections:
|
||||
return (True, target_collections, f"fileStatus: {datei_status}")
|
||||
else:
|
||||
# File is new/changed but no collections found
|
||||
self._log(f"⚠️ fileStatus '{datei_status}' but no collections found - skipping sync")
|
||||
return (False, [], f"fileStatus: {datei_status}, but no collections")
|
||||
|
||||
# ═══════════════════════════════════════════════════════════════
|
||||
# CASE 1: Document is already in xAI AND collections are set
|
||||
# ═══════════════════════════════════════════════════════════════
|
||||
if xai_file_id:
|
||||
self._log(f"✅ Document already synced to xAI with {len(target_collections)} collection(s)")
|
||||
|
||||
# Check if file content was changed (hash comparison)
|
||||
current_hash = file_md5 or file_sha
|
||||
|
||||
if current_hash and xai_synced_hash:
|
||||
if current_hash != xai_synced_hash:
|
||||
self._log(f"🔄 Hash change detected! RESYNC required")
|
||||
self._log(f" Old: {xai_synced_hash[:16]}...")
|
||||
self._log(f" New: {current_hash[:16]}...")
|
||||
return (True, target_collections, "File content changed (hash mismatch)")
|
||||
else:
|
||||
self._log(f"✅ Hash identical - no change")
|
||||
else:
|
||||
self._log(f"⚠️ No hash values available for comparison")
|
||||
|
||||
return (False, target_collections, "Already synced, no change detected")
|
||||
|
||||
# ═══════════════════════════════════════════════════════════════
|
||||
# CASE 2: Document has xaiFileId but collections is empty/None
|
||||
# ═══════════════════════════════════════════════════════════════
|
||||
# ═══════════════════════════════════════════════════════════════
|
||||
# CASE 3: Collections present but no status/hash trigger
|
||||
# ═══════════════════════════════════════════════════════════════
|
||||
self._log(f"✅ Document is linked to {len(target_collections)} entity/ies with collections")
|
||||
return (True, target_collections, "Linked to entities that require collections")
|
||||
|
||||
async def _get_required_collections_from_relations(
|
||||
self,
|
||||
document_id: str,
|
||||
entity_type: str = 'Document'
|
||||
) -> List[str]:
|
||||
"""
|
||||
Determine all xAI collection IDs of CAIKnowledge entities linked to this document.
|
||||
|
||||
Checks CAIKnowledgeCDokumente junction table:
|
||||
- Status 'active' + datenbankId: Returns collection ID
|
||||
- Status 'new': Returns "NEW:{knowledge_id}" marker (collection must be created first)
|
||||
- Other statuses (paused, deactivated): Skips
|
||||
|
||||
Args:
|
||||
document_id: Document ID
|
||||
entity_type: Entity type (e.g., 'CDokumente')
|
||||
|
||||
Returns:
|
||||
List of collection IDs or markers:
|
||||
- Normal IDs: "abc123..." (existing collections)
|
||||
- New markers: "NEW:kb-id..." (collection needs to be created via knowledge sync)
|
||||
"""
|
||||
collections = set()
|
||||
|
||||
self._log(f"🔍 Checking relations of {entity_type} {document_id}...")
|
||||
|
||||
# ═══════════════════════════════════════════════════════════════
|
||||
# SPECIAL HANDLING: CAIKnowledge via Junction Table
|
||||
# ═══════════════════════════════════════════════════════════════
|
||||
try:
|
||||
junction_entries = await self.espocrm.get_junction_entries(
|
||||
'CAIKnowledgeCDokumente',
|
||||
'cDokumenteId',
|
||||
document_id
|
||||
)
|
||||
|
||||
if junction_entries:
|
||||
self._log(f" 📋 Found {len(junction_entries)} CAIKnowledge link(s)")
|
||||
|
||||
for junction in junction_entries:
|
||||
knowledge_id = junction.get('cAIKnowledgeId')
|
||||
if not knowledge_id:
|
||||
continue
|
||||
|
||||
try:
|
||||
knowledge = await self.espocrm.get_entity('CAIKnowledge', knowledge_id)
|
||||
activation_status = knowledge.get('aktivierungsstatus')
|
||||
collection_id = knowledge.get('datenbankId')
|
||||
|
||||
if activation_status == 'active' and collection_id:
|
||||
# Existing collection - use it
|
||||
collections.add(collection_id)
|
||||
self._log(f" ✅ CAIKnowledge {knowledge_id}: {collection_id} (active)")
|
||||
elif activation_status == 'new':
|
||||
# Collection doesn't exist yet - return special marker
|
||||
# Format: "NEW:{knowledge_id}" signals to caller: trigger knowledge sync first
|
||||
collections.add(f"NEW:{knowledge_id}")
|
||||
self._log(f" 🆕 CAIKnowledge {knowledge_id}: status='new' → collection must be created first")
|
||||
else:
|
||||
self._log(f" ⏭️ CAIKnowledge {knowledge_id}: status={activation_status}, datenbankId={collection_id or 'N/A'}")
|
||||
|
||||
except Exception as e:
|
||||
self._log(f" ⚠️ Failed to load CAIKnowledge {knowledge_id}: {e}", level='warn')
|
||||
|
||||
except Exception as e:
|
||||
self._log(f" ⚠️ Failed to check CAIKnowledge junction: {e}", level='warn')
|
||||
|
||||
result = list(collections)
|
||||
self._log(f"📊 Gesamt: {len(result)} eindeutige Collection(s) gefunden")
|
||||
|
||||
return result
|
||||
|
||||
async def get_document_download_info(self, document_id: str, entity_type: str = 'CDokumente') -> Optional[Dict[str, Any]]:
|
||||
"""
|
||||
Holt Download-Informationen für ein Document
|
||||
|
||||
Args:
|
||||
document_id: ID des Documents
|
||||
entity_type: Entity-Type (CDokumente oder Document)
|
||||
|
||||
Returns:
|
||||
Dict mit:
|
||||
- attachment_id: ID des Attachments
|
||||
- download_url: URL zum Download
|
||||
- filename: Dateiname
|
||||
- mime_type: MIME-Type
|
||||
- size: Dateigröße in Bytes
|
||||
"""
|
||||
try:
|
||||
# Hole vollständiges Document
|
||||
doc = await self.espocrm.get_entity(entity_type, document_id)
|
||||
|
||||
# EspoCRM Documents können Files auf verschiedene Arten speichern:
|
||||
# CDokumente: dokumentId/dokumentName (Custom Entity)
|
||||
# Document: fileId/fileName ODER attachmentsIds
|
||||
|
||||
attachment_id = None
|
||||
filename = None
|
||||
|
||||
# Prüfe zuerst dokumentId (CDokumente Custom Entity)
|
||||
if doc.get('dokumentId'):
|
||||
attachment_id = doc.get('dokumentId')
|
||||
filename = doc.get('dokumentName')
|
||||
self._log(f"📎 CDokumente verwendet dokumentId: {attachment_id}")
|
||||
|
||||
# Fallback: fileId (Standard Document Entity)
|
||||
elif doc.get('fileId'):
|
||||
attachment_id = doc.get('fileId')
|
||||
filename = doc.get('fileName')
|
||||
self._log(f"📎 Document verwendet fileId: {attachment_id}")
|
||||
|
||||
# Fallback 2: attachmentsIds (z.B. bei zusätzlichen Attachments)
|
||||
elif doc.get('attachmentsIds'):
|
||||
attachment_ids = doc.get('attachmentsIds')
|
||||
if attachment_ids:
|
||||
attachment_id = attachment_ids[0]
|
||||
self._log(f"📎 Document verwendet attachmentsIds: {attachment_id}")
|
||||
|
||||
if not attachment_id:
|
||||
self._log(f"⚠️ {entity_type} {document_id} hat weder dokumentId, fileId noch attachmentsIds", level='warn')
|
||||
self._log(f" Verfügbare Felder: {list(doc.keys())}")
|
||||
return None
|
||||
|
||||
# Hole Attachment-Details
|
||||
attachment = await self.espocrm.get_entity('Attachment', attachment_id)
|
||||
|
||||
# Filename: Nutze dokumentName/fileName falls vorhanden, sonst aus Attachment
|
||||
final_filename = filename or attachment.get('name', 'unknown')
|
||||
|
||||
# URL-decode filename (fixes special chars like §, ä, ö, ü, etc.)
|
||||
# EspoCRM stores filenames URL-encoded: %C2%A7 → §
|
||||
final_filename = unquote(final_filename)
|
||||
|
||||
return {
|
||||
'attachment_id': attachment_id,
|
||||
'download_url': f"/api/v1/Attachment/file/{attachment_id}",
|
||||
'filename': final_filename,
|
||||
'mime_type': attachment.get('type', 'application/octet-stream'),
|
||||
'size': attachment.get('size', 0)
|
||||
}
|
||||
|
||||
except Exception as e:
|
||||
self._log(f"❌ Fehler beim Laden von Download-Info: {e}", level='error')
|
||||
return None
|
||||
|
||||
async def generate_thumbnail(self, file_path: str, mime_type: str, max_width: int = 600, max_height: int = 800) -> Optional[bytes]:
|
||||
"""
|
||||
Generiert Vorschaubild (Preview) für ein Document im WebP-Format
|
||||
|
||||
Unterstützt:
|
||||
- PDF: Erste Seite als Bild
|
||||
- DOCX/DOC: Konvertierung zu PDF, dann erste Seite
|
||||
- Images: Resize auf Preview-Größe
|
||||
- Andere: Platzhalter-Icon basierend auf MIME-Type
|
||||
|
||||
Args:
|
||||
file_path: Pfad zur Datei (lokal)
|
||||
mime_type: MIME-Type des Documents
|
||||
max_width: Maximale Breite (default: 600px)
|
||||
max_height: Maximale Höhe (default: 800px)
|
||||
|
||||
Returns:
|
||||
Preview als WebP bytes oder None bei Fehler
|
||||
"""
|
||||
self._log(f"🖼️ Preview-Generierung für {mime_type} (max: {max_width}x{max_height})")
|
||||
|
||||
try:
|
||||
from PIL import Image
|
||||
import io
|
||||
|
||||
thumbnail = None
|
||||
|
||||
# PDF-Handling
|
||||
if mime_type == 'application/pdf':
|
||||
try:
|
||||
from pdf2image import convert_from_path
|
||||
self._log(" Converting PDF page 1 to image...")
|
||||
images = convert_from_path(file_path, first_page=1, last_page=1, dpi=150)
|
||||
if images:
|
||||
thumbnail = images[0]
|
||||
except ImportError:
|
||||
self._log("⚠️ pdf2image nicht installiert - überspringe PDF-Preview", level='warn')
|
||||
return None
|
||||
except Exception as e:
|
||||
self._log(f"⚠️ PDF-Konvertierung fehlgeschlagen: {e}", level='warn')
|
||||
return None
|
||||
|
||||
# DOCX/DOC-Handling
|
||||
elif mime_type in ['application/vnd.openxmlformats-officedocument.wordprocessingml.document',
|
||||
'application/msword']:
|
||||
try:
|
||||
import tempfile
|
||||
import os
|
||||
from docx2pdf import convert
|
||||
from pdf2image import convert_from_path
|
||||
|
||||
self._log(" Converting DOCX → PDF → Image...")
|
||||
|
||||
# Temporäres PDF erstellen
|
||||
with tempfile.NamedTemporaryFile(suffix='.pdf', delete=False) as tmp:
|
||||
pdf_path = tmp.name
|
||||
|
||||
# DOCX → PDF (benötigt LibreOffice)
|
||||
convert(file_path, pdf_path)
|
||||
|
||||
# PDF → Image
|
||||
images = convert_from_path(pdf_path, first_page=1, last_page=1, dpi=150)
|
||||
if images:
|
||||
thumbnail = images[0]
|
||||
|
||||
# Cleanup
|
||||
os.remove(pdf_path)
|
||||
|
||||
except ImportError:
|
||||
self._log("⚠️ docx2pdf nicht installiert - überspringe DOCX-Preview", level='warn')
|
||||
return None
|
||||
except Exception as e:
|
||||
self._log(f"⚠️ DOCX-Konvertierung fehlgeschlagen: {e}", level='warn')
|
||||
return None
|
||||
|
||||
# Image-Handling
|
||||
elif mime_type.startswith('image/'):
|
||||
try:
|
||||
self._log(" Processing image file...")
|
||||
thumbnail = Image.open(file_path)
|
||||
except Exception as e:
|
||||
self._log(f"⚠️ Image-Laden fehlgeschlagen: {e}", level='warn')
|
||||
return None
|
||||
|
||||
else:
|
||||
self._log(f"⚠️ Keine Preview-Generierung für MIME-Type: {mime_type}", level='warn')
|
||||
return None
|
||||
|
||||
if not thumbnail:
|
||||
return None
|
||||
|
||||
# Resize auf max dimensions (behält Aspect Ratio)
|
||||
thumbnail.thumbnail((max_width, max_height), Image.Resampling.LANCZOS)
|
||||
|
||||
# Convert zu WebP bytes
|
||||
buffer = io.BytesIO()
|
||||
thumbnail.save(buffer, format='WEBP', quality=85)
|
||||
webp_bytes = buffer.getvalue()
|
||||
|
||||
self._log(f"✅ Preview generiert: {len(webp_bytes)} bytes WebP")
|
||||
return webp_bytes
|
||||
|
||||
except Exception as e:
|
||||
self._log(f"❌ Fehler bei Preview-Generierung: {e}", level='error')
|
||||
import traceback
|
||||
self._log(traceback.format_exc(), level='debug')
|
||||
return None
|
||||
|
||||
async def update_sync_metadata(
|
||||
self,
|
||||
document_id: str,
|
||||
xai_file_id: Optional[str] = None,
|
||||
collection_ids: Optional[List[str]] = None,
|
||||
file_hash: Optional[str] = None,
|
||||
preview_data: Optional[bytes] = None,
|
||||
reset_file_status: bool = False,
|
||||
entity_type: str = 'CDokumente'
|
||||
) -> None:
|
||||
"""
|
||||
Updated Document-Metadaten nach erfolgreichem xAI-Sync oder Preview-Generierung
|
||||
|
||||
Args:
|
||||
document_id: EspoCRM Document ID
|
||||
xai_file_id: xAI File ID (optional - setzt nur wenn vorhanden)
|
||||
collection_ids: Liste der xAI Collection IDs (optional)
|
||||
file_hash: MD5/SHA Hash des gesyncten Files
|
||||
preview_data: Vorschaubild (WebP) als bytes
|
||||
reset_file_status: Ob fileStatus/dateiStatus zurückgesetzt werden soll
|
||||
entity_type: Entity-Type (CDokumente oder Document)
|
||||
"""
|
||||
try:
|
||||
update_data = {}
|
||||
|
||||
# Nur xAI-Felder updaten wenn vorhanden
|
||||
if xai_file_id:
|
||||
# CDokumente verwendet xaiId, Document verwendet xaiFileId
|
||||
if entity_type == 'CDokumente':
|
||||
update_data['xaiId'] = xai_file_id
|
||||
else:
|
||||
update_data['xaiFileId'] = xai_file_id
|
||||
|
||||
if collection_ids is not None:
|
||||
update_data['xaiCollections'] = collection_ids
|
||||
|
||||
# fileStatus auf "unchanged" setzen wenn Dokument verarbeitet/clean ist
|
||||
if reset_file_status:
|
||||
if entity_type == 'CDokumente':
|
||||
update_data['fileStatus'] = 'unchanged'
|
||||
else:
|
||||
# Document Entity hat kein fileStatus, nur dateiStatus
|
||||
update_data['dateiStatus'] = 'unchanged'
|
||||
|
||||
# xaiSyncStatus auf "clean" setzen wenn xAI-Sync erfolgreich war
|
||||
if xai_file_id:
|
||||
update_data['xaiSyncStatus'] = 'clean'
|
||||
|
||||
# Hash speichern für zukünftige Change Detection
|
||||
if file_hash:
|
||||
update_data['xaiSyncedHash'] = file_hash
|
||||
|
||||
# Preview als Attachment hochladen (falls vorhanden)
|
||||
if preview_data:
|
||||
await self._upload_preview_to_espocrm(document_id, preview_data, entity_type)
|
||||
|
||||
# Nur updaten wenn es etwas zu updaten gibt
|
||||
if update_data:
|
||||
await self.espocrm.update_entity(entity_type, document_id, update_data)
|
||||
self._log(f"✅ Sync-Metadaten aktualisiert für {entity_type} {document_id}: {list(update_data.keys())}")
|
||||
|
||||
except Exception as e:
|
||||
self._log(f"❌ Fehler beim Update von Sync-Metadaten: {e}", level='error')
|
||||
raise
|
||||
|
||||
async def _upload_preview_to_espocrm(self, document_id: str, preview_data: bytes, entity_type: str = 'CDokumente') -> None:
|
||||
"""
|
||||
Lädt Preview-Image als Attachment zu EspoCRM hoch
|
||||
|
||||
Args:
|
||||
document_id: Document ID
|
||||
preview_data: WebP Preview als bytes
|
||||
entity_type: Entity-Type (CDokumente oder Document)
|
||||
"""
|
||||
try:
|
||||
self._log(f"📤 Uploading preview image to {entity_type} ({len(preview_data)} bytes)...")
|
||||
|
||||
# EspoCRM erwartet base64-encoded file im Format: data:mime/type;base64,xxxxx
|
||||
import base64
|
||||
import aiohttp
|
||||
|
||||
# Base64-encode preview data
|
||||
base64_data = base64.b64encode(preview_data).decode('ascii')
|
||||
file_data_uri = f"data:image/webp;base64,{base64_data}"
|
||||
|
||||
# Upload via JSON POST mit base64-encoded file field
|
||||
url = self.espocrm.api_base_url.rstrip('/') + '/Attachment'
|
||||
headers = {
|
||||
'X-Api-Key': self.espocrm.api_key,
|
||||
'Content-Type': 'application/json'
|
||||
}
|
||||
|
||||
payload = {
|
||||
'name': 'preview.webp',
|
||||
'type': 'image/webp',
|
||||
'role': 'Attachment',
|
||||
'field': 'preview',
|
||||
'relatedType': entity_type,
|
||||
'relatedId': document_id,
|
||||
'file': file_data_uri
|
||||
}
|
||||
|
||||
self._log(f"📤 Posting to {url} with base64-encoded file ({len(base64_data)} chars)")
|
||||
self._log(f" relatedType={entity_type}, relatedId={document_id}, field=preview")
|
||||
|
||||
timeout = aiohttp.ClientTimeout(total=30)
|
||||
async with aiohttp.ClientSession(timeout=timeout) as session:
|
||||
async with session.post(url, headers=headers, json=payload) as response:
|
||||
self._log(f"Upload response status: {response.status}")
|
||||
|
||||
if response.status >= 400:
|
||||
error_text = await response.text()
|
||||
self._log(f"❌ Upload failed: {error_text}", level='error')
|
||||
raise Exception(f"Upload error {response.status}: {error_text}")
|
||||
|
||||
result = await response.json()
|
||||
attachment_id = result.get('id')
|
||||
self._log(f"✅ Preview Attachment created: {attachment_id}")
|
||||
|
||||
# Update Entity mit previewId
|
||||
self._log(f"📝 Updating {entity_type} with previewId...")
|
||||
await self.espocrm.update_entity(entity_type, document_id, {
|
||||
'previewId': attachment_id,
|
||||
'previewName': 'preview.webp'
|
||||
})
|
||||
self._log(f"✅ {entity_type} previewId/previewName aktualisiert")
|
||||
|
||||
except Exception as e:
|
||||
self._log(f"❌ Fehler beim Preview-Upload: {e}", level='error')
|
||||
# Don't raise - Preview ist optional, Sync sollte trotzdem erfolgreich sein
|
||||
@@ -2,21 +2,20 @@
|
||||
import aiohttp
|
||||
import asyncio
|
||||
import logging
|
||||
import redis
|
||||
import os
|
||||
import time
|
||||
from typing import Optional, Dict, Any, List
|
||||
import os
|
||||
|
||||
logger = logging.getLogger(__name__)
|
||||
|
||||
|
||||
class EspoCRMError(Exception):
|
||||
"""Base exception for EspoCRM API errors"""
|
||||
pass
|
||||
|
||||
|
||||
class EspoCRMAuthError(EspoCRMError):
|
||||
"""Authentication error"""
|
||||
pass
|
||||
from services.exceptions import (
|
||||
EspoCRMAPIError,
|
||||
EspoCRMAuthError,
|
||||
EspoCRMTimeoutError,
|
||||
RetryableError,
|
||||
ValidationError
|
||||
)
|
||||
from services.redis_client import get_redis_client
|
||||
from services.config import ESPOCRM_CONFIG, API_CONFIG
|
||||
from services.logging_utils import get_service_logger
|
||||
|
||||
|
||||
class EspoCRMAPI:
|
||||
@@ -32,7 +31,6 @@ class EspoCRMAPI:
|
||||
- ESPOCRM_API_BASE_URL (e.g., https://crm.bitbylaw.com/api/v1)
|
||||
- ESPOCRM_API_KEY (Marvin API key)
|
||||
- ESPOCRM_API_TIMEOUT_SECONDS (optional, default: 30)
|
||||
- REDIS_HOST, REDIS_PORT, REDIS_DB_ADVOWARE_CACHE (for caching)
|
||||
"""
|
||||
|
||||
def __init__(self, context=None):
|
||||
@@ -43,47 +41,38 @@ class EspoCRMAPI:
|
||||
context: Motia FlowContext for logging (optional)
|
||||
"""
|
||||
self.context = context
|
||||
self._log("EspoCRMAPI initializing", level='debug')
|
||||
self.logger = get_service_logger('espocrm', context)
|
||||
self.logger.debug("EspoCRMAPI initializing")
|
||||
|
||||
# Load configuration from environment
|
||||
self.api_base_url = os.getenv('ESPOCRM_API_BASE_URL', 'https://crm.bitbylaw.com/api/v1')
|
||||
self.api_key = os.getenv('ESPOCRM_API_KEY', '')
|
||||
self.api_timeout_seconds = int(os.getenv('ESPOCRM_API_TIMEOUT_SECONDS', '30'))
|
||||
self.api_timeout_seconds = int(os.getenv('ESPOCRM_API_TIMEOUT_SECONDS', str(API_CONFIG.default_timeout_seconds)))
|
||||
|
||||
if not self.api_key:
|
||||
raise EspoCRMAuthError("ESPOCRM_API_KEY not configured in environment")
|
||||
|
||||
self._log(f"EspoCRM API initialized with base URL: {self.api_base_url}")
|
||||
self.logger.info(f"EspoCRM API initialized with base URL: {self.api_base_url}")
|
||||
|
||||
# Optional Redis for caching/rate limiting
|
||||
try:
|
||||
redis_host = os.getenv('REDIS_HOST', 'localhost')
|
||||
redis_port = int(os.getenv('REDIS_PORT', '6379'))
|
||||
redis_db = int(os.getenv('REDIS_DB_ADVOWARE_CACHE', '1'))
|
||||
redis_timeout = int(os.getenv('REDIS_TIMEOUT_SECONDS', '5'))
|
||||
self._session: Optional[aiohttp.ClientSession] = None
|
||||
self._entity_defs_cache: Dict[str, Dict[str, Any]] = {}
|
||||
self._entity_defs_cache_ttl_seconds = int(os.getenv('ESPOCRM_METADATA_TTL_SECONDS', '300'))
|
||||
|
||||
self.redis_client = redis.Redis(
|
||||
host=redis_host,
|
||||
port=redis_port,
|
||||
db=redis_db,
|
||||
socket_timeout=redis_timeout,
|
||||
socket_connect_timeout=redis_timeout,
|
||||
decode_responses=True
|
||||
)
|
||||
self.redis_client.ping()
|
||||
self._log("Connected to Redis for EspoCRM operations")
|
||||
except Exception as e:
|
||||
self._log(f"Could not connect to Redis: {e}. Continuing without caching.", level='warning')
|
||||
self.redis_client = None
|
||||
# Metadata cache (complete metadata loaded once)
|
||||
self._metadata_cache: Optional[Dict[str, Any]] = None
|
||||
self._metadata_cache_ts: float = 0
|
||||
|
||||
def _log(self, message: str, level: str = 'info'):
|
||||
"""Log message via context.logger if available, otherwise use module logger"""
|
||||
if self.context and hasattr(self.context, 'logger'):
|
||||
log_func = getattr(self.context.logger, level, self.context.logger.info)
|
||||
log_func(f"[EspoCRM] {message}")
|
||||
# Optional Redis for caching/rate limiting (centralized)
|
||||
self.redis_client = get_redis_client(strict=False)
|
||||
if self.redis_client:
|
||||
self.logger.info("Connected to Redis for EspoCRM operations")
|
||||
else:
|
||||
log_func = getattr(logger, level, logger.info)
|
||||
log_func(f"[EspoCRM] {message}")
|
||||
self.logger.warning("⚠️ Redis unavailable - caching disabled")
|
||||
|
||||
def _log(self, message: str, level: str = 'info') -> None:
|
||||
"""Delegate to IntegrationLogger with optional level"""
|
||||
log_func = getattr(self.logger, level, self.logger.info)
|
||||
log_func(message)
|
||||
|
||||
def _get_headers(self) -> Dict[str, str]:
|
||||
"""Generate request headers with API key"""
|
||||
@@ -93,6 +82,86 @@ class EspoCRMAPI:
|
||||
'Accept': 'application/json'
|
||||
}
|
||||
|
||||
async def _get_session(self) -> aiohttp.ClientSession:
|
||||
if self._session is None or self._session.closed:
|
||||
self._session = aiohttp.ClientSession()
|
||||
return self._session
|
||||
|
||||
async def close(self) -> None:
|
||||
if self._session and not self._session.closed:
|
||||
await self._session.close()
|
||||
|
||||
async def get_metadata(self) -> Dict[str, Any]:
|
||||
"""
|
||||
Get complete EspoCRM metadata (cached).
|
||||
|
||||
Loads once and caches for TTL duration.
|
||||
Much faster than individual entity def calls.
|
||||
|
||||
Returns:
|
||||
Complete metadata dict with entityDefs, clientDefs, etc.
|
||||
"""
|
||||
now = time.monotonic()
|
||||
|
||||
# Return cached if still valid
|
||||
if (self._metadata_cache is not None and
|
||||
(now - self._metadata_cache_ts) < self._entity_defs_cache_ttl_seconds):
|
||||
return self._metadata_cache
|
||||
|
||||
# Load fresh metadata
|
||||
try:
|
||||
self._log("📥 Loading complete EspoCRM metadata...", level='debug')
|
||||
metadata = await self.api_call("/Metadata", method='GET')
|
||||
|
||||
if not isinstance(metadata, dict):
|
||||
self._log("⚠️ Metadata response is not a dict, using empty", level='warn')
|
||||
metadata = {}
|
||||
|
||||
# Cache it
|
||||
self._metadata_cache = metadata
|
||||
self._metadata_cache_ts = now
|
||||
|
||||
entity_count = len(metadata.get('entityDefs', {}))
|
||||
self._log(f"✅ Metadata cached: {entity_count} entity definitions", level='debug')
|
||||
|
||||
return metadata
|
||||
|
||||
except Exception as e:
|
||||
self._log(f"❌ Failed to load metadata: {e}", level='error')
|
||||
# Return empty dict as fallback
|
||||
return {}
|
||||
|
||||
async def get_entity_def(self, entity_type: str) -> Dict[str, Any]:
|
||||
"""
|
||||
Get entity definition for a specific entity type (cached via metadata).
|
||||
|
||||
Uses complete metadata cache - much faster and correct API usage.
|
||||
|
||||
Args:
|
||||
entity_type: Entity type (e.g., 'Document', 'CDokumente', 'Account')
|
||||
|
||||
Returns:
|
||||
Entity definition dict with fields, links, etc.
|
||||
"""
|
||||
try:
|
||||
metadata = await self.get_metadata()
|
||||
entity_defs = metadata.get('entityDefs', {})
|
||||
|
||||
if not isinstance(entity_defs, dict):
|
||||
self._log(f"⚠️ entityDefs is not a dict for {entity_type}", level='warn')
|
||||
return {}
|
||||
|
||||
entity_def = entity_defs.get(entity_type, {})
|
||||
|
||||
if not entity_def:
|
||||
self._log(f"⚠️ No entity definition found for '{entity_type}'", level='debug')
|
||||
|
||||
return entity_def
|
||||
|
||||
except Exception as e:
|
||||
self._log(f"⚠️ Could not load entity def for {entity_type}: {e}", level='warn')
|
||||
return {}
|
||||
|
||||
async def api_call(
|
||||
self,
|
||||
endpoint: str,
|
||||
@@ -115,7 +184,9 @@ class EspoCRMAPI:
|
||||
Parsed JSON response or None
|
||||
|
||||
Raises:
|
||||
EspoCRMError: On API errors
|
||||
EspoCRMAuthError: Authentication failed
|
||||
EspoCRMTimeoutError: Request timed out
|
||||
EspoCRMAPIError: Other API errors
|
||||
"""
|
||||
# Ensure endpoint starts with /
|
||||
if not endpoint.startswith('/'):
|
||||
@@ -127,45 +198,62 @@ class EspoCRMAPI:
|
||||
total=timeout_seconds or self.api_timeout_seconds
|
||||
)
|
||||
|
||||
self._log(f"API call: {method} {url}", level='debug')
|
||||
if params:
|
||||
self._log(f"Params: {params}", level='debug')
|
||||
|
||||
async with aiohttp.ClientSession(timeout=effective_timeout) as session:
|
||||
try:
|
||||
session = await self._get_session()
|
||||
try:
|
||||
with self.logger.api_call(endpoint, method):
|
||||
async with session.request(
|
||||
method,
|
||||
url,
|
||||
headers=headers,
|
||||
params=params,
|
||||
json=json_data
|
||||
json=json_data,
|
||||
timeout=effective_timeout
|
||||
) as response:
|
||||
# Log response status
|
||||
self._log(f"Response status: {response.status}", level='debug')
|
||||
|
||||
# Handle errors
|
||||
if response.status == 401:
|
||||
raise EspoCRMAuthError("Authentication failed - check API key")
|
||||
raise EspoCRMAuthError(
|
||||
"Authentication failed - check API key",
|
||||
status_code=401
|
||||
)
|
||||
elif response.status == 403:
|
||||
raise EspoCRMError("Access forbidden")
|
||||
raise EspoCRMAPIError(
|
||||
"Access forbidden",
|
||||
status_code=403
|
||||
)
|
||||
elif response.status == 404:
|
||||
raise EspoCRMError(f"Resource not found: {endpoint}")
|
||||
raise EspoCRMAPIError(
|
||||
f"Resource not found: {endpoint}",
|
||||
status_code=404
|
||||
)
|
||||
elif response.status >= 500:
|
||||
error_text = await response.text()
|
||||
raise RetryableError(
|
||||
f"Server error {response.status}: {error_text}"
|
||||
)
|
||||
elif response.status >= 400:
|
||||
error_text = await response.text()
|
||||
raise EspoCRMError(f"API error {response.status}: {error_text}")
|
||||
raise EspoCRMAPIError(
|
||||
f"API error {response.status}: {error_text}",
|
||||
status_code=response.status,
|
||||
response_body=error_text
|
||||
)
|
||||
|
||||
# Parse response
|
||||
if response.content_type == 'application/json':
|
||||
result = await response.json()
|
||||
self._log(f"Response received", level='debug')
|
||||
return result
|
||||
else:
|
||||
# For DELETE or other non-JSON responses
|
||||
return None
|
||||
|
||||
except aiohttp.ClientError as e:
|
||||
self._log(f"API call failed: {e}", level='error')
|
||||
raise EspoCRMError(f"Request failed: {e}") from e
|
||||
except asyncio.TimeoutError:
|
||||
raise EspoCRMTimeoutError(
|
||||
f"Request timed out after {effective_timeout.total}s",
|
||||
status_code=408
|
||||
)
|
||||
except aiohttp.ClientError as e:
|
||||
self.logger.error(f"API call failed: {e}")
|
||||
raise EspoCRMAPIError(f"Request failed: {str(e)}")
|
||||
|
||||
async def get_entity(self, entity_type: str, entity_id: str) -> Dict[str, Any]:
|
||||
"""
|
||||
@@ -221,6 +309,36 @@ class EspoCRMAPI:
|
||||
self._log(f"Listing {entity_type} entities")
|
||||
return await self.api_call(f"/{entity_type}", method='GET', params=params)
|
||||
|
||||
async def list_related(
|
||||
self,
|
||||
entity_type: str,
|
||||
entity_id: str,
|
||||
link: str,
|
||||
where: Optional[List[Dict]] = None,
|
||||
select: Optional[str] = None,
|
||||
order_by: Optional[str] = None,
|
||||
order: Optional[str] = None,
|
||||
offset: int = 0,
|
||||
max_size: int = 50
|
||||
) -> Dict[str, Any]:
|
||||
params = {
|
||||
'offset': offset,
|
||||
'maxSize': max_size
|
||||
}
|
||||
|
||||
if where:
|
||||
import json
|
||||
params['where'] = where if isinstance(where, str) else json.dumps(where)
|
||||
if select:
|
||||
params['select'] = select
|
||||
if order_by:
|
||||
params['orderBy'] = order_by
|
||||
if order:
|
||||
params['order'] = order
|
||||
|
||||
self._log(f"Listing related {entity_type}/{entity_id}/{link}")
|
||||
return await self.api_call(f"/{entity_type}/{entity_id}/{link}", method='GET', params=params)
|
||||
|
||||
async def create_entity(
|
||||
self,
|
||||
entity_type: str,
|
||||
@@ -298,3 +416,316 @@ class EspoCRMAPI:
|
||||
|
||||
result = await self.list_entities(entity_type, where=where)
|
||||
return result.get('list', [])
|
||||
|
||||
async def upload_attachment(
|
||||
self,
|
||||
file_content: bytes,
|
||||
filename: str,
|
||||
parent_type: str,
|
||||
parent_id: str,
|
||||
field: str,
|
||||
mime_type: str = 'application/octet-stream',
|
||||
role: str = 'Attachment'
|
||||
) -> Dict[str, Any]:
|
||||
"""
|
||||
Upload an attachment to EspoCRM.
|
||||
|
||||
Args:
|
||||
file_content: File content as bytes
|
||||
filename: Name of the file
|
||||
parent_type: Parent entity type (e.g., 'Document')
|
||||
parent_id: Parent entity ID
|
||||
field: Field name for the attachment (e.g., 'preview')
|
||||
mime_type: MIME type of the file
|
||||
role: Attachment role (default: 'Attachment')
|
||||
|
||||
Returns:
|
||||
Attachment entity data
|
||||
"""
|
||||
self._log(f"Uploading attachment: {filename} ({len(file_content)} bytes) to {parent_type}/{parent_id}/{field}")
|
||||
|
||||
url = self.api_base_url.rstrip('/') + '/Attachment'
|
||||
headers = {
|
||||
'X-Api-Key': self.api_key,
|
||||
# Content-Type wird automatisch von aiohttp gesetzt für FormData
|
||||
}
|
||||
|
||||
# Erstelle FormData
|
||||
form_data = aiohttp.FormData()
|
||||
form_data.add_field('file', file_content, filename=filename, content_type=mime_type)
|
||||
form_data.add_field('parentType', parent_type)
|
||||
form_data.add_field('parentId', parent_id)
|
||||
form_data.add_field('field', field)
|
||||
form_data.add_field('role', role)
|
||||
form_data.add_field('name', filename)
|
||||
|
||||
self._log(f"Upload params: parentType={parent_type}, parentId={parent_id}, field={field}, role={role}")
|
||||
|
||||
effective_timeout = aiohttp.ClientTimeout(total=self.api_timeout_seconds)
|
||||
|
||||
session = await self._get_session()
|
||||
try:
|
||||
async with session.post(url, headers=headers, data=form_data, timeout=effective_timeout) as response:
|
||||
self._log(f"Upload response status: {response.status}")
|
||||
|
||||
if response.status == 401:
|
||||
raise EspoCRMAuthError("Authentication failed - check API key")
|
||||
elif response.status == 403:
|
||||
raise EspoCRMError("Access forbidden")
|
||||
elif response.status == 404:
|
||||
raise EspoCRMError(f"Attachment endpoint not found")
|
||||
elif response.status >= 400:
|
||||
error_text = await response.text()
|
||||
self._log(f"❌ Upload failed with {response.status}. Response: {error_text}", level='error')
|
||||
raise EspoCRMError(f"Upload error {response.status}: {error_text}")
|
||||
|
||||
# Parse response
|
||||
if response.content_type == 'application/json':
|
||||
result = await response.json()
|
||||
attachment_id = result.get('id')
|
||||
self._log(f"✅ Attachment uploaded successfully: {attachment_id}")
|
||||
return result
|
||||
else:
|
||||
response_text = await response.text()
|
||||
self._log(f"⚠️ Non-JSON response: {response_text[:200]}", level='warn')
|
||||
return {'success': True, 'response': response_text}
|
||||
|
||||
except aiohttp.ClientError as e:
|
||||
self._log(f"Upload failed: {e}", level='error')
|
||||
raise EspoCRMError(f"Upload request failed: {e}") from e
|
||||
|
||||
async def download_attachment(self, attachment_id: str) -> bytes:
|
||||
"""
|
||||
Download an attachment from EspoCRM.
|
||||
|
||||
Args:
|
||||
attachment_id: Attachment ID
|
||||
|
||||
Returns:
|
||||
File content as bytes
|
||||
"""
|
||||
self._log(f"Downloading attachment: {attachment_id}")
|
||||
|
||||
url = self.api_base_url.rstrip('/') + f'/Attachment/file/{attachment_id}'
|
||||
headers = {
|
||||
'X-Api-Key': self.api_key,
|
||||
}
|
||||
|
||||
effective_timeout = aiohttp.ClientTimeout(total=self.api_timeout_seconds)
|
||||
|
||||
session = await self._get_session()
|
||||
try:
|
||||
async with session.get(url, headers=headers, timeout=effective_timeout) as response:
|
||||
if response.status == 401:
|
||||
raise EspoCRMAuthError("Authentication failed - check API key")
|
||||
elif response.status == 403:
|
||||
raise EspoCRMError("Access forbidden")
|
||||
elif response.status == 404:
|
||||
raise EspoCRMError(f"Attachment not found: {attachment_id}")
|
||||
elif response.status >= 400:
|
||||
error_text = await response.text()
|
||||
raise EspoCRMError(f"Download error {response.status}: {error_text}")
|
||||
|
||||
content = await response.read()
|
||||
self._log(f"✅ Downloaded {len(content)} bytes")
|
||||
return content
|
||||
|
||||
except aiohttp.ClientError as e:
|
||||
self._log(f"Download failed: {e}", level='error')
|
||||
raise EspoCRMError(f"Download request failed: {e}") from e
|
||||
|
||||
# ========== Junction Table Operations ==========
|
||||
|
||||
async def get_junction_entries(
|
||||
self,
|
||||
junction_entity: str,
|
||||
filter_field: str,
|
||||
filter_value: str,
|
||||
max_size: int = 1000
|
||||
) -> List[Dict[str, Any]]:
|
||||
"""
|
||||
Load junction table entries with filtering.
|
||||
|
||||
Args:
|
||||
junction_entity: Junction entity name (e.g., 'CAIKnowledgeCDokumente')
|
||||
filter_field: Field to filter on (e.g., 'cAIKnowledgeId')
|
||||
filter_value: Value to match
|
||||
max_size: Maximum entries to return
|
||||
|
||||
Returns:
|
||||
List of junction records with ALL additionalColumns
|
||||
|
||||
Example:
|
||||
entries = await espocrm.get_junction_entries(
|
||||
'CAIKnowledgeCDokumente',
|
||||
'cAIKnowledgeId',
|
||||
'kb-123'
|
||||
)
|
||||
"""
|
||||
self._log(f"Loading junction entries: {junction_entity} where {filter_field}={filter_value}")
|
||||
|
||||
result = await self.list_entities(
|
||||
junction_entity,
|
||||
where=[{
|
||||
'type': 'equals',
|
||||
'attribute': filter_field,
|
||||
'value': filter_value
|
||||
}],
|
||||
max_size=max_size
|
||||
)
|
||||
|
||||
entries = result.get('list', [])
|
||||
self._log(f"✅ Loaded {len(entries)} junction entries")
|
||||
return entries
|
||||
|
||||
async def update_junction_entry(
|
||||
self,
|
||||
junction_entity: str,
|
||||
junction_id: str,
|
||||
fields: Dict[str, Any]
|
||||
) -> None:
|
||||
"""
|
||||
Update junction table entry.
|
||||
|
||||
Args:
|
||||
junction_entity: Junction entity name
|
||||
junction_id: Junction entry ID
|
||||
fields: Fields to update
|
||||
|
||||
Example:
|
||||
await espocrm.update_junction_entry(
|
||||
'CAIKnowledgeCDokumente',
|
||||
'jct-123',
|
||||
{'syncstatus': 'synced', 'lastSync': '2026-03-11T20:00:00Z'}
|
||||
)
|
||||
"""
|
||||
await self.update_entity(junction_entity, junction_id, fields)
|
||||
|
||||
async def get_knowledge_documents_with_junction(
|
||||
self,
|
||||
knowledge_id: str
|
||||
) -> List[Dict[str, Any]]:
|
||||
"""
|
||||
Get all documents linked to a CAIKnowledge entry with junction data.
|
||||
|
||||
Uses custom EspoCRM endpoint: GET /JunctionData/CAIKnowledge/{knowledge_id}/dokumentes
|
||||
|
||||
Returns enriched list with:
|
||||
- junctionId: Junction table ID
|
||||
- cAIKnowledgeId, cDokumenteId: Junction keys
|
||||
- aiDocumentId: XAI document ID from junction
|
||||
- syncstatus: Sync status from junction (new, synced, failed, unclean)
|
||||
- lastSync: Last sync timestamp from junction
|
||||
- documentId, documentName: Document info
|
||||
- blake3hash: Blake3 hash from document entity
|
||||
- documentCreatedAt, documentModifiedAt: Document timestamps
|
||||
|
||||
This consolidates multiple API calls into one efficient query.
|
||||
|
||||
Args:
|
||||
knowledge_id: CAIKnowledge entity ID
|
||||
|
||||
Returns:
|
||||
List of document dicts with junction data
|
||||
|
||||
Example:
|
||||
docs = await espocrm.get_knowledge_documents_with_junction('69b1b03582bb6e2da')
|
||||
for doc in docs:
|
||||
print(f"{doc['documentName']}: {doc['syncstatus']}")
|
||||
"""
|
||||
# JunctionData uses API Gateway URL, not direct EspoCRM
|
||||
# Use gateway URL from env or construct from ESPOCRM_API_BASE_URL
|
||||
gateway_url = os.getenv('ESPOCRM_GATEWAY_URL', 'https://api.bitbylaw.com/vmh/crm')
|
||||
url = f"{gateway_url}/JunctionData/CAIKnowledge/{knowledge_id}/dokumentes"
|
||||
|
||||
self._log(f"GET {url}")
|
||||
|
||||
try:
|
||||
session = await self._get_session()
|
||||
timeout = aiohttp.ClientTimeout(total=self.api_timeout_seconds)
|
||||
|
||||
async with session.get(url, headers=self._get_headers(), timeout=timeout) as response:
|
||||
self._log(f"Response status: {response.status}")
|
||||
|
||||
if response.status == 404:
|
||||
# Knowledge base not found or no documents linked
|
||||
return []
|
||||
|
||||
if response.status >= 400:
|
||||
error_text = await response.text()
|
||||
raise EspoCRMAPIError(f"JunctionData GET failed: {response.status} - {error_text}")
|
||||
|
||||
result = await response.json()
|
||||
documents = result.get('list', [])
|
||||
|
||||
self._log(f"✅ Loaded {len(documents)} document(s) with junction data")
|
||||
return documents
|
||||
|
||||
except asyncio.TimeoutError:
|
||||
raise EspoCRMTimeoutError(f"Timeout getting junction data for knowledge {knowledge_id}")
|
||||
except aiohttp.ClientError as e:
|
||||
raise EspoCRMAPIError(f"Network error getting junction data: {e}")
|
||||
|
||||
async def update_knowledge_document_junction(
|
||||
self,
|
||||
knowledge_id: str,
|
||||
document_id: str,
|
||||
fields: Dict[str, Any],
|
||||
update_last_sync: bool = True
|
||||
) -> Dict[str, Any]:
|
||||
"""
|
||||
Update junction columns for a specific document link.
|
||||
|
||||
Uses custom EspoCRM endpoint:
|
||||
PUT /JunctionData/CAIKnowledge/{knowledge_id}/dokumentes/{document_id}
|
||||
|
||||
Args:
|
||||
knowledge_id: CAIKnowledge entity ID
|
||||
document_id: CDokumente entity ID
|
||||
fields: Junction fields to update (aiDocumentId, syncstatus, etc.)
|
||||
update_last_sync: Whether to update lastSync timestamp (default: True)
|
||||
|
||||
Returns:
|
||||
Updated junction data
|
||||
|
||||
Example:
|
||||
await espocrm.update_knowledge_document_junction(
|
||||
'69b1b03582bb6e2da',
|
||||
'69a68b556a39771bf',
|
||||
{
|
||||
'aiDocumentId': 'xai-file-abc123',
|
||||
'syncstatus': 'synced'
|
||||
},
|
||||
update_last_sync=True
|
||||
)
|
||||
"""
|
||||
# JunctionData uses API Gateway URL, not direct EspoCRM
|
||||
gateway_url = os.getenv('ESPOCRM_GATEWAY_URL', 'https://api.bitbylaw.com/vmh/crm')
|
||||
url = f"{gateway_url}/JunctionData/CAIKnowledge/{knowledge_id}/dokumentes/{document_id}"
|
||||
|
||||
payload = {**fields}
|
||||
if update_last_sync:
|
||||
payload['updateLastSync'] = True
|
||||
|
||||
self._log(f"PUT {url}")
|
||||
self._log(f" Payload: {payload}")
|
||||
|
||||
try:
|
||||
session = await self._get_session()
|
||||
timeout = aiohttp.ClientTimeout(total=self.api_timeout_seconds)
|
||||
|
||||
async with session.put(url, headers=self._get_headers(), json=payload, timeout=timeout) as response:
|
||||
self._log(f"Response status: {response.status}")
|
||||
|
||||
if response.status >= 400:
|
||||
error_text = await response.text()
|
||||
raise EspoCRMAPIError(f"JunctionData PUT failed: {response.status} - {error_text}")
|
||||
|
||||
result = await response.json()
|
||||
self._log(f"✅ Junction updated: junctionId={result.get('junctionId')}")
|
||||
return result
|
||||
|
||||
except asyncio.TimeoutError:
|
||||
raise EspoCRMTimeoutError(f"Timeout updating junction data")
|
||||
except aiohttp.ClientError as e:
|
||||
raise EspoCRMAPIError(f"Network error updating junction data: {e}")
|
||||
|
||||
@@ -8,7 +8,15 @@ from typing import Dict, Any, Optional, List
|
||||
from datetime import datetime
|
||||
import logging
|
||||
|
||||
logger = logging.getLogger(__name__)
|
||||
from services.models import (
|
||||
AdvowareBeteiligteCreate,
|
||||
AdvowareBeteiligteUpdate,
|
||||
EspoCRMBeteiligteCreate,
|
||||
validate_beteiligte_advoware,
|
||||
validate_beteiligte_espocrm
|
||||
)
|
||||
from services.exceptions import ValidationError
|
||||
from services.config import FEATURE_FLAGS
|
||||
|
||||
|
||||
class BeteiligteMapper:
|
||||
@@ -27,6 +35,9 @@ class BeteiligteMapper:
|
||||
|
||||
Returns:
|
||||
Dict mit Stammdaten für Advoware API (POST/PUT /api/v1/advonet/Beteiligte)
|
||||
|
||||
Raises:
|
||||
ValidationError: Bei Validierungsfehlern (wenn strict_validation aktiviert)
|
||||
"""
|
||||
logger.debug(f"Mapping EspoCRM → Advoware STAMMDATEN: {espo_entity.get('id')}")
|
||||
|
||||
@@ -78,6 +89,14 @@ class BeteiligteMapper:
|
||||
|
||||
logger.debug(f"Mapped to Advoware STAMMDATEN: name={advo_data.get('name')}, vorname={advo_data.get('vorname')}, rechtsform={advo_data.get('rechtsform')}")
|
||||
|
||||
# Optional: Validiere mit Pydantic wenn aktiviert
|
||||
if FEATURE_FLAGS.strict_validation:
|
||||
try:
|
||||
validate_beteiligte_advoware(advo_data)
|
||||
except ValidationError as e:
|
||||
logger.warning(f"Validation warning: {e}")
|
||||
# Continue anyway - validation ist optional
|
||||
|
||||
return advo_data
|
||||
|
||||
@staticmethod
|
||||
|
||||
217
services/exceptions.py
Normal file
217
services/exceptions.py
Normal file
@@ -0,0 +1,217 @@
|
||||
"""
|
||||
Custom Exception Classes für BitByLaw Integration
|
||||
|
||||
Hierarchie:
|
||||
- IntegrationError (Base)
|
||||
- APIError
|
||||
- AdvowareAPIError
|
||||
- AdvowareAuthError
|
||||
- AdvowareTimeoutError
|
||||
- EspoCRMAPIError
|
||||
- EspoCRMAuthError
|
||||
- EspoCRMTimeoutError
|
||||
- SyncError
|
||||
- LockAcquisitionError
|
||||
- ValidationError
|
||||
- ConflictError
|
||||
- RetryableError
|
||||
- NonRetryableError
|
||||
"""
|
||||
|
||||
from typing import Optional, Dict, Any
|
||||
|
||||
|
||||
class IntegrationError(Exception):
|
||||
"""Base exception for all integration errors"""
|
||||
|
||||
def __init__(self, message: str, details: Optional[Dict[str, Any]] = None):
|
||||
super().__init__(message)
|
||||
self.message = message
|
||||
self.details = details or {}
|
||||
|
||||
|
||||
# ========== API Errors ==========
|
||||
|
||||
class APIError(IntegrationError):
|
||||
"""Base class for all API-related errors"""
|
||||
|
||||
def __init__(
|
||||
self,
|
||||
message: str,
|
||||
status_code: Optional[int] = None,
|
||||
response_body: Optional[str] = None,
|
||||
details: Optional[Dict[str, Any]] = None
|
||||
):
|
||||
super().__init__(message, details)
|
||||
self.status_code = status_code
|
||||
self.response_body = response_body
|
||||
|
||||
|
||||
class AdvowareAPIError(APIError):
|
||||
"""Advoware API error"""
|
||||
pass
|
||||
|
||||
|
||||
class AdvowareAuthError(AdvowareAPIError):
|
||||
"""Advoware authentication error"""
|
||||
pass
|
||||
|
||||
|
||||
class AdvowareTimeoutError(AdvowareAPIError):
|
||||
"""Advoware API timeout"""
|
||||
pass
|
||||
|
||||
|
||||
class EspoCRMAPIError(APIError):
|
||||
"""EspoCRM API error"""
|
||||
pass
|
||||
|
||||
|
||||
class EspoCRMAuthError(EspoCRMAPIError):
|
||||
"""EspoCRM authentication error"""
|
||||
pass
|
||||
|
||||
|
||||
class EspoCRMTimeoutError(EspoCRMAPIError):
|
||||
"""EspoCRM API timeout"""
|
||||
pass
|
||||
|
||||
|
||||
# ========== Sync Errors ==========
|
||||
|
||||
class SyncError(IntegrationError):
|
||||
"""Base class for synchronization errors"""
|
||||
pass
|
||||
|
||||
|
||||
class LockAcquisitionError(SyncError):
|
||||
"""Failed to acquire distributed lock"""
|
||||
|
||||
def __init__(self, entity_id: str, lock_key: str, message: Optional[str] = None):
|
||||
super().__init__(
|
||||
message or f"Could not acquire lock for entity {entity_id}",
|
||||
details={"entity_id": entity_id, "lock_key": lock_key}
|
||||
)
|
||||
self.entity_id = entity_id
|
||||
self.lock_key = lock_key
|
||||
|
||||
|
||||
class ValidationError(SyncError):
|
||||
"""Data validation error"""
|
||||
|
||||
def __init__(self, message: str, field: Optional[str] = None, value: Any = None):
|
||||
super().__init__(
|
||||
message,
|
||||
details={"field": field, "value": value}
|
||||
)
|
||||
self.field = field
|
||||
self.value = value
|
||||
|
||||
|
||||
class ConflictError(SyncError):
|
||||
"""Data conflict during synchronization"""
|
||||
|
||||
def __init__(
|
||||
self,
|
||||
message: str,
|
||||
entity_id: str,
|
||||
source_system: Optional[str] = None,
|
||||
target_system: Optional[str] = None
|
||||
):
|
||||
super().__init__(
|
||||
message,
|
||||
details={
|
||||
"entity_id": entity_id,
|
||||
"source_system": source_system,
|
||||
"target_system": target_system
|
||||
}
|
||||
)
|
||||
self.entity_id = entity_id
|
||||
|
||||
|
||||
# ========== Retry Classification ==========
|
||||
|
||||
class RetryableError(IntegrationError):
|
||||
"""Error that should trigger retry logic"""
|
||||
|
||||
def __init__(
|
||||
self,
|
||||
message: str,
|
||||
retry_after_seconds: Optional[int] = None,
|
||||
details: Optional[Dict[str, Any]] = None
|
||||
):
|
||||
super().__init__(message, details)
|
||||
self.retry_after_seconds = retry_after_seconds
|
||||
|
||||
|
||||
class NonRetryableError(IntegrationError):
|
||||
"""Error that should NOT trigger retry (e.g., validation errors)"""
|
||||
pass
|
||||
|
||||
|
||||
# ========== Redis Errors ==========
|
||||
|
||||
class RedisError(IntegrationError):
|
||||
"""Redis connection or operation error"""
|
||||
|
||||
def __init__(self, message: str, operation: Optional[str] = None):
|
||||
super().__init__(message, details={"operation": operation})
|
||||
self.operation = operation
|
||||
|
||||
|
||||
class RedisConnectionError(RedisError):
|
||||
"""Redis connection failed"""
|
||||
pass
|
||||
|
||||
|
||||
# ========== Helper Functions ==========
|
||||
|
||||
def is_retryable(error: Exception) -> bool:
|
||||
"""
|
||||
Determine if an error should trigger retry logic.
|
||||
|
||||
Args:
|
||||
error: Exception to check
|
||||
|
||||
Returns:
|
||||
True if error is retryable
|
||||
"""
|
||||
if isinstance(error, NonRetryableError):
|
||||
return False
|
||||
|
||||
if isinstance(error, RetryableError):
|
||||
return True
|
||||
|
||||
if isinstance(error, (AdvowareTimeoutError, EspoCRMTimeoutError)):
|
||||
return True
|
||||
|
||||
if isinstance(error, ValidationError):
|
||||
return False
|
||||
|
||||
# Default: assume retryable for API errors
|
||||
if isinstance(error, APIError):
|
||||
return True
|
||||
|
||||
return False
|
||||
|
||||
|
||||
def get_retry_delay(error: Exception, attempt: int) -> int:
|
||||
"""
|
||||
Calculate retry delay based on error type and attempt number.
|
||||
|
||||
Args:
|
||||
error: The error that occurred
|
||||
attempt: Current retry attempt (0-indexed)
|
||||
|
||||
Returns:
|
||||
Delay in seconds
|
||||
"""
|
||||
if isinstance(error, RetryableError) and error.retry_after_seconds:
|
||||
return error.retry_after_seconds
|
||||
|
||||
# Exponential backoff: [1, 5, 15, 60, 240] minutes
|
||||
backoff_minutes = [1, 5, 15, 60, 240]
|
||||
if attempt < len(backoff_minutes):
|
||||
return backoff_minutes[attempt] * 60
|
||||
|
||||
return backoff_minutes[-1] * 60
|
||||
@@ -24,8 +24,6 @@ from services.kommunikation_mapper import (
|
||||
from services.advoware_service import AdvowareService
|
||||
from services.espocrm import EspoCRMAPI
|
||||
|
||||
logger = logging.getLogger(__name__)
|
||||
|
||||
|
||||
class KommunikationSyncManager:
|
||||
"""Manager für Kommunikation-Synchronisation"""
|
||||
|
||||
218
services/langchain_xai_service.py
Normal file
218
services/langchain_xai_service.py
Normal file
@@ -0,0 +1,218 @@
|
||||
"""LangChain xAI Integration Service
|
||||
|
||||
Service für LangChain ChatXAI Integration mit File Search Binding.
|
||||
Analog zu xai_service.py für xAI Files API.
|
||||
"""
|
||||
import os
|
||||
from typing import Dict, List, Any, Optional, AsyncIterator
|
||||
from services.logging_utils import get_service_logger
|
||||
|
||||
|
||||
class LangChainXAIService:
|
||||
"""
|
||||
Wrapper für LangChain ChatXAI mit Motia-Integration.
|
||||
|
||||
Benötigte Umgebungsvariablen:
|
||||
- XAI_API_KEY: API Key für xAI (für ChatXAI model)
|
||||
|
||||
Usage:
|
||||
service = LangChainXAIService(ctx)
|
||||
model = service.get_chat_model(model="grok-4-1-fast-reasoning")
|
||||
model_with_tools = service.bind_file_search(model, collection_id)
|
||||
result = await service.invoke_chat(model_with_tools, messages)
|
||||
"""
|
||||
|
||||
def __init__(self, ctx=None):
|
||||
"""
|
||||
Initialize LangChain xAI Service.
|
||||
|
||||
Args:
|
||||
ctx: Optional Motia context for logging
|
||||
|
||||
Raises:
|
||||
ValueError: If XAI_API_KEY not configured
|
||||
"""
|
||||
self.api_key = os.getenv('XAI_API_KEY', '')
|
||||
self.ctx = ctx
|
||||
self.logger = get_service_logger('langchain_xai', ctx)
|
||||
|
||||
if not self.api_key:
|
||||
raise ValueError("XAI_API_KEY not configured in environment")
|
||||
|
||||
def _log(self, msg: str, level: str = 'info') -> None:
|
||||
"""Delegate logging to service logger"""
|
||||
log_func = getattr(self.logger, level, self.logger.info)
|
||||
log_func(msg)
|
||||
|
||||
def get_chat_model(
|
||||
self,
|
||||
model: str = "grok-4-1-fast-reasoning",
|
||||
temperature: float = 0.7,
|
||||
max_tokens: Optional[int] = None
|
||||
):
|
||||
"""
|
||||
Initialisiert ChatXAI Model.
|
||||
|
||||
Args:
|
||||
model: Model name (default: grok-4-1-fast-reasoning)
|
||||
temperature: Sampling temperature 0.0-1.0
|
||||
max_tokens: Optional max tokens for response
|
||||
|
||||
Returns:
|
||||
ChatXAI model instance
|
||||
|
||||
Raises:
|
||||
ImportError: If langchain_xai not installed
|
||||
"""
|
||||
try:
|
||||
from langchain_xai import ChatXAI
|
||||
except ImportError:
|
||||
raise ImportError(
|
||||
"langchain_xai not installed. "
|
||||
"Run: pip install langchain-xai>=0.2.0"
|
||||
)
|
||||
|
||||
self._log(f"🤖 Initializing ChatXAI: model={model}, temp={temperature}")
|
||||
|
||||
kwargs = {
|
||||
"model": model,
|
||||
"api_key": self.api_key,
|
||||
"temperature": temperature
|
||||
}
|
||||
if max_tokens:
|
||||
kwargs["max_tokens"] = max_tokens
|
||||
|
||||
return ChatXAI(**kwargs)
|
||||
|
||||
def bind_tools(
|
||||
self,
|
||||
model,
|
||||
collection_id: Optional[str] = None,
|
||||
enable_web_search: bool = False,
|
||||
web_search_config: Optional[Dict[str, Any]] = None,
|
||||
max_num_results: int = 10
|
||||
):
|
||||
"""
|
||||
Bindet xAI Tools (file_search und/oder web_search) an Model.
|
||||
|
||||
Args:
|
||||
model: ChatXAI model instance
|
||||
collection_id: Optional xAI Collection ID für file_search
|
||||
enable_web_search: Enable web search tool (default: False)
|
||||
web_search_config: Optional web search configuration:
|
||||
{
|
||||
'allowed_domains': ['example.com'], # Max 5 domains
|
||||
'excluded_domains': ['spam.com'], # Max 5 domains
|
||||
'enable_image_understanding': True
|
||||
}
|
||||
max_num_results: Max results from file search (default: 10)
|
||||
|
||||
Returns:
|
||||
Model with requested tools bound (file_search and/or web_search)
|
||||
"""
|
||||
tools = []
|
||||
|
||||
# Add file_search tool if collection_id provided
|
||||
if collection_id:
|
||||
self._log(f"🔍 Binding file_search: collection={collection_id}")
|
||||
tools.append({
|
||||
"type": "file_search",
|
||||
"vector_store_ids": [collection_id],
|
||||
"max_num_results": max_num_results
|
||||
})
|
||||
|
||||
# Add web_search tool if enabled
|
||||
if enable_web_search:
|
||||
self._log("🌐 Binding web_search")
|
||||
web_search_tool = {"type": "web_search"}
|
||||
|
||||
# Add optional web search filters
|
||||
if web_search_config:
|
||||
if 'allowed_domains' in web_search_config:
|
||||
domains = web_search_config['allowed_domains'][:5] # Max 5
|
||||
web_search_tool['filters'] = {'allowed_domains': domains}
|
||||
self._log(f" Allowed domains: {domains}")
|
||||
elif 'excluded_domains' in web_search_config:
|
||||
domains = web_search_config['excluded_domains'][:5] # Max 5
|
||||
web_search_tool['filters'] = {'excluded_domains': domains}
|
||||
self._log(f" Excluded domains: {domains}")
|
||||
|
||||
if web_search_config.get('enable_image_understanding'):
|
||||
web_search_tool['enable_image_understanding'] = True
|
||||
self._log(" Image understanding: enabled")
|
||||
|
||||
tools.append(web_search_tool)
|
||||
|
||||
if not tools:
|
||||
self._log("⚠️ No tools to bind (no collection_id and web_search disabled)", level='warn')
|
||||
return model
|
||||
|
||||
self._log(f"🔧 Binding {len(tools)} tool(s) to model")
|
||||
return model.bind_tools(tools)
|
||||
|
||||
def bind_file_search(
|
||||
self,
|
||||
model,
|
||||
collection_id: str,
|
||||
max_num_results: int = 10
|
||||
):
|
||||
"""
|
||||
Legacy method: Bindet nur file_search Tool an Model.
|
||||
|
||||
Use bind_tools() for more flexibility.
|
||||
"""
|
||||
return self.bind_tools(
|
||||
model=model,
|
||||
collection_id=collection_id,
|
||||
max_num_results=max_num_results
|
||||
)
|
||||
|
||||
async def invoke_chat(
|
||||
self,
|
||||
model,
|
||||
messages: List[Dict[str, Any]]
|
||||
) -> Any:
|
||||
"""
|
||||
Non-streaming Chat Completion.
|
||||
|
||||
Args:
|
||||
model: ChatXAI model (with or without tools)
|
||||
messages: List of message dicts [{"role": "user", "content": "..."}]
|
||||
|
||||
Returns:
|
||||
LangChain AIMessage with response
|
||||
|
||||
Raises:
|
||||
Exception: If API call fails
|
||||
"""
|
||||
self._log(f"💬 Invoking chat: {len(messages)} messages", level='debug')
|
||||
|
||||
result = await model.ainvoke(messages)
|
||||
|
||||
self._log(f"✅ Response received: {len(result.content)} chars", level='debug')
|
||||
return result
|
||||
|
||||
async def astream_chat(
|
||||
self,
|
||||
model,
|
||||
messages: List[Dict[str, Any]]
|
||||
) -> AsyncIterator:
|
||||
"""
|
||||
Streaming Chat Completion.
|
||||
|
||||
Args:
|
||||
model: ChatXAI model (with or without tools)
|
||||
messages: List of message dicts
|
||||
|
||||
Yields:
|
||||
Chunks from streaming response
|
||||
|
||||
Example:
|
||||
async for chunk in service.astream_chat(model, messages):
|
||||
delta = chunk.content if hasattr(chunk, "content") else ""
|
||||
# Process delta...
|
||||
"""
|
||||
self._log(f"💬 Streaming chat: {len(messages)} messages", level='debug')
|
||||
|
||||
async for chunk in model.astream(messages):
|
||||
yield chunk
|
||||
416
services/logging_utils.py
Normal file
416
services/logging_utils.py
Normal file
@@ -0,0 +1,416 @@
|
||||
"""
|
||||
Konsistenter Logging Wrapper für BitByLaw Integration
|
||||
|
||||
Vereinheitlicht Logging über:
|
||||
- Standard Python Logger
|
||||
- Motia FlowContext Logger
|
||||
- Structured Logging
|
||||
|
||||
Usage Guidelines:
|
||||
=================
|
||||
|
||||
FOR SERVICES: Use get_service_logger('service_name', context)
|
||||
-----------------------------------------------------------------
|
||||
Example:
|
||||
from services.logging_utils import get_service_logger
|
||||
|
||||
class XAIService:
|
||||
def __init__(self, ctx=None):
|
||||
self.logger = get_service_logger('xai', ctx)
|
||||
|
||||
def upload(self):
|
||||
self.logger.info("Uploading file...")
|
||||
|
||||
FOR STEPS: Use ctx.logger directly (preferred)
|
||||
-----------------------------------------------------------------
|
||||
Steps already have ctx.logger available - use it directly:
|
||||
async def handler(event_data, ctx: FlowContext):
|
||||
ctx.logger.info("Processing event")
|
||||
|
||||
Alternative: Use get_step_logger() for additional loggers:
|
||||
step_logger = get_step_logger('beteiligte_sync', ctx)
|
||||
|
||||
FOR SYNC UTILS: Inherit from BaseSyncUtils (provides self.logger)
|
||||
-----------------------------------------------------------------
|
||||
from services.sync_utils_base import BaseSyncUtils
|
||||
|
||||
class MySync(BaseSyncUtils):
|
||||
def __init__(self, espocrm, redis, context):
|
||||
super().__init__(espocrm, redis, context)
|
||||
# self.logger is now available
|
||||
|
||||
def sync(self):
|
||||
self._log("Syncing...", level='info')
|
||||
|
||||
FOR STANDALONE UTILITIES: Use get_logger()
|
||||
-----------------------------------------------------------------
|
||||
from services.logging_utils import get_logger
|
||||
|
||||
logger = get_logger('my_module', context)
|
||||
logger.info("Processing...")
|
||||
|
||||
CONSISTENCY RULES:
|
||||
==================
|
||||
✅ Services: get_service_logger('service_name', ctx)
|
||||
✅ Steps: ctx.logger (direct) or get_step_logger('step_name', ctx)
|
||||
✅ Sync Utils: Inherit from BaseSyncUtils → use self._log() or self.logger
|
||||
✅ Standalone: get_logger('module_name', ctx)
|
||||
|
||||
❌ DO NOT: Use module-level logging.getLogger(__name__)
|
||||
❌ DO NOT: Mix get_logger() and get_service_logger() in same module
|
||||
"""
|
||||
|
||||
import logging
|
||||
import time
|
||||
from typing import Optional, Any, Dict
|
||||
from contextlib import contextmanager
|
||||
from datetime import datetime
|
||||
|
||||
|
||||
class IntegrationLogger:
|
||||
"""
|
||||
Unified Logger mit Support für:
|
||||
- Motia FlowContext
|
||||
- Standard Python Logging
|
||||
- Structured Logging
|
||||
- Performance Tracking
|
||||
"""
|
||||
|
||||
def __init__(
|
||||
self,
|
||||
name: str,
|
||||
context: Optional[Any] = None,
|
||||
extra_fields: Optional[Dict[str, Any]] = None
|
||||
):
|
||||
"""
|
||||
Initialize logger.
|
||||
|
||||
Args:
|
||||
name: Logger name (z.B. 'advoware.api')
|
||||
context: Optional Motia FlowContext
|
||||
extra_fields: Optional extra fields für structured logging
|
||||
"""
|
||||
self.name = name
|
||||
self.context = context
|
||||
self.extra_fields = extra_fields or {}
|
||||
self._standard_logger = logging.getLogger(name)
|
||||
|
||||
def _format_message(self, message: str, **kwargs) -> str:
|
||||
"""
|
||||
Formatiert Log-Message mit optionalen Feldern.
|
||||
|
||||
Args:
|
||||
message: Base message
|
||||
**kwargs: Extra fields
|
||||
|
||||
Returns:
|
||||
Formatted message
|
||||
"""
|
||||
if not kwargs and not self.extra_fields:
|
||||
return message
|
||||
|
||||
# Merge extra fields
|
||||
fields = {**self.extra_fields, **kwargs}
|
||||
|
||||
if fields:
|
||||
field_str = " | ".join(f"{k}={v}" for k, v in fields.items())
|
||||
return f"{message} | {field_str}"
|
||||
|
||||
return message
|
||||
|
||||
def _log(
|
||||
self,
|
||||
level: str,
|
||||
message: str,
|
||||
exc_info: bool = False,
|
||||
**kwargs
|
||||
) -> None:
|
||||
"""
|
||||
Internal logging method.
|
||||
|
||||
Args:
|
||||
level: Log level (debug, info, warning, error, critical)
|
||||
message: Log message
|
||||
exc_info: Include exception info
|
||||
**kwargs: Extra fields for structured logging
|
||||
"""
|
||||
formatted_msg = self._format_message(message, **kwargs)
|
||||
|
||||
# Log to FlowContext if available
|
||||
if self.context and hasattr(self.context, 'logger'):
|
||||
try:
|
||||
log_func = getattr(self.context.logger, level, self.context.logger.info)
|
||||
log_func(formatted_msg)
|
||||
except Exception:
|
||||
# Fallback to standard logger
|
||||
pass
|
||||
|
||||
# Always log to standard Python logger
|
||||
log_func = getattr(self._standard_logger, level, self._standard_logger.info)
|
||||
log_func(formatted_msg, exc_info=exc_info)
|
||||
|
||||
def debug(self, message: str, **kwargs) -> None:
|
||||
"""Log debug message"""
|
||||
self._log('debug', message, **kwargs)
|
||||
|
||||
def info(self, message: str, **kwargs) -> None:
|
||||
"""Log info message"""
|
||||
self._log('info', message, **kwargs)
|
||||
|
||||
def warning(self, message: str, **kwargs) -> None:
|
||||
"""Log warning message"""
|
||||
self._log('warning', message, **kwargs)
|
||||
|
||||
def warn(self, message: str, **kwargs) -> None:
|
||||
"""Alias for warning"""
|
||||
self.warning(message, **kwargs)
|
||||
|
||||
def error(self, message: str, exc_info: bool = True, **kwargs) -> None:
|
||||
"""Log error message (with exception info by default)"""
|
||||
self._log('error', message, exc_info=exc_info, **kwargs)
|
||||
|
||||
def critical(self, message: str, exc_info: bool = True, **kwargs) -> None:
|
||||
"""Log critical message"""
|
||||
self._log('critical', message, exc_info=exc_info, **kwargs)
|
||||
|
||||
def exception(self, message: str, **kwargs) -> None:
|
||||
"""Log exception with traceback"""
|
||||
self._log('error', message, exc_info=True, **kwargs)
|
||||
|
||||
@contextmanager
|
||||
def operation(self, operation_name: str, **context_fields):
|
||||
"""
|
||||
Context manager für Operations mit automatischem Timing.
|
||||
|
||||
Args:
|
||||
operation_name: Name der Operation
|
||||
**context_fields: Context fields für logging
|
||||
|
||||
Example:
|
||||
with logger.operation('sync_beteiligte', entity_id='123'):
|
||||
# Do sync
|
||||
pass
|
||||
"""
|
||||
start_time = time.time()
|
||||
self.info(f"▶️ Starting: {operation_name}", **context_fields)
|
||||
|
||||
try:
|
||||
yield
|
||||
duration_ms = int((time.time() - start_time) * 1000)
|
||||
self.info(
|
||||
f"✅ Completed: {operation_name}",
|
||||
duration_ms=duration_ms,
|
||||
**context_fields
|
||||
)
|
||||
except Exception as e:
|
||||
duration_ms = int((time.time() - start_time) * 1000)
|
||||
self.error(
|
||||
f"❌ Failed: {operation_name} - {str(e)}",
|
||||
duration_ms=duration_ms,
|
||||
error_type=type(e).__name__,
|
||||
**context_fields
|
||||
)
|
||||
raise
|
||||
|
||||
@contextmanager
|
||||
def api_call(self, endpoint: str, method: str = 'GET', **context_fields):
|
||||
"""
|
||||
Context manager speziell für API-Calls.
|
||||
|
||||
Args:
|
||||
endpoint: API endpoint
|
||||
method: HTTP method
|
||||
**context_fields: Extra context
|
||||
|
||||
Example:
|
||||
with logger.api_call('/api/v1/Beteiligte', method='POST'):
|
||||
result = await api.post(...)
|
||||
"""
|
||||
start_time = time.time()
|
||||
self.debug(f"API Call: {method} {endpoint}", **context_fields)
|
||||
|
||||
try:
|
||||
yield
|
||||
duration_ms = int((time.time() - start_time) * 1000)
|
||||
self.debug(
|
||||
f"API Success: {method} {endpoint}",
|
||||
duration_ms=duration_ms,
|
||||
**context_fields
|
||||
)
|
||||
except Exception as e:
|
||||
duration_ms = int((time.time() - start_time) * 1000)
|
||||
self.error(
|
||||
f"API Error: {method} {endpoint} - {str(e)}",
|
||||
duration_ms=duration_ms,
|
||||
error_type=type(e).__name__,
|
||||
**context_fields
|
||||
)
|
||||
raise
|
||||
|
||||
def with_context(self, **extra_fields) -> 'IntegrationLogger':
|
||||
"""
|
||||
Erstellt neuen Logger mit zusätzlichen Context-Feldern.
|
||||
|
||||
Args:
|
||||
**extra_fields: Additional context fields
|
||||
|
||||
Returns:
|
||||
New logger instance with merged context
|
||||
"""
|
||||
merged_fields = {**self.extra_fields, **extra_fields}
|
||||
return IntegrationLogger(
|
||||
name=self.name,
|
||||
context=self.context,
|
||||
extra_fields=merged_fields
|
||||
)
|
||||
|
||||
|
||||
# ========== Factory Functions ==========
|
||||
|
||||
def get_logger(
|
||||
name: str,
|
||||
context: Optional[Any] = None,
|
||||
**extra_fields
|
||||
) -> IntegrationLogger:
|
||||
"""
|
||||
Factory function für Logger.
|
||||
|
||||
Args:
|
||||
name: Logger name
|
||||
context: Optional Motia FlowContext
|
||||
**extra_fields: Extra context fields
|
||||
|
||||
Returns:
|
||||
Configured logger
|
||||
|
||||
Example:
|
||||
logger = get_logger('advoware.sync', context=ctx, entity_id='123')
|
||||
logger.info("Starting sync")
|
||||
"""
|
||||
return IntegrationLogger(name, context, extra_fields)
|
||||
|
||||
|
||||
def get_service_logger(
|
||||
service_name: str,
|
||||
context: Optional[Any] = None
|
||||
) -> IntegrationLogger:
|
||||
"""
|
||||
Factory für Service-Logger.
|
||||
|
||||
Args:
|
||||
service_name: Service name (z.B. 'advoware', 'espocrm')
|
||||
context: Optional FlowContext
|
||||
|
||||
Returns:
|
||||
Service logger
|
||||
"""
|
||||
return IntegrationLogger(f"services.{service_name}", context)
|
||||
|
||||
|
||||
def get_step_logger(
|
||||
step_name: str,
|
||||
context: Optional[Any] = None
|
||||
) -> IntegrationLogger:
|
||||
"""
|
||||
Factory für Step-Logger.
|
||||
|
||||
Args:
|
||||
step_name: Step name
|
||||
context: FlowContext (required for steps)
|
||||
|
||||
Returns:
|
||||
Step logger
|
||||
"""
|
||||
return IntegrationLogger(f"steps.{step_name}", context)
|
||||
|
||||
|
||||
# ========== Decorator for Logging ==========
|
||||
|
||||
def log_operation(operation_name: str):
|
||||
"""
|
||||
Decorator für automatisches Operation-Logging.
|
||||
|
||||
Args:
|
||||
operation_name: Name der Operation
|
||||
|
||||
Example:
|
||||
@log_operation('sync_beteiligte')
|
||||
async def sync_entity(entity_id: str):
|
||||
...
|
||||
"""
|
||||
def decorator(func):
|
||||
async def async_wrapper(*args, **kwargs):
|
||||
# Try to find context in args
|
||||
context = None
|
||||
for arg in args:
|
||||
if hasattr(arg, 'logger'):
|
||||
context = arg
|
||||
break
|
||||
|
||||
logger = get_logger(func.__module__, context)
|
||||
|
||||
with logger.operation(operation_name):
|
||||
return await func(*args, **kwargs)
|
||||
|
||||
def sync_wrapper(*args, **kwargs):
|
||||
context = None
|
||||
for arg in args:
|
||||
if hasattr(arg, 'logger'):
|
||||
context = arg
|
||||
break
|
||||
|
||||
logger = get_logger(func.__module__, context)
|
||||
|
||||
with logger.operation(operation_name):
|
||||
return func(*args, **kwargs)
|
||||
|
||||
# Return appropriate wrapper
|
||||
import asyncio
|
||||
if asyncio.iscoroutinefunction(func):
|
||||
return async_wrapper
|
||||
else:
|
||||
return sync_wrapper
|
||||
|
||||
return decorator
|
||||
|
||||
|
||||
# ========== Performance Tracking ==========
|
||||
|
||||
class PerformanceTracker:
|
||||
"""Track performance metrics for operations"""
|
||||
|
||||
def __init__(self, logger: IntegrationLogger):
|
||||
self.logger = logger
|
||||
self.metrics: Dict[str, list] = {}
|
||||
|
||||
def record(self, operation: str, duration_ms: int) -> None:
|
||||
"""Record operation duration"""
|
||||
if operation not in self.metrics:
|
||||
self.metrics[operation] = []
|
||||
self.metrics[operation].append(duration_ms)
|
||||
|
||||
def get_stats(self, operation: str) -> Dict[str, float]:
|
||||
"""Get statistics for operation"""
|
||||
if operation not in self.metrics:
|
||||
return {}
|
||||
|
||||
durations = self.metrics[operation]
|
||||
return {
|
||||
'count': len(durations),
|
||||
'avg_ms': sum(durations) / len(durations),
|
||||
'min_ms': min(durations),
|
||||
'max_ms': max(durations),
|
||||
'total_ms': sum(durations)
|
||||
}
|
||||
|
||||
def log_summary(self) -> None:
|
||||
"""Log summary of all operations"""
|
||||
self.logger.info("=== Performance Summary ===")
|
||||
for operation, durations in self.metrics.items():
|
||||
stats = self.get_stats(operation)
|
||||
self.logger.info(
|
||||
f"{operation}: {stats['count']} calls, "
|
||||
f"avg {stats['avg_ms']:.1f}ms, "
|
||||
f"min {stats['min_ms']:.1f}ms, "
|
||||
f"max {stats['max_ms']:.1f}ms"
|
||||
)
|
||||
315
services/models.py
Normal file
315
services/models.py
Normal file
@@ -0,0 +1,315 @@
|
||||
"""
|
||||
Pydantic Models für Datenvalidierung
|
||||
|
||||
Definiert strenge Schemas für:
|
||||
- Advoware Entities
|
||||
- EspoCRM Entities
|
||||
- Sync Operations
|
||||
"""
|
||||
|
||||
from pydantic import BaseModel, Field, field_validator, ConfigDict
|
||||
from typing import Optional, Literal
|
||||
from datetime import date, datetime
|
||||
from enum import Enum
|
||||
|
||||
|
||||
# ========== Enums ==========
|
||||
|
||||
class Rechtsform(str, Enum):
|
||||
"""Legal forms for Beteiligte"""
|
||||
NATUERLICHE_PERSON = ""
|
||||
GMBH = "GmbH"
|
||||
AG = "AG"
|
||||
GMBH_CO_KG = "GmbH & Co. KG"
|
||||
KG = "KG"
|
||||
OHG = "OHG"
|
||||
EV = "e.V."
|
||||
EINZELUNTERNEHMEN = "Einzelunternehmen"
|
||||
FREIBERUFLER = "Freiberufler"
|
||||
|
||||
|
||||
class SyncStatus(str, Enum):
|
||||
"""Sync status for EspoCRM entities (Beteiligte)"""
|
||||
PENDING_SYNC = "pending_sync"
|
||||
SYNCING = "syncing"
|
||||
CLEAN = "clean"
|
||||
FAILED = "failed"
|
||||
CONFLICT = "conflict"
|
||||
PERMANENTLY_FAILED = "permanently_failed"
|
||||
|
||||
|
||||
class FileStatus(str, Enum):
|
||||
"""Valid values for CDokumente.fileStatus field"""
|
||||
NEW = "new"
|
||||
CHANGED = "changed"
|
||||
SYNCED = "synced"
|
||||
|
||||
def __str__(self) -> str:
|
||||
return self.value
|
||||
|
||||
|
||||
class XAISyncStatus(str, Enum):
|
||||
"""Valid values for CDokumente.xaiSyncStatus field"""
|
||||
NO_SYNC = "no_sync" # Entity has no xAI collections
|
||||
PENDING_SYNC = "pending_sync" # Sync in progress (locked)
|
||||
CLEAN = "clean" # Synced successfully
|
||||
UNCLEAN = "unclean" # Needs re-sync (file changed)
|
||||
FAILED = "failed" # Sync failed (see xaiSyncError)
|
||||
|
||||
def __str__(self) -> str:
|
||||
return self.value
|
||||
|
||||
|
||||
class SalutationType(str, Enum):
|
||||
"""Salutation types"""
|
||||
HERR = "Herr"
|
||||
FRAU = "Frau"
|
||||
DIVERS = "Divers"
|
||||
FIRMA = ""
|
||||
|
||||
|
||||
class AIKnowledgeActivationStatus(str, Enum):
|
||||
"""Activation status for CAIKnowledge collections"""
|
||||
NEW = "new" # Collection noch nicht in XAI erstellt
|
||||
ACTIVE = "active" # Collection aktiv, Sync läuft
|
||||
PAUSED = "paused" # Collection existiert, aber kein Sync
|
||||
DEACTIVATED = "deactivated" # Collection aus XAI gelöscht
|
||||
|
||||
def __str__(self) -> str:
|
||||
return self.value
|
||||
|
||||
|
||||
class AIKnowledgeSyncStatus(str, Enum):
|
||||
"""Sync status for CAIKnowledge"""
|
||||
UNCLEAN = "unclean" # Änderungen pending
|
||||
PENDING_SYNC = "pending_sync" # Sync läuft (locked)
|
||||
SYNCED = "synced" # Alles synced
|
||||
FAILED = "failed" # Sync fehlgeschlagen
|
||||
|
||||
def __str__(self) -> str:
|
||||
return self.value
|
||||
|
||||
|
||||
class JunctionSyncStatus(str, Enum):
|
||||
"""Sync status for junction tables (CAIKnowledgeCDokumente)"""
|
||||
NEW = "new"
|
||||
UNCLEAN = "unclean"
|
||||
SYNCED = "synced"
|
||||
FAILED = "failed"
|
||||
UNSUPPORTED = "unsupported"
|
||||
|
||||
def __str__(self) -> str:
|
||||
return self.value
|
||||
|
||||
|
||||
# ========== Advoware Models ==========
|
||||
|
||||
class AdvowareBeteiligteBase(BaseModel):
|
||||
"""Base Model für Advoware Beteiligte (POST/PUT)"""
|
||||
|
||||
model_config = ConfigDict(str_strip_whitespace=True, validate_assignment=True)
|
||||
|
||||
name: str = Field(..., min_length=1, max_length=200)
|
||||
vorname: Optional[str] = Field(None, max_length=100)
|
||||
rechtsform: str = Field(default="")
|
||||
anrede: Optional[str] = Field(None, max_length=50)
|
||||
titel: Optional[str] = Field(None, max_length=50)
|
||||
bAnrede: Optional[str] = Field(None, max_length=200, description="Briefanrede")
|
||||
zusatz: Optional[str] = Field(None, max_length=200)
|
||||
geburtsdatum: Optional[date] = None
|
||||
|
||||
@field_validator('name')
|
||||
@classmethod
|
||||
def validate_name(cls, v: str) -> str:
|
||||
if not v or not v.strip():
|
||||
raise ValueError('Name darf nicht leer sein')
|
||||
return v.strip()
|
||||
|
||||
@field_validator('geburtsdatum')
|
||||
@classmethod
|
||||
def validate_birthdate(cls, v: Optional[date]) -> Optional[date]:
|
||||
if v and v > date.today():
|
||||
raise ValueError('Geburtsdatum kann nicht in der Zukunft liegen')
|
||||
if v and v.year < 1900:
|
||||
raise ValueError('Geburtsdatum vor 1900 nicht erlaubt')
|
||||
return v
|
||||
|
||||
|
||||
class AdvowareBeteiligteRead(AdvowareBeteiligteBase):
|
||||
"""Advoware Beteiligte Response (GET)"""
|
||||
|
||||
betNr: int = Field(..., ge=1)
|
||||
rowId: str = Field(..., description="Change detection ID")
|
||||
|
||||
# Optional fields die Advoware zurückgibt
|
||||
strasse: Optional[str] = None
|
||||
plz: Optional[str] = None
|
||||
ort: Optional[str] = None
|
||||
land: Optional[str] = None
|
||||
|
||||
|
||||
class AdvowareBeteiligteCreate(AdvowareBeteiligteBase):
|
||||
"""Advoware Beteiligte für POST"""
|
||||
pass
|
||||
|
||||
|
||||
class AdvowareBeteiligteUpdate(AdvowareBeteiligteBase):
|
||||
"""Advoware Beteiligte für PUT"""
|
||||
pass
|
||||
|
||||
|
||||
# ========== EspoCRM Models ==========
|
||||
|
||||
class EspoCRMBeteiligteBase(BaseModel):
|
||||
"""Base Model für EspoCRM CBeteiligte"""
|
||||
|
||||
model_config = ConfigDict(str_strip_whitespace=True, validate_assignment=True)
|
||||
|
||||
name: str = Field(..., min_length=1, max_length=255)
|
||||
firstName: Optional[str] = Field(None, max_length=100)
|
||||
lastName: Optional[str] = Field(None, max_length=100)
|
||||
firmenname: Optional[str] = Field(None, max_length=255)
|
||||
rechtsform: str = Field(default="")
|
||||
salutationName: Optional[str] = None
|
||||
titel: Optional[str] = Field(None, max_length=100)
|
||||
briefAnrede: Optional[str] = Field(None, max_length=255)
|
||||
zusatz: Optional[str] = Field(None, max_length=255)
|
||||
dateOfBirth: Optional[date] = None
|
||||
|
||||
@field_validator('name')
|
||||
@classmethod
|
||||
def validate_name(cls, v: str) -> str:
|
||||
if not v or not v.strip():
|
||||
raise ValueError('Name darf nicht leer sein')
|
||||
return v.strip()
|
||||
|
||||
@field_validator('dateOfBirth')
|
||||
@classmethod
|
||||
def validate_birthdate(cls, v: Optional[date]) -> Optional[date]:
|
||||
if v and v > date.today():
|
||||
raise ValueError('Geburtsdatum kann nicht in der Zukunft liegen')
|
||||
if v and v.year < 1900:
|
||||
raise ValueError('Geburtsdatum vor 1900 nicht erlaubt')
|
||||
return v
|
||||
|
||||
@field_validator('firstName', 'lastName')
|
||||
@classmethod
|
||||
def validate_person_fields(cls, v: Optional[str]) -> Optional[str]:
|
||||
"""Validiere dass Person-Felder nur bei natürlichen Personen gesetzt sind"""
|
||||
if v:
|
||||
return v.strip()
|
||||
return None
|
||||
|
||||
|
||||
class EspoCRMBeteiligteRead(EspoCRMBeteiligteBase):
|
||||
"""EspoCRM CBeteiligte Response (GET)"""
|
||||
|
||||
id: str = Field(..., min_length=1)
|
||||
betnr: Optional[int] = Field(None, ge=1)
|
||||
advowareRowId: Optional[str] = None
|
||||
syncStatus: SyncStatus = Field(default=SyncStatus.PENDING_SYNC)
|
||||
syncRetryCount: int = Field(default=0, ge=0, le=10)
|
||||
syncErrorMessage: Optional[str] = None
|
||||
advowareLastSync: Optional[datetime] = None
|
||||
syncNextRetry: Optional[datetime] = None
|
||||
syncAutoResetAt: Optional[datetime] = None
|
||||
|
||||
|
||||
class EspoCRMBeteiligteCreate(EspoCRMBeteiligteBase):
|
||||
"""EspoCRM CBeteiligte für POST"""
|
||||
|
||||
syncStatus: SyncStatus = Field(default=SyncStatus.PENDING_SYNC)
|
||||
|
||||
|
||||
class EspoCRMBeteiligteUpdate(BaseModel):
|
||||
"""EspoCRM CBeteiligte für PUT (alle Felder optional)"""
|
||||
|
||||
model_config = ConfigDict(str_strip_whitespace=True, validate_assignment=True)
|
||||
|
||||
name: Optional[str] = Field(None, min_length=1, max_length=255)
|
||||
firstName: Optional[str] = Field(None, max_length=100)
|
||||
lastName: Optional[str] = Field(None, max_length=100)
|
||||
firmenname: Optional[str] = Field(None, max_length=255)
|
||||
rechtsform: Optional[str] = None
|
||||
salutationName: Optional[str] = None
|
||||
titel: Optional[str] = Field(None, max_length=100)
|
||||
briefAnrede: Optional[str] = Field(None, max_length=255)
|
||||
zusatz: Optional[str] = Field(None, max_length=255)
|
||||
dateOfBirth: Optional[date] = None
|
||||
betnr: Optional[int] = Field(None, ge=1)
|
||||
advowareRowId: Optional[str] = None
|
||||
syncStatus: Optional[SyncStatus] = None
|
||||
syncRetryCount: Optional[int] = Field(None, ge=0, le=10)
|
||||
syncErrorMessage: Optional[str] = Field(None, max_length=2000)
|
||||
advowareLastSync: Optional[datetime] = None
|
||||
syncNextRetry: Optional[datetime] = None
|
||||
|
||||
def model_dump_clean(self) -> dict:
|
||||
"""Gibt nur nicht-None Werte zurück (für PATCH-ähnliches Update)"""
|
||||
return {k: v for k, v in self.model_dump().items() if v is not None}
|
||||
|
||||
|
||||
# ========== Sync Operation Models ==========
|
||||
|
||||
class SyncOperation(BaseModel):
|
||||
"""Model für Sync-Operation Tracking"""
|
||||
|
||||
entity_id: str
|
||||
action: Literal["create", "update", "delete", "sync_check"]
|
||||
source: Literal["webhook", "cron", "api", "manual"]
|
||||
timestamp: datetime = Field(default_factory=datetime.utcnow)
|
||||
entity_type: str = "CBeteiligte"
|
||||
|
||||
|
||||
class SyncResult(BaseModel):
|
||||
"""Result einer Sync-Operation"""
|
||||
|
||||
success: bool
|
||||
entity_id: str
|
||||
action: str
|
||||
message: Optional[str] = None
|
||||
error: Optional[str] = None
|
||||
details: Optional[dict] = None
|
||||
duration_ms: Optional[int] = None
|
||||
|
||||
|
||||
# ========== Validation Helpers ==========
|
||||
|
||||
def validate_beteiligte_advoware(data: dict) -> AdvowareBeteiligteCreate:
|
||||
"""
|
||||
Validiert Advoware Beteiligte Daten.
|
||||
|
||||
Args:
|
||||
data: Dict mit Advoware Daten
|
||||
|
||||
Returns:
|
||||
Validiertes Model
|
||||
|
||||
Raises:
|
||||
ValidationError: Bei Validierungsfehlern
|
||||
"""
|
||||
try:
|
||||
return AdvowareBeteiligteCreate.model_validate(data)
|
||||
except Exception as e:
|
||||
from services.exceptions import ValidationError
|
||||
raise ValidationError(f"Invalid Advoware data: {e}")
|
||||
|
||||
|
||||
def validate_beteiligte_espocrm(data: dict) -> EspoCRMBeteiligteCreate:
|
||||
"""
|
||||
Validiert EspoCRM Beteiligte Daten.
|
||||
|
||||
Args:
|
||||
data: Dict mit EspoCRM Daten
|
||||
|
||||
Returns:
|
||||
Validiertes Model
|
||||
|
||||
Raises:
|
||||
ValidationError: Bei Validierungsfehlern
|
||||
"""
|
||||
try:
|
||||
return EspoCRMBeteiligteCreate.model_validate(data)
|
||||
except Exception as e:
|
||||
from services.exceptions import ValidationError
|
||||
raise ValidationError(f"Invalid EspoCRM data: {e}")
|
||||
202
services/redis_client.py
Normal file
202
services/redis_client.py
Normal file
@@ -0,0 +1,202 @@
|
||||
"""
|
||||
Redis Client Factory
|
||||
|
||||
Centralized Redis client management with:
|
||||
- Singleton pattern
|
||||
- Connection pooling
|
||||
- Automatic reconnection
|
||||
- Health checks
|
||||
"""
|
||||
|
||||
import redis
|
||||
import os
|
||||
from typing import Optional
|
||||
from services.exceptions import RedisConnectionError
|
||||
from services.logging_utils import get_service_logger
|
||||
|
||||
|
||||
class RedisClientFactory:
|
||||
"""
|
||||
Singleton factory for Redis clients.
|
||||
|
||||
Benefits:
|
||||
- Centralized configuration
|
||||
- Connection pooling
|
||||
- Lazy initialization
|
||||
- Better error handling
|
||||
"""
|
||||
|
||||
_instance: Optional[redis.Redis] = None
|
||||
_connection_pool: Optional[redis.ConnectionPool] = None
|
||||
_logger = None
|
||||
|
||||
@classmethod
|
||||
def _get_logger(cls):
|
||||
"""Get logger instance (lazy initialization)"""
|
||||
if cls._logger is None:
|
||||
cls._logger = get_service_logger('redis_factory', None)
|
||||
return cls._logger
|
||||
|
||||
@classmethod
|
||||
def get_client(cls, strict: bool = False) -> Optional[redis.Redis]:
|
||||
"""
|
||||
Return Redis client (creates if needed).
|
||||
|
||||
Args:
|
||||
strict: If True, raises exception on connection failures.
|
||||
If False, returns None (for optional Redis usage).
|
||||
|
||||
Returns:
|
||||
Redis client or None (if strict=False and connection fails)
|
||||
|
||||
Raises:
|
||||
RedisConnectionError: If strict=True and connection fails
|
||||
"""
|
||||
logger = cls._get_logger()
|
||||
if cls._instance is None:
|
||||
try:
|
||||
cls._instance = cls._create_client()
|
||||
logger.info("Redis client created successfully")
|
||||
except Exception as e:
|
||||
logger.error(f"Failed to create Redis client: {e}")
|
||||
if strict:
|
||||
raise RedisConnectionError(
|
||||
f"Could not connect to Redis: {e}",
|
||||
operation="get_client"
|
||||
)
|
||||
logger.warning("Redis unavailable - continuing without caching")
|
||||
return None
|
||||
|
||||
return cls._instance
|
||||
|
||||
@classmethod
|
||||
def _create_client(cls) -> redis.Redis:
|
||||
"""
|
||||
Create new Redis client with connection pool.
|
||||
|
||||
Returns:
|
||||
Configured Redis client
|
||||
|
||||
Raises:
|
||||
redis.ConnectionError: On connection problems
|
||||
"""
|
||||
logger = cls._get_logger()
|
||||
# Load configuration from environment
|
||||
redis_host = os.getenv('REDIS_HOST', 'localhost')
|
||||
redis_port = int(os.getenv('REDIS_PORT', '6379'))
|
||||
redis_db = int(os.getenv('REDIS_DB_ADVOWARE_CACHE', '1'))
|
||||
redis_timeout = int(os.getenv('REDIS_TIMEOUT_SECONDS', '5'))
|
||||
redis_max_connections = int(os.getenv('REDIS_MAX_CONNECTIONS', '50'))
|
||||
|
||||
logger.info(
|
||||
f"Creating Redis client: {redis_host}:{redis_port} "
|
||||
f"(db={redis_db}, timeout={redis_timeout}s)"
|
||||
)
|
||||
|
||||
# Create connection pool
|
||||
if cls._connection_pool is None:
|
||||
cls._connection_pool = redis.ConnectionPool(
|
||||
host=redis_host,
|
||||
port=redis_port,
|
||||
db=redis_db,
|
||||
socket_timeout=redis_timeout,
|
||||
socket_connect_timeout=redis_timeout,
|
||||
max_connections=redis_max_connections,
|
||||
decode_responses=True # Auto-decode bytes to strings
|
||||
)
|
||||
|
||||
# Create client from pool
|
||||
client = redis.Redis(connection_pool=cls._connection_pool)
|
||||
|
||||
# Verify connection
|
||||
client.ping()
|
||||
|
||||
return client
|
||||
|
||||
@classmethod
|
||||
def reset(cls) -> None:
|
||||
"""
|
||||
Reset factory state (mainly for tests).
|
||||
|
||||
Closes existing connections and resets singleton.
|
||||
"""
|
||||
logger = cls._get_logger()
|
||||
if cls._instance:
|
||||
try:
|
||||
cls._instance.close()
|
||||
except Exception as e:
|
||||
logger.warning(f"Error closing Redis client: {e}")
|
||||
|
||||
if cls._connection_pool:
|
||||
try:
|
||||
cls._connection_pool.disconnect()
|
||||
except Exception as e:
|
||||
logger.warning(f"Error closing connection pool: {e}")
|
||||
|
||||
cls._instance = None
|
||||
cls._connection_pool = None
|
||||
logger.info("Redis factory reset")
|
||||
|
||||
@classmethod
|
||||
def health_check(cls) -> bool:
|
||||
"""
|
||||
Check Redis connection.
|
||||
|
||||
Returns:
|
||||
True if Redis is reachable, False otherwise
|
||||
"""
|
||||
logger = cls._get_logger()
|
||||
try:
|
||||
client = cls.get_client(strict=False)
|
||||
if client is None:
|
||||
return False
|
||||
|
||||
client.ping()
|
||||
return True
|
||||
except Exception as e:
|
||||
logger.warning(f"Redis health check failed: {e}")
|
||||
return False
|
||||
|
||||
@classmethod
|
||||
def get_info(cls) -> Optional[dict]:
|
||||
"""
|
||||
Return Redis server info (for monitoring).
|
||||
|
||||
Returns:
|
||||
Redis info dict or None on error
|
||||
"""
|
||||
logger = cls._get_logger()
|
||||
try:
|
||||
client = cls.get_client(strict=False)
|
||||
if client is None:
|
||||
return None
|
||||
|
||||
return client.info()
|
||||
except Exception as e:
|
||||
logger.error(f"Failed to get Redis info: {e}")
|
||||
return None
|
||||
|
||||
|
||||
# ========== Convenience Functions ==========
|
||||
|
||||
def get_redis_client(strict: bool = False) -> Optional[redis.Redis]:
|
||||
"""
|
||||
Convenience function for Redis client.
|
||||
|
||||
Args:
|
||||
strict: If True, raises exception on error
|
||||
|
||||
Returns:
|
||||
Redis client or None
|
||||
"""
|
||||
return RedisClientFactory.get_client(strict=strict)
|
||||
|
||||
|
||||
def is_redis_available() -> bool:
|
||||
"""
|
||||
Check if Redis is available.
|
||||
|
||||
Returns:
|
||||
True if Redis is reachable
|
||||
"""
|
||||
return RedisClientFactory.health_check()
|
||||
144
services/sync_utils_base.py
Normal file
144
services/sync_utils_base.py
Normal file
@@ -0,0 +1,144 @@
|
||||
"""
|
||||
Base Sync Utilities
|
||||
|
||||
Gemeinsame Funktionalität für alle Sync-Operationen:
|
||||
- Redis Distributed Locking
|
||||
- Context-aware Logging
|
||||
- EspoCRM API Helpers
|
||||
"""
|
||||
|
||||
from typing import Dict, Any, Optional
|
||||
from datetime import datetime
|
||||
import pytz
|
||||
|
||||
from services.exceptions import RedisConnectionError, LockAcquisitionError
|
||||
from services.redis_client import get_redis_client
|
||||
from services.config import SYNC_CONFIG, get_lock_key
|
||||
from services.logging_utils import get_service_logger
|
||||
|
||||
import redis
|
||||
|
||||
|
||||
class BaseSyncUtils:
|
||||
"""Base-Klasse mit gemeinsamer Sync-Funktionalität"""
|
||||
|
||||
def __init__(self, espocrm_api, redis_client: Optional[redis.Redis] = None, context=None):
|
||||
"""
|
||||
Args:
|
||||
espocrm_api: EspoCRM API client instance
|
||||
redis_client: Optional Redis client (wird sonst über Factory initialisiert)
|
||||
context: Optional Motia FlowContext für Logging
|
||||
"""
|
||||
self.espocrm = espocrm_api
|
||||
self.context = context
|
||||
self.logger = get_service_logger('sync_utils', context)
|
||||
|
||||
# Use provided Redis client or get from factory
|
||||
self.redis = redis_client or get_redis_client(strict=False)
|
||||
|
||||
if not self.redis:
|
||||
self.logger.error(
|
||||
"⚠️ WARNUNG: Redis nicht verfügbar! "
|
||||
"Distributed Locking deaktiviert - Race Conditions möglich!"
|
||||
)
|
||||
|
||||
def _log(self, message: str, level: str = 'info') -> None:
|
||||
"""Delegate logging to the logger with optional level"""
|
||||
log_func = getattr(self.logger, level, self.logger.info)
|
||||
log_func(message)
|
||||
|
||||
def _get_lock_key(self, entity_id: str) -> str:
|
||||
"""
|
||||
Erzeugt Redis Lock-Key für eine Entity
|
||||
|
||||
Muss in Subklassen überschrieben werden, um entity-spezifische Prefixes zu nutzen.
|
||||
z.B. 'sync_lock:cbeteiligte:{entity_id}' oder 'sync_lock:document:{entity_id}'
|
||||
"""
|
||||
raise NotImplementedError("Subclass must implement _get_lock_key()")
|
||||
|
||||
def _acquire_redis_lock(self, lock_key: str) -> bool:
|
||||
"""
|
||||
Atomic Redis lock acquisition
|
||||
|
||||
Args:
|
||||
lock_key: Redis key für den Lock
|
||||
|
||||
Returns:
|
||||
True wenn Lock erfolgreich, False wenn bereits locked
|
||||
|
||||
Raises:
|
||||
LockAcquisitionError: Bei kritischen Lock-Problemen (wenn strict mode)
|
||||
"""
|
||||
if not self.redis:
|
||||
self.logger.error(
|
||||
"CRITICAL: Distributed Locking deaktiviert - Redis nicht verfügbar!"
|
||||
)
|
||||
# In production: Dies könnte zu Race Conditions führen!
|
||||
# Für jetzt erlauben wir Fortsetzung, aber mit Warning
|
||||
return True
|
||||
|
||||
try:
|
||||
acquired = self.redis.set(
|
||||
lock_key,
|
||||
"locked",
|
||||
nx=True,
|
||||
ex=SYNC_CONFIG.lock_ttl_seconds
|
||||
)
|
||||
return bool(acquired)
|
||||
except redis.RedisError as e:
|
||||
self.logger.error(f"Redis lock error: {e}")
|
||||
# Bei Redis-Fehler: Lock erlauben, um Deadlocks zu vermeiden
|
||||
return True
|
||||
|
||||
def _release_redis_lock(self, lock_key: str) -> None:
|
||||
"""
|
||||
Redis lock freigeben
|
||||
|
||||
Args:
|
||||
lock_key: Redis key für den Lock
|
||||
"""
|
||||
if not self.redis:
|
||||
return
|
||||
|
||||
try:
|
||||
self.redis.delete(lock_key)
|
||||
except redis.RedisError as e:
|
||||
self.logger.error(f"Redis unlock error: {e}")
|
||||
|
||||
def _get_espocrm_datetime(self, dt: Optional[datetime] = None) -> str:
|
||||
"""
|
||||
Formatiert datetime für EspoCRM (ohne Timezone!)
|
||||
|
||||
Args:
|
||||
dt: Optional datetime object (default: now UTC)
|
||||
|
||||
Returns:
|
||||
String im Format 'YYYY-MM-DD HH:MM:SS'
|
||||
"""
|
||||
if dt is None:
|
||||
dt = datetime.now(pytz.UTC)
|
||||
elif dt.tzinfo is None:
|
||||
dt = pytz.UTC.localize(dt)
|
||||
|
||||
return dt.strftime('%Y-%m-%d %H:%M:%S')
|
||||
|
||||
async def acquire_sync_lock(self, entity_id: str, **kwargs) -> bool:
|
||||
"""
|
||||
Erwirbt Sync-Lock für eine Entity
|
||||
|
||||
Muss in Subklassen implementiert werden, um entity-spezifische
|
||||
Status-Updates durchzuführen.
|
||||
|
||||
Returns:
|
||||
True wenn Lock erfolgreich, False wenn bereits locked
|
||||
"""
|
||||
raise NotImplementedError("Subclass must implement acquire_sync_lock()")
|
||||
|
||||
async def release_sync_lock(self, entity_id: str, **kwargs) -> None:
|
||||
"""
|
||||
Gibt Sync-Lock frei und setzt finalen Status
|
||||
|
||||
Muss in Subklassen implementiert werden, um entity-spezifische
|
||||
Status-Updates durchzuführen.
|
||||
"""
|
||||
raise NotImplementedError("Subclass must implement release_sync_lock()")
|
||||
529
services/xai_service.py
Normal file
529
services/xai_service.py
Normal file
@@ -0,0 +1,529 @@
|
||||
"""xAI Files & Collections Service"""
|
||||
import os
|
||||
import asyncio
|
||||
import aiohttp
|
||||
from typing import Optional, List, Dict, Tuple
|
||||
from services.logging_utils import get_service_logger
|
||||
|
||||
XAI_FILES_URL = "https://api.x.ai"
|
||||
XAI_MANAGEMENT_URL = "https://management-api.x.ai"
|
||||
|
||||
|
||||
class XAIService:
|
||||
"""
|
||||
Client für xAI Files API und Collections Management API.
|
||||
|
||||
Benötigte Umgebungsvariablen:
|
||||
- XAI_API_KEY – regulärer API-Key für File-Uploads (api.x.ai)
|
||||
- XAI_MANAGEMENT_KEY – Management-API-Key für Collection-Operationen (management-api.x.ai)
|
||||
"""
|
||||
|
||||
def __init__(self, ctx=None):
|
||||
self.api_key = os.getenv('XAI_API_KEY', '')
|
||||
self.management_key = os.getenv('XAI_MANAGEMENT_KEY', '')
|
||||
self.ctx = ctx
|
||||
self.logger = get_service_logger('xai', ctx)
|
||||
self._session: Optional[aiohttp.ClientSession] = None
|
||||
|
||||
if not self.api_key:
|
||||
raise ValueError("XAI_API_KEY not configured in environment")
|
||||
if not self.management_key:
|
||||
raise ValueError("XAI_MANAGEMENT_KEY not configured in environment")
|
||||
|
||||
def _log(self, msg: str, level: str = 'info') -> None:
|
||||
"""Delegate logging to service logger"""
|
||||
log_func = getattr(self.logger, level, self.logger.info)
|
||||
log_func(msg)
|
||||
|
||||
async def _get_session(self) -> aiohttp.ClientSession:
|
||||
if self._session is None or self._session.closed:
|
||||
self._session = aiohttp.ClientSession(
|
||||
timeout=aiohttp.ClientTimeout(total=120)
|
||||
)
|
||||
return self._session
|
||||
|
||||
async def close(self) -> None:
|
||||
if self._session and not self._session.closed:
|
||||
await self._session.close()
|
||||
|
||||
async def upload_file(
|
||||
self,
|
||||
file_content: bytes,
|
||||
filename: str,
|
||||
mime_type: str = 'application/octet-stream'
|
||||
) -> str:
|
||||
"""
|
||||
Lädt eine Datei zur xAI Files API hoch (multipart/form-data).
|
||||
|
||||
POST https://api.x.ai/v1/files
|
||||
|
||||
Returns:
|
||||
xAI file_id (str)
|
||||
|
||||
Raises:
|
||||
RuntimeError: bei HTTP-Fehler oder fehlendem file_id in der Antwort
|
||||
"""
|
||||
# Normalize MIME type: xAI needs correct Content-Type for proper processing
|
||||
# If generic octet-stream but file is clearly a PDF, fix it
|
||||
if mime_type == 'application/octet-stream' and filename.lower().endswith('.pdf'):
|
||||
mime_type = 'application/pdf'
|
||||
self._log(f"⚠️ Corrected MIME type to application/pdf for {filename}")
|
||||
|
||||
self._log(f"📤 Uploading {len(file_content)} bytes to xAI: {filename} ({mime_type})")
|
||||
|
||||
session = await self._get_session()
|
||||
url = f"{XAI_FILES_URL}/v1/files"
|
||||
headers = {"Authorization": f"Bearer {self.api_key}"}
|
||||
|
||||
# Create multipart form with explicit UTF-8 filename encoding
|
||||
# aiohttp automatically URL-encodes filenames with special chars,
|
||||
# but xAI expects raw UTF-8 in the filename parameter
|
||||
form = aiohttp.FormData(quote_fields=False)
|
||||
form.add_field(
|
||||
'file',
|
||||
file_content,
|
||||
filename=filename,
|
||||
content_type=mime_type
|
||||
)
|
||||
# CRITICAL: purpose="file_search" enables proper PDF processing
|
||||
# Without this, xAI throws "internal error" on complex PDFs
|
||||
form.add_field('purpose', 'file_search')
|
||||
|
||||
async with session.post(url, data=form, headers=headers) as response:
|
||||
try:
|
||||
data = await response.json()
|
||||
except Exception:
|
||||
raw = await response.text()
|
||||
data = {"_raw": raw}
|
||||
|
||||
if response.status not in (200, 201):
|
||||
raise RuntimeError(
|
||||
f"xAI file upload failed ({response.status}): {data}"
|
||||
)
|
||||
|
||||
file_id = data.get('id') or data.get('file_id')
|
||||
if not file_id:
|
||||
raise RuntimeError(
|
||||
f"No file_id in xAI upload response: {data}"
|
||||
)
|
||||
|
||||
self._log(f"✅ xAI file uploaded: {file_id}")
|
||||
return file_id
|
||||
|
||||
async def add_to_collection(self, collection_id: str, file_id: str) -> None:
|
||||
"""
|
||||
Fügt eine Datei einer xAI-Collection (Vector Store) hinzu.
|
||||
|
||||
POST https://api.x.ai/v1/vector_stores/{vector_store_id}/files
|
||||
|
||||
Uses the OpenAI-compatible API pattern for adding files to vector stores.
|
||||
This triggers proper indexing and processing.
|
||||
|
||||
Raises:
|
||||
RuntimeError: bei HTTP-Fehler
|
||||
"""
|
||||
self._log(f"📚 Adding file {file_id} to collection {collection_id}")
|
||||
|
||||
session = await self._get_session()
|
||||
# Use the OpenAI-compatible endpoint (not management API)
|
||||
url = f"{XAI_FILES_URL}/v1/vector_stores/{collection_id}/files"
|
||||
headers = {
|
||||
"Authorization": f"Bearer {self.api_key}",
|
||||
"Content-Type": "application/json",
|
||||
}
|
||||
|
||||
payload = {"file_id": file_id}
|
||||
|
||||
async with session.post(url, json=payload, headers=headers) as response:
|
||||
if response.status not in (200, 201):
|
||||
raw = await response.text()
|
||||
raise RuntimeError(
|
||||
f"Failed to add file to collection {collection_id} ({response.status}): {raw}"
|
||||
)
|
||||
|
||||
self._log(f"✅ File {file_id} added to collection {collection_id}")
|
||||
|
||||
async def remove_from_collection(self, collection_id: str, file_id: str) -> None:
|
||||
"""
|
||||
Entfernt eine Datei aus einer xAI-Collection.
|
||||
Die Datei selbst wird NICHT gelöscht – sie kann in anderen Collections sein.
|
||||
|
||||
DELETE https://management-api.x.ai/v1/collections/{collection_id}/documents/{file_id}
|
||||
|
||||
Raises:
|
||||
RuntimeError: bei HTTP-Fehler
|
||||
"""
|
||||
self._log(f"🗑️ Removing file {file_id} from collection {collection_id}")
|
||||
|
||||
session = await self._get_session()
|
||||
url = f"{XAI_MANAGEMENT_URL}/v1/collections/{collection_id}/documents/{file_id}"
|
||||
headers = {"Authorization": f"Bearer {self.management_key}"}
|
||||
|
||||
async with session.delete(url, headers=headers) as response:
|
||||
if response.status not in (200, 204):
|
||||
raw = await response.text()
|
||||
raise RuntimeError(
|
||||
f"Failed to remove file from collection {collection_id} ({response.status}): {raw}"
|
||||
)
|
||||
|
||||
self._log(f"✅ File {file_id} removed from collection {collection_id}")
|
||||
|
||||
async def add_to_collections(self, collection_ids: List[str], file_id: str) -> List[str]:
|
||||
"""
|
||||
Fügt eine Datei zu mehreren Collections hinzu.
|
||||
|
||||
Returns:
|
||||
Liste der erfolgreich hinzugefügten Collection-IDs
|
||||
"""
|
||||
added = []
|
||||
for collection_id in collection_ids:
|
||||
try:
|
||||
await self.add_to_collection(collection_id, file_id)
|
||||
added.append(collection_id)
|
||||
except Exception as e:
|
||||
self._log(
|
||||
f"⚠️ Fehler beim Hinzufügen zu Collection {collection_id}: {e}",
|
||||
level='warn'
|
||||
)
|
||||
return added
|
||||
|
||||
async def remove_from_collections(self, collection_ids: List[str], file_id: str) -> None:
|
||||
"""Entfernt eine Datei aus mehreren Collections (ignoriert Fehler pro Collection)."""
|
||||
for collection_id in collection_ids:
|
||||
try:
|
||||
await self.remove_from_collection(collection_id, file_id)
|
||||
except Exception as e:
|
||||
self._log(
|
||||
f"⚠️ Fehler beim Entfernen aus Collection {collection_id}: {e}",
|
||||
level='warn'
|
||||
)
|
||||
|
||||
# ========== Collection Management ==========
|
||||
|
||||
async def create_collection(
|
||||
self,
|
||||
name: str,
|
||||
metadata: Optional[Dict[str, str]] = None,
|
||||
field_definitions: Optional[List[Dict]] = None
|
||||
) -> Dict:
|
||||
"""
|
||||
Erstellt eine neue xAI Collection.
|
||||
|
||||
POST https://management-api.x.ai/v1/collections
|
||||
|
||||
Args:
|
||||
name: Collection name
|
||||
metadata: Optional metadata dict
|
||||
field_definitions: Optional field definitions for metadata fields
|
||||
|
||||
Returns:
|
||||
Collection object mit 'id' field
|
||||
|
||||
Raises:
|
||||
RuntimeError: bei HTTP-Fehler
|
||||
"""
|
||||
self._log(f"📚 Creating collection: {name}")
|
||||
|
||||
# Standard field definitions für document metadata
|
||||
if field_definitions is None:
|
||||
field_definitions = [
|
||||
{"key": "document_name", "inject_into_chunk": True},
|
||||
{"key": "description", "inject_into_chunk": True},
|
||||
{"key": "created_at", "inject_into_chunk": False},
|
||||
{"key": "modified_at", "inject_into_chunk": False},
|
||||
{"key": "espocrm_id", "inject_into_chunk": False}
|
||||
]
|
||||
|
||||
session = await self._get_session()
|
||||
url = f"{XAI_MANAGEMENT_URL}/v1/collections"
|
||||
headers = {
|
||||
"Authorization": f"Bearer {self.management_key}",
|
||||
"Content-Type": "application/json"
|
||||
}
|
||||
|
||||
body = {
|
||||
"collection_name": name,
|
||||
"field_definitions": field_definitions
|
||||
}
|
||||
|
||||
# Add metadata if provided
|
||||
if metadata:
|
||||
body["metadata"] = metadata
|
||||
|
||||
async with session.post(url, json=body, headers=headers) as response:
|
||||
if response.status not in (200, 201):
|
||||
raw = await response.text()
|
||||
raise RuntimeError(
|
||||
f"Failed to create collection ({response.status}): {raw}"
|
||||
)
|
||||
|
||||
data = await response.json()
|
||||
|
||||
# API returns 'collection_id' not 'id'
|
||||
collection_id = data.get('collection_id') or data.get('id')
|
||||
self._log(f"✅ Collection created: {collection_id}")
|
||||
return data
|
||||
|
||||
async def get_collection(self, collection_id: str) -> Optional[Dict]:
|
||||
"""
|
||||
Holt Collection-Details.
|
||||
|
||||
GET https://management-api.x.ai/v1/collections/{collection_id}
|
||||
|
||||
Returns:
|
||||
Collection object or None if not found
|
||||
|
||||
Raises:
|
||||
RuntimeError: bei HTTP-Fehler (außer 404)
|
||||
"""
|
||||
self._log(f"📄 Getting collection: {collection_id}")
|
||||
|
||||
session = await self._get_session()
|
||||
url = f"{XAI_MANAGEMENT_URL}/v1/collections/{collection_id}"
|
||||
headers = {"Authorization": f"Bearer {self.management_key}"}
|
||||
|
||||
async with session.get(url, headers=headers) as response:
|
||||
if response.status == 404:
|
||||
self._log(f"⚠️ Collection not found: {collection_id}", level='warn')
|
||||
return None
|
||||
|
||||
if response.status not in (200,):
|
||||
raw = await response.text()
|
||||
raise RuntimeError(
|
||||
f"Failed to get collection ({response.status}): {raw}"
|
||||
)
|
||||
|
||||
data = await response.json()
|
||||
|
||||
self._log(f"✅ Collection retrieved: {data.get('collection_name', 'N/A')}")
|
||||
return data
|
||||
|
||||
async def delete_collection(self, collection_id: str) -> None:
|
||||
"""
|
||||
Löscht eine XAI Collection.
|
||||
|
||||
DELETE https://management-api.x.ai/v1/collections/{collection_id}
|
||||
|
||||
NOTE: Documents in der Collection werden NICHT gelöscht!
|
||||
Sie können noch in anderen Collections sein.
|
||||
|
||||
Raises:
|
||||
RuntimeError: bei HTTP-Fehler
|
||||
"""
|
||||
self._log(f"🗑️ Deleting collection {collection_id}")
|
||||
|
||||
session = await self._get_session()
|
||||
url = f"{XAI_MANAGEMENT_URL}/v1/collections/{collection_id}"
|
||||
headers = {"Authorization": f"Bearer {self.management_key}"}
|
||||
|
||||
async with session.delete(url, headers=headers) as response:
|
||||
if response.status not in (200, 204):
|
||||
raw = await response.text()
|
||||
raise RuntimeError(
|
||||
f"Failed to delete collection {collection_id} ({response.status}): {raw}"
|
||||
)
|
||||
|
||||
self._log(f"✅ Collection deleted: {collection_id}")
|
||||
|
||||
async def list_collection_documents(self, collection_id: str) -> List[Dict]:
|
||||
"""
|
||||
Listet alle Dokumente in einer Collection.
|
||||
|
||||
GET https://management-api.x.ai/v1/collections/{collection_id}/documents
|
||||
|
||||
Returns:
|
||||
List von normalized document objects:
|
||||
[
|
||||
{
|
||||
'file_id': 'file_...',
|
||||
'filename': 'doc.pdf',
|
||||
'blake3_hash': 'hex_string', # Plain hex, kein prefix
|
||||
'size_bytes': 12345,
|
||||
'content_type': 'application/pdf',
|
||||
'fields': {}, # Custom metadata
|
||||
'status': 'DOCUMENT_STATUS_...'
|
||||
}
|
||||
]
|
||||
|
||||
Raises:
|
||||
RuntimeError: bei HTTP-Fehler
|
||||
"""
|
||||
self._log(f"📋 Listing documents in collection {collection_id}")
|
||||
|
||||
session = await self._get_session()
|
||||
url = f"{XAI_MANAGEMENT_URL}/v1/collections/{collection_id}/documents"
|
||||
headers = {"Authorization": f"Bearer {self.management_key}"}
|
||||
|
||||
async with session.get(url, headers=headers) as response:
|
||||
if response.status not in (200,):
|
||||
raw = await response.text()
|
||||
raise RuntimeError(
|
||||
f"Failed to list documents ({response.status}): {raw}"
|
||||
)
|
||||
|
||||
data = await response.json()
|
||||
|
||||
# API gibt Liste zurück oder dict mit 'documents' key
|
||||
if isinstance(data, list):
|
||||
raw_documents = data
|
||||
elif isinstance(data, dict) and 'documents' in data:
|
||||
raw_documents = data['documents']
|
||||
else:
|
||||
raw_documents = []
|
||||
|
||||
# Normalize nested structure: file_metadata -> top-level
|
||||
normalized = []
|
||||
for doc in raw_documents:
|
||||
file_meta = doc.get('file_metadata', {})
|
||||
normalized.append({
|
||||
'file_id': file_meta.get('file_id'),
|
||||
'filename': file_meta.get('name'),
|
||||
'blake3_hash': file_meta.get('hash'), # Plain hex string
|
||||
'size_bytes': int(file_meta.get('size_bytes', 0)) if file_meta.get('size_bytes') else 0,
|
||||
'content_type': file_meta.get('content_type'),
|
||||
'created_at': file_meta.get('created_at'),
|
||||
'fields': doc.get('fields', {}),
|
||||
'status': doc.get('status')
|
||||
})
|
||||
|
||||
self._log(f"✅ Listed {len(normalized)} documents")
|
||||
return normalized
|
||||
|
||||
async def get_collection_document(self, collection_id: str, file_id: str) -> Optional[Dict]:
|
||||
"""
|
||||
Holt Dokument-Details aus einer XAI Collection.
|
||||
|
||||
GET https://management-api.x.ai/v1/collections/{collection_id}/documents/{file_id}
|
||||
|
||||
Returns:
|
||||
Normalized dict mit document info:
|
||||
{
|
||||
'file_id': 'file_xyz',
|
||||
'filename': 'document.pdf',
|
||||
'blake3_hash': 'hex_string', # Plain hex, kein prefix
|
||||
'size_bytes': 12345,
|
||||
'content_type': 'application/pdf',
|
||||
'fields': {...} # Custom metadata
|
||||
}
|
||||
|
||||
Returns None if not found.
|
||||
"""
|
||||
self._log(f"📄 Getting document {file_id} from collection {collection_id}")
|
||||
|
||||
session = await self._get_session()
|
||||
url = f"{XAI_MANAGEMENT_URL}/v1/collections/{collection_id}/documents/{file_id}"
|
||||
headers = {"Authorization": f"Bearer {self.management_key}"}
|
||||
|
||||
async with session.get(url, headers=headers) as response:
|
||||
if response.status == 404:
|
||||
return None
|
||||
|
||||
if response.status not in (200,):
|
||||
raw = await response.text()
|
||||
raise RuntimeError(
|
||||
f"Failed to get document from collection ({response.status}): {raw}"
|
||||
)
|
||||
|
||||
data = await response.json()
|
||||
|
||||
# Normalize nested structure
|
||||
file_meta = data.get('file_metadata', {})
|
||||
normalized = {
|
||||
'file_id': file_meta.get('file_id'),
|
||||
'filename': file_meta.get('name'),
|
||||
'blake3_hash': file_meta.get('hash'), # Plain hex
|
||||
'size_bytes': int(file_meta.get('size_bytes', 0)) if file_meta.get('size_bytes') else 0,
|
||||
'content_type': file_meta.get('content_type'),
|
||||
'created_at': file_meta.get('created_at'),
|
||||
'fields': data.get('fields', {}),
|
||||
'status': data.get('status')
|
||||
}
|
||||
|
||||
self._log(f"✅ Document info retrieved: {normalized.get('filename', 'N/A')}")
|
||||
return normalized
|
||||
|
||||
async def update_document_metadata(
|
||||
self,
|
||||
collection_id: str,
|
||||
file_id: str,
|
||||
metadata: Dict[str, str]
|
||||
) -> None:
|
||||
"""
|
||||
Aktualisiert nur Metadaten eines Documents (kein File-Upload).
|
||||
|
||||
PATCH https://management-api.x.ai/v1/collections/{collection_id}/documents/{file_id}
|
||||
|
||||
Args:
|
||||
collection_id: XAI Collection ID
|
||||
file_id: XAI file_id
|
||||
metadata: Updated metadata fields
|
||||
|
||||
Raises:
|
||||
RuntimeError: bei HTTP-Fehler
|
||||
"""
|
||||
self._log(f"📝 Updating metadata for document {file_id}")
|
||||
|
||||
session = await self._get_session()
|
||||
url = f"{XAI_MANAGEMENT_URL}/v1/collections/{collection_id}/documents/{file_id}"
|
||||
headers = {
|
||||
"Authorization": f"Bearer {self.management_key}",
|
||||
"Content-Type": "application/json"
|
||||
}
|
||||
|
||||
body = {"fields": metadata}
|
||||
|
||||
async with session.patch(url, json=body, headers=headers) as response:
|
||||
if response.status not in (200, 204):
|
||||
raw = await response.text()
|
||||
raise RuntimeError(
|
||||
f"Failed to update document metadata ({response.status}): {raw}"
|
||||
)
|
||||
|
||||
self._log(f"✅ Metadata updated for {file_id}")
|
||||
|
||||
def is_mime_type_supported(self, mime_type: str) -> bool:
|
||||
"""
|
||||
Prüft, ob XAI diesen MIME-Type unterstützt.
|
||||
|
||||
Args:
|
||||
mime_type: MIME type string
|
||||
|
||||
Returns:
|
||||
True wenn unterstützt, False sonst
|
||||
"""
|
||||
# Liste der unterstützten MIME-Types basierend auf XAI Dokumentation
|
||||
supported_types = {
|
||||
# Documents
|
||||
'application/pdf',
|
||||
'application/msword',
|
||||
'application/vnd.openxmlformats-officedocument.wordprocessingml.document',
|
||||
'application/vnd.ms-excel',
|
||||
'application/vnd.openxmlformats-officedocument.spreadsheetml.sheet',
|
||||
'application/vnd.oasis.opendocument.text',
|
||||
'application/epub+zip',
|
||||
'application/vnd.openxmlformats-officedocument.presentationml.presentation',
|
||||
|
||||
# Text
|
||||
'text/plain',
|
||||
'text/html',
|
||||
'text/markdown',
|
||||
'text/csv',
|
||||
'text/xml',
|
||||
|
||||
# Code
|
||||
'text/javascript',
|
||||
'application/json',
|
||||
'application/xml',
|
||||
'text/x-python',
|
||||
'text/x-java-source',
|
||||
'text/x-c',
|
||||
'text/x-c++src',
|
||||
|
||||
# Other
|
||||
'application/zip',
|
||||
}
|
||||
|
||||
# Normalisiere MIME-Type (lowercase, strip whitespace)
|
||||
normalized = mime_type.lower().strip()
|
||||
|
||||
return normalized in supported_types
|
||||
@@ -17,7 +17,7 @@ from calendar_sync_utils import (
|
||||
import math
|
||||
import time
|
||||
from datetime import datetime
|
||||
from typing import Any
|
||||
from typing import Any, Dict
|
||||
from motia import queue, FlowContext
|
||||
from pydantic import BaseModel, Field
|
||||
from services.advoware_service import AdvowareService
|
||||
@@ -33,7 +33,7 @@ config = {
|
||||
}
|
||||
|
||||
|
||||
async def handler(input_data: dict, ctx: FlowContext):
|
||||
async def handler(input_data: Dict[str, Any], ctx: FlowContext) -> None:
|
||||
"""
|
||||
Handler that fetches all employees, sorts by last sync time,
|
||||
and emits calendar_sync_employee events for the oldest ones.
|
||||
@@ -7,7 +7,7 @@ Supports syncing a single employee or all employees.
|
||||
import sys
|
||||
from pathlib import Path
|
||||
sys.path.insert(0, str(Path(__file__).parent))
|
||||
from calendar_sync_utils import get_redis_client, set_employee_lock, log_operation
|
||||
from calendar_sync_utils import get_redis_client, set_employee_lock, get_logger
|
||||
|
||||
from motia import http, ApiRequest, ApiResponse, FlowContext
|
||||
|
||||
@@ -41,7 +41,7 @@ async def handler(request: ApiRequest, ctx: FlowContext) -> ApiResponse:
|
||||
status=400,
|
||||
body={
|
||||
'error': 'kuerzel required',
|
||||
'message': 'Bitte kuerzel im Body angeben'
|
||||
'message': 'Please provide kuerzel in body'
|
||||
}
|
||||
)
|
||||
|
||||
@@ -49,7 +49,7 @@ async def handler(request: ApiRequest, ctx: FlowContext) -> ApiResponse:
|
||||
|
||||
if kuerzel_upper == 'ALL':
|
||||
# Emit sync-all event
|
||||
log_operation('info', "Calendar Sync API: Emitting sync-all event", context=ctx)
|
||||
ctx.logger.info("Calendar Sync API: Emitting sync-all event")
|
||||
await ctx.enqueue({
|
||||
"topic": "calendar_sync_all",
|
||||
"data": {
|
||||
@@ -60,7 +60,7 @@ async def handler(request: ApiRequest, ctx: FlowContext) -> ApiResponse:
|
||||
status=200,
|
||||
body={
|
||||
'status': 'triggered',
|
||||
'message': 'Calendar sync wurde für alle Mitarbeiter ausgelöst',
|
||||
'message': 'Calendar sync triggered for all employees',
|
||||
'triggered_by': 'api'
|
||||
}
|
||||
)
|
||||
@@ -69,7 +69,7 @@ async def handler(request: ApiRequest, ctx: FlowContext) -> ApiResponse:
|
||||
redis_client = get_redis_client(ctx)
|
||||
|
||||
if not set_employee_lock(redis_client, kuerzel_upper, 'api', ctx):
|
||||
log_operation('info', f"Calendar Sync API: Sync already active for {kuerzel_upper}, skipping", context=ctx)
|
||||
ctx.logger.info(f"Calendar Sync API: Sync already active for {kuerzel_upper}, skipping")
|
||||
return ApiResponse(
|
||||
status=409,
|
||||
body={
|
||||
@@ -80,7 +80,7 @@ async def handler(request: ApiRequest, ctx: FlowContext) -> ApiResponse:
|
||||
}
|
||||
)
|
||||
|
||||
log_operation('info', f"Calendar Sync API called for {kuerzel_upper}", context=ctx)
|
||||
ctx.logger.info(f"Calendar Sync API called for {kuerzel_upper}")
|
||||
|
||||
# Lock successfully set, now emit event
|
||||
await ctx.enqueue({
|
||||
@@ -95,14 +95,14 @@ async def handler(request: ApiRequest, ctx: FlowContext) -> ApiResponse:
|
||||
status=200,
|
||||
body={
|
||||
'status': 'triggered',
|
||||
'message': f'Calendar sync was triggered for {kuerzel_upper}',
|
||||
'message': f'Calendar sync triggered for {kuerzel_upper}',
|
||||
'kuerzel': kuerzel_upper,
|
||||
'triggered_by': 'api'
|
||||
}
|
||||
)
|
||||
|
||||
except Exception as e:
|
||||
log_operation('error', f"Error in API trigger: {e}", context=ctx)
|
||||
ctx.logger.error(f"Error in API trigger: {e}")
|
||||
return ApiResponse(
|
||||
status=500,
|
||||
body={
|
||||
@@ -9,6 +9,7 @@ from pathlib import Path
|
||||
sys.path.insert(0, str(Path(__file__).parent))
|
||||
from calendar_sync_utils import log_operation
|
||||
|
||||
from typing import Dict, Any
|
||||
from motia import cron, FlowContext
|
||||
|
||||
|
||||
@@ -17,16 +18,19 @@ config = {
|
||||
'description': 'Runs calendar sync automatically every 15 minutes',
|
||||
'flows': ['advoware-calendar-sync'],
|
||||
'triggers': [
|
||||
cron("0 */15 * * * *") # Every 15 minutes (6-field: sec min hour day month weekday)
|
||||
cron("0 15 1 * * *") # Every 15 minutes at second 0 (6-field: sec min hour day month weekday)
|
||||
],
|
||||
'enqueues': ['calendar_sync_all']
|
||||
}
|
||||
|
||||
|
||||
async def handler(input_data: dict, ctx: FlowContext):
|
||||
async def handler(input_data: None, ctx: FlowContext) -> None:
|
||||
"""Cron handler that triggers the calendar sync cascade."""
|
||||
try:
|
||||
log_operation('info', "Calendar Sync Cron: Starting to emit sync-all event", context=ctx)
|
||||
ctx.logger.info("=" * 80)
|
||||
ctx.logger.info("🕐 CALENDAR SYNC CRON: STARTING")
|
||||
ctx.logger.info("=" * 80)
|
||||
ctx.logger.info("Emitting sync-all event")
|
||||
|
||||
# Enqueue sync-all event
|
||||
await ctx.enqueue({
|
||||
@@ -36,15 +40,11 @@ async def handler(input_data: dict, ctx: FlowContext):
|
||||
}
|
||||
})
|
||||
|
||||
log_operation('info', "Calendar Sync Cron: Emitted sync-all event", context=ctx)
|
||||
return {
|
||||
'status': 'completed',
|
||||
'triggered_by': 'cron'
|
||||
}
|
||||
ctx.logger.info("✅ Calendar sync-all event emitted successfully")
|
||||
ctx.logger.info("=" * 80)
|
||||
|
||||
except Exception as e:
|
||||
log_operation('error', f"Fehler beim Cron-Job: {e}", context=ctx)
|
||||
return {
|
||||
'status': 'error',
|
||||
'error': str(e)
|
||||
}
|
||||
ctx.logger.error("=" * 80)
|
||||
ctx.logger.error("❌ ERROR: CALENDAR SYNC CRON")
|
||||
ctx.logger.error(f"Error: {e}")
|
||||
ctx.logger.error("=" * 80)
|
||||
@@ -14,6 +14,7 @@ import asyncio
|
||||
import os
|
||||
import datetime
|
||||
from datetime import timedelta
|
||||
from typing import Dict, Any
|
||||
import pytz
|
||||
import backoff
|
||||
import time
|
||||
@@ -64,7 +65,8 @@ async def enforce_global_rate_limit(context=None):
|
||||
socket_timeout=int(os.getenv('REDIS_TIMEOUT_SECONDS', '5'))
|
||||
)
|
||||
|
||||
lua_script = """
|
||||
try:
|
||||
lua_script = """
|
||||
local key = KEYS[1]
|
||||
local current_time_ms = tonumber(ARGV[1])
|
||||
local max_tokens = tonumber(ARGV[2])
|
||||
@@ -96,7 +98,6 @@ async def enforce_global_rate_limit(context=None):
|
||||
end
|
||||
"""
|
||||
|
||||
try:
|
||||
script = redis_client.register_script(lua_script)
|
||||
|
||||
while True:
|
||||
@@ -120,6 +121,12 @@ async def enforce_global_rate_limit(context=None):
|
||||
|
||||
except Exception as e:
|
||||
log_operation('error', f"Rate limiting failed: {e}. Proceeding without limit.", context=context)
|
||||
finally:
|
||||
# Always close Redis connection to prevent resource leaks
|
||||
try:
|
||||
redis_client.close()
|
||||
except Exception:
|
||||
pass
|
||||
|
||||
|
||||
@backoff.on_exception(backoff.expo, HttpError, max_tries=4, base=3,
|
||||
@@ -945,18 +952,19 @@ config = {
|
||||
}
|
||||
|
||||
|
||||
async def handler(input_data: dict, ctx: FlowContext):
|
||||
async def handler(input_data: Dict[str, Any], ctx: FlowContext) -> None:
|
||||
"""Main event handler for calendar sync."""
|
||||
start_time = time.time()
|
||||
|
||||
kuerzel = input_data.get('kuerzel')
|
||||
if not kuerzel:
|
||||
log_operation('error', "No kuerzel provided in event", context=ctx)
|
||||
return {'status': 400, 'body': {'error': 'No kuerzel provided'}}
|
||||
return
|
||||
|
||||
log_operation('info', f"Starting calendar sync for employee {kuerzel}", context=ctx)
|
||||
|
||||
redis_client = get_redis_client(ctx)
|
||||
service = None
|
||||
|
||||
try:
|
||||
log_operation('debug', "Initializing Advoware service", context=ctx)
|
||||
@@ -1052,6 +1060,19 @@ async def handler(input_data: dict, ctx: FlowContext):
|
||||
log_operation('error', f"Sync failed for {kuerzel}: {e}", context=ctx)
|
||||
log_operation('info', f"Handler duration (failed): {time.time() - start_time}", context=ctx)
|
||||
return {'status': 500, 'body': {'error': str(e)}}
|
||||
|
||||
finally:
|
||||
# Always close resources to prevent memory leaks
|
||||
if service is not None:
|
||||
try:
|
||||
service.close()
|
||||
except Exception as e:
|
||||
log_operation('debug', f"Error closing Google service: {e}", context=ctx)
|
||||
|
||||
try:
|
||||
redis_client.close()
|
||||
except Exception as e:
|
||||
log_operation('debug', f"Error closing Redis client: {e}", context=ctx)
|
||||
|
||||
# Ensure lock is always released
|
||||
clear_employee_lock(redis_client, kuerzel, ctx)
|
||||
@@ -3,50 +3,44 @@ Calendar Sync Utilities
|
||||
|
||||
Shared utility functions for calendar synchronization between Google Calendar and Advoware.
|
||||
"""
|
||||
import logging
|
||||
import asyncpg
|
||||
import os
|
||||
import redis
|
||||
import time
|
||||
from typing import Optional, Any, List
|
||||
from googleapiclient.discovery import build
|
||||
from google.oauth2 import service_account
|
||||
|
||||
# Configure logging
|
||||
logger = logging.getLogger(__name__)
|
||||
from services.logging_utils import get_service_logger
|
||||
|
||||
|
||||
def log_operation(level: str, message: str, context=None, **context_vars):
|
||||
"""Centralized logging with context, supporting file and console logging."""
|
||||
context_str = ' '.join(f"{k}={v}" for k, v in context_vars.items() if v is not None)
|
||||
full_message = f"{message} {context_str}".strip()
|
||||
def get_logger(context=None):
|
||||
"""Get logger for calendar sync operations"""
|
||||
return get_service_logger('calendar_sync', context)
|
||||
|
||||
# Use ctx.logger if context is available (Motia III FlowContext)
|
||||
if context and hasattr(context, 'logger'):
|
||||
if level == 'info':
|
||||
context.logger.info(full_message)
|
||||
elif level == 'warning':
|
||||
context.logger.warning(full_message)
|
||||
elif level == 'error':
|
||||
context.logger.error(full_message)
|
||||
elif level == 'debug':
|
||||
context.logger.debug(full_message)
|
||||
|
||||
def log_operation(level: str, message: str, context=None, **extra):
|
||||
"""
|
||||
Log calendar sync operations with structured context.
|
||||
|
||||
Args:
|
||||
level: Log level ('debug', 'info', 'warning', 'error')
|
||||
message: Log message
|
||||
context: FlowContext if available
|
||||
**extra: Additional key-value pairs to log
|
||||
"""
|
||||
logger = get_logger(context)
|
||||
log_func = getattr(logger, level.lower(), logger.info)
|
||||
|
||||
if extra:
|
||||
extra_str = " | " + " | ".join(f"{k}={v}" for k, v in extra.items())
|
||||
log_func(message + extra_str)
|
||||
else:
|
||||
# Fallback to standard logger
|
||||
if level == 'info':
|
||||
logger.info(full_message)
|
||||
elif level == 'warning':
|
||||
logger.warning(full_message)
|
||||
elif level == 'error':
|
||||
logger.error(full_message)
|
||||
elif level == 'debug':
|
||||
logger.debug(full_message)
|
||||
|
||||
# Also log to console for journalctl visibility
|
||||
print(f"[{level.upper()}] {full_message}")
|
||||
log_func(message)
|
||||
|
||||
|
||||
async def connect_db(context=None):
|
||||
"""Connect to Postgres DB from environment variables."""
|
||||
logger = get_logger(context)
|
||||
try:
|
||||
conn = await asyncpg.connect(
|
||||
host=os.getenv('POSTGRES_HOST', 'localhost'),
|
||||
@@ -57,12 +51,13 @@ async def connect_db(context=None):
|
||||
)
|
||||
return conn
|
||||
except Exception as e:
|
||||
log_operation('error', f"Failed to connect to DB: {e}", context=context)
|
||||
logger.error(f"Failed to connect to DB: {e}")
|
||||
raise
|
||||
|
||||
|
||||
async def get_google_service(context=None):
|
||||
"""Initialize Google Calendar service."""
|
||||
logger = get_logger(context)
|
||||
try:
|
||||
service_account_path = os.getenv('GOOGLE_CALENDAR_SERVICE_ACCOUNT_PATH', 'service-account.json')
|
||||
if not os.path.exists(service_account_path):
|
||||
@@ -75,48 +70,53 @@ async def get_google_service(context=None):
|
||||
service = build('calendar', 'v3', credentials=creds)
|
||||
return service
|
||||
except Exception as e:
|
||||
log_operation('error', f"Failed to initialize Google service: {e}", context=context)
|
||||
logger.error(f"Failed to initialize Google service: {e}")
|
||||
raise
|
||||
|
||||
|
||||
def get_redis_client(context=None):
|
||||
def get_redis_client(context=None) -> redis.Redis:
|
||||
"""Initialize Redis client for calendar sync operations."""
|
||||
logger = get_logger(context)
|
||||
try:
|
||||
redis_client = redis.Redis(
|
||||
host=os.getenv('REDIS_HOST', 'localhost'),
|
||||
port=int(os.getenv('REDIS_PORT', '6379')),
|
||||
db=int(os.getenv('REDIS_DB_CALENDAR_SYNC', '2')),
|
||||
socket_timeout=int(os.getenv('REDIS_TIMEOUT_SECONDS', '5'))
|
||||
socket_timeout=int(os.getenv('REDIS_TIMEOUT_SECONDS', '5')),
|
||||
decode_responses=True
|
||||
)
|
||||
return redis_client
|
||||
except Exception as e:
|
||||
log_operation('error', f"Failed to initialize Redis client: {e}", context=context)
|
||||
logger.error(f"Failed to initialize Redis client: {e}")
|
||||
raise
|
||||
|
||||
|
||||
async def get_advoware_employees(advoware, context=None):
|
||||
async def get_advoware_employees(advoware, context=None) -> List[Any]:
|
||||
"""Fetch list of employees from Advoware."""
|
||||
logger = get_logger(context)
|
||||
try:
|
||||
result = await advoware.api_call('api/v1/advonet/Mitarbeiter', method='GET', params={'aktiv': 'true'})
|
||||
employees = result if isinstance(result, list) else []
|
||||
log_operation('info', f"Fetched {len(employees)} Advoware employees", context=context)
|
||||
logger.info(f"Fetched {len(employees)} Advoware employees")
|
||||
return employees
|
||||
except Exception as e:
|
||||
log_operation('error', f"Failed to fetch Advoware employees: {e}", context=context)
|
||||
logger.error(f"Failed to fetch Advoware employees: {e}")
|
||||
raise
|
||||
|
||||
|
||||
def set_employee_lock(redis_client, kuerzel: str, triggered_by: str, context=None) -> bool:
|
||||
def set_employee_lock(redis_client: redis.Redis, kuerzel: str, triggered_by: str, context=None) -> bool:
|
||||
"""Set lock for employee sync operation."""
|
||||
logger = get_logger(context)
|
||||
employee_lock_key = f'calendar_sync_lock_{kuerzel}'
|
||||
if redis_client.set(employee_lock_key, triggered_by, ex=1800, nx=True) is None:
|
||||
log_operation('info', f"Sync already active for {kuerzel}, skipping", context=context)
|
||||
logger.info(f"Sync already active for {kuerzel}, skipping")
|
||||
return False
|
||||
return True
|
||||
|
||||
|
||||
def clear_employee_lock(redis_client, kuerzel: str, context=None):
|
||||
def clear_employee_lock(redis_client: redis.Redis, kuerzel: str, context=None) -> None:
|
||||
"""Clear lock for employee sync operation and update last-synced timestamp."""
|
||||
logger = get_logger(context)
|
||||
try:
|
||||
employee_lock_key = f'calendar_sync_lock_{kuerzel}'
|
||||
employee_last_synced_key = f'calendar_sync_last_synced_{kuerzel}'
|
||||
@@ -128,6 +128,6 @@ def clear_employee_lock(redis_client, kuerzel: str, context=None):
|
||||
# Delete the lock
|
||||
redis_client.delete(employee_lock_key)
|
||||
|
||||
log_operation('debug', f"Cleared lock and updated last-synced for {kuerzel} to {current_time}", context=context)
|
||||
logger.debug(f"Cleared lock and updated last-synced for {kuerzel} to {current_time}")
|
||||
except Exception as e:
|
||||
log_operation('warning', f"Failed to clear lock and update last-synced for {kuerzel}: {e}", context=context)
|
||||
logger.warning(f"Failed to clear lock and update last-synced for {kuerzel}: {e}")
|
||||
@@ -32,23 +32,33 @@ async def handler(request: ApiRequest, ctx: FlowContext[Any]) -> ApiResponse:
|
||||
body={'error': 'Endpoint required as query parameter'}
|
||||
)
|
||||
|
||||
ctx.logger.info("=" * 80)
|
||||
ctx.logger.info("🔄 ADVOWARE PROXY: DELETE REQUEST")
|
||||
ctx.logger.info("=" * 80)
|
||||
ctx.logger.info(f"Endpoint: {endpoint}")
|
||||
ctx.logger.info("=" * 80)
|
||||
|
||||
# Initialize Advoware client
|
||||
advoware = AdvowareAPI(ctx)
|
||||
|
||||
# Forward all query params except 'endpoint'
|
||||
params = {k: v for k, v in request.query_params.items() if k != 'endpoint'}
|
||||
|
||||
ctx.logger.info(f"Proxying DELETE request to Advoware: {endpoint}")
|
||||
result = await advoware.api_call(
|
||||
endpoint,
|
||||
method='DELETE',
|
||||
params=params
|
||||
)
|
||||
|
||||
ctx.logger.info("✅ Proxy DELETE erfolgreich")
|
||||
return ApiResponse(status=200, body={'result': result})
|
||||
|
||||
except Exception as e:
|
||||
ctx.logger.error(f"Proxy error: {e}")
|
||||
ctx.logger.error("=" * 80)
|
||||
ctx.logger.error("❌ ADVOWARE PROXY DELETE FEHLER")
|
||||
ctx.logger.error(f"Endpoint: {request.query_params.get('endpoint', 'N/A')}")
|
||||
ctx.logger.error(f"Error: {e}")
|
||||
ctx.logger.error("=" * 80)
|
||||
return ApiResponse(
|
||||
status=500,
|
||||
body={'error': 'Internal server error', 'details': str(e)}
|
||||
@@ -32,23 +32,33 @@ async def handler(request: ApiRequest, ctx: FlowContext[Any]) -> ApiResponse:
|
||||
body={'error': 'Endpoint required as query parameter'}
|
||||
)
|
||||
|
||||
ctx.logger.info("=" * 80)
|
||||
ctx.logger.info("🔄 ADVOWARE PROXY: GET REQUEST")
|
||||
ctx.logger.info("=" * 80)
|
||||
ctx.logger.info(f"Endpoint: {endpoint}")
|
||||
ctx.logger.info("=" * 80)
|
||||
|
||||
# Initialize Advoware client
|
||||
advoware = AdvowareAPI(ctx)
|
||||
|
||||
# Forward all query params except 'endpoint'
|
||||
params = {k: v for k, v in request.query_params.items() if k != 'endpoint'}
|
||||
|
||||
ctx.logger.info(f"Proxying GET request to Advoware: {endpoint}")
|
||||
result = await advoware.api_call(
|
||||
endpoint,
|
||||
method='GET',
|
||||
params=params
|
||||
)
|
||||
|
||||
ctx.logger.info("✅ Proxy GET erfolgreich")
|
||||
return ApiResponse(status=200, body={'result': result})
|
||||
|
||||
except Exception as e:
|
||||
ctx.logger.error(f"Proxy error: {e}")
|
||||
ctx.logger.error("=" * 80)
|
||||
ctx.logger.error("❌ ADVOWARE PROXY GET FEHLER")
|
||||
ctx.logger.error(f"Endpoint: {request.query_params.get('endpoint', 'N/A')}")
|
||||
ctx.logger.error(f"Error: {e}")
|
||||
ctx.logger.error("=" * 80)
|
||||
return ApiResponse(
|
||||
status=500,
|
||||
body={'error': 'Internal server error', 'details': str(e)}
|
||||
@@ -34,6 +34,12 @@ async def handler(request: ApiRequest, ctx: FlowContext[Any]) -> ApiResponse:
|
||||
body={'error': 'Endpoint required as query parameter'}
|
||||
)
|
||||
|
||||
ctx.logger.info("=" * 80)
|
||||
ctx.logger.info("🔄 ADVOWARE PROXY: POST REQUEST")
|
||||
ctx.logger.info("=" * 80)
|
||||
ctx.logger.info(f"Endpoint: {endpoint}")
|
||||
ctx.logger.info("=" * 80)
|
||||
|
||||
# Initialize Advoware client
|
||||
advoware = AdvowareAPI(ctx)
|
||||
|
||||
@@ -43,7 +49,6 @@ async def handler(request: ApiRequest, ctx: FlowContext[Any]) -> ApiResponse:
|
||||
# Get request body
|
||||
json_data = request.body
|
||||
|
||||
ctx.logger.info(f"Proxying POST request to Advoware: {endpoint}")
|
||||
result = await advoware.api_call(
|
||||
endpoint,
|
||||
method='POST',
|
||||
@@ -51,10 +56,15 @@ async def handler(request: ApiRequest, ctx: FlowContext[Any]) -> ApiResponse:
|
||||
json_data=json_data
|
||||
)
|
||||
|
||||
ctx.logger.info("✅ Proxy POST erfolgreich")
|
||||
return ApiResponse(status=200, body={'result': result})
|
||||
|
||||
except Exception as e:
|
||||
ctx.logger.error(f"Proxy error: {e}")
|
||||
ctx.logger.error("=" * 80)
|
||||
ctx.logger.error("❌ ADVOWARE PROXY POST FEHLER")
|
||||
ctx.logger.error(f"Endpoint: {request.query_params.get('endpoint', 'N/A')}")
|
||||
ctx.logger.error(f"Error: {e}")
|
||||
ctx.logger.error("=" * 80)
|
||||
return ApiResponse(
|
||||
status=500,
|
||||
body={'error': 'Internal server error', 'details': str(e)}
|
||||
@@ -34,6 +34,12 @@ async def handler(request: ApiRequest, ctx: FlowContext[Any]) -> ApiResponse:
|
||||
body={'error': 'Endpoint required as query parameter'}
|
||||
)
|
||||
|
||||
ctx.logger.info("=" * 80)
|
||||
ctx.logger.info("🔄 ADVOWARE PROXY: PUT REQUEST")
|
||||
ctx.logger.info("=" * 80)
|
||||
ctx.logger.info(f"Endpoint: {endpoint}")
|
||||
ctx.logger.info("=" * 80)
|
||||
|
||||
# Initialize Advoware client
|
||||
advoware = AdvowareAPI(ctx)
|
||||
|
||||
@@ -43,7 +49,6 @@ async def handler(request: ApiRequest, ctx: FlowContext[Any]) -> ApiResponse:
|
||||
# Get request body
|
||||
json_data = request.body
|
||||
|
||||
ctx.logger.info(f"Proxying PUT request to Advoware: {endpoint}")
|
||||
result = await advoware.api_call(
|
||||
endpoint,
|
||||
method='PUT',
|
||||
@@ -51,10 +56,15 @@ async def handler(request: ApiRequest, ctx: FlowContext[Any]) -> ApiResponse:
|
||||
json_data=json_data
|
||||
)
|
||||
|
||||
ctx.logger.info("✅ Proxy PUT erfolgreich")
|
||||
return ApiResponse(status=200, body={'result': result})
|
||||
|
||||
except Exception as e:
|
||||
ctx.logger.error(f"Proxy error: {e}")
|
||||
ctx.logger.error("=" * 80)
|
||||
ctx.logger.error("❌ ADVOWARE PROXY PUT FEHLER")
|
||||
ctx.logger.error(f"Endpoint: {request.query_params.get('endpoint', 'N/A')}")
|
||||
ctx.logger.error(f"Error: {e}")
|
||||
ctx.logger.error("=" * 80)
|
||||
return ApiResponse(
|
||||
status=500,
|
||||
body={'error': 'Internal server error', 'details': str(e)}
|
||||
90
src/steps/vmh/aiknowledge_full_sync_cron_step.py
Normal file
90
src/steps/vmh/aiknowledge_full_sync_cron_step.py
Normal file
@@ -0,0 +1,90 @@
|
||||
"""AI Knowledge Daily Sync - Cron Job"""
|
||||
from typing import Any
|
||||
from motia import FlowContext, cron
|
||||
|
||||
|
||||
config = {
|
||||
"name": "AI Knowledge Daily Sync",
|
||||
"description": "Daily sync of all CAIKnowledge entities (catches missed webhooks, Blake3 verification included)",
|
||||
"flows": ["aiknowledge-full-sync"],
|
||||
"triggers": [
|
||||
cron("0 0 2 * * *"), # Daily at 2:00 AM
|
||||
],
|
||||
"enqueues": ["aiknowledge.sync"],
|
||||
}
|
||||
|
||||
|
||||
async def handler(input_data: None, ctx: FlowContext[Any]) -> None:
|
||||
"""
|
||||
Daily sync handler - ensures all active knowledge bases are synchronized.
|
||||
|
||||
Loads all CAIKnowledge entities that need sync and emits events.
|
||||
Blake3 hash verification is always performed (hash available from JunctionData API).
|
||||
Runs every day at 02:00:00.
|
||||
"""
|
||||
from services.espocrm import EspoCRMAPI
|
||||
from services.models import AIKnowledgeActivationStatus, AIKnowledgeSyncStatus
|
||||
|
||||
ctx.logger.info("=" * 80)
|
||||
ctx.logger.info("🌙 DAILY AI KNOWLEDGE SYNC STARTED")
|
||||
ctx.logger.info("=" * 80)
|
||||
|
||||
espocrm = EspoCRMAPI(ctx)
|
||||
|
||||
try:
|
||||
# Load all CAIKnowledge entities with status 'active' that need sync
|
||||
result = await espocrm.list_entities(
|
||||
'CAIKnowledge',
|
||||
where=[
|
||||
{
|
||||
'type': 'equals',
|
||||
'attribute': 'aktivierungsstatus',
|
||||
'value': AIKnowledgeActivationStatus.ACTIVE.value
|
||||
},
|
||||
{
|
||||
'type': 'in',
|
||||
'attribute': 'syncStatus',
|
||||
'value': [
|
||||
AIKnowledgeSyncStatus.UNCLEAN.value,
|
||||
AIKnowledgeSyncStatus.FAILED.value
|
||||
]
|
||||
}
|
||||
],
|
||||
select='id,name,syncStatus',
|
||||
max_size=1000 # Adjust if you have more
|
||||
)
|
||||
|
||||
entities = result.get('list', [])
|
||||
total = len(entities)
|
||||
|
||||
ctx.logger.info(f"📊 Found {total} knowledge bases needing sync")
|
||||
|
||||
if total == 0:
|
||||
ctx.logger.info("✅ All knowledge bases are synced")
|
||||
ctx.logger.info("=" * 80)
|
||||
return
|
||||
|
||||
# Enqueue sync events for all (Blake3 verification always enabled)
|
||||
for i, entity in enumerate(entities, 1):
|
||||
await ctx.enqueue({
|
||||
'topic': 'aiknowledge.sync',
|
||||
'data': {
|
||||
'knowledge_id': entity['id'],
|
||||
'source': 'daily_cron'
|
||||
}
|
||||
})
|
||||
ctx.logger.info(
|
||||
f"📤 [{i}/{total}] Enqueued: {entity['name']} "
|
||||
f"(syncStatus={entity.get('syncStatus')})"
|
||||
)
|
||||
|
||||
ctx.logger.info("=" * 80)
|
||||
ctx.logger.info(f"✅ Daily sync complete: {total} events enqueued")
|
||||
ctx.logger.info("=" * 80)
|
||||
|
||||
except Exception as e:
|
||||
ctx.logger.error("=" * 80)
|
||||
ctx.logger.error("❌ FULL SYNC FAILED")
|
||||
ctx.logger.error("=" * 80)
|
||||
ctx.logger.error(f"Error: {e}", exc_info=True)
|
||||
raise
|
||||
89
src/steps/vmh/aiknowledge_sync_event_step.py
Normal file
89
src/steps/vmh/aiknowledge_sync_event_step.py
Normal file
@@ -0,0 +1,89 @@
|
||||
"""AI Knowledge Sync Event Handler"""
|
||||
from typing import Dict, Any
|
||||
from redis import Redis
|
||||
from motia import FlowContext, queue
|
||||
|
||||
|
||||
config = {
|
||||
"name": "AI Knowledge Sync",
|
||||
"description": "Synchronizes CAIKnowledge entities with XAI Collections",
|
||||
"flows": ["vmh-aiknowledge"],
|
||||
"triggers": [
|
||||
queue("aiknowledge.sync")
|
||||
],
|
||||
}
|
||||
|
||||
|
||||
async def handler(event_data: Dict[str, Any], ctx: FlowContext[Any]) -> None:
|
||||
"""
|
||||
Event handler for AI Knowledge synchronization.
|
||||
|
||||
Emitted by:
|
||||
- Webhook on CAIKnowledge update
|
||||
- Daily full sync cron job
|
||||
|
||||
Args:
|
||||
event_data: Event payload with knowledge_id
|
||||
ctx: Motia context
|
||||
"""
|
||||
from services.redis_client import RedisClientFactory
|
||||
from services.aiknowledge_sync_utils import AIKnowledgeSync
|
||||
|
||||
ctx.logger.info("=" * 80)
|
||||
ctx.logger.info("🔄 AI KNOWLEDGE SYNC STARTED")
|
||||
ctx.logger.info("=" * 80)
|
||||
|
||||
# Extract data
|
||||
knowledge_id = event_data.get('knowledge_id')
|
||||
source = event_data.get('source', 'unknown')
|
||||
|
||||
if not knowledge_id:
|
||||
ctx.logger.error("❌ Missing knowledge_id in event data")
|
||||
return
|
||||
|
||||
ctx.logger.info(f"📋 Knowledge ID: {knowledge_id}")
|
||||
ctx.logger.info(f"📋 Source: {source}")
|
||||
ctx.logger.info("=" * 80)
|
||||
|
||||
# Get Redis for locking
|
||||
redis_client = RedisClientFactory.get_client(strict=False)
|
||||
|
||||
# Initialize sync utils
|
||||
sync_utils = AIKnowledgeSync(ctx, redis_client)
|
||||
|
||||
# Acquire lock
|
||||
lock_acquired = await sync_utils.acquire_sync_lock(knowledge_id)
|
||||
|
||||
if not lock_acquired:
|
||||
ctx.logger.warn(f"⏸️ Lock already held for {knowledge_id}, skipping")
|
||||
ctx.logger.info(" (Will be retried by Motia queue)")
|
||||
raise RuntimeError(f"Lock busy for {knowledge_id}") # Motia will retry
|
||||
|
||||
try:
|
||||
# Perform sync (Blake3 hash verification always enabled)
|
||||
await sync_utils.sync_knowledge_to_xai(knowledge_id, ctx)
|
||||
|
||||
ctx.logger.info("=" * 80)
|
||||
ctx.logger.info("✅ AI KNOWLEDGE SYNC COMPLETED")
|
||||
ctx.logger.info("=" * 80)
|
||||
|
||||
# Release lock with success=True
|
||||
await sync_utils.release_sync_lock(knowledge_id, success=True)
|
||||
|
||||
except Exception as e:
|
||||
ctx.logger.error("=" * 80)
|
||||
ctx.logger.error("❌ AI KNOWLEDGE SYNC FAILED")
|
||||
ctx.logger.error("=" * 80)
|
||||
ctx.logger.error(f"Error: {e}")
|
||||
ctx.logger.error(f"Knowledge ID: {knowledge_id}")
|
||||
ctx.logger.error("=" * 80)
|
||||
|
||||
# Release lock with failure
|
||||
await sync_utils.release_sync_lock(
|
||||
knowledge_id,
|
||||
success=False,
|
||||
error_message=str(e)
|
||||
)
|
||||
|
||||
# Re-raise to let Motia retry
|
||||
raise
|
||||
@@ -11,30 +11,29 @@ Verarbeitet:
|
||||
"""
|
||||
|
||||
from typing import Dict, Any, Optional
|
||||
from motia import FlowContext
|
||||
from motia import FlowContext, queue
|
||||
from services.advoware import AdvowareAPI
|
||||
from services.espocrm import EspoCRMAPI
|
||||
from services.bankverbindungen_mapper import BankverbindungenMapper
|
||||
from services.notification_utils import NotificationManager
|
||||
from services.redis_client import get_redis_client
|
||||
import json
|
||||
import redis
|
||||
import os
|
||||
|
||||
config = {
|
||||
"name": "VMH Bankverbindungen Sync Handler",
|
||||
"description": "Zentraler Sync-Handler für Bankverbindungen (Webhooks + Cron Events)",
|
||||
"flows": ["vmh-bankverbindungen"],
|
||||
"triggers": [
|
||||
{"type": "queue", "topic": "vmh.bankverbindungen.create"},
|
||||
{"type": "queue", "topic": "vmh.bankverbindungen.update"},
|
||||
{"type": "queue", "topic": "vmh.bankverbindungen.delete"},
|
||||
{"type": "queue", "topic": "vmh.bankverbindungen.sync_check"}
|
||||
queue("vmh.bankverbindungen.create"),
|
||||
queue("vmh.bankverbindungen.update"),
|
||||
queue("vmh.bankverbindungen.delete"),
|
||||
queue("vmh.bankverbindungen.sync_check")
|
||||
],
|
||||
"enqueues": []
|
||||
}
|
||||
|
||||
|
||||
async def handler(event_data: Dict[str, Any], ctx: FlowContext[Any]):
|
||||
async def handler(event_data: Dict[str, Any], ctx: FlowContext[Any]) -> None:
|
||||
"""Zentraler Sync-Handler für Bankverbindungen"""
|
||||
|
||||
entity_id = event_data.get('entity_id')
|
||||
@@ -47,20 +46,11 @@ async def handler(event_data: Dict[str, Any], ctx: FlowContext[Any]):
|
||||
|
||||
ctx.logger.info(f"🔄 Bankverbindungen Sync gestartet: {action.upper()} | Entity: {entity_id} | Source: {source}")
|
||||
|
||||
# Shared Redis client
|
||||
redis_host = os.getenv('REDIS_HOST', 'localhost')
|
||||
redis_port = int(os.getenv('REDIS_PORT', '6379'))
|
||||
redis_db = int(os.getenv('REDIS_DB_ADVOWARE_CACHE', '1'))
|
||||
# Shared Redis client (centralized factory)
|
||||
redis_client = get_redis_client(strict=False)
|
||||
|
||||
redis_client = redis.Redis(
|
||||
host=redis_host,
|
||||
port=redis_port,
|
||||
db=redis_db,
|
||||
decode_responses=True
|
||||
)
|
||||
|
||||
# APIs initialisieren
|
||||
espocrm = EspoCRMAPI()
|
||||
# APIs initialisieren (mit Context für besseres Logging)
|
||||
espocrm = EspoCRMAPI(ctx)
|
||||
advoware = AdvowareAPI(ctx)
|
||||
mapper = BankverbindungenMapper()
|
||||
notification_mgr = NotificationManager(espocrm_api=espocrm, context=ctx)
|
||||
@@ -130,7 +120,7 @@ async def handler(event_data: Dict[str, Any], ctx: FlowContext[Any]):
|
||||
pass
|
||||
|
||||
|
||||
async def handle_create(entity_id, betnr, espo_entity, espocrm, advoware, mapper, ctx, redis_client, lock_key):
|
||||
async def handle_create(entity_id, betnr, espo_entity, espocrm, advoware, mapper, ctx, redis_client, lock_key) -> None:
|
||||
"""Erstellt neue Bankverbindung in Advoware"""
|
||||
try:
|
||||
ctx.logger.info(f"🔨 CREATE Bankverbindung in Advoware für Beteiligter {betnr}...")
|
||||
@@ -176,7 +166,7 @@ async def handle_create(entity_id, betnr, espo_entity, espocrm, advoware, mapper
|
||||
redis_client.delete(lock_key)
|
||||
|
||||
|
||||
async def handle_update(entity_id, betnr, advoware_id, espo_entity, espocrm, notification_mgr, ctx, redis_client, lock_key):
|
||||
async def handle_update(entity_id, betnr, advoware_id, espo_entity, espocrm, notification_mgr, ctx, redis_client, lock_key) -> None:
|
||||
"""Update nicht möglich - Sendet Notification an User"""
|
||||
try:
|
||||
ctx.logger.warn(f"⚠️ UPDATE: Advoware API unterstützt kein PUT für Bankverbindungen")
|
||||
@@ -219,7 +209,7 @@ async def handle_update(entity_id, betnr, advoware_id, espo_entity, espocrm, not
|
||||
redis_client.delete(lock_key)
|
||||
|
||||
|
||||
async def handle_delete(entity_id, betnr, advoware_id, espo_entity, espocrm, notification_mgr, ctx, redis_client, lock_key):
|
||||
async def handle_delete(entity_id, betnr, advoware_id, espo_entity, espocrm, notification_mgr, ctx, redis_client, lock_key) -> None:
|
||||
"""Delete nicht möglich - Sendet Notification an User"""
|
||||
try:
|
||||
ctx.logger.warn(f"⚠️ DELETE: Advoware API unterstützt kein DELETE für Bankverbindungen")
|
||||
@@ -19,20 +19,20 @@ config = {
|
||||
"description": "Prüft alle 15 Minuten welche Beteiligte synchronisiert werden müssen",
|
||||
"flows": ["vmh-beteiligte"],
|
||||
"triggers": [
|
||||
cron("0 */15 * * * *") # Alle 15 Minuten (6-field format!)
|
||||
cron("0 */15 1 * * *") # Alle 15 Minuten (6-field format!)
|
||||
],
|
||||
"enqueues": ["vmh.beteiligte.sync_check"]
|
||||
}
|
||||
|
||||
|
||||
async def handler(input_data: Dict[str, Any], ctx: FlowContext):
|
||||
async def handler(input_data: Dict[str, Any], ctx: FlowContext) -> None:
|
||||
"""
|
||||
Cron-Handler: Findet alle Beteiligte die Sync benötigen und emittiert Events
|
||||
"""
|
||||
ctx.logger.info("🕐 Beteiligte Sync Cron gestartet")
|
||||
|
||||
try:
|
||||
espocrm = EspoCRMAPI()
|
||||
espocrm = EspoCRMAPI(ctx)
|
||||
|
||||
# Berechne Threshold für "veraltete" Syncs (24 Stunden)
|
||||
threshold = datetime.datetime.now() - datetime.timedelta(hours=24)
|
||||
@@ -54,7 +54,7 @@ async def handler(input_data: Dict[str, Any], ctx: FlowContext):
|
||||
]
|
||||
}
|
||||
|
||||
unclean_result = await espocrm.search_entities('CBeteiligte', unclean_filter, max_size=100)
|
||||
unclean_result = await espocrm.list_entities('CBeteiligte', where=unclean_filter['where'], max_size=100)
|
||||
unclean_entities = unclean_result.get('list', [])
|
||||
|
||||
ctx.logger.info(f"📊 Gefunden: {len(unclean_entities)} Entities mit Status pending/dirty/failed")
|
||||
@@ -73,7 +73,7 @@ async def handler(input_data: Dict[str, Any], ctx: FlowContext):
|
||||
]
|
||||
}
|
||||
|
||||
reset_result = await espocrm.search_entities('CBeteiligte', permanently_failed_filter, max_size=50)
|
||||
reset_result = await espocrm.list_entities('CBeteiligte', where=permanently_failed_filter['where'], max_size=50)
|
||||
reset_entities = reset_result.get('list', [])
|
||||
|
||||
# Reset permanently_failed entities
|
||||
@@ -111,7 +111,7 @@ async def handler(input_data: Dict[str, Any], ctx: FlowContext):
|
||||
]
|
||||
}
|
||||
|
||||
stale_result = await espocrm.search_entities('CBeteiligte', stale_filter, max_size=50)
|
||||
stale_result = await espocrm.list_entities('CBeteiligte', where=stale_filter['where'], max_size=50)
|
||||
stale_entities = stale_result.get('list', [])
|
||||
|
||||
ctx.logger.info(f"📊 Gefunden: {len(stale_entities)} Entities mit veraltetem Sync (> 24h)")
|
||||
@@ -11,56 +11,66 @@ Verarbeitet:
|
||||
"""
|
||||
|
||||
from typing import Dict, Any, Optional
|
||||
from motia import FlowContext
|
||||
from motia import FlowContext, queue
|
||||
from services.advoware import AdvowareAPI
|
||||
from services.advoware_service import AdvowareService
|
||||
from services.espocrm import EspoCRMAPI
|
||||
from services.espocrm_mapper import BeteiligteMapper
|
||||
from services.beteiligte_sync_utils import BeteiligteSync
|
||||
from services.redis_client import get_redis_client
|
||||
from services.exceptions import (
|
||||
AdvowareAPIError,
|
||||
EspoCRMAPIError,
|
||||
SyncError,
|
||||
RetryableError,
|
||||
is_retryable
|
||||
)
|
||||
from services.logging_utils import get_step_logger
|
||||
import json
|
||||
import redis
|
||||
import os
|
||||
|
||||
config = {
|
||||
"name": "VMH Beteiligte Sync Handler",
|
||||
"description": "Zentraler Sync-Handler für Beteiligte (Webhooks + Cron Events)",
|
||||
"flows": ["vmh-beteiligte"],
|
||||
"triggers": [
|
||||
{"type": "queue", "topic": "vmh.beteiligte.create"},
|
||||
{"type": "queue", "topic": "vmh.beteiligte.update"},
|
||||
{"type": "queue", "topic": "vmh.beteiligte.delete"},
|
||||
{"type": "queue", "topic": "vmh.beteiligte.sync_check"}
|
||||
queue("vmh.beteiligte.create"),
|
||||
queue("vmh.beteiligte.update"),
|
||||
queue("vmh.beteiligte.delete"),
|
||||
queue("vmh.beteiligte.sync_check")
|
||||
],
|
||||
"enqueues": []
|
||||
}
|
||||
|
||||
|
||||
async def handler(event_data: Dict[str, Any], ctx: FlowContext[Any]):
|
||||
"""Zentraler Sync-Handler für Beteiligte"""
|
||||
entity_id = event_data.entity_id
|
||||
action = event_data.action
|
||||
source = event_data.source
|
||||
async def handler(event_data: Dict[str, Any], ctx: FlowContext[Any]) -> None:
|
||||
"""
|
||||
Zentraler Sync-Handler für Beteiligte
|
||||
|
||||
Args:
|
||||
event_data: Event data mit entity_id, action, source
|
||||
ctx: Motia FlowContext
|
||||
"""
|
||||
entity_id = event_data.get('entity_id')
|
||||
action = event_data.get('action')
|
||||
source = event_data.get('source')
|
||||
|
||||
step_logger = get_step_logger('beteiligte_sync', ctx)
|
||||
|
||||
if not entity_id:
|
||||
ctx.logger.error("Keine entity_id im Event gefunden")
|
||||
step_logger.error("Keine entity_id im Event gefunden")
|
||||
return
|
||||
|
||||
ctx.logger.info(f"🔄 Sync-Handler gestartet: {action.upper()} | Entity: {entity_id} | Source: {source}")
|
||||
step_logger.info("=" * 80)
|
||||
step_logger.info(f"🔄 BETEILIGTE SYNC HANDLER: {action.upper()}")
|
||||
step_logger.info("=" * 80)
|
||||
step_logger.info(f"Entity: {entity_id} | Source: {source}")
|
||||
step_logger.info("=" * 80)
|
||||
|
||||
# Shared Redis client for distributed locking
|
||||
redis_host = os.getenv('REDIS_HOST', 'localhost')
|
||||
redis_port = int(os.getenv('REDIS_PORT', '6379'))
|
||||
redis_db = int(os.getenv('REDIS_DB_ADVOWARE_CACHE', '1'))
|
||||
|
||||
redis_client = redis.Redis(
|
||||
host=redis_host,
|
||||
port=redis_port,
|
||||
db=redis_db,
|
||||
decode_responses=True
|
||||
)
|
||||
# Get shared Redis client (centralized)
|
||||
redis_client = get_redis_client(strict=False)
|
||||
|
||||
# APIs initialisieren
|
||||
espocrm = EspoCRMAPI()
|
||||
espocrm = EspoCRMAPI(ctx)
|
||||
advoware = AdvowareAPI(ctx)
|
||||
sync_utils = BeteiligteSync(espocrm, redis_client, ctx)
|
||||
mapper = BeteiligteMapper()
|
||||
@@ -164,7 +174,7 @@ async def handler(event_data: Dict[str, Any], ctx: FlowContext[Any]):
|
||||
ctx.logger.error(traceback.format_exc())
|
||||
|
||||
|
||||
async def handle_create(entity_id, espo_entity, espocrm, advoware, sync_utils, mapper, ctx):
|
||||
async def handle_create(entity_id, espo_entity, espocrm, advoware, sync_utils, mapper, ctx) -> None:
|
||||
"""Erstellt neuen Beteiligten in Advoware"""
|
||||
try:
|
||||
ctx.logger.info(f"🔨 CREATE in Advoware...")
|
||||
@@ -223,7 +233,7 @@ async def handle_create(entity_id, espo_entity, espocrm, advoware, sync_utils, m
|
||||
await sync_utils.release_sync_lock(entity_id, 'failed', str(e), increment_retry=True)
|
||||
|
||||
|
||||
async def handle_update(entity_id, betnr, espo_entity, espocrm, advoware, sync_utils, mapper, ctx):
|
||||
async def handle_update(entity_id, betnr, espo_entity, espocrm, advoware, sync_utils, mapper, ctx) -> None:
|
||||
"""Synchronisiert existierenden Beteiligten"""
|
||||
try:
|
||||
ctx.logger.info(f"🔍 Fetch von Advoware betNr={betnr}...")
|
||||
394
src/steps/vmh/document_sync_event_step.py
Normal file
394
src/steps/vmh/document_sync_event_step.py
Normal file
@@ -0,0 +1,394 @@
|
||||
"""
|
||||
VMH Document Sync Handler
|
||||
|
||||
Zentraler Sync-Handler für Documents mit xAI Collections
|
||||
|
||||
Verarbeitet:
|
||||
- vmh.document.create: Neu in EspoCRM → Prüfe ob xAI-Sync nötig
|
||||
- vmh.document.update: Geändert in EspoCRM → Prüfe ob xAI-Sync/Update nötig
|
||||
- vmh.document.delete: Gelöscht in EspoCRM → Remove from xAI Collections
|
||||
"""
|
||||
|
||||
from typing import Dict, Any
|
||||
from motia import FlowContext, queue
|
||||
from services.espocrm import EspoCRMAPI
|
||||
from services.document_sync_utils import DocumentSync
|
||||
from services.xai_service import XAIService
|
||||
from services.redis_client import get_redis_client
|
||||
import hashlib
|
||||
import json
|
||||
|
||||
config = {
|
||||
"name": "VMH Document Sync Handler",
|
||||
"description": "Zentraler Sync-Handler für Documents mit xAI Collections",
|
||||
"flows": ["vmh-documents"],
|
||||
"triggers": [
|
||||
queue("vmh.document.create"),
|
||||
queue("vmh.document.update"),
|
||||
queue("vmh.document.delete")
|
||||
],
|
||||
"enqueues": []
|
||||
}
|
||||
|
||||
|
||||
async def handler(event_data: Dict[str, Any], ctx: FlowContext[Any]) -> None:
|
||||
"""Zentraler Sync-Handler für Documents"""
|
||||
entity_id = event_data.get('entity_id')
|
||||
entity_type = event_data.get('entity_type', 'CDokumente') # Default: CDokumente
|
||||
action = event_data.get('action')
|
||||
source = event_data.get('source')
|
||||
|
||||
if not entity_id:
|
||||
ctx.logger.error("Keine entity_id im Event gefunden")
|
||||
return
|
||||
|
||||
ctx.logger.info("=" * 80)
|
||||
ctx.logger.info(f"🔄 DOCUMENT SYNC HANDLER GESTARTET")
|
||||
ctx.logger.info("=" * 80)
|
||||
ctx.logger.info(f"Entity Type: {entity_type}")
|
||||
ctx.logger.info(f"Action: {action.upper()}")
|
||||
ctx.logger.info(f"Document ID: {entity_id}")
|
||||
ctx.logger.info(f"Source: {source}")
|
||||
ctx.logger.info("=" * 80)
|
||||
|
||||
# Shared Redis client for distributed locking (centralized factory)
|
||||
redis_client = get_redis_client(strict=False)
|
||||
|
||||
# APIs initialisieren (mit Context für besseres Logging)
|
||||
espocrm = EspoCRMAPI(ctx)
|
||||
sync_utils = DocumentSync(espocrm, redis_client, ctx)
|
||||
xai_service = XAIService(ctx)
|
||||
|
||||
try:
|
||||
# 1. ACQUIRE LOCK (verhindert parallele Syncs)
|
||||
lock_acquired = await sync_utils.acquire_sync_lock(entity_id, entity_type)
|
||||
|
||||
if not lock_acquired:
|
||||
ctx.logger.warn(f"⏸️ Sync bereits aktiv für {entity_type} {entity_id}, überspringe")
|
||||
return
|
||||
|
||||
# Lock erfolgreich acquired - MUSS im finally block released werden!
|
||||
try:
|
||||
# 2. FETCH VOLLSTÄNDIGES DOCUMENT VON ESPOCRM
|
||||
try:
|
||||
document = await espocrm.get_entity(entity_type, entity_id)
|
||||
except Exception as e:
|
||||
ctx.logger.error(f"❌ Fehler beim Laden von {entity_type}: {e}")
|
||||
await sync_utils.release_sync_lock(entity_id, success=False, error_message=str(e), entity_type=entity_type)
|
||||
return
|
||||
|
||||
ctx.logger.info(f"📋 {entity_type} geladen:")
|
||||
ctx.logger.info(f" Name: {document.get('name', 'N/A')}")
|
||||
ctx.logger.info(f" Type: {document.get('type', 'N/A')}")
|
||||
ctx.logger.info(f" fileStatus: {document.get('fileStatus', 'N/A')}")
|
||||
ctx.logger.info(f" xaiFileId: {document.get('xaiFileId') or document.get('xaiId', 'N/A')}")
|
||||
ctx.logger.info(f" xaiCollections: {document.get('xaiCollections', [])}")
|
||||
|
||||
# 3. BESTIMME SYNC-AKTION BASIEREND AUF ACTION
|
||||
|
||||
if action == 'delete':
|
||||
await handle_delete(entity_id, document, sync_utils, xai_service, ctx, entity_type)
|
||||
|
||||
elif action in ['create', 'update']:
|
||||
await handle_create_or_update(entity_id, document, sync_utils, xai_service, ctx, entity_type)
|
||||
|
||||
else:
|
||||
ctx.logger.warn(f"⚠️ Unbekannte Action: {action}")
|
||||
await sync_utils.release_sync_lock(entity_id, success=False, error_message=f"Unbekannte Action: {action}", entity_type=entity_type)
|
||||
|
||||
except Exception as e:
|
||||
# Unerwarteter Fehler während Sync - GARANTIERE Lock-Release
|
||||
ctx.logger.error(f"❌ Unerwarteter Fehler im Sync-Handler: {e}")
|
||||
import traceback
|
||||
ctx.logger.error(traceback.format_exc())
|
||||
|
||||
try:
|
||||
await sync_utils.release_sync_lock(
|
||||
entity_id,
|
||||
success=False,
|
||||
error_message=str(e)[:2000],
|
||||
entity_type=entity_type
|
||||
)
|
||||
except Exception as release_error:
|
||||
# Selbst Lock-Release failed - logge kritischen Fehler
|
||||
ctx.logger.critical(f"🚨 CRITICAL: Lock-Release failed für Document {entity_id}: {release_error}")
|
||||
# Force Redis lock release
|
||||
try:
|
||||
lock_key = f"sync_lock:document:{entity_id}"
|
||||
redis_client.delete(lock_key)
|
||||
ctx.logger.info(f"✅ Redis lock manuell released: {lock_key}")
|
||||
except:
|
||||
pass
|
||||
|
||||
except Exception as e:
|
||||
# Fehler VOR Lock-Acquire - kein Lock-Release nötig
|
||||
ctx.logger.error(f"❌ Fehler vor Lock-Acquire: {e}")
|
||||
import traceback
|
||||
ctx.logger.error(traceback.format_exc())
|
||||
|
||||
|
||||
async def handle_create_or_update(entity_id: str, document: Dict[str, Any], sync_utils: DocumentSync, xai_service: XAIService, ctx: FlowContext[Any], entity_type: str = 'CDokumente') -> None:
|
||||
"""
|
||||
Behandelt Create/Update von Documents
|
||||
|
||||
Entscheidet ob xAI-Sync nötig ist und führt diesen durch
|
||||
"""
|
||||
try:
|
||||
ctx.logger.info("")
|
||||
ctx.logger.info("=" * 80)
|
||||
ctx.logger.info("🔍 ANALYSE: Braucht dieses Document xAI-Sync?")
|
||||
ctx.logger.info("=" * 80)
|
||||
|
||||
# Datei-Status für Preview-Generierung (verschiedene Feld-Namen unterstützen)
|
||||
datei_status = document.get('fileStatus') or document.get('dateiStatus')
|
||||
|
||||
# Entscheidungslogik: Soll dieses Document zu xAI?
|
||||
needs_sync, collection_ids, reason = await sync_utils.should_sync_to_xai(document)
|
||||
|
||||
ctx.logger.info(f"📊 Entscheidung: {'✅ SYNC NÖTIG' if needs_sync else '⏭️ KEIN SYNC NÖTIG'}")
|
||||
ctx.logger.info(f" Grund: {reason}")
|
||||
ctx.logger.info(f" File-Status: {datei_status or 'N/A'}")
|
||||
|
||||
if collection_ids:
|
||||
ctx.logger.info(f" Collections: {collection_ids}")
|
||||
|
||||
# ═══════════════════════════════════════════════════════════════
|
||||
# CHECK: Knowledge Bases mit Status "new" (noch keine Collection)
|
||||
# ═══════════════════════════════════════════════════════════════
|
||||
new_knowledge_bases = [cid for cid in collection_ids if cid.startswith('NEW:')]
|
||||
if new_knowledge_bases:
|
||||
ctx.logger.info("")
|
||||
ctx.logger.info("=" * 80)
|
||||
ctx.logger.info("🆕 DOKUMENT IST MIT KNOWLEDGE BASE(S) VERKNÜPFT (Status: new)")
|
||||
ctx.logger.info("=" * 80)
|
||||
|
||||
for new_kb in new_knowledge_bases:
|
||||
kb_id = new_kb[4:] # Remove "NEW:" prefix
|
||||
ctx.logger.info(f"📋 CAIKnowledge {kb_id}")
|
||||
ctx.logger.info(f" Status: new → Collection muss zuerst erstellt werden")
|
||||
|
||||
# Trigger Knowledge Sync
|
||||
ctx.logger.info(f"📤 Triggering aiknowledge.sync event...")
|
||||
await ctx.emit('aiknowledge.sync', {
|
||||
'entity_id': kb_id,
|
||||
'entity_type': 'CAIKnowledge',
|
||||
'triggered_by': 'document_sync',
|
||||
'document_id': entity_id
|
||||
})
|
||||
ctx.logger.info(f"✅ Event emitted for {kb_id}")
|
||||
|
||||
# Release lock and skip document sync - knowledge sync will handle documents
|
||||
ctx.logger.info("")
|
||||
ctx.logger.info("=" * 80)
|
||||
ctx.logger.info("✅ KNOWLEDGE SYNC GETRIGGERT")
|
||||
ctx.logger.info(" Document Sync wird übersprungen")
|
||||
ctx.logger.info(" (Knowledge Sync erstellt Collection und synchronisiert dann Dokumente)")
|
||||
ctx.logger.info("=" * 80)
|
||||
|
||||
await sync_utils.release_sync_lock(entity_id, success=True, entity_type=entity_type)
|
||||
return
|
||||
|
||||
# ═══════════════════════════════════════════════════════════════
|
||||
# PREVIEW-GENERIERUNG bei neuen/geänderten Dateien
|
||||
# ═══════════════════════════════════════════════════════════════
|
||||
|
||||
# Case-insensitive check für Datei-Status
|
||||
datei_status_lower = (datei_status or '').lower()
|
||||
if datei_status_lower in ['neu', 'geändert', 'new', 'changed']:
|
||||
ctx.logger.info("")
|
||||
ctx.logger.info("=" * 80)
|
||||
ctx.logger.info("🖼️ PREVIEW-GENERIERUNG STARTEN")
|
||||
ctx.logger.info(f" Datei-Status: {datei_status}")
|
||||
ctx.logger.info("=" * 80)
|
||||
|
||||
try:
|
||||
# 1. Hole Download-Informationen
|
||||
download_info = await sync_utils.get_document_download_info(entity_id, entity_type)
|
||||
|
||||
if not download_info:
|
||||
ctx.logger.warn("⚠️ Keine Download-Info verfügbar - überspringe Preview")
|
||||
else:
|
||||
ctx.logger.info(f"📥 Datei-Info:")
|
||||
ctx.logger.info(f" Filename: {download_info['filename']}")
|
||||
ctx.logger.info(f" MIME-Type: {download_info['mime_type']}")
|
||||
ctx.logger.info(f" Size: {download_info['size']} bytes")
|
||||
|
||||
# 2. Download File von EspoCRM
|
||||
ctx.logger.info(f"📥 Downloading file...")
|
||||
espocrm = sync_utils.espocrm
|
||||
file_content = await espocrm.download_attachment(download_info['attachment_id'])
|
||||
ctx.logger.info(f"✅ Downloaded {len(file_content)} bytes")
|
||||
|
||||
# 3. Speichere temporär für Preview-Generierung
|
||||
import tempfile
|
||||
import os
|
||||
|
||||
with tempfile.NamedTemporaryFile(delete=False, suffix=f"_{download_info['filename']}") as tmp_file:
|
||||
tmp_file.write(file_content)
|
||||
tmp_path = tmp_file.name
|
||||
|
||||
try:
|
||||
# 4. Generiere Preview
|
||||
ctx.logger.info(f"🖼️ Generating preview (600x800 WebP)...")
|
||||
preview_data = await sync_utils.generate_thumbnail(
|
||||
tmp_path,
|
||||
download_info['mime_type'],
|
||||
max_width=600,
|
||||
max_height=800
|
||||
)
|
||||
|
||||
if preview_data:
|
||||
ctx.logger.info(f"✅ Preview generated: {len(preview_data)} bytes WebP")
|
||||
|
||||
# 5. Upload Preview zu EspoCRM und reset file status
|
||||
ctx.logger.info(f"📤 Uploading preview to EspoCRM...")
|
||||
await sync_utils.update_sync_metadata(
|
||||
entity_id,
|
||||
preview_data=preview_data,
|
||||
reset_file_status=True, # Reset status nach Preview-Generierung
|
||||
entity_type=entity_type
|
||||
)
|
||||
ctx.logger.info(f"✅ Preview uploaded successfully")
|
||||
else:
|
||||
ctx.logger.warn("⚠️ Preview-Generierung lieferte keine Daten")
|
||||
# Auch bei fehlgeschlagener Preview-Generierung Status zurücksetzen
|
||||
await sync_utils.update_sync_metadata(
|
||||
entity_id,
|
||||
reset_file_status=True,
|
||||
entity_type=entity_type
|
||||
)
|
||||
|
||||
finally:
|
||||
# Cleanup temp file
|
||||
try:
|
||||
os.remove(tmp_path)
|
||||
except:
|
||||
pass
|
||||
|
||||
except Exception as e:
|
||||
ctx.logger.error(f"❌ Fehler bei Preview-Generierung: {e}")
|
||||
import traceback
|
||||
ctx.logger.error(traceback.format_exc())
|
||||
# Continue - Preview ist optional
|
||||
|
||||
ctx.logger.info("")
|
||||
ctx.logger.info("=" * 80)
|
||||
ctx.logger.info("✅ PREVIEW-VERARBEITUNG ABGESCHLOSSEN")
|
||||
ctx.logger.info("=" * 80)
|
||||
|
||||
# ═══════════════════════════════════════════════════════════════
|
||||
# xAI SYNC (falls erforderlich)
|
||||
# ═══════════════════════════════════════════════════════════════
|
||||
|
||||
if not needs_sync:
|
||||
ctx.logger.info("✅ Kein xAI-Sync erforderlich, Lock wird released")
|
||||
# Wenn Preview generiert wurde aber kein xAI sync nötig,
|
||||
# wurde Status bereits in Preview-Schritt zurückgesetzt
|
||||
await sync_utils.release_sync_lock(entity_id, success=True, entity_type=entity_type)
|
||||
return
|
||||
|
||||
# ═══════════════════════════════════════════════════════════════
|
||||
# xAI SYNC DURCHFÜHREN
|
||||
# ═══════════════════════════════════════════════════════════════
|
||||
|
||||
ctx.logger.info("")
|
||||
ctx.logger.info("=" * 80)
|
||||
ctx.logger.info("🤖 xAI SYNC STARTEN")
|
||||
ctx.logger.info("=" * 80)
|
||||
|
||||
# 1. Hole Download-Informationen (falls nicht schon aus Preview-Schritt vorhanden)
|
||||
download_info = await sync_utils.get_document_download_info(entity_id, entity_type)
|
||||
if not download_info:
|
||||
raise Exception("Konnte Download-Info nicht ermitteln – Datei fehlt?")
|
||||
|
||||
ctx.logger.info(f"📥 Datei: {download_info['filename']} ({download_info['size']} bytes, {download_info['mime_type']})")
|
||||
|
||||
# 2. Download Datei von EspoCRM
|
||||
espocrm = sync_utils.espocrm
|
||||
file_content = await espocrm.download_attachment(download_info['attachment_id'])
|
||||
ctx.logger.info(f"✅ Downloaded {len(file_content)} bytes")
|
||||
|
||||
# 3. MD5-Hash berechnen für Change-Detection
|
||||
file_hash = hashlib.md5(file_content).hexdigest()
|
||||
ctx.logger.info(f"🔑 MD5: {file_hash}")
|
||||
|
||||
# 4. Upload zu xAI
|
||||
# Immer neu hochladen wenn needs_sync=True (neues File oder Hash geändert)
|
||||
ctx.logger.info("📤 Uploading to xAI...")
|
||||
xai_file_id = await xai_service.upload_file(
|
||||
file_content,
|
||||
download_info['filename'],
|
||||
download_info['mime_type']
|
||||
)
|
||||
ctx.logger.info(f"✅ xAI file_id: {xai_file_id}")
|
||||
|
||||
# 5. Zu allen Ziel-Collections hinzufügen
|
||||
ctx.logger.info(f"📚 Füge zu {len(collection_ids)} Collection(s) hinzu...")
|
||||
added_collections = await xai_service.add_to_collections(collection_ids, xai_file_id)
|
||||
ctx.logger.info(f"✅ In {len(added_collections)}/{len(collection_ids)} Collections eingetragen")
|
||||
|
||||
# 6. EspoCRM Metadaten aktualisieren und Lock freigeben
|
||||
await sync_utils.update_sync_metadata(
|
||||
entity_id,
|
||||
xai_file_id=xai_file_id,
|
||||
collection_ids=added_collections,
|
||||
file_hash=file_hash,
|
||||
entity_type=entity_type
|
||||
)
|
||||
await sync_utils.release_sync_lock(
|
||||
entity_id,
|
||||
success=True,
|
||||
entity_type=entity_type
|
||||
)
|
||||
|
||||
ctx.logger.info("=" * 80)
|
||||
ctx.logger.info("✅ DOCUMENT SYNC ABGESCHLOSSEN")
|
||||
ctx.logger.info("=" * 80)
|
||||
|
||||
except Exception as e:
|
||||
ctx.logger.error(f"❌ Fehler bei Create/Update: {e}")
|
||||
import traceback
|
||||
ctx.logger.error(traceback.format_exc())
|
||||
await sync_utils.release_sync_lock(entity_id, success=False, error_message=str(e))
|
||||
|
||||
|
||||
async def handle_delete(entity_id: str, document: Dict[str, Any], sync_utils: DocumentSync, xai_service: XAIService, ctx: FlowContext[Any], entity_type: str = 'CDokumente') -> None:
|
||||
"""
|
||||
Behandelt Delete von Documents
|
||||
|
||||
Entfernt Document aus xAI Collections (aber löscht File nicht - kann in anderen Collections sein)
|
||||
"""
|
||||
try:
|
||||
ctx.logger.info("")
|
||||
ctx.logger.info("=" * 80)
|
||||
ctx.logger.info("🗑️ DOCUMENT DELETE - xAI CLEANUP")
|
||||
ctx.logger.info("=" * 80)
|
||||
|
||||
xai_file_id = document.get('xaiFileId') or document.get('xaiId')
|
||||
xai_collections = document.get('xaiCollections') or []
|
||||
|
||||
if not xai_file_id or not xai_collections:
|
||||
ctx.logger.info("⏭️ Document war nicht in xAI gesynct, nichts zu tun")
|
||||
await sync_utils.release_sync_lock(entity_id, success=True, entity_type=entity_type)
|
||||
return
|
||||
|
||||
ctx.logger.info(f"📋 Document Info:")
|
||||
ctx.logger.info(f" xaiFileId: {xai_file_id}")
|
||||
ctx.logger.info(f" Collections: {xai_collections}")
|
||||
|
||||
ctx.logger.info(f"🗑️ Entferne aus {len(xai_collections)} Collection(s)...")
|
||||
await xai_service.remove_from_collections(xai_collections, xai_file_id)
|
||||
ctx.logger.info(f"✅ File aus {len(xai_collections)} Collection(s) entfernt")
|
||||
ctx.logger.info(" (File selbst bleibt in xAI – kann in anderen Collections sein)")
|
||||
|
||||
await sync_utils.release_sync_lock(entity_id, success=True, entity_type=entity_type)
|
||||
|
||||
ctx.logger.info("=" * 80)
|
||||
ctx.logger.info("✅ DELETE ABGESCHLOSSEN")
|
||||
ctx.logger.info("=" * 80)
|
||||
|
||||
except Exception as e:
|
||||
ctx.logger.error(f"❌ Fehler bei Delete: {e}")
|
||||
import traceback
|
||||
ctx.logger.error(traceback.format_exc())
|
||||
await sync_utils.release_sync_lock(entity_id, success=False, error_message=str(e), entity_type=entity_type)
|
||||
91
src/steps/vmh/webhook/aiknowledge_update_api_step.py
Normal file
91
src/steps/vmh/webhook/aiknowledge_update_api_step.py
Normal file
@@ -0,0 +1,91 @@
|
||||
"""VMH Webhook - AI Knowledge Update"""
|
||||
from typing import Any
|
||||
from motia import FlowContext, http, ApiRequest, ApiResponse
|
||||
|
||||
|
||||
config = {
|
||||
"name": "VMH Webhook AI Knowledge Update",
|
||||
"description": "Receives update webhooks from EspoCRM for CAIKnowledge entities",
|
||||
"flows": ["vmh-aiknowledge"],
|
||||
"triggers": [
|
||||
http("POST", "/vmh/webhook/aiknowledge/update")
|
||||
],
|
||||
"enqueues": ["aiknowledge.sync"],
|
||||
}
|
||||
|
||||
|
||||
async def handler(request: ApiRequest, ctx: FlowContext[Any]) -> ApiResponse:
|
||||
"""
|
||||
Webhook handler for CAIKnowledge updates in EspoCRM.
|
||||
|
||||
Triggered when:
|
||||
- activationStatus changes
|
||||
- syncStatus changes (e.g., set to 'unclean')
|
||||
- Documents linked/unlinked
|
||||
"""
|
||||
try:
|
||||
ctx.logger.info("=" * 80)
|
||||
ctx.logger.info("🔔 AI Knowledge Update Webhook")
|
||||
ctx.logger.info("=" * 80)
|
||||
|
||||
# Extract payload
|
||||
payload = request.body
|
||||
|
||||
# Handle case where payload is a list (e.g., from array-based webhook)
|
||||
if isinstance(payload, list):
|
||||
if not payload:
|
||||
ctx.logger.error("❌ Empty payload list")
|
||||
return ApiResponse(
|
||||
status=400,
|
||||
body={'success': False, 'error': 'Empty payload'}
|
||||
)
|
||||
payload = payload[0] # Take first item
|
||||
|
||||
# Ensure payload is a dict
|
||||
if not isinstance(payload, dict):
|
||||
ctx.logger.error(f"❌ Invalid payload type: {type(payload)}")
|
||||
return ApiResponse(
|
||||
status=400,
|
||||
body={'success': False, 'error': f'Invalid payload type: {type(payload).__name__}'}
|
||||
)
|
||||
|
||||
# Validate required fields
|
||||
knowledge_id = payload.get('entity_id') or payload.get('id')
|
||||
entity_type = payload.get('entity_type', 'CAIKnowledge')
|
||||
action = payload.get('action', 'update')
|
||||
|
||||
if not knowledge_id:
|
||||
ctx.logger.error("❌ Missing entity_id in payload")
|
||||
return ApiResponse(
|
||||
status=400,
|
||||
body={'success': False, 'error': 'Missing entity_id'}
|
||||
)
|
||||
|
||||
ctx.logger.info(f"📋 Entity Type: {entity_type}")
|
||||
ctx.logger.info(f"📋 Entity ID: {knowledge_id}")
|
||||
ctx.logger.info(f"📋 Action: {action}")
|
||||
|
||||
# Enqueue sync event
|
||||
await ctx.enqueue({
|
||||
'topic': 'aiknowledge.sync',
|
||||
'data': {
|
||||
'knowledge_id': knowledge_id,
|
||||
'source': 'webhook',
|
||||
'action': action
|
||||
}
|
||||
})
|
||||
|
||||
ctx.logger.info(f"✅ Sync event enqueued for {knowledge_id}")
|
||||
ctx.logger.info("=" * 80)
|
||||
|
||||
return ApiResponse(
|
||||
status=200,
|
||||
body={'success': True, 'knowledge_id': knowledge_id}
|
||||
)
|
||||
|
||||
except Exception as e:
|
||||
ctx.logger.error(f"❌ Webhook error: {e}")
|
||||
return ApiResponse(
|
||||
status=500,
|
||||
body={'success': False, 'error': str(e)}
|
||||
)
|
||||
@@ -7,7 +7,7 @@ from motia import FlowContext, http, ApiRequest, ApiResponse
|
||||
|
||||
config = {
|
||||
"name": "VMH Webhook Bankverbindungen Create",
|
||||
"description": "Empfängt Create-Webhooks von EspoCRM für Bankverbindungen",
|
||||
"description": "Receives create webhooks from EspoCRM for Bankverbindungen",
|
||||
"flows": ["vmh-bankverbindungen"],
|
||||
"triggers": [
|
||||
http("POST", "/vmh/webhook/bankverbindungen/create")
|
||||
@@ -23,10 +23,13 @@ async def handler(request: ApiRequest, ctx: FlowContext[Any]) -> ApiResponse:
|
||||
try:
|
||||
payload = request.body or []
|
||||
|
||||
ctx.logger.info("VMH Webhook Bankverbindungen Create empfangen")
|
||||
ctx.logger.info("=" * 80)
|
||||
ctx.logger.info("📥 VMH WEBHOOK: BANKVERBINDUNGEN CREATE")
|
||||
ctx.logger.info("=" * 80)
|
||||
ctx.logger.info(f"Payload: {json.dumps(payload, indent=2, ensure_ascii=False)}")
|
||||
ctx.logger.info("=" * 80)
|
||||
|
||||
# Sammle alle IDs aus dem Batch
|
||||
# Collect all IDs from batch
|
||||
entity_ids = set()
|
||||
|
||||
if isinstance(payload, list):
|
||||
@@ -36,7 +39,7 @@ async def handler(request: ApiRequest, ctx: FlowContext[Any]) -> ApiResponse:
|
||||
elif isinstance(payload, dict) and 'id' in payload:
|
||||
entity_ids.add(payload['id'])
|
||||
|
||||
ctx.logger.info(f"{len(entity_ids)} IDs zum Create-Sync gefunden")
|
||||
ctx.logger.info(f"{len(entity_ids)} IDs found for create sync")
|
||||
|
||||
# Emit events
|
||||
for entity_id in entity_ids:
|
||||
@@ -50,7 +53,8 @@ async def handler(request: ApiRequest, ctx: FlowContext[Any]) -> ApiResponse:
|
||||
}
|
||||
})
|
||||
|
||||
ctx.logger.info(f"VMH Create Webhook verarbeitet: {len(entity_ids)} Events emittiert")
|
||||
ctx.logger.info("✅ VMH Create Webhook processed: "
|
||||
f"{len(entity_ids)} events emitted")
|
||||
|
||||
return ApiResponse(
|
||||
status=200,
|
||||
@@ -62,7 +66,10 @@ async def handler(request: ApiRequest, ctx: FlowContext[Any]) -> ApiResponse:
|
||||
)
|
||||
|
||||
except Exception as e:
|
||||
ctx.logger.error(f"Fehler beim Verarbeiten des VMH Create Webhooks: {e}")
|
||||
ctx.logger.error("=" * 80)
|
||||
ctx.logger.error("❌ ERROR: BANKVERBINDUNGEN CREATE WEBHOOK")
|
||||
ctx.logger.error(f"Error: {e}")
|
||||
ctx.logger.error("=" * 80)
|
||||
return ApiResponse(
|
||||
status=500,
|
||||
body={'error': 'Internal server error', 'details': str(e)}
|
||||
@@ -7,7 +7,7 @@ from motia import FlowContext, http, ApiRequest, ApiResponse
|
||||
|
||||
config = {
|
||||
"name": "VMH Webhook Bankverbindungen Delete",
|
||||
"description": "Empfängt Delete-Webhooks von EspoCRM für Bankverbindungen",
|
||||
"description": "Receives delete webhooks from EspoCRM for Bankverbindungen",
|
||||
"flows": ["vmh-bankverbindungen"],
|
||||
"triggers": [
|
||||
http("POST", "/vmh/webhook/bankverbindungen/delete")
|
||||
@@ -23,10 +23,13 @@ async def handler(request: ApiRequest, ctx: FlowContext[Any]) -> ApiResponse:
|
||||
try:
|
||||
payload = request.body or []
|
||||
|
||||
ctx.logger.info("VMH Webhook Bankverbindungen Delete empfangen")
|
||||
ctx.logger.info("=" * 80)
|
||||
ctx.logger.info("📥 VMH WEBHOOK: BANKVERBINDUNGEN DELETE")
|
||||
ctx.logger.info("=" * 80)
|
||||
ctx.logger.info(f"Payload: {json.dumps(payload, indent=2, ensure_ascii=False)}")
|
||||
ctx.logger.info("=" * 80)
|
||||
|
||||
# Sammle alle IDs
|
||||
# Collect all IDs
|
||||
entity_ids = set()
|
||||
|
||||
if isinstance(payload, list):
|
||||
@@ -36,7 +39,7 @@ async def handler(request: ApiRequest, ctx: FlowContext[Any]) -> ApiResponse:
|
||||
elif isinstance(payload, dict) and 'id' in payload:
|
||||
entity_ids.add(payload['id'])
|
||||
|
||||
ctx.logger.info(f"{len(entity_ids)} IDs zum Delete-Sync gefunden")
|
||||
ctx.logger.info(f"{len(entity_ids)} IDs found for delete sync")
|
||||
|
||||
# Emit events
|
||||
for entity_id in entity_ids:
|
||||
@@ -50,7 +53,8 @@ async def handler(request: ApiRequest, ctx: FlowContext[Any]) -> ApiResponse:
|
||||
}
|
||||
})
|
||||
|
||||
ctx.logger.info(f"VMH Delete Webhook verarbeitet: {len(entity_ids)} Events emittiert")
|
||||
ctx.logger.info("✅ VMH Delete Webhook processed: "
|
||||
f"{len(entity_ids)} events emitted")
|
||||
|
||||
return ApiResponse(
|
||||
status=200,
|
||||
@@ -62,7 +66,10 @@ async def handler(request: ApiRequest, ctx: FlowContext[Any]) -> ApiResponse:
|
||||
)
|
||||
|
||||
except Exception as e:
|
||||
ctx.logger.error(f"Fehler beim Verarbeiten des VMH Delete Webhooks: {e}")
|
||||
ctx.logger.error("=" * 80)
|
||||
ctx.logger.error("❌ ERROR: BANKVERBINDUNGEN DELETE WEBHOOK")
|
||||
ctx.logger.error(f"Error: {e}")
|
||||
ctx.logger.error("=" * 80)
|
||||
return ApiResponse(
|
||||
status=500,
|
||||
body={'error': 'Internal server error', 'details': str(e)}
|
||||
@@ -7,7 +7,7 @@ from motia import FlowContext, http, ApiRequest, ApiResponse
|
||||
|
||||
config = {
|
||||
"name": "VMH Webhook Bankverbindungen Update",
|
||||
"description": "Empfängt Update-Webhooks von EspoCRM für Bankverbindungen",
|
||||
"description": "Receives update webhooks from EspoCRM for Bankverbindungen",
|
||||
"flows": ["vmh-bankverbindungen"],
|
||||
"triggers": [
|
||||
http("POST", "/vmh/webhook/bankverbindungen/update")
|
||||
@@ -23,10 +23,13 @@ async def handler(request: ApiRequest, ctx: FlowContext[Any]) -> ApiResponse:
|
||||
try:
|
||||
payload = request.body or []
|
||||
|
||||
ctx.logger.info("VMH Webhook Bankverbindungen Update empfangen")
|
||||
ctx.logger.info("=" * 80)
|
||||
ctx.logger.info("📥 VMH WEBHOOK: BANKVERBINDUNGEN UPDATE")
|
||||
ctx.logger.info("=" * 80)
|
||||
ctx.logger.info(f"Payload: {json.dumps(payload, indent=2, ensure_ascii=False)}")
|
||||
ctx.logger.info("=" * 80)
|
||||
|
||||
# Sammle alle IDs
|
||||
# Collect all IDs
|
||||
entity_ids = set()
|
||||
|
||||
if isinstance(payload, list):
|
||||
@@ -36,7 +39,7 @@ async def handler(request: ApiRequest, ctx: FlowContext[Any]) -> ApiResponse:
|
||||
elif isinstance(payload, dict) and 'id' in payload:
|
||||
entity_ids.add(payload['id'])
|
||||
|
||||
ctx.logger.info(f"{len(entity_ids)} IDs zum Update-Sync gefunden")
|
||||
ctx.logger.info(f"{len(entity_ids)} IDs found for update sync")
|
||||
|
||||
# Emit events
|
||||
for entity_id in entity_ids:
|
||||
@@ -50,7 +53,8 @@ async def handler(request: ApiRequest, ctx: FlowContext[Any]) -> ApiResponse:
|
||||
}
|
||||
})
|
||||
|
||||
ctx.logger.info(f"VMH Update Webhook verarbeitet: {len(entity_ids)} Events emittiert")
|
||||
ctx.logger.info("✅ VMH Update Webhook processed: "
|
||||
f"{len(entity_ids)} events emitted")
|
||||
|
||||
return ApiResponse(
|
||||
status=200,
|
||||
@@ -62,7 +66,10 @@ async def handler(request: ApiRequest, ctx: FlowContext[Any]) -> ApiResponse:
|
||||
)
|
||||
|
||||
except Exception as e:
|
||||
ctx.logger.error(f"Fehler beim Verarbeiten des VMH Update Webhooks: {e}")
|
||||
ctx.logger.error("=" * 80)
|
||||
ctx.logger.error("❌ ERROR: BANKVERBINDUNGEN UPDATE WEBHOOK")
|
||||
ctx.logger.error(f"Error: {e}")
|
||||
ctx.logger.error("=" * 80)
|
||||
return ApiResponse(
|
||||
status=500,
|
||||
body={'error': 'Internal server error', 'details': str(e)}
|
||||
@@ -7,7 +7,7 @@ from motia import FlowContext, http, ApiRequest, ApiResponse
|
||||
|
||||
config = {
|
||||
"name": "VMH Webhook Beteiligte Create",
|
||||
"description": "Empfängt Create-Webhooks von EspoCRM für Beteiligte",
|
||||
"description": "Receives create webhooks from EspoCRM for Beteiligte",
|
||||
"flows": ["vmh-beteiligte"],
|
||||
"triggers": [
|
||||
http("POST", "/vmh/webhook/beteiligte/create")
|
||||
@@ -26,10 +26,13 @@ async def handler(request: ApiRequest, ctx: FlowContext[Any]) -> ApiResponse:
|
||||
try:
|
||||
payload = request.body or []
|
||||
|
||||
ctx.logger.info("VMH Webhook Beteiligte Create empfangen")
|
||||
ctx.logger.info("=" * 80)
|
||||
ctx.logger.info("📥 VMH WEBHOOK: BETEILIGTE CREATE")
|
||||
ctx.logger.info("=" * 80)
|
||||
ctx.logger.info(f"Payload: {json.dumps(payload, indent=2, ensure_ascii=False)}")
|
||||
ctx.logger.info("=" * 80)
|
||||
|
||||
# Sammle alle IDs aus dem Batch
|
||||
# Collect all IDs from batch
|
||||
entity_ids = set()
|
||||
|
||||
if isinstance(payload, list):
|
||||
@@ -39,9 +42,9 @@ async def handler(request: ApiRequest, ctx: FlowContext[Any]) -> ApiResponse:
|
||||
elif isinstance(payload, dict) and 'id' in payload:
|
||||
entity_ids.add(payload['id'])
|
||||
|
||||
ctx.logger.info(f"{len(entity_ids)} IDs zum Create-Sync gefunden")
|
||||
ctx.logger.info(f"{len(entity_ids)} IDs found for create sync")
|
||||
|
||||
# Emit events für Queue-Processing (Deduplizierung erfolgt im Event-Handler via Lock)
|
||||
# Emit events for queue processing (deduplication via lock in event handler)
|
||||
for entity_id in entity_ids:
|
||||
await ctx.enqueue({
|
||||
'topic': 'vmh.beteiligte.create',
|
||||
@@ -53,7 +56,8 @@ async def handler(request: ApiRequest, ctx: FlowContext[Any]) -> ApiResponse:
|
||||
}
|
||||
})
|
||||
|
||||
ctx.logger.info(f"VMH Create Webhook verarbeitet: {len(entity_ids)} Events emittiert")
|
||||
ctx.logger.info("✅ VMH Create Webhook processed: "
|
||||
f"{len(entity_ids)} events emitted")
|
||||
|
||||
return ApiResponse(
|
||||
status=200,
|
||||
@@ -65,7 +69,14 @@ async def handler(request: ApiRequest, ctx: FlowContext[Any]) -> ApiResponse:
|
||||
)
|
||||
|
||||
except Exception as e:
|
||||
ctx.logger.error(f"Fehler beim Verarbeiten des VMH Create Webhooks: {e}")
|
||||
ctx.logger.error("=" * 80)
|
||||
ctx.logger.error("❌ ERROR: VMH CREATE WEBHOOK")
|
||||
ctx.logger.error("=" * 80)
|
||||
ctx.logger.error(f"Error: {e}")
|
||||
ctx.logger.error(f"Entity IDs attempted: {list(entity_ids) if 'entity_ids' in locals() else 'N/A'}")
|
||||
ctx.logger.error(f"Full Payload: {json.dumps(request.body, indent=2, ensure_ascii=False)}")
|
||||
ctx.logger.error(f"Timestamp: {datetime.datetime.now().isoformat()}")
|
||||
ctx.logger.error("=" * 80)
|
||||
return ApiResponse(
|
||||
status=500,
|
||||
body={
|
||||
@@ -7,7 +7,7 @@ from motia import FlowContext, http, ApiRequest, ApiResponse
|
||||
|
||||
config = {
|
||||
"name": "VMH Webhook Beteiligte Delete",
|
||||
"description": "Empfängt Delete-Webhooks von EspoCRM für Beteiligte",
|
||||
"description": "Receives delete webhooks from EspoCRM for Beteiligte",
|
||||
"flows": ["vmh-beteiligte"],
|
||||
"triggers": [
|
||||
http("POST", "/vmh/webhook/beteiligte/delete")
|
||||
@@ -23,10 +23,13 @@ async def handler(request: ApiRequest, ctx: FlowContext[Any]) -> ApiResponse:
|
||||
try:
|
||||
payload = request.body or []
|
||||
|
||||
ctx.logger.info("VMH Webhook Beteiligte Delete empfangen")
|
||||
ctx.logger.info("=" * 80)
|
||||
ctx.logger.info("📥 VMH WEBHOOK: BETEILIGTE DELETE")
|
||||
ctx.logger.info("=" * 80)
|
||||
ctx.logger.info(f"Payload: {json.dumps(payload, indent=2, ensure_ascii=False)}")
|
||||
ctx.logger.info("=" * 80)
|
||||
|
||||
# Sammle alle IDs aus dem Batch
|
||||
# Collect all IDs from batch
|
||||
entity_ids = set()
|
||||
|
||||
if isinstance(payload, list):
|
||||
@@ -36,9 +39,9 @@ async def handler(request: ApiRequest, ctx: FlowContext[Any]) -> ApiResponse:
|
||||
elif isinstance(payload, dict) and 'id' in payload:
|
||||
entity_ids.add(payload['id'])
|
||||
|
||||
ctx.logger.info(f"{len(entity_ids)} IDs zum Delete-Sync gefunden")
|
||||
ctx.logger.info(f"{len(entity_ids)} IDs found for delete sync")
|
||||
|
||||
# Emit events für Queue-Processing
|
||||
# Emit events for queue processing
|
||||
for entity_id in entity_ids:
|
||||
await ctx.enqueue({
|
||||
'topic': 'vmh.beteiligte.delete',
|
||||
@@ -50,7 +53,8 @@ async def handler(request: ApiRequest, ctx: FlowContext[Any]) -> ApiResponse:
|
||||
}
|
||||
})
|
||||
|
||||
ctx.logger.info(f"VMH Delete Webhook verarbeitet: {len(entity_ids)} Events emittiert")
|
||||
ctx.logger.info("✅ VMH Delete Webhook processed: "
|
||||
f"{len(entity_ids)} events emitted")
|
||||
|
||||
return ApiResponse(
|
||||
status=200,
|
||||
@@ -62,7 +66,10 @@ async def handler(request: ApiRequest, ctx: FlowContext[Any]) -> ApiResponse:
|
||||
)
|
||||
|
||||
except Exception as e:
|
||||
ctx.logger.error(f"Fehler beim Delete-Webhook: {e}")
|
||||
ctx.logger.error("=" * 80)
|
||||
ctx.logger.error("❌ ERROR: BETEILIGTE DELETE WEBHOOK")
|
||||
ctx.logger.error(f"Error: {e}")
|
||||
ctx.logger.error("=" * 80)
|
||||
return ApiResponse(
|
||||
status=500,
|
||||
body={'error': 'Internal server error', 'details': str(e)}
|
||||
@@ -7,7 +7,7 @@ from motia import FlowContext, http, ApiRequest, ApiResponse
|
||||
|
||||
config = {
|
||||
"name": "VMH Webhook Beteiligte Update",
|
||||
"description": "Empfängt Update-Webhooks von EspoCRM für Beteiligte",
|
||||
"description": "Receives update webhooks from EspoCRM for Beteiligte",
|
||||
"flows": ["vmh-beteiligte"],
|
||||
"triggers": [
|
||||
http("POST", "/vmh/webhook/beteiligte/update")
|
||||
@@ -20,16 +20,19 @@ async def handler(request: ApiRequest, ctx: FlowContext[Any]) -> ApiResponse:
|
||||
"""
|
||||
Webhook handler for Beteiligte updates in EspoCRM.
|
||||
|
||||
Note: Loop-Prevention ist auf EspoCRM-Seite implementiert.
|
||||
rowId-Updates triggern keine Webhooks mehr, daher keine Filterung nötig.
|
||||
Note: Loop prevention is implemented on EspoCRM side.
|
||||
rowId updates no longer trigger webhooks, so no filtering needed.
|
||||
"""
|
||||
try:
|
||||
payload = request.body or []
|
||||
|
||||
ctx.logger.info("VMH Webhook Beteiligte Update empfangen")
|
||||
ctx.logger.info("=" * 80)
|
||||
ctx.logger.info("📥 VMH WEBHOOK: BETEILIGTE UPDATE")
|
||||
ctx.logger.info("=" * 80)
|
||||
ctx.logger.info(f"Payload: {json.dumps(payload, indent=2, ensure_ascii=False)}")
|
||||
ctx.logger.info("=" * 80)
|
||||
|
||||
# Sammle alle IDs aus dem Batch
|
||||
# Collect all IDs from batch
|
||||
entity_ids = set()
|
||||
|
||||
if isinstance(payload, list):
|
||||
@@ -39,9 +42,9 @@ async def handler(request: ApiRequest, ctx: FlowContext[Any]) -> ApiResponse:
|
||||
elif isinstance(payload, dict) and 'id' in payload:
|
||||
entity_ids.add(payload['id'])
|
||||
|
||||
ctx.logger.info(f"{len(entity_ids)} IDs zum Update-Sync gefunden")
|
||||
ctx.logger.info(f"{len(entity_ids)} IDs found for update sync")
|
||||
|
||||
# Emit events für Queue-Processing
|
||||
# Emit events for queue processing
|
||||
for entity_id in entity_ids:
|
||||
await ctx.enqueue({
|
||||
'topic': 'vmh.beteiligte.update',
|
||||
@@ -53,7 +56,8 @@ async def handler(request: ApiRequest, ctx: FlowContext[Any]) -> ApiResponse:
|
||||
}
|
||||
})
|
||||
|
||||
ctx.logger.info(f"VMH Update Webhook verarbeitet: {len(entity_ids)} Events emittiert")
|
||||
ctx.logger.info("✅ VMH Update Webhook processed: "
|
||||
f"{len(entity_ids)} events emitted")
|
||||
|
||||
return ApiResponse(
|
||||
status=200,
|
||||
@@ -65,7 +69,14 @@ async def handler(request: ApiRequest, ctx: FlowContext[Any]) -> ApiResponse:
|
||||
)
|
||||
|
||||
except Exception as e:
|
||||
ctx.logger.error(f"Fehler beim Verarbeiten des VMH Update Webhooks: {e}")
|
||||
ctx.logger.error("=" * 80)
|
||||
ctx.logger.error("❌ ERROR: VMH UPDATE WEBHOOK")
|
||||
ctx.logger.error("=" * 80)
|
||||
ctx.logger.error(f"Error: {e}")
|
||||
ctx.logger.error(f"Entity IDs attempted: {list(entity_ids) if 'entity_ids' in locals() else 'N/A'}")
|
||||
ctx.logger.error(f"Full Payload: {json.dumps(request.body, indent=2, ensure_ascii=False)}")
|
||||
ctx.logger.error(f"Timestamp: {datetime.datetime.now().isoformat()}")
|
||||
ctx.logger.error("=" * 80)
|
||||
return ApiResponse(
|
||||
status=500,
|
||||
body={
|
||||
91
src/steps/vmh/webhook/document_create_api_step.py
Normal file
91
src/steps/vmh/webhook/document_create_api_step.py
Normal file
@@ -0,0 +1,91 @@
|
||||
"""VMH Webhook - Document Create"""
|
||||
import json
|
||||
import datetime
|
||||
from typing import Any
|
||||
from motia import FlowContext, http, ApiRequest, ApiResponse
|
||||
|
||||
|
||||
config = {
|
||||
"name": "VMH Webhook Document Create",
|
||||
"description": "Empfängt Create-Webhooks von EspoCRM für Documents",
|
||||
"flows": ["vmh-documents"],
|
||||
"triggers": [
|
||||
http("POST", "/vmh/webhook/document/create")
|
||||
],
|
||||
"enqueues": ["vmh.document.create"],
|
||||
}
|
||||
|
||||
|
||||
async def handler(request: ApiRequest, ctx: FlowContext[Any]) -> ApiResponse:
|
||||
"""
|
||||
Webhook handler for Document creation in EspoCRM.
|
||||
|
||||
Receives batch or single entity notifications and emits queue events
|
||||
for each entity ID to be synced to xAI.
|
||||
"""
|
||||
try:
|
||||
payload = request.body or []
|
||||
|
||||
ctx.logger.info("=" * 80)
|
||||
ctx.logger.info("📥 VMH WEBHOOK: DOCUMENT CREATE")
|
||||
ctx.logger.info("=" * 80)
|
||||
ctx.logger.debug(f"Payload: {json.dumps(payload, indent=2, ensure_ascii=False)}")
|
||||
|
||||
# Collect all IDs from batch
|
||||
entity_ids = set()
|
||||
entity_type = 'CDokumente' # Default
|
||||
|
||||
if isinstance(payload, list):
|
||||
for entity in payload:
|
||||
if isinstance(entity, dict) and 'id' in entity:
|
||||
entity_ids.add(entity['id'])
|
||||
# Take entityType from first entity if present
|
||||
if entity_type == 'CDokumente':
|
||||
entity_type = entity.get('entityType', 'CDokumente')
|
||||
elif isinstance(payload, dict) and 'id' in payload:
|
||||
entity_ids.add(payload['id'])
|
||||
entity_type = payload.get('entityType', 'CDokumente')
|
||||
|
||||
ctx.logger.info(f"{len(entity_ids)} document IDs found for create sync")
|
||||
|
||||
# Emit events for queue processing (deduplication via lock in event handler)
|
||||
for entity_id in entity_ids:
|
||||
await ctx.enqueue({
|
||||
'topic': 'vmh.document.create',
|
||||
'data': {
|
||||
'entity_id': entity_id,
|
||||
'entity_type': entity_type,
|
||||
'action': 'create',
|
||||
'timestamp': payload[0].get('modifiedAt') if isinstance(payload, list) and payload else None
|
||||
}
|
||||
})
|
||||
|
||||
ctx.logger.info("✅ Document Create Webhook processed: "
|
||||
f"{len(entity_ids)} events emitted")
|
||||
|
||||
return ApiResponse(
|
||||
status=200,
|
||||
body={
|
||||
'success': True,
|
||||
'message': f'{len(entity_ids)} document(s) enqueued for sync',
|
||||
'entity_ids': list(entity_ids)
|
||||
}
|
||||
)
|
||||
|
||||
except Exception as e:
|
||||
ctx.logger.error("=" * 80)
|
||||
ctx.logger.error("❌ ERROR: DOCUMENT CREATE WEBHOOK")
|
||||
ctx.logger.error("=" * 80)
|
||||
ctx.logger.error(f"Error: {e}")
|
||||
ctx.logger.error(f"Entity IDs attempted: {list(entity_ids) if 'entity_ids' in locals() else 'N/A'}")
|
||||
ctx.logger.error(f"Full Payload: {json.dumps(request.body, indent=2, ensure_ascii=False)}")
|
||||
ctx.logger.error(f"Timestamp: {datetime.datetime.now().isoformat()}")
|
||||
ctx.logger.error("=" * 80)
|
||||
|
||||
return ApiResponse(
|
||||
status=500,
|
||||
body={
|
||||
'success': False,
|
||||
'error': str(e)
|
||||
}
|
||||
)
|
||||
91
src/steps/vmh/webhook/document_delete_api_step.py
Normal file
91
src/steps/vmh/webhook/document_delete_api_step.py
Normal file
@@ -0,0 +1,91 @@
|
||||
"""VMH Webhook - Document Delete"""
|
||||
import json
|
||||
import datetime
|
||||
from typing import Any
|
||||
from motia import FlowContext, http, ApiRequest, ApiResponse
|
||||
|
||||
|
||||
config = {
|
||||
"name": "VMH Webhook Document Delete",
|
||||
"description": "Empfängt Delete-Webhooks von EspoCRM für Documents",
|
||||
"flows": ["vmh-documents"],
|
||||
"triggers": [
|
||||
http("POST", "/vmh/webhook/document/delete")
|
||||
],
|
||||
"enqueues": ["vmh.document.delete"],
|
||||
}
|
||||
|
||||
|
||||
async def handler(request: ApiRequest, ctx: FlowContext[Any]) -> ApiResponse:
|
||||
"""
|
||||
Webhook handler for Document deletion in EspoCRM.
|
||||
|
||||
Receives batch or single entity notifications and emits queue events
|
||||
for each entity ID to be removed from xAI.
|
||||
"""
|
||||
try:
|
||||
payload = request.body or []
|
||||
|
||||
ctx.logger.info("=" * 80)
|
||||
ctx.logger.info("📥 VMH WEBHOOK: DOCUMENT DELETE")
|
||||
ctx.logger.info("=" * 80)
|
||||
ctx.logger.debug(f"Payload: {json.dumps(payload, indent=2, ensure_ascii=False)}")
|
||||
|
||||
# Collect all IDs from batch
|
||||
entity_ids = set()
|
||||
entity_type = 'CDokumente' # Default
|
||||
|
||||
if isinstance(payload, list):
|
||||
for entity in payload:
|
||||
if isinstance(entity, dict) and 'id' in entity:
|
||||
entity_ids.add(entity['id'])
|
||||
# Take entityType from first entity if present
|
||||
if entity_type == 'CDokumente':
|
||||
entity_type = entity.get('entityType', 'CDokumente')
|
||||
elif isinstance(payload, dict) and 'id' in payload:
|
||||
entity_ids.add(payload['id'])
|
||||
entity_type = payload.get('entityType', 'CDokumente')
|
||||
|
||||
ctx.logger.info(f"{len(entity_ids)} document IDs found for delete sync")
|
||||
|
||||
# Emit events for queue processing
|
||||
for entity_id in entity_ids:
|
||||
await ctx.enqueue({
|
||||
'topic': 'vmh.document.delete',
|
||||
'data': {
|
||||
'entity_id': entity_id,
|
||||
'entity_type': entity_type,
|
||||
'action': 'delete',
|
||||
'timestamp': payload[0].get('deletedAt') if isinstance(payload, list) and payload else None
|
||||
}
|
||||
})
|
||||
|
||||
ctx.logger.info("✅ Document Delete Webhook processed: "
|
||||
f"{len(entity_ids)} events emitted")
|
||||
|
||||
return ApiResponse(
|
||||
status=200,
|
||||
body={
|
||||
'success': True,
|
||||
'message': f'{len(entity_ids)} document(s) enqueued for deletion',
|
||||
'entity_ids': list(entity_ids)
|
||||
}
|
||||
)
|
||||
|
||||
except Exception as e:
|
||||
ctx.logger.error("=" * 80)
|
||||
ctx.logger.error("❌ ERROR: DOCUMENT DELETE WEBHOOK")
|
||||
ctx.logger.error("=" * 80)
|
||||
ctx.logger.error(f"Error: {e}")
|
||||
ctx.logger.error(f"Entity IDs attempted: {list(entity_ids) if 'entity_ids' in locals() else 'N/A'}")
|
||||
ctx.logger.error(f"Full Payload: {json.dumps(request.body, indent=2, ensure_ascii=False)}")
|
||||
ctx.logger.error(f"Timestamp: {datetime.datetime.now().isoformat()}")
|
||||
ctx.logger.error("=" * 80)
|
||||
|
||||
return ApiResponse(
|
||||
status=500,
|
||||
body={
|
||||
'success': False,
|
||||
'error': str(e)
|
||||
}
|
||||
)
|
||||
91
src/steps/vmh/webhook/document_update_api_step.py
Normal file
91
src/steps/vmh/webhook/document_update_api_step.py
Normal file
@@ -0,0 +1,91 @@
|
||||
"""VMH Webhook - Document Update"""
|
||||
import json
|
||||
import datetime
|
||||
from typing import Any
|
||||
from motia import FlowContext, http, ApiRequest, ApiResponse
|
||||
|
||||
|
||||
config = {
|
||||
"name": "VMH Webhook Document Update",
|
||||
"description": "Empfängt Update-Webhooks von EspoCRM für Documents",
|
||||
"flows": ["vmh-documents"],
|
||||
"triggers": [
|
||||
http("POST", "/vmh/webhook/document/update")
|
||||
],
|
||||
"enqueues": ["vmh.document.update"],
|
||||
}
|
||||
|
||||
|
||||
async def handler(request: ApiRequest, ctx: FlowContext[Any]) -> ApiResponse:
|
||||
"""
|
||||
Webhook handler for Document updates in EspoCRM.
|
||||
|
||||
Receives batch or single entity notifications and emits queue events
|
||||
for each entity ID to be synced to xAI.
|
||||
"""
|
||||
try:
|
||||
payload = request.body or []
|
||||
|
||||
ctx.logger.info("=" * 80)
|
||||
ctx.logger.info("📥 VMH WEBHOOK: DOCUMENT UPDATE")
|
||||
ctx.logger.info("=" * 80)
|
||||
ctx.logger.debug(f"Payload: {json.dumps(payload, indent=2, ensure_ascii=False)}")
|
||||
|
||||
# Collect all IDs from batch
|
||||
entity_ids = set()
|
||||
entity_type = 'CDokumente' # Default
|
||||
|
||||
if isinstance(payload, list):
|
||||
for entity in payload:
|
||||
if isinstance(entity, dict) and 'id' in entity:
|
||||
entity_ids.add(entity['id'])
|
||||
# Take entityType from first entity if present
|
||||
if entity_type == 'CDokumente':
|
||||
entity_type = entity.get('entityType', 'CDokumente')
|
||||
elif isinstance(payload, dict) and 'id' in payload:
|
||||
entity_ids.add(payload['id'])
|
||||
entity_type = payload.get('entityType', 'CDokumente')
|
||||
|
||||
ctx.logger.info(f"{len(entity_ids)} document IDs found for update sync")
|
||||
|
||||
# Emit events for queue processing
|
||||
for entity_id in entity_ids:
|
||||
await ctx.enqueue({
|
||||
'topic': 'vmh.document.update',
|
||||
'data': {
|
||||
'entity_id': entity_id,
|
||||
'entity_type': entity_type,
|
||||
'action': 'update',
|
||||
'timestamp': payload[0].get('modifiedAt') if isinstance(payload, list) and payload else None
|
||||
}
|
||||
})
|
||||
|
||||
ctx.logger.info("✅ Document Update Webhook processed: "
|
||||
f"{len(entity_ids)} events emitted")
|
||||
|
||||
return ApiResponse(
|
||||
status=200,
|
||||
body={
|
||||
'success': True,
|
||||
'message': f'{len(entity_ids)} document(s) enqueued for sync',
|
||||
'entity_ids': list(entity_ids)
|
||||
}
|
||||
)
|
||||
|
||||
except Exception as e:
|
||||
ctx.logger.error("=" * 80)
|
||||
ctx.logger.error("❌ ERROR: DOCUMENT UPDATE WEBHOOK")
|
||||
ctx.logger.error("=" * 80)
|
||||
ctx.logger.error(f"Error: {e}")
|
||||
ctx.logger.error(f"Entity IDs attempted: {list(entity_ids) if 'entity_ids' in locals() else 'N/A'}")
|
||||
ctx.logger.error(f"Full Payload: {json.dumps(request.body, indent=2, ensure_ascii=False)}")
|
||||
ctx.logger.error(f"Timestamp: {datetime.datetime.now().isoformat()}")
|
||||
ctx.logger.error("=" * 80)
|
||||
|
||||
return ApiResponse(
|
||||
status=500,
|
||||
body={
|
||||
'success': False,
|
||||
'error': str(e)
|
||||
}
|
||||
)
|
||||
110
tests/README.md
Normal file
110
tests/README.md
Normal file
@@ -0,0 +1,110 @@
|
||||
# Test Scripts
|
||||
|
||||
This directory contains test scripts for the Motia III xAI Collections integration.
|
||||
|
||||
## Test Files
|
||||
|
||||
### `test_xai_collections_api.py`
|
||||
Tests xAI Collections API authentication and basic operations.
|
||||
|
||||
**Usage:**
|
||||
```bash
|
||||
cd /opt/motia-iii/bitbylaw
|
||||
python tests/test_xai_collections_api.py
|
||||
```
|
||||
|
||||
**Required Environment Variables:**
|
||||
- `XAI_MANAGEMENT_API_KEY` - xAI Management API key for collection operations
|
||||
- `XAI_API_KEY` - xAI Regular API key for file operations
|
||||
|
||||
**Tests:**
|
||||
- ✅ Management API authentication
|
||||
- ✅ Regular API authentication
|
||||
- ✅ Collection listing
|
||||
- ✅ Collection creation
|
||||
- ✅ File upload
|
||||
- ✅ Collection deletion
|
||||
- ✅ Error handling
|
||||
|
||||
### `test_preview_upload.py`
|
||||
Tests preview/thumbnail upload to EspoCRM CDokumente entity.
|
||||
|
||||
**Usage:**
|
||||
```bash
|
||||
cd /opt/motia-iii/bitbylaw
|
||||
python tests/test_preview_upload.py
|
||||
```
|
||||
|
||||
**Required Environment Variables:**
|
||||
- `ESPOCRM_URL` - EspoCRM instance URL (default: https://crm.bitbylaw.com)
|
||||
- `ESPOCRM_API_KEY` - EspoCRM API key
|
||||
|
||||
**Tests:**
|
||||
- ✅ Preview image generation (WebP format, 600x800px)
|
||||
- ✅ Base64 Data URI encoding
|
||||
- ✅ Attachment upload via JSON POST
|
||||
- ✅ Entity update with previewId/previewName
|
||||
|
||||
**Status:** ✅ Successfully tested - Attachment ID `69a71194c7c6baebf` created
|
||||
|
||||
### `test_thumbnail_generation.py`
|
||||
Tests thumbnail generation for various document types.
|
||||
|
||||
**Usage:**
|
||||
```bash
|
||||
cd /opt/motia-iii/bitbylaw
|
||||
python tests/test_thumbnail_generation.py
|
||||
```
|
||||
|
||||
**Supported Formats:**
|
||||
- PDF → WebP (first page)
|
||||
- DOCX/DOC → PDF → WebP
|
||||
- Images (JPEG, PNG, etc.) → WebP resize
|
||||
|
||||
**Dependencies:**
|
||||
- `python3-pil` - PIL/Pillow for image processing
|
||||
- `poppler-utils` - PDF rendering
|
||||
- `libreoffice` - DOCX to PDF conversion
|
||||
- `pdf2image` - PDF to image conversion
|
||||
|
||||
## Running Tests
|
||||
|
||||
### All Tests
|
||||
```bash
|
||||
cd /opt/motia-iii/bitbylaw
|
||||
python -m pytest tests/ -v
|
||||
```
|
||||
|
||||
### Individual Tests
|
||||
```bash
|
||||
cd /opt/motia-iii/bitbylaw
|
||||
python tests/test_xai_collections_api.py
|
||||
python tests/test_preview_upload.py
|
||||
python tests/test_thumbnail_generation.py
|
||||
```
|
||||
|
||||
## Environment Setup
|
||||
|
||||
Create `.env` file in `/opt/motia-iii/bitbylaw/`:
|
||||
```bash
|
||||
# xAI Collections API
|
||||
XAI_MANAGEMENT_API_KEY=xai-token-xxx...
|
||||
XAI_API_KEY=xai-xxx...
|
||||
|
||||
# EspoCRM API
|
||||
ESPOCRM_URL=https://crm.bitbylaw.com
|
||||
ESPOCRM_API_KEY=xxx...
|
||||
|
||||
# Redis (for locking)
|
||||
REDIS_HOST=localhost
|
||||
REDIS_PORT=6379
|
||||
REDIS_DB_ADVOWARE_CACHE=1
|
||||
```
|
||||
|
||||
## Test Results
|
||||
|
||||
Last test run: Successfully validated preview upload functionality
|
||||
- Preview upload works with base64 Data URI format
|
||||
- Attachment created with ID: `69a71194c7c6baebf`
|
||||
- CDokumente entity updated with previewId/previewName
|
||||
- WebP format at 600x800px confirmed working
|
||||
279
tests/test_preview_upload.py
Executable file
279
tests/test_preview_upload.py
Executable file
@@ -0,0 +1,279 @@
|
||||
#!/usr/bin/env python3
|
||||
"""
|
||||
Test Script: Preview Image Upload zu EspoCRM
|
||||
|
||||
Testet das Hochladen eines Preview-Bildes (WebP) als Attachment
|
||||
zu einem CDokumente Entity via EspoCRM API.
|
||||
|
||||
Usage:
|
||||
python test_preview_upload.py <document_id>
|
||||
|
||||
Example:
|
||||
python test_preview_upload.py 69a68906ac3d0fd25
|
||||
"""
|
||||
|
||||
import asyncio
|
||||
import aiohttp
|
||||
import base64
|
||||
import os
|
||||
import sys
|
||||
from io import BytesIO
|
||||
from PIL import Image
|
||||
|
||||
|
||||
# EspoCRM Config (aus Environment oder hardcoded für Test)
|
||||
ESPOCRM_API_BASE_URL = os.getenv('ESPOCRM_API_BASE_URL', 'https://crm.bitbylaw.com/api/v1')
|
||||
ESPOCRM_API_KEY = os.getenv('ESPOCRM_API_KEY', '')
|
||||
|
||||
# Test-Parameter
|
||||
ENTITY_TYPE = 'CDokumente'
|
||||
FIELD_NAME = 'preview'
|
||||
|
||||
|
||||
def generate_test_webp(text: str = "TEST PREVIEW", size: tuple = (600, 800)) -> bytes:
|
||||
"""
|
||||
Generiert ein einfaches Test-WebP-Bild
|
||||
|
||||
Args:
|
||||
text: Text der im Bild angezeigt wird
|
||||
size: Größe des Bildes (width, height)
|
||||
|
||||
Returns:
|
||||
WebP image als bytes
|
||||
"""
|
||||
print(f"📐 Generating test image ({size[0]}x{size[1]})...")
|
||||
|
||||
# Erstelle einfaches Bild mit Text
|
||||
img = Image.new('RGB', size, color='lightblue')
|
||||
|
||||
# Optional: Füge Text hinzu (benötigt PIL ImageDraw)
|
||||
try:
|
||||
from PIL import ImageDraw, ImageFont
|
||||
draw = ImageDraw.Draw(img)
|
||||
|
||||
# Versuche ein größeres Font zu laden
|
||||
try:
|
||||
font = ImageFont.truetype("/usr/share/fonts/truetype/dejavu/DejaVuSans-Bold.ttf", 40)
|
||||
except:
|
||||
font = ImageFont.load_default()
|
||||
|
||||
# Text zentriert
|
||||
bbox = draw.textbbox((0, 0), text, font=font)
|
||||
text_width = bbox[2] - bbox[0]
|
||||
text_height = bbox[3] - bbox[1]
|
||||
x = (size[0] - text_width) // 2
|
||||
y = (size[1] - text_height) // 2
|
||||
|
||||
draw.text((x, y), text, fill='darkblue', font=font)
|
||||
except Exception as e:
|
||||
print(f"⚠️ Text rendering failed: {e}")
|
||||
|
||||
# Konvertiere zu WebP
|
||||
buffer = BytesIO()
|
||||
img.save(buffer, format='WEBP', quality=85)
|
||||
webp_bytes = buffer.getvalue()
|
||||
|
||||
print(f"✅ Test image generated: {len(webp_bytes)} bytes")
|
||||
return webp_bytes
|
||||
|
||||
|
||||
async def upload_preview_to_espocrm(
|
||||
document_id: str,
|
||||
preview_data: bytes,
|
||||
entity_type: str = 'CDokumente'
|
||||
) -> dict:
|
||||
"""
|
||||
Upload Preview zu EspoCRM Attachment API
|
||||
|
||||
Args:
|
||||
document_id: ID des CDokumente/Document Entity
|
||||
preview_data: WebP image als bytes
|
||||
entity_type: Entity-Type (CDokumente oder Document)
|
||||
|
||||
Returns:
|
||||
Response dict mit Attachment ID
|
||||
"""
|
||||
print(f"\n📤 Uploading preview to {entity_type}/{document_id}...")
|
||||
print(f" Preview size: {len(preview_data)} bytes")
|
||||
|
||||
# Base64-encode
|
||||
base64_data = base64.b64encode(preview_data).decode('ascii')
|
||||
file_data_uri = f"data:image/webp;base64,{base64_data}"
|
||||
|
||||
print(f" Base64 encoded: {len(base64_data)} chars")
|
||||
|
||||
# API Request
|
||||
url = ESPOCRM_API_BASE_URL.rstrip('/') + '/Attachment'
|
||||
headers = {
|
||||
'X-Api-Key': ESPOCRM_API_KEY,
|
||||
'Content-Type': 'application/json'
|
||||
}
|
||||
|
||||
payload = {
|
||||
'name': 'preview.webp',
|
||||
'type': 'image/webp',
|
||||
'role': 'Attachment',
|
||||
'field': FIELD_NAME,
|
||||
'relatedType': entity_type,
|
||||
'relatedId': document_id,
|
||||
'file': file_data_uri
|
||||
}
|
||||
|
||||
print(f"\n🌐 POST {url}")
|
||||
print(f" Headers: X-Api-Key={ESPOCRM_API_KEY[:20]}...")
|
||||
print(f" Payload keys: {list(payload.keys())}")
|
||||
print(f" - name: {payload['name']}")
|
||||
print(f" - type: {payload['type']}")
|
||||
print(f" - role: {payload['role']}")
|
||||
print(f" - field: {payload['field']}")
|
||||
print(f" - relatedType: {payload['relatedType']}")
|
||||
print(f" - relatedId: {payload['relatedId']}")
|
||||
print(f" - file: data:image/webp;base64,... ({len(base64_data)} chars)")
|
||||
|
||||
timeout = aiohttp.ClientTimeout(total=30)
|
||||
async with aiohttp.ClientSession(timeout=timeout) as session:
|
||||
async with session.post(url, headers=headers, json=payload) as response:
|
||||
print(f"\n📥 Response Status: {response.status}")
|
||||
print(f" Content-Type: {response.content_type}")
|
||||
|
||||
response_text = await response.text()
|
||||
|
||||
if response.status >= 400:
|
||||
print(f"\n❌ Upload FAILED!")
|
||||
print(f" Status: {response.status}")
|
||||
print(f" Response: {response_text}")
|
||||
raise Exception(f"Upload error {response.status}: {response_text}")
|
||||
|
||||
# Parse JSON response
|
||||
result = await response.json()
|
||||
attachment_id = result.get('id')
|
||||
|
||||
print(f"\n✅ Upload SUCCESSFUL!")
|
||||
print(f" Attachment ID: {attachment_id}")
|
||||
print(f" Full response: {result}")
|
||||
|
||||
return result
|
||||
|
||||
|
||||
async def update_entity_with_preview(
|
||||
document_id: str,
|
||||
attachment_id: str,
|
||||
entity_type: str = 'CDokumente'
|
||||
) -> dict:
|
||||
"""
|
||||
Update Entity mit previewId und previewName
|
||||
|
||||
Args:
|
||||
document_id: Entity ID
|
||||
attachment_id: Attachment ID vom Upload
|
||||
entity_type: Entity-Type
|
||||
|
||||
Returns:
|
||||
Updated entity data
|
||||
"""
|
||||
print(f"\n📝 Updating {entity_type}/{document_id} with previewId...")
|
||||
|
||||
url = f"{ESPOCRM_API_BASE_URL.rstrip('/')}/{entity_type}/{document_id}"
|
||||
headers = {
|
||||
'X-Api-Key': ESPOCRM_API_KEY,
|
||||
'Content-Type': 'application/json'
|
||||
}
|
||||
|
||||
payload = {
|
||||
'previewId': attachment_id,
|
||||
'previewName': 'preview.webp'
|
||||
}
|
||||
|
||||
print(f" PUT {url}")
|
||||
print(f" Payload: {payload}")
|
||||
|
||||
timeout = aiohttp.ClientTimeout(total=30)
|
||||
async with aiohttp.ClientSession(timeout=timeout) as session:
|
||||
async with session.put(url, headers=headers, json=payload) as response:
|
||||
print(f" Response Status: {response.status}")
|
||||
|
||||
if response.status >= 400:
|
||||
response_text = await response.text()
|
||||
print(f"\n❌ Update FAILED!")
|
||||
print(f" Status: {response.status}")
|
||||
print(f" Response: {response_text}")
|
||||
raise Exception(f"Update error {response.status}: {response_text}")
|
||||
|
||||
result = await response.json()
|
||||
print(f"\n✅ Entity updated successfully!")
|
||||
print(f" previewId: {result.get('previewId')}")
|
||||
print(f" previewName: {result.get('previewName')}")
|
||||
|
||||
return result
|
||||
|
||||
|
||||
async def main():
|
||||
"""Main test flow"""
|
||||
print("=" * 80)
|
||||
print("🖼️ ESPOCRM PREVIEW UPLOAD TEST")
|
||||
print("=" * 80)
|
||||
|
||||
# Check arguments
|
||||
if len(sys.argv) < 2:
|
||||
print("\n❌ Error: Document ID required!")
|
||||
print(f"\nUsage: {sys.argv[0]} <document_id>")
|
||||
print(f"Example: {sys.argv[0]} 69a68906ac3d0fd25")
|
||||
sys.exit(1)
|
||||
|
||||
document_id = sys.argv[1]
|
||||
|
||||
# Check API key
|
||||
if not ESPOCRM_API_KEY:
|
||||
print("\n❌ Error: ESPOCRM_API_KEY environment variable not set!")
|
||||
sys.exit(1)
|
||||
|
||||
print(f"\n📋 Test Parameters:")
|
||||
print(f" API Base URL: {ESPOCRM_API_BASE_URL}")
|
||||
print(f" API Key: {ESPOCRM_API_KEY[:20]}...")
|
||||
print(f" Entity Type: {ENTITY_TYPE}")
|
||||
print(f" Document ID: {document_id}")
|
||||
print(f" Field: {FIELD_NAME}")
|
||||
|
||||
try:
|
||||
# Step 1: Generate test image
|
||||
print("\n" + "=" * 80)
|
||||
print("STEP 1: Generate Test Image")
|
||||
print("=" * 80)
|
||||
preview_data = generate_test_webp(f"Preview Test\n{document_id[:8]}", size=(600, 800))
|
||||
|
||||
# Step 2: Upload to EspoCRM
|
||||
print("\n" + "=" * 80)
|
||||
print("STEP 2: Upload to EspoCRM Attachment API")
|
||||
print("=" * 80)
|
||||
result = await upload_preview_to_espocrm(document_id, preview_data, ENTITY_TYPE)
|
||||
attachment_id = result.get('id')
|
||||
|
||||
# Step 3: Update Entity
|
||||
print("\n" + "=" * 80)
|
||||
print("STEP 3: Update Entity with Preview Reference")
|
||||
print("=" * 80)
|
||||
await update_entity_with_preview(document_id, attachment_id, ENTITY_TYPE)
|
||||
|
||||
# Success summary
|
||||
print("\n" + "=" * 80)
|
||||
print("✅ TEST SUCCESSFUL!")
|
||||
print("=" * 80)
|
||||
print(f"\n📊 Summary:")
|
||||
print(f" - Attachment ID: {attachment_id}")
|
||||
print(f" - Entity: {ENTITY_TYPE}/{document_id}")
|
||||
print(f" - Preview Size: {len(preview_data)} bytes")
|
||||
print(f"\n🔗 View in EspoCRM:")
|
||||
print(f" {ESPOCRM_API_BASE_URL.replace('/api/v1', '')}/#CDokumente/view/{document_id}")
|
||||
|
||||
except Exception as e:
|
||||
print("\n" + "=" * 80)
|
||||
print("❌ TEST FAILED!")
|
||||
print("=" * 80)
|
||||
print(f"\nError: {e}")
|
||||
import traceback
|
||||
traceback.print_exc()
|
||||
sys.exit(1)
|
||||
|
||||
|
||||
if __name__ == '__main__':
|
||||
asyncio.run(main())
|
||||
253
tests/test_thumbnail_generation.py
Normal file
253
tests/test_thumbnail_generation.py
Normal file
@@ -0,0 +1,253 @@
|
||||
#!/usr/bin/env python3
|
||||
"""
|
||||
Test script for Document Thumbnail Generation
|
||||
Tests the complete flow:
|
||||
1. Create a test document in EspoCRM
|
||||
2. Upload a file attachment
|
||||
3. Trigger the webhook (or wait for automatic trigger)
|
||||
4. Verify preview generation
|
||||
"""
|
||||
|
||||
import asyncio
|
||||
import aiohttp
|
||||
import os
|
||||
import sys
|
||||
import json
|
||||
from pathlib import Path
|
||||
from io import BytesIO
|
||||
from PIL import Image
|
||||
|
||||
# Add bitbylaw to path
|
||||
sys.path.insert(0, str(Path(__file__).parent))
|
||||
|
||||
from services.espocrm import EspoCRMAPI
|
||||
|
||||
|
||||
async def create_test_image(width: int = 800, height: int = 600) -> bytes:
|
||||
"""Create a simple test PNG image"""
|
||||
img = Image.new('RGB', (width, height), color='lightblue')
|
||||
|
||||
# Add some text/pattern so it's not just a solid color
|
||||
from PIL import ImageDraw, ImageFont
|
||||
draw = ImageDraw.Draw(img)
|
||||
|
||||
# Draw some shapes
|
||||
draw.rectangle([50, 50, width-50, height-50], outline='darkblue', width=5)
|
||||
draw.ellipse([width//4, height//4, 3*width//4, 3*height//4], outline='red', width=3)
|
||||
|
||||
# Add text
|
||||
try:
|
||||
font = ImageFont.truetype("/usr/share/fonts/truetype/dejavu/DejaVuSans-Bold.ttf", 48)
|
||||
except:
|
||||
font = None
|
||||
|
||||
text = "TEST IMAGE\nFor Thumbnail\nGeneration"
|
||||
draw.text((width//2, height//2), text, fill='black', anchor='mm', font=font, align='center')
|
||||
|
||||
# Save to bytes
|
||||
buffer = BytesIO()
|
||||
img.save(buffer, format='PNG')
|
||||
return buffer.getvalue()
|
||||
|
||||
|
||||
async def create_test_document(espocrm: EspoCRMAPI) -> str:
|
||||
"""Create a test document in EspoCRM"""
|
||||
print("\n📄 Creating test document in EspoCRM...")
|
||||
|
||||
document_data = {
|
||||
"name": f"Test Thumbnail Generation {asyncio.get_event_loop().time()}",
|
||||
"status": "Active",
|
||||
"dateiStatus": "Neu", # This should trigger preview generation
|
||||
"type": "Image",
|
||||
"description": "Automated test document for thumbnail generation"
|
||||
}
|
||||
|
||||
result = await espocrm.create_entity("Document", document_data)
|
||||
doc_id = result.get("id")
|
||||
|
||||
print(f"✅ Document created: {doc_id}")
|
||||
print(f" Name: {result.get('name')}")
|
||||
print(f" Datei-Status: {result.get('dateiStatus')}")
|
||||
|
||||
return doc_id
|
||||
|
||||
|
||||
async def upload_test_file(espocrm: EspoCRMAPI, doc_id: str) -> str:
|
||||
"""Upload a test image file to the document"""
|
||||
print(f"\n📤 Uploading test image to document {doc_id}...")
|
||||
|
||||
# Create test image
|
||||
image_data = await create_test_image(1200, 900)
|
||||
print(f" Generated test image: {len(image_data)} bytes")
|
||||
|
||||
# Upload to EspoCRM
|
||||
attachment = await espocrm.upload_attachment(
|
||||
file_content=image_data,
|
||||
filename="test_image.png",
|
||||
parent_type="Document",
|
||||
parent_id=doc_id,
|
||||
field="file",
|
||||
mime_type="image/png",
|
||||
role="Attachment"
|
||||
)
|
||||
|
||||
attachment_id = attachment.get("id")
|
||||
print(f"✅ File uploaded: {attachment_id}")
|
||||
print(f" Filename: {attachment.get('name')}")
|
||||
print(f" Size: {attachment.get('size')} bytes")
|
||||
|
||||
return attachment_id
|
||||
|
||||
|
||||
async def trigger_webhook(doc_id: str, action: str = "update"):
|
||||
"""Manually trigger the document webhook"""
|
||||
print(f"\n🔔 Triggering webhook for document {doc_id}...")
|
||||
|
||||
webhook_url = f"http://localhost:7777/vmh/webhook/document/{action}"
|
||||
payload = {
|
||||
"entityType": "Document",
|
||||
"entity": {
|
||||
"id": doc_id,
|
||||
"entityType": "Document"
|
||||
},
|
||||
"data": {
|
||||
"entity": {
|
||||
"id": doc_id
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
async with aiohttp.ClientSession() as session:
|
||||
async with session.post(webhook_url, json=payload) as response:
|
||||
status = response.status
|
||||
text = await response.text()
|
||||
|
||||
if status == 200:
|
||||
print(f"✅ Webhook triggered successfully")
|
||||
print(f" Response: {text}")
|
||||
else:
|
||||
print(f"❌ Webhook failed: {status}")
|
||||
print(f" Response: {text}")
|
||||
|
||||
return status == 200
|
||||
|
||||
|
||||
async def check_preview_generated(espocrm: EspoCRMAPI, doc_id: str, max_wait: int = 30):
|
||||
"""Check if preview was generated (poll for a few seconds)"""
|
||||
print(f"\n🔍 Checking for preview generation (max {max_wait}s)...")
|
||||
|
||||
for i in range(max_wait):
|
||||
await asyncio.sleep(1)
|
||||
|
||||
# Get document
|
||||
doc = await espocrm.get_entity("Document", doc_id)
|
||||
|
||||
# Check if preview field is populated
|
||||
preview_id = doc.get("previewId")
|
||||
if preview_id:
|
||||
print(f"\n✅ Preview generated!")
|
||||
print(f" Preview Attachment ID: {preview_id}")
|
||||
print(f" Preview Name: {doc.get('previewName')}")
|
||||
print(f" Preview Type: {doc.get('previewType')}")
|
||||
|
||||
# Try to download and check the preview
|
||||
try:
|
||||
preview_data = await espocrm.download_attachment(preview_id)
|
||||
print(f" Preview Size: {len(preview_data)} bytes")
|
||||
|
||||
# Verify it's a WebP image
|
||||
from PIL import Image
|
||||
img = Image.open(BytesIO(preview_data))
|
||||
print(f" Preview Format: {img.format}")
|
||||
print(f" Preview Dimensions: {img.width}x{img.height}")
|
||||
|
||||
if img.format == "WEBP":
|
||||
print(" ✅ Format is WebP as expected")
|
||||
if img.width <= 600 and img.height <= 800:
|
||||
print(" ✅ Dimensions within expected range")
|
||||
|
||||
except Exception as e:
|
||||
print(f" ⚠️ Could not verify preview: {e}")
|
||||
|
||||
return True
|
||||
|
||||
if (i + 1) % 5 == 0:
|
||||
print(f" Still waiting... ({i + 1}s)")
|
||||
|
||||
print(f"\n❌ Preview not generated after {max_wait}s")
|
||||
return False
|
||||
|
||||
|
||||
async def cleanup_test_document(espocrm: EspoCRMAPI, doc_id: str):
|
||||
"""Delete the test document"""
|
||||
print(f"\n🗑️ Cleaning up test document {doc_id}...")
|
||||
try:
|
||||
await espocrm.delete_entity("Document", doc_id)
|
||||
print("✅ Test document deleted")
|
||||
except Exception as e:
|
||||
print(f"⚠️ Could not delete test document: {e}")
|
||||
|
||||
|
||||
async def main():
|
||||
print("=" * 80)
|
||||
print("THUMBNAIL GENERATION TEST")
|
||||
print("=" * 80)
|
||||
|
||||
# Initialize EspoCRM API
|
||||
espocrm = EspoCRMAPI()
|
||||
|
||||
doc_id = None
|
||||
try:
|
||||
# Step 1: Create test document
|
||||
doc_id = await create_test_document(espocrm)
|
||||
|
||||
# Step 2: Upload test file
|
||||
attachment_id = await upload_test_file(espocrm, doc_id)
|
||||
|
||||
# Step 3: Update document to trigger webhook (set dateiStatus to trigger sync)
|
||||
print(f"\n🔄 Updating document to trigger webhook...")
|
||||
await espocrm.update_entity("Document", doc_id, {
|
||||
"dateiStatus": "Neu" # This should trigger the webhook
|
||||
})
|
||||
print("✅ Document updated")
|
||||
|
||||
# Step 4: Wait a bit for webhook to be processed
|
||||
print("\n⏳ Waiting 3 seconds for webhook processing...")
|
||||
await asyncio.sleep(3)
|
||||
|
||||
# Step 5: Check if preview was generated
|
||||
success = await check_preview_generated(espocrm, doc_id, max_wait=20)
|
||||
|
||||
# Summary
|
||||
print("\n" + "=" * 80)
|
||||
if success:
|
||||
print("✅ TEST PASSED - Preview generation successful!")
|
||||
else:
|
||||
print("❌ TEST FAILED - Preview was not generated")
|
||||
print("\nCheck logs with:")
|
||||
print(" sudo journalctl -u motia.service --since '2 minutes ago' | grep -E '(PREVIEW|Document)'")
|
||||
print("=" * 80)
|
||||
|
||||
# Ask if we should clean up
|
||||
print(f"\nTest document ID: {doc_id}")
|
||||
cleanup = input("\nDelete test document? (y/N): ").strip().lower()
|
||||
if cleanup == 'y':
|
||||
await cleanup_test_document(espocrm, doc_id)
|
||||
else:
|
||||
print(f"ℹ️ Test document kept: {doc_id}")
|
||||
print(f" View in EspoCRM: https://crm.bitbylaw.com/#Document/view/{doc_id}")
|
||||
|
||||
except Exception as e:
|
||||
print(f"\n❌ Test failed with error: {e}")
|
||||
import traceback
|
||||
traceback.print_exc()
|
||||
|
||||
if doc_id:
|
||||
print(f"\nTest document ID: {doc_id}")
|
||||
cleanup = input("\nDelete test document? (y/N): ").strip().lower()
|
||||
if cleanup == 'y':
|
||||
await cleanup_test_document(espocrm, doc_id)
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
asyncio.run(main())
|
||||
788
tests/test_xai_collections_api.py
Executable file
788
tests/test_xai_collections_api.py
Executable file
@@ -0,0 +1,788 @@
|
||||
#!/usr/bin/env python3
|
||||
"""
|
||||
xAI Collections API Test Script
|
||||
|
||||
Tests all critical operations for our document sync requirements:
|
||||
1. File upload and ID behavior (collection-specific vs global?)
|
||||
2. Same file in multiple collections (shared file_id?)
|
||||
3. CRUD operations on collections
|
||||
4. CRUD operations on documents
|
||||
5. Response structures and metadata
|
||||
6. Update/versioning behavior
|
||||
|
||||
Usage:
|
||||
export XAI_API_KEY="xai-..."
|
||||
python test_xai_collections_api.py
|
||||
"""
|
||||
|
||||
import os
|
||||
import sys
|
||||
import json
|
||||
import asyncio
|
||||
import aiohttp
|
||||
from typing import Optional, Dict, Any, List
|
||||
from datetime import datetime
|
||||
import tempfile
|
||||
|
||||
# Configuration
|
||||
XAI_MANAGEMENT_URL = os.getenv("XAI_MANAGEMENT_URL", "https://management-api.x.ai")
|
||||
XAI_FILES_URL = os.getenv("XAI_FILES_URL", "https://api.x.ai")
|
||||
XAI_MANAGEMENT_KEY = os.getenv("XAI_MANAGEMENT_KEY", "") # Management API Key
|
||||
XAI_API_KEY = os.getenv("XAI_API_KEY", "") # Regular API Key for file upload
|
||||
|
||||
if not XAI_MANAGEMENT_KEY:
|
||||
print("❌ ERROR: XAI_MANAGEMENT_KEY environment variable not set!")
|
||||
print(" export XAI_MANAGEMENT_KEY='xai-token-...'")
|
||||
sys.exit(1)
|
||||
|
||||
if not XAI_API_KEY:
|
||||
print("❌ ERROR: XAI_API_KEY environment variable not set!")
|
||||
print(" export XAI_API_KEY='xai-...'")
|
||||
sys.exit(1)
|
||||
|
||||
|
||||
class Colors:
|
||||
"""ANSI color codes for terminal output"""
|
||||
HEADER = '\033[95m'
|
||||
BLUE = '\033[94m'
|
||||
CYAN = '\033[96m'
|
||||
GREEN = '\033[92m'
|
||||
YELLOW = '\033[93m'
|
||||
RED = '\033[91m'
|
||||
BOLD = '\033[1m'
|
||||
UNDERLINE = '\033[4m'
|
||||
END = '\033[0m'
|
||||
|
||||
|
||||
def print_header(text: str):
|
||||
print(f"\n{Colors.BOLD}{Colors.CYAN}{'='*70}{Colors.END}")
|
||||
print(f"{Colors.BOLD}{Colors.CYAN}{text}{Colors.END}")
|
||||
print(f"{Colors.BOLD}{Colors.CYAN}{'='*70}{Colors.END}\n")
|
||||
|
||||
|
||||
def print_success(text: str):
|
||||
print(f"{Colors.GREEN}✅ {text}{Colors.END}")
|
||||
|
||||
|
||||
def print_error(text: str):
|
||||
print(f"{Colors.RED}❌ {text}{Colors.END}")
|
||||
|
||||
|
||||
def print_info(text: str):
|
||||
print(f"{Colors.BLUE}ℹ️ {text}{Colors.END}")
|
||||
|
||||
|
||||
def print_warning(text: str):
|
||||
print(f"{Colors.YELLOW}⚠️ {text}{Colors.END}")
|
||||
|
||||
|
||||
def print_json(data: Any, title: Optional[str] = None):
|
||||
if title:
|
||||
print(f"{Colors.BOLD}{title}:{Colors.END}")
|
||||
print(json.dumps(data, indent=2, ensure_ascii=False))
|
||||
|
||||
|
||||
class XAICollectionsTestClient:
|
||||
"""Test client for xAI Collections API"""
|
||||
|
||||
def __init__(self):
|
||||
self.management_url = XAI_MANAGEMENT_URL
|
||||
self.files_url = XAI_FILES_URL
|
||||
self.management_key = XAI_MANAGEMENT_KEY
|
||||
self.api_key = XAI_API_KEY
|
||||
self.session: Optional[aiohttp.ClientSession] = None
|
||||
|
||||
# Test state
|
||||
self.created_collections: List[str] = []
|
||||
self.uploaded_files: List[str] = []
|
||||
self.test_results: Dict[str, bool] = {}
|
||||
|
||||
async def __aenter__(self):
|
||||
# Session without default Content-Type (set per-request)
|
||||
self.session = aiohttp.ClientSession(
|
||||
timeout=aiohttp.ClientTimeout(total=30)
|
||||
)
|
||||
return self
|
||||
|
||||
async def __aexit__(self, *args):
|
||||
if self.session:
|
||||
await self.session.close()
|
||||
|
||||
async def _request(self, method: str, path: str, use_files_api: bool = False, **kwargs) -> tuple[int, Any]:
|
||||
"""Make HTTP request and return (status, response_data)"""
|
||||
base_url = self.files_url if use_files_api else self.management_url
|
||||
url = f"{base_url}{path}"
|
||||
|
||||
# Set headers per-request
|
||||
if 'headers' not in kwargs:
|
||||
kwargs['headers'] = {}
|
||||
|
||||
# Set authorization
|
||||
if use_files_api:
|
||||
kwargs['headers']['Authorization'] = f"Bearer {self.api_key}"
|
||||
else:
|
||||
kwargs['headers']['Authorization'] = f"Bearer {self.management_key}"
|
||||
|
||||
# Set Content-Type for JSON requests
|
||||
if 'json' in kwargs:
|
||||
kwargs['headers']['Content-Type'] = 'application/json'
|
||||
|
||||
print_info(f"{method} {url}")
|
||||
print_info(f"Headers: {kwargs.get('headers', {})}")
|
||||
|
||||
try:
|
||||
async with self.session.request(method, url, **kwargs) as response:
|
||||
status = response.status
|
||||
|
||||
try:
|
||||
data = await response.json()
|
||||
except:
|
||||
text = await response.text()
|
||||
data = {"_raw_text": text} if text else {}
|
||||
|
||||
if status < 400:
|
||||
print_success(f"Response: {status}")
|
||||
else:
|
||||
print_error(f"Response: {status}")
|
||||
|
||||
return status, data
|
||||
|
||||
except Exception as e:
|
||||
print_error(f"Request failed: {e}")
|
||||
return 0, {"error": str(e)}
|
||||
|
||||
# ========================================================================
|
||||
# COLLECTION OPERATIONS
|
||||
# ========================================================================
|
||||
|
||||
async def create_collection(self, name: str, metadata: Optional[Dict] = None) -> tuple[int, Any]:
|
||||
"""POST /v1/collections"""
|
||||
payload = {
|
||||
"collection_name": name, # xAI uses "collection_name" not "name"
|
||||
"metadata": metadata or {}
|
||||
}
|
||||
status, data = await self._request("POST", "/v1/collections", json=payload)
|
||||
|
||||
if status == 200 or status == 201:
|
||||
# Try different possible field names for collection ID
|
||||
collection_id = data.get("id") or data.get("collection_id") or data.get("collectionId")
|
||||
if collection_id:
|
||||
self.created_collections.append(collection_id)
|
||||
print_success(f"Created collection: {collection_id}")
|
||||
|
||||
return status, data
|
||||
|
||||
async def get_collection(self, collection_id: str) -> tuple[int, Any]:
|
||||
"""GET /v1/collections/{collection_id}"""
|
||||
return await self._request("GET", f"/v1/collections/{collection_id}")
|
||||
|
||||
async def list_collections(self) -> tuple[int, Any]:
|
||||
"""GET /v1/collections"""
|
||||
return await self._request("GET", "/v1/collections")
|
||||
|
||||
async def update_collection(self, collection_id: str, name: Optional[str] = None,
|
||||
metadata: Optional[Dict] = None) -> tuple[int, Any]:
|
||||
"""PUT /v1/collections/{collection_id}"""
|
||||
payload = {}
|
||||
if name:
|
||||
payload["collection_name"] = name # xAI uses "collection_name"
|
||||
if metadata:
|
||||
payload["metadata"] = metadata
|
||||
|
||||
return await self._request("PUT", f"/v1/collections/{collection_id}", json=payload)
|
||||
|
||||
async def delete_collection(self, collection_id: str) -> tuple[int, Any]:
|
||||
"""DELETE /v1/collections/{collection_id}"""
|
||||
status, data = await self._request("DELETE", f"/v1/collections/{collection_id}")
|
||||
|
||||
if status == 200 or status == 204:
|
||||
if collection_id in self.created_collections:
|
||||
self.created_collections.remove(collection_id)
|
||||
|
||||
return status, data
|
||||
|
||||
# ========================================================================
|
||||
# FILE OPERATIONS (multiple upload methods)
|
||||
# ========================================================================
|
||||
|
||||
async def upload_file_multipart(self, content: bytes, filename: str,
|
||||
mime_type: str = "text/plain") -> tuple[int, Any]:
|
||||
"""
|
||||
Method 0: Multipart form-data upload (what the server actually expects!)
|
||||
POST /v1/files with multipart/form-data
|
||||
"""
|
||||
print_info("METHOD 0: Multipart Form-Data Upload (POST /v1/files)")
|
||||
|
||||
# Create multipart form data
|
||||
form = aiohttp.FormData()
|
||||
form.add_field('file', content, filename=filename, content_type=mime_type)
|
||||
|
||||
print_info(f"Uploading {len(content)} bytes as multipart/form-data")
|
||||
|
||||
# Use _request but with form data instead of json
|
||||
base_url = self.files_url
|
||||
url = f"{base_url}/v1/files"
|
||||
|
||||
headers = {
|
||||
"Authorization": f"Bearer {self.api_key}"
|
||||
# Do NOT set Content-Type - aiohttp will set it with boundary
|
||||
}
|
||||
|
||||
print_info(f"POST {url}")
|
||||
print_info(f"Headers: {headers}")
|
||||
|
||||
try:
|
||||
async with self.session.request("POST", url, data=form, headers=headers) as response:
|
||||
status = response.status
|
||||
|
||||
try:
|
||||
data = await response.json()
|
||||
except:
|
||||
text = await response.text()
|
||||
data = {"_raw_text": text} if text else {}
|
||||
|
||||
if status < 400:
|
||||
print_success(f"Response: {status}")
|
||||
else:
|
||||
print_error(f"Response: {status}")
|
||||
|
||||
return status, data
|
||||
|
||||
except Exception as e:
|
||||
print_error(f"Request failed: {e}")
|
||||
return 0, {"error": str(e)}
|
||||
|
||||
async def upload_file_direct(self, content: bytes, filename: str,
|
||||
mime_type: str = "text/plain") -> tuple[int, Any]:
|
||||
"""
|
||||
Method 1: Direct upload to xAI Files API
|
||||
POST /v1/files with JSON body containing base64-encoded data
|
||||
"""
|
||||
import base64
|
||||
|
||||
print_info("METHOD 1: Direct Upload (POST /v1/files with JSON)")
|
||||
|
||||
# Encode file content as base64
|
||||
data_b64 = base64.b64encode(content).decode('ascii')
|
||||
|
||||
payload = {
|
||||
"name": filename,
|
||||
"content_type": mime_type,
|
||||
"data": data_b64
|
||||
}
|
||||
|
||||
print_info(f"Uploading {len(content)} bytes as base64 ({len(data_b64)} chars)")
|
||||
|
||||
status, data = await self._request(
|
||||
"POST",
|
||||
"/v1/files",
|
||||
use_files_api=True,
|
||||
json=payload
|
||||
)
|
||||
|
||||
return status, data
|
||||
|
||||
async def upload_file_chunked(self, content: bytes, filename: str,
|
||||
mime_type: str = "text/plain") -> tuple[int, Any]:
|
||||
"""
|
||||
Method 2: Initialize + Chunk streaming upload
|
||||
POST /v1/files:initialize → POST /v1/files:uploadChunks
|
||||
"""
|
||||
import base64
|
||||
|
||||
print_info("METHOD 2: Initialize + Chunk Streaming")
|
||||
|
||||
# Step 1: Initialize upload
|
||||
print_info("Step 1: Initialize upload")
|
||||
init_payload = {
|
||||
"name": filename,
|
||||
"content_type": mime_type
|
||||
}
|
||||
|
||||
status, data = await self._request(
|
||||
"POST",
|
||||
"/v1/files:initialize",
|
||||
use_files_api=True,
|
||||
json=init_payload
|
||||
)
|
||||
|
||||
print_json(data, "Initialize Response")
|
||||
|
||||
if status not in [200, 201]:
|
||||
print_error("Failed to initialize upload")
|
||||
return status, data
|
||||
|
||||
file_id = data.get("file_id")
|
||||
if not file_id:
|
||||
print_error("No file_id in initialize response")
|
||||
return status, data
|
||||
|
||||
print_success(f"Initialized upload with file_id: {file_id}")
|
||||
|
||||
# Step 2: Upload chunks
|
||||
print_info(f"Step 2: Upload {len(content)} bytes in chunks")
|
||||
|
||||
# Encode content as base64 for chunk upload
|
||||
chunk_b64 = base64.b64encode(content).decode('ascii')
|
||||
|
||||
chunk_payload = {
|
||||
"file_id": file_id,
|
||||
"chunk": chunk_b64
|
||||
}
|
||||
|
||||
status, data = await self._request(
|
||||
"POST",
|
||||
"/v1/files:uploadChunks",
|
||||
use_files_api=True,
|
||||
json=chunk_payload
|
||||
)
|
||||
|
||||
print_json(data, "Upload Chunks Response")
|
||||
|
||||
if status in [200, 201]:
|
||||
print_success(f"Uploaded file chunks: {file_id}")
|
||||
self.uploaded_files.append(file_id)
|
||||
|
||||
return status, data
|
||||
|
||||
async def upload_file(self, content: bytes, filename: str,
|
||||
mime_type: str = "text/plain") -> tuple[int, Any]:
|
||||
"""
|
||||
Try multiple upload methods until one succeeds
|
||||
"""
|
||||
print_info("Trying upload methods...")
|
||||
|
||||
# Try Method 0: Multipart form-data (what the server really wants!)
|
||||
status0, data0 = await self.upload_file_multipart(content, filename, mime_type)
|
||||
|
||||
if status0 in [200, 201]:
|
||||
file_id = data0.get("id") or data0.get("file_id") # Try both field names
|
||||
if file_id:
|
||||
self.uploaded_files.append(file_id)
|
||||
print_success(f"✅ Multipart upload succeeded: {file_id}")
|
||||
return status0, data0
|
||||
else:
|
||||
print_error("No 'id' or 'file_id' in response")
|
||||
print_json(data0, "Response data")
|
||||
|
||||
print_warning(f"Multipart upload failed ({status0}), trying JSON upload...")
|
||||
|
||||
# Try Method 1: Direct upload with JSON
|
||||
status1, data1 = await self.upload_file_direct(content, filename, mime_type)
|
||||
|
||||
if status1 in [200, 201]:
|
||||
file_id = data1.get("file_id")
|
||||
if file_id:
|
||||
self.uploaded_files.append(file_id)
|
||||
print_success(f"✅ Direct upload succeeded: {file_id}")
|
||||
return status1, data1
|
||||
|
||||
print_warning(f"Direct upload failed ({status1}), trying chunked upload...")
|
||||
|
||||
# Try Method 2: Initialize + Chunks
|
||||
status2, data2 = await self.upload_file_chunked(content, filename, mime_type)
|
||||
|
||||
if status2 in [200, 201]:
|
||||
print_success("✅ Chunked upload succeeded")
|
||||
return status2, data2
|
||||
|
||||
print_error("❌ All upload methods failed")
|
||||
return status0, data0 # Return multipart method's error
|
||||
|
||||
# ========================================================================
|
||||
# COLLECTION DOCUMENT OPERATIONS
|
||||
# ========================================================================
|
||||
|
||||
async def add_document_to_collection(self, collection_id: str,
|
||||
file_id: str) -> tuple[int, Any]:
|
||||
"""POST /v1/collections/{collection_id}/documents/{file_id}"""
|
||||
return await self._request("POST",
|
||||
f"/v1/collections/{collection_id}/documents/{file_id}")
|
||||
|
||||
async def get_collection_documents(self, collection_id: str) -> tuple[int, Any]:
|
||||
"""GET /v1/collections/{collection_id}/documents"""
|
||||
return await self._request("GET",
|
||||
f"/v1/collections/{collection_id}/documents")
|
||||
|
||||
async def get_collection_document(self, collection_id: str,
|
||||
file_id: str) -> tuple[int, Any]:
|
||||
"""GET /v1/collections/{collection_id}/documents/{file_id}"""
|
||||
return await self._request("GET",
|
||||
f"/v1/collections/{collection_id}/documents/{file_id}")
|
||||
|
||||
async def update_collection_document(self, collection_id: str, file_id: str,
|
||||
metadata: Dict) -> tuple[int, Any]:
|
||||
"""PATCH /v1/collections/{collection_id}/documents/{file_id}"""
|
||||
return await self._request("PATCH",
|
||||
f"/v1/collections/{collection_id}/documents/{file_id}",
|
||||
json={"metadata": metadata})
|
||||
|
||||
async def remove_document_from_collection(self, collection_id: str,
|
||||
file_id: str) -> tuple[int, Any]:
|
||||
"""DELETE /v1/collections/{collection_id}/documents/{file_id}"""
|
||||
return await self._request("DELETE",
|
||||
f"/v1/collections/{collection_id}/documents/{file_id}")
|
||||
|
||||
async def batch_get_documents(self, collection_id: str,
|
||||
file_ids: List[str]) -> tuple[int, Any]:
|
||||
"""GET /v1/collections/{collection_id}/documents:batchGet"""
|
||||
params = {"fileIds": ",".join(file_ids)}
|
||||
return await self._request("GET",
|
||||
f"/v1/collections/{collection_id}/documents:batchGet",
|
||||
params=params)
|
||||
|
||||
# ========================================================================
|
||||
# TEST SCENARIOS
|
||||
# ========================================================================
|
||||
|
||||
async def test_basic_collection_crud(self):
|
||||
"""Test 1: Basic Collection CRUD operations"""
|
||||
print_header("TEST 1: Basic Collection CRUD")
|
||||
|
||||
# Create
|
||||
print_info("Creating collection...")
|
||||
status, data = await self.create_collection(
|
||||
name="Test Collection 1",
|
||||
metadata={"test": True, "purpose": "API testing"}
|
||||
)
|
||||
print_json(data, "Response")
|
||||
|
||||
if status not in [200, 201]:
|
||||
print_error("Failed to create collection")
|
||||
self.test_results["collection_crud"] = False
|
||||
return None
|
||||
|
||||
# Try different possible field names for collection ID
|
||||
collection_id = data.get("id") or data.get("collection_id") or data.get("collectionId")
|
||||
if not collection_id:
|
||||
print_error("No collection ID field in response")
|
||||
print_json(data, "Response Data")
|
||||
self.test_results["collection_crud"] = False
|
||||
return None
|
||||
|
||||
print_success(f"Collection created: {collection_id}")
|
||||
|
||||
# Read
|
||||
print_info("Reading collection...")
|
||||
status, data = await self.get_collection(collection_id)
|
||||
print_json(data, "Response")
|
||||
|
||||
# Update
|
||||
print_info("Updating collection...")
|
||||
status, data = await self.update_collection(
|
||||
collection_id,
|
||||
name="Test Collection 1 (Updated)",
|
||||
metadata={"test": True, "updated": True}
|
||||
)
|
||||
print_json(data, "Response")
|
||||
|
||||
self.test_results["collection_crud"] = True
|
||||
return collection_id
|
||||
|
||||
async def test_file_upload_and_structure(self, collection_id: str):
|
||||
"""Test 2: File upload (two-step process)"""
|
||||
print_header("TEST 2: File Upload (Two-Step) & Response Structure")
|
||||
|
||||
# Create test file content
|
||||
test_content = b"""
|
||||
This is a test document for xAI Collections API testing.
|
||||
|
||||
Topic: German Contract Law
|
||||
|
||||
Key Points:
|
||||
- Contracts require offer and acceptance
|
||||
- Consideration is necessary
|
||||
- Written form may be required for certain contracts
|
||||
|
||||
This document contains sufficient content for testing.
|
||||
"""
|
||||
|
||||
# STEP 1: Upload file to Files API
|
||||
print_info("STEP 1: Uploading file to Files API (api.x.ai)...")
|
||||
status, data = await self.upload_file(
|
||||
content=test_content,
|
||||
filename="test_document.txt",
|
||||
mime_type="text/plain"
|
||||
)
|
||||
print_json(data, "Files API Upload Response")
|
||||
|
||||
if status not in [200, 201]:
|
||||
print_error("File upload to Files API failed")
|
||||
self.test_results["file_upload"] = False
|
||||
return None
|
||||
|
||||
# Try both field names: 'id' (Files API) or 'file_id' (Collections API)
|
||||
file_id = data.get("id") or data.get("file_id")
|
||||
if not file_id:
|
||||
print_error("No 'id' or 'file_id' field in response")
|
||||
print_json(data, "Response for debugging")
|
||||
self.test_results["file_upload"] = False
|
||||
return None
|
||||
|
||||
print_success(f"File uploaded to Files API: {file_id}")
|
||||
|
||||
# STEP 2: Add file to collection using Management API
|
||||
print_info("STEP 2: Adding file to collection (management-api.x.ai)...")
|
||||
status2, data2 = await self.add_document_to_collection(collection_id, file_id)
|
||||
print_json(data2, "Add to Collection Response")
|
||||
|
||||
if status2 not in [200, 201]:
|
||||
print_error("Failed to add file to collection")
|
||||
self.test_results["file_upload"] = False
|
||||
return None
|
||||
|
||||
print_success(f"File added to collection: {file_id}")
|
||||
self.test_results["file_upload"] = True
|
||||
return file_id
|
||||
|
||||
async def test_document_in_collection(self, collection_id: str, file_id: str):
|
||||
"""Test 3: Verify document is in collection and get details"""
|
||||
print_header("TEST 3: Verify Document in Collection")
|
||||
|
||||
# Verify by listing documents
|
||||
print_info("Listing collection documents...")
|
||||
status, data = await self.get_collection_documents(collection_id)
|
||||
print_json(data, "Collection Documents")
|
||||
|
||||
if status not in [200, 201]:
|
||||
print_error("Failed to list documents")
|
||||
self.test_results["add_to_collection"] = False
|
||||
return False
|
||||
|
||||
# Get specific document
|
||||
print_info("Getting specific document...")
|
||||
status, data = await self.get_collection_document(collection_id, file_id)
|
||||
print_json(data, "Document Details")
|
||||
|
||||
if status not in [200, 201]:
|
||||
print_error("Failed to get document details")
|
||||
self.test_results["add_to_collection"] = False
|
||||
return False
|
||||
|
||||
print_success("Document verified in collection")
|
||||
self.test_results["add_to_collection"] = True
|
||||
return True
|
||||
|
||||
async def test_shared_file_across_collections(self, file_id: str):
|
||||
"""Test 4: CRITICAL - Can same file_id be used in multiple collections?"""
|
||||
print_header("TEST 4: Shared File Across Collections (CRITICAL)")
|
||||
|
||||
# Create second collection
|
||||
print_info("Creating second collection...")
|
||||
status, data = await self.create_collection(
|
||||
name="Test Collection 2",
|
||||
metadata={"test": True, "purpose": "Multi-collection test"}
|
||||
)
|
||||
|
||||
if status not in [200, 201]:
|
||||
print_error("Failed to create second collection")
|
||||
self.test_results["shared_file"] = False
|
||||
return
|
||||
|
||||
collection2_id = data.get("collection_id") or data.get("id")
|
||||
print_success(f"Collection 2 created: {collection2_id}")
|
||||
|
||||
# Try to add SAME file_id to second collection
|
||||
print_info(f"Adding SAME file_id {file_id} to collection 2...")
|
||||
|
||||
status, data = await self.add_document_to_collection(collection2_id, file_id)
|
||||
print_json(data, "Response from adding existing file_id to second collection")
|
||||
|
||||
if status not in [200, 201]:
|
||||
print_error("Failed to add same file to second collection")
|
||||
print_warning("⚠️ Files might be collection-specific (BAD for our use case)")
|
||||
self.test_results["shared_file"] = False
|
||||
return
|
||||
|
||||
print_success("✅ SAME FILE_ID CAN BE USED IN MULTIPLE COLLECTIONS!")
|
||||
print_success("✅ This is PERFECT for our architecture!")
|
||||
|
||||
# Verify both collections have the file
|
||||
print_info("Verifying file in both collections...")
|
||||
|
||||
status1, data1 = await self.get_collection_documents(self.created_collections[0])
|
||||
status2, data2 = await self.get_collection_documents(collection2_id)
|
||||
|
||||
print_json(data1, "Collection 1 Documents")
|
||||
print_json(data2, "Collection 2 Documents")
|
||||
|
||||
# Extract file_ids from both collections to verify they match
|
||||
docs1 = data1.get("documents", [])
|
||||
docs2 = data2.get("documents", [])
|
||||
|
||||
file_ids_1 = [d.get("file_metadata", {}).get("file_id") for d in docs1]
|
||||
file_ids_2 = [d.get("file_metadata", {}).get("file_id") for d in docs2]
|
||||
|
||||
if file_id in file_ids_1 and file_id in file_ids_2:
|
||||
print_success(f"✅ CONFIRMED: file_id {file_id} is IDENTICAL in both collections!")
|
||||
print_info(" → We can store ONE xaiFileId per document!")
|
||||
print_info(" → Simply track which collections contain it!")
|
||||
|
||||
self.test_results["shared_file"] = True
|
||||
|
||||
async def test_document_update(self, collection_id: str, file_id: str):
|
||||
"""Test 5: Update document metadata"""
|
||||
print_header("TEST 5: Update Document Metadata")
|
||||
|
||||
print_info("Updating document metadata...")
|
||||
status, data = await self.update_collection_document(
|
||||
collection_id,
|
||||
file_id,
|
||||
metadata={"updated_at": datetime.now().isoformat(), "version": 2}
|
||||
)
|
||||
print_json(data, "Update Response")
|
||||
|
||||
if status not in [200, 201]:
|
||||
print_error("Failed to update document")
|
||||
self.test_results["document_update"] = False
|
||||
return
|
||||
|
||||
print_success("Document metadata updated")
|
||||
self.test_results["document_update"] = True
|
||||
|
||||
async def test_document_removal(self):
|
||||
"""Test 6: Remove document from collection"""
|
||||
print_header("TEST 6: Remove Document from Collection")
|
||||
|
||||
if len(self.created_collections) < 2 or not self.uploaded_files:
|
||||
print_warning("Skipping - need at least 2 collections and 1 file")
|
||||
return
|
||||
|
||||
collection_id = self.created_collections[0]
|
||||
file_id = self.uploaded_files[0]
|
||||
|
||||
print_info(f"Removing file {file_id} from collection {collection_id}...")
|
||||
status, data = await self.remove_document_from_collection(collection_id, file_id)
|
||||
print_json(data, "Response")
|
||||
|
||||
if status not in [200, 204]:
|
||||
print_error("Failed to remove document")
|
||||
self.test_results["document_removal"] = False
|
||||
return
|
||||
|
||||
print_success("Document removed from collection")
|
||||
|
||||
# Verify removal
|
||||
print_info("Verifying removal...")
|
||||
status, data = await self.get_collection_documents(collection_id)
|
||||
print_json(data, "Remaining Documents")
|
||||
|
||||
self.test_results["document_removal"] = True
|
||||
|
||||
async def test_batch_get(self):
|
||||
"""Test 7: Batch get documents"""
|
||||
print_header("TEST 7: Batch Get Documents")
|
||||
|
||||
if not self.created_collections or not self.uploaded_files:
|
||||
print_warning("Skipping - need collections and files")
|
||||
return
|
||||
|
||||
collection_id = self.created_collections[-1] # Use last collection
|
||||
file_ids = self.uploaded_files
|
||||
|
||||
if not file_ids:
|
||||
print_warning("No file IDs to batch get")
|
||||
return
|
||||
|
||||
print_info(f"Batch getting {len(file_ids)} documents...")
|
||||
status, data = await self.batch_get_documents(collection_id, file_ids)
|
||||
print_json(data, "Batch Response")
|
||||
|
||||
self.test_results["batch_get"] = status in [200, 201]
|
||||
|
||||
async def cleanup(self):
|
||||
"""Clean up all created test resources"""
|
||||
print_header("CLEANUP: Deleting Test Resources")
|
||||
|
||||
# Delete collections (should cascade delete documents?)
|
||||
for collection_id in list(self.created_collections):
|
||||
print_info(f"Deleting collection {collection_id}...")
|
||||
await self.delete_collection(collection_id)
|
||||
|
||||
print_success("Cleanup complete")
|
||||
|
||||
def print_summary(self):
|
||||
"""Print test results summary"""
|
||||
print_header("TEST RESULTS SUMMARY")
|
||||
|
||||
total = len(self.test_results)
|
||||
passed = sum(1 for v in self.test_results.values() if v)
|
||||
|
||||
for test_name, result in self.test_results.items():
|
||||
status = "✅ PASS" if result else "❌ FAIL"
|
||||
print(f"{status} - {test_name}")
|
||||
|
||||
print(f"\n{Colors.BOLD}Total: {passed}/{total} tests passed{Colors.END}\n")
|
||||
|
||||
# Critical findings
|
||||
print_header("CRITICAL FINDINGS")
|
||||
|
||||
if "shared_file" in self.test_results:
|
||||
if self.test_results["shared_file"]:
|
||||
print_success("✅ Same file CAN be used in multiple collections")
|
||||
print_info(" → We can use a SINGLE xaiFileId per document!")
|
||||
print_info(" → Much simpler architecture!")
|
||||
else:
|
||||
print_error("❌ Files seem to be collection-specific")
|
||||
print_warning(" → More complex mapping required")
|
||||
print_warning(" → Each collection might need separate file upload")
|
||||
|
||||
|
||||
async def main():
|
||||
"""Run all tests"""
|
||||
print_header("xAI Collections API Test Suite")
|
||||
print_info(f"Management URL: {XAI_MANAGEMENT_URL}")
|
||||
print_info(f"Files URL: {XAI_FILES_URL}")
|
||||
print_info(f"Management Key: {XAI_MANAGEMENT_KEY[:20]}...{XAI_MANAGEMENT_KEY[-4:]}")
|
||||
print_info(f"API Key: {XAI_API_KEY[:20]}...{XAI_API_KEY[-4:]}")
|
||||
|
||||
async with XAICollectionsTestClient() as client:
|
||||
try:
|
||||
# Test 1: Basic Collection CRUD
|
||||
collection_id = await client.test_basic_collection_crud()
|
||||
|
||||
if not collection_id:
|
||||
print_error("Cannot continue without collection. Stopping.")
|
||||
return
|
||||
|
||||
# Test 2: File Upload (now two-step process)
|
||||
file_id = await client.test_file_upload_and_structure(collection_id)
|
||||
|
||||
if not file_id:
|
||||
print_error("File upload failed. Continuing with remaining tests...")
|
||||
else:
|
||||
# Test 3: Verify document in collection
|
||||
await client.test_document_in_collection(collection_id, file_id)
|
||||
|
||||
# Test 4: CRITICAL - Shared file test
|
||||
await client.test_shared_file_across_collections(file_id)
|
||||
|
||||
# Test 5: Update document
|
||||
await client.test_document_update(collection_id, file_id)
|
||||
|
||||
# Test 6: Remove document
|
||||
await client.test_document_removal()
|
||||
|
||||
# Test 7: Batch get
|
||||
await client.test_batch_get()
|
||||
|
||||
# Cleanup
|
||||
await client.cleanup()
|
||||
|
||||
# Print summary
|
||||
client.print_summary()
|
||||
|
||||
except Exception as e:
|
||||
print_error(f"Test suite failed: {e}")
|
||||
import traceback
|
||||
traceback.print_exc()
|
||||
|
||||
# Try cleanup anyway
|
||||
try:
|
||||
await client.cleanup()
|
||||
except:
|
||||
pass
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
asyncio.run(main())
|
||||
Reference in New Issue
Block a user