Implement EspoCRM-based sync strategy for Beteiligte entities
- Add SYNC_STRATEGY_ESPOCRM_BASED.md detailing the sync flows and status management. - Create utilities for sync operations in services/beteiligte_sync_utils.py, including locking, timestamp comparison, conflict resolution, and notification handling. - Implement entity mapping between EspoCRM and Advoware in services/espocrm_mapper.py. - Develop a cron job for periodic sync checks in steps/vmh/beteiligte_sync_cron_step.py, emitting events for entities needing synchronization.
This commit is contained in:
326
bitbylaw/IMPLEMENTATION_COMPLETE.md
Normal file
326
bitbylaw/IMPLEMENTATION_COMPLETE.md
Normal file
@@ -0,0 +1,326 @@
|
||||
# Beteiligte Sync Implementation - Fertig! ✅
|
||||
|
||||
**Stand**: 7. Februar 2026
|
||||
**Status**: Vollständig implementiert, ready for testing
|
||||
|
||||
---
|
||||
|
||||
## 📦 Implementierte Module
|
||||
|
||||
### 1. **services/espocrm_mapper.py** ✅
|
||||
**Zweck**: Entity-Transformation zwischen EspoCRM ↔ Advoware
|
||||
|
||||
**Funktionen**:
|
||||
- `map_cbeteiligte_to_advoware(espo_entity)` - EspoCRM → Advoware
|
||||
- `map_advoware_to_cbeteiligte(advo_entity)` - Advoware → EspoCRM
|
||||
- `get_changed_fields(espo, advo)` - Diff-Vergleich
|
||||
|
||||
**Features**:
|
||||
- Unterscheidet Person vs. Firma
|
||||
- Mapped Namen, Kontaktdaten, Handelsregister
|
||||
- Transformiert emailAddressData/phoneNumberData Arrays
|
||||
- Normalisiert Rechtsform und Anrede
|
||||
|
||||
---
|
||||
|
||||
### 2. **services/beteiligte_sync_utils.py** ✅
|
||||
**Zweck**: Sync-Utility-Funktionen
|
||||
|
||||
**Funktionen**:
|
||||
- `acquire_sync_lock(entity_id)` - Atomares Lock via syncStatus
|
||||
- `release_sync_lock(entity_id, status, error, retry)` - Lock freigeben + Update
|
||||
- `parse_timestamp(ts)` - Parse EspoCRM/Advoware Timestamps
|
||||
- `compare_timestamps(espo, advo, last_sync)` - Returns: espocrm_newer | advoware_newer | conflict | no_change
|
||||
- `send_notification(entity_id, type, data)` - EspoCRM In-App Notification
|
||||
- `handle_advoware_deleted(entity_id, error)` - Soft-Delete + Notification
|
||||
- `resolve_conflict_espocrm_wins(entity_id, ...)` - Konfliktauflösung
|
||||
|
||||
**Features**:
|
||||
- Race-Condition-Prevention durch syncStatus="syncing"
|
||||
- Automatische syncRetryCount Increment bei Fehlern
|
||||
- EspoCRM Notifications (🔔 Bell-Icon)
|
||||
- Timestamp-Normalisierung für beide Systeme
|
||||
|
||||
---
|
||||
|
||||
### 3. **steps/vmh/beteiligte_sync_event_step.py** ✅
|
||||
**Zweck**: Zentraler Sync-Handler (Webhooks + Cron)
|
||||
|
||||
**Config**:
|
||||
```python
|
||||
subscribes: [
|
||||
'vmh.beteiligte.create',
|
||||
'vmh.beteiligte.update',
|
||||
'vmh.beteiligte.delete',
|
||||
'vmh.beteiligte.sync_check' # Von Cron
|
||||
]
|
||||
```
|
||||
|
||||
**Ablauf**:
|
||||
1. Acquire Lock (syncStatus → syncing)
|
||||
2. Fetch Entity von EspoCRM
|
||||
3. Bestimme Aktion:
|
||||
- **Kein betnr** → `handle_create()` - Neu in Advoware
|
||||
- **Hat betnr** → `handle_update()` - Sync mit Timestamp-Vergleich
|
||||
4. Release Lock mit finalem Status
|
||||
|
||||
**handle_create()**:
|
||||
- Transform zu Advoware Format
|
||||
- POST /api/v1/advonet/Beteiligte
|
||||
- Update EspoCRM mit neuer betnr
|
||||
- Status → clean
|
||||
|
||||
**handle_update()**:
|
||||
- Fetch von Advoware (betNr)
|
||||
- 404 → `handle_advoware_deleted()` (Soft-Delete)
|
||||
- Timestamp-Vergleich:
|
||||
- `espocrm_newer` → PUT zu Advoware
|
||||
- `advoware_newer` → PUT zu EspoCRM
|
||||
- `conflict` → **EspoCRM WINS** → Überschreibe Advoware → Notification
|
||||
- `no_change` → Skip
|
||||
|
||||
**Error Handling**:
|
||||
- Try/Catch um alle Operationen
|
||||
- Bei Fehler: syncStatus=failed, syncErrorMessage, syncRetryCount++
|
||||
- Redis Queue Cleanup
|
||||
|
||||
---
|
||||
|
||||
### 4. **steps/vmh/beteiligte_sync_cron_step.py** ✅
|
||||
**Zweck**: Cron-Job der Events emittiert
|
||||
|
||||
**Config**:
|
||||
```python
|
||||
schedule: '*/15 * * * *' # Alle 15 Minuten
|
||||
emits: ['vmh.beteiligte.sync_check']
|
||||
```
|
||||
|
||||
**Ablauf**:
|
||||
1. Query 1: Entities mit Status `pending_sync`, `dirty`, `failed` (max 100)
|
||||
2. Query 2: `clean` Entities mit `advowareLastSync < NOW() - 24h` (max 50)
|
||||
3. Kombiniere + Dedupliziere
|
||||
4. Emittiere `vmh.beteiligte.sync_check` Event für JEDEN Beteiligten
|
||||
5. Log: Anzahl emittierter Events
|
||||
|
||||
**Vorteile**:
|
||||
- Kein Batch-Processing
|
||||
- Events werden einzeln vom normalen Handler verarbeitet
|
||||
- Code-Wiederverwendung (gleicher Handler wie Webhooks)
|
||||
|
||||
---
|
||||
|
||||
## 🔄 Sync-Flows
|
||||
|
||||
### Flow A: EspoCRM Create/Update → Advoware (Webhook)
|
||||
|
||||
```
|
||||
User ändert in EspoCRM
|
||||
↓
|
||||
EspoCRM Webhook → /vmh/webhook/beteiligte/update
|
||||
↓
|
||||
beteiligte_update_api_step.py → Emit 'vmh.beteiligte.update'
|
||||
↓
|
||||
beteiligte_sync_event_step.py → handler()
|
||||
↓
|
||||
Acquire Lock → Fetch EspoCRM → Timestamp-Check
|
||||
↓
|
||||
Update Advoware (oder Konflikt → EspoCRM wins)
|
||||
↓
|
||||
Release Lock → Status: clean
|
||||
```
|
||||
|
||||
**Timing**: 2-5 Sekunden
|
||||
|
||||
---
|
||||
|
||||
### Flow B: Advoware → EspoCRM Check (Cron)
|
||||
|
||||
```
|
||||
Cron (alle 15 Min)
|
||||
↓
|
||||
beteiligte_sync_cron_step.py
|
||||
↓
|
||||
Query EspoCRM: Unclean + Stale Entities
|
||||
↓
|
||||
Emit 'vmh.beteiligte.sync_check' für jeden
|
||||
↓
|
||||
beteiligte_sync_event_step.py → handler()
|
||||
↓
|
||||
GLEICHE Logik wie Flow A!
|
||||
```
|
||||
|
||||
**Timing**: Alle 15 Minuten
|
||||
|
||||
---
|
||||
|
||||
## 🎯 Status-Übergänge
|
||||
|
||||
```
|
||||
pending_sync → syncing → clean (Create erfolgreich)
|
||||
pending_sync → syncing → failed (Create fehlgeschlagen)
|
||||
|
||||
clean → dirty → syncing → clean (Update erfolgreich)
|
||||
clean → syncing → conflict → clean (Konflikt → EspoCRM wins)
|
||||
|
||||
dirty → syncing → deleted_in_advoware (404 von Advoware)
|
||||
|
||||
failed → syncing → clean (Retry erfolgreich)
|
||||
failed → syncing → failed (Retry fehlgeschlagen, retryCount++)
|
||||
|
||||
deleted_in_advoware (Soft-Delete, bleibt bis manuelle Aktion)
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## 📋 Testing Checklist
|
||||
|
||||
### Unit Tests
|
||||
- [x] Mapper Import
|
||||
- [x] Sync Utils Import
|
||||
- [x] Event Step Config Load
|
||||
- [x] Cron Step Config Load
|
||||
- [x] Mapper Transform Person
|
||||
- [x] Mapper Transform Firma
|
||||
|
||||
### Integration Tests (TODO)
|
||||
- [ ] Create: Neuer Beteiligter in EspoCRM → Advoware
|
||||
- [ ] Update: Änderung in EspoCRM → Advoware
|
||||
- [ ] Conflict: Beide geändert → EspoCRM wins
|
||||
- [ ] Advoware newer: Advoware → EspoCRM
|
||||
- [ ] 404 Handling: Soft-Delete + Notification
|
||||
- [ ] Cron: Query + Event Emission
|
||||
- [ ] Notification: In-App Notification sichtbar
|
||||
|
||||
---
|
||||
|
||||
## 🚀 Deployment
|
||||
|
||||
### Voraussetzungen
|
||||
✅ EspoCRM Felder angelegt (syncStatus, betnr, advowareLastSync, advowareDeletedAt, syncErrorMessage, syncRetryCount)
|
||||
✅ Webhooks aktiviert (Create/Update/Delete)
|
||||
⏳ Motia Workbench Restart (damit Steps geladen werden)
|
||||
|
||||
### Schritte
|
||||
1. **Motia Restart**: `systemctl restart motia` (oder wie auch immer)
|
||||
2. **Verify Steps**:
|
||||
```bash
|
||||
# Check ob Steps geladen wurden
|
||||
curl http://localhost:PORT/api/flows/vmh/steps
|
||||
```
|
||||
3. **Test Webhook**: Ändere einen Beteiligten in EspoCRM
|
||||
4. **Check Logs**: Motia Workbench Logs → Event Handler Output
|
||||
5. **Verify Advoware**: Prüfe ob betNr gesetzt wurde
|
||||
6. **Test Cron**: Warte 15 Min oder trigger manuell
|
||||
|
||||
---
|
||||
|
||||
## 🔧 Configuration
|
||||
|
||||
### Environment Variables (bereits gesetzt)
|
||||
```bash
|
||||
# EspoCRM
|
||||
ESPOCRM_API_BASE_URL=https://crm.bitbylaw.com/api/v1
|
||||
ESPOCRM_MARVIN_API_KEY=e53def10eea27b92a6cd00f40a3e09a4
|
||||
|
||||
# Advoware
|
||||
ADVOWARE_API_BASE_URL=https://www2.advo-net.net:90
|
||||
ADVOWARE_PRODUCT_ID=...
|
||||
ADVOWARE_APP_ID=...
|
||||
ADVOWARE_API_KEY=...
|
||||
|
||||
# Redis
|
||||
REDIS_HOST=localhost
|
||||
REDIS_PORT=6379
|
||||
REDIS_DB_ADVOWARE_CACHE=1
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## 📊 Monitoring
|
||||
|
||||
### EspoCRM Queries
|
||||
|
||||
**Entities die Sync benötigen**:
|
||||
```javascript
|
||||
GET /api/v1/CBeteiligte?where=[
|
||||
{type: 'in', attribute: 'syncStatus',
|
||||
value: ['pending_sync', 'dirty', 'failed']}
|
||||
]
|
||||
```
|
||||
|
||||
**Konflikte**:
|
||||
```javascript
|
||||
GET /api/v1/CBeteiligte?where=[
|
||||
{type: 'equals', attribute: 'syncStatus', value: 'conflict'}
|
||||
]
|
||||
```
|
||||
|
||||
**Soft-Deletes**:
|
||||
```javascript
|
||||
GET /api/v1/CBeteiligte?where=[
|
||||
{type: 'equals', attribute: 'syncStatus', value: 'deleted_in_advoware'}
|
||||
]
|
||||
```
|
||||
|
||||
**Sync-Fehler**:
|
||||
```javascript
|
||||
GET /api/v1/CBeteiligte?where=[
|
||||
{type: 'isNotNull', attribute: 'syncErrorMessage'}
|
||||
]
|
||||
```
|
||||
|
||||
### Motia Logs
|
||||
```bash
|
||||
# Event Handler Logs
|
||||
tail -f /path/to/motia/logs/events.log | grep "Beteiligte"
|
||||
|
||||
# Cron Logs
|
||||
tail -f /path/to/motia/logs/cron.log | grep "Sync Cron"
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## 🐛 Troubleshooting
|
||||
|
||||
### Problem: Lock bleibt auf "syncing" hängen
|
||||
**Ursache**: Handler-Crash während Sync
|
||||
**Lösung**: Manuell Status auf "failed" setzen:
|
||||
```python
|
||||
PUT /api/v1/CBeteiligte/{id}
|
||||
{"syncStatus": "failed", "syncErrorMessage": "Manual reset"}
|
||||
```
|
||||
|
||||
### Problem: Notifications werden nicht angezeigt
|
||||
**Ursache**: userId fehlt oder falsch
|
||||
**Check**:
|
||||
```python
|
||||
GET /api/v1/Notification?where=[{type: 'equals', attribute: 'relatedType', value: 'CBeteiligte'}]
|
||||
```
|
||||
|
||||
### Problem: Cron emittiert keine Events
|
||||
**Ursache**: Query findet keine Entities
|
||||
**Debug**: Führe Cron-Handler manuell aus und checke Logs
|
||||
|
||||
---
|
||||
|
||||
## 📈 Performance
|
||||
|
||||
**Erwartete Last**:
|
||||
- Webhooks: ~10-50 pro Tag (User-Änderungen)
|
||||
- Cron: Alle 15 Min → ~96 Runs/Tag
|
||||
- Events pro Cron: 0-100 (typisch 5-20)
|
||||
|
||||
**Optimization**:
|
||||
- Cron Max Entities: 150 total (100 unclean + 50 stale)
|
||||
- Event-Processing: Parallel (Motia-Standard)
|
||||
- Redis Caching: Token + Deduplication
|
||||
|
||||
---
|
||||
|
||||
## ✅ Done!
|
||||
|
||||
**Implementiert**: 4 Module, ~800 Lines of Code
|
||||
**Status**: Ready for Testing
|
||||
**Next Steps**: Deploy + Integration Testing + Monitoring Setup
|
||||
|
||||
🎉 **Viel Erfolg beim Testing!**
|
||||
Reference in New Issue
Block a user