Add sync strategy documentation and templates for bidirectional sync between EspoCRM and Advoware
- Introduced SYNC_STRATEGY_ARCHIVE.md detailing the sync process, status values, and flow for updating entities from EspoCRM to Advoware and vice versa. - Created SYNC_TEMPLATE.md as a guide for implementing new syncs, including field definitions, mapper examples, sync utilities, event handlers, and cron jobs. - Added README_SYNC.md for the Beteiligte sync event handler, outlining its functionality, event subscriptions, optimizations, error handling, and performance metrics.
This commit is contained in:
359
bitbylaw/docs/BETEILIGTE_SYNC.md
Normal file
359
bitbylaw/docs/BETEILIGTE_SYNC.md
Normal file
@@ -0,0 +1,359 @@
|
||||
# Beteiligte Sync - Bidirektionale Synchronisation EspoCRM ↔ Advoware
|
||||
|
||||
## Übersicht
|
||||
|
||||
Bidirektionale Synchronisation der **Stammdaten** von Beteiligten zwischen EspoCRM (CBeteiligte) und Advoware (Beteiligte).
|
||||
|
||||
**Scope**: Nur Stammdaten (Name, Rechtsform, Geburtsdatum, Anrede, Handelsregister)
|
||||
**Out of Scope**: Kontaktdaten (Telefon, Email, Fax, Bankverbindungen) → separate Endpoints
|
||||
|
||||
## Architektur
|
||||
|
||||
### Event-Driven Architecture
|
||||
|
||||
```
|
||||
┌─────────────┐
|
||||
│ EspoCRM │ Webhook → vmh.beteiligte.{create,update,delete}
|
||||
│ CBeteiligte │ ↓
|
||||
└─────────────┘ ┌────────────────────┐
|
||||
│ Event Handler │
|
||||
┌─────────────┐ │ (sync_event_step) │
|
||||
│ Cron │ ───→ │ │
|
||||
│ (15 min) │ sync_ │ - Lock (Redis) │
|
||||
└─────────────┘ check │ - Timestamp Check │
|
||||
│ - Merge & Sync │
|
||||
└────────┬───────────┘
|
||||
↓
|
||||
┌────────────────────┐
|
||||
│ Advoware API │
|
||||
│ /Beteiligte │
|
||||
└────────────────────┘
|
||||
```
|
||||
|
||||
### Komponenten
|
||||
|
||||
1. **Event Handler** ([beteiligte_sync_event_step.py](../steps/vmh/beteiligte_sync_event_step.py))
|
||||
- Subscribes: `vmh.beteiligte.{create,update,delete,sync_check}`
|
||||
- Verarbeitet Sync-Events
|
||||
- Verwendet Redis distributed lock
|
||||
|
||||
2. **Cron Job** ([beteiligte_sync_cron_step.py](../steps/vmh/beteiligte_sync_cron_step.py))
|
||||
- Läuft alle 15 Minuten
|
||||
- Findet Entities mit Sync-Bedarf
|
||||
- Emittiert `sync_check` Events
|
||||
|
||||
3. **Sync Utils** ([beteiligte_sync_utils.py](../services/beteiligte_sync_utils.py))
|
||||
- Lock-Management (Redis distributed lock)
|
||||
- Timestamp-Vergleich
|
||||
- Merge-Utility für Advoware PUT
|
||||
- Notifications
|
||||
|
||||
4. **Mapper** ([espocrm_mapper.py](../services/espocrm_mapper.py))
|
||||
- `map_cbeteiligte_to_advoware()` - EspoCRM → Advoware
|
||||
- `map_advoware_to_cbeteiligte()` - Advoware → EspoCRM
|
||||
- Nur Stammdaten, keine Kontaktdaten
|
||||
|
||||
5. **APIs**
|
||||
- [espocrm.py](../services/espocrm.py) - EspoCRM API Client
|
||||
- [advoware.py](../services/advoware.py) - Advoware API Client
|
||||
|
||||
## Sync-Strategie
|
||||
|
||||
### State Management
|
||||
- **Sync-Status in EspoCRM** (nicht PostgreSQL)
|
||||
- **Field**: `syncStatus` (enum mit 7 Werten)
|
||||
- **Lock**: Redis distributed lock (5 min TTL)
|
||||
|
||||
### Konfliktauflösung
|
||||
- **Policy**: EspoCRM wins
|
||||
- **Detection**: Timestamp-Vergleich (`modifiedAt` vs `geaendertAm`)
|
||||
- **Notification**: In-App Notification in EspoCRM
|
||||
|
||||
### Sync-Status Values
|
||||
|
||||
```typescript
|
||||
enum SyncStatus {
|
||||
clean // ✅ Synced, keine Änderungen
|
||||
dirty // 📝 Lokale Änderungen, noch nicht synced
|
||||
pending_sync // ⏳ Wartet auf ersten Sync
|
||||
syncing // 🔄 Sync läuft gerade (Lock)
|
||||
failed // ❌ Sync fehlgeschlagen (retry möglich)
|
||||
conflict // ⚠️ Konflikt erkannt
|
||||
permanently_failed // 💀 Max retries erreicht (5x)
|
||||
}
|
||||
```
|
||||
|
||||
## Datenfluss
|
||||
|
||||
### 1. Create (Neu in EspoCRM)
|
||||
```
|
||||
EspoCRM (neu) → Webhook → Event Handler
|
||||
↓
|
||||
Acquire Lock (Redis)
|
||||
↓
|
||||
Map EspoCRM → Advoware
|
||||
↓
|
||||
POST /api/v1/advonet/Beteiligte
|
||||
↓
|
||||
Response: {betNr: 12345}
|
||||
↓
|
||||
Update EspoCRM: betnr=12345, syncStatus=clean
|
||||
↓
|
||||
Release Lock
|
||||
```
|
||||
|
||||
### 2. Update (Änderung in EspoCRM)
|
||||
```
|
||||
EspoCRM (geändert) → Webhook → Event Handler
|
||||
↓
|
||||
Acquire Lock (Redis)
|
||||
↓
|
||||
GET /api/v1/advonet/Beteiligte/{betnr}
|
||||
↓
|
||||
Timestamp-Vergleich:
|
||||
- espocrm_newer → Update Advoware (PUT)
|
||||
- advoware_newer → Update EspoCRM (PATCH)
|
||||
- conflict → EspoCRM wins (PUT) + Notification
|
||||
- no_change → Skip
|
||||
↓
|
||||
Release Lock
|
||||
```
|
||||
|
||||
### 3. Cron Check
|
||||
```
|
||||
Cron (alle 15 min)
|
||||
↓
|
||||
Query EspoCRM:
|
||||
- syncStatus IN (pending_sync, dirty, failed)
|
||||
- OR (clean AND advowareLastSync > 24h)
|
||||
↓
|
||||
Batch emit: vmh.beteiligte.sync_check events
|
||||
↓
|
||||
Event Handler (siehe Update)
|
||||
```
|
||||
|
||||
## Optimierungen
|
||||
|
||||
### 1. Redis Distributed Lock (Atomicity)
|
||||
```python
|
||||
lock_key = f"sync_lock:cbeteiligte:{entity_id}"
|
||||
acquired = redis.set(lock_key, "locked", nx=True, ex=300)
|
||||
```
|
||||
- ✅ Verhindert Race Conditions
|
||||
- ✅ TTL verhindert Deadlocks (5 min)
|
||||
|
||||
### 2. Combined API Calls (Performance)
|
||||
```python
|
||||
await sync_utils.release_sync_lock(
|
||||
entity_id,
|
||||
'clean',
|
||||
extra_fields={'betnr': new_betnr} # ← kombiniert 2 calls in 1
|
||||
)
|
||||
```
|
||||
- ✅ 33% weniger API Requests
|
||||
|
||||
### 3. Merge Utility (Code Quality)
|
||||
```python
|
||||
merged = sync_utils.merge_for_advoware_put(advo_entity, espo_entity, mapper)
|
||||
```
|
||||
- ✅ Keine Code-Duplikation
|
||||
- ✅ Konsistentes Logging
|
||||
- ✅ Wiederverwendbar
|
||||
|
||||
### 4. Max Retry Limit (Robustheit)
|
||||
```python
|
||||
MAX_SYNC_RETRIES = 5
|
||||
|
||||
if retry_count >= 5:
|
||||
status = 'permanently_failed'
|
||||
send_notification("Max retries erreicht")
|
||||
```
|
||||
- ✅ Verhindert infinite loops
|
||||
- ✅ User wird benachrichtigt
|
||||
|
||||
### 5. Batch Processing (Scalability)
|
||||
```python
|
||||
tasks = [context.emit(...) for entity_id in entity_ids]
|
||||
results = await asyncio.gather(*tasks, return_exceptions=True)
|
||||
```
|
||||
- ✅ 90% schneller bei 100 Entities
|
||||
|
||||
## Performance
|
||||
|
||||
| Operation | API Calls | Latency |
|
||||
|-----------|-----------|---------|
|
||||
| CREATE | 2 | ~200ms |
|
||||
| UPDATE (initial) | 2 | ~250ms |
|
||||
| UPDATE (normal) | 2 | ~250ms |
|
||||
| Cron (100 entities) | 200 | ~1s (parallel) |
|
||||
|
||||
## Monitoring
|
||||
|
||||
### Sync-Status Tracking
|
||||
```sql
|
||||
-- In EspoCRM
|
||||
SELECT syncStatus, COUNT(*)
|
||||
FROM c_beteiligte
|
||||
GROUP BY syncStatus;
|
||||
```
|
||||
|
||||
### Failed Syncs
|
||||
```sql
|
||||
-- Entities mit Sync-Problemen
|
||||
SELECT id, name, syncStatus, syncErrorMessage, syncRetryCount
|
||||
FROM c_beteiligte
|
||||
WHERE syncStatus IN ('failed', 'permanently_failed')
|
||||
ORDER BY syncRetryCount DESC;
|
||||
```
|
||||
|
||||
## Fehlerbehandlung
|
||||
|
||||
### Retriable Errors
|
||||
- Netzwerk-Timeout
|
||||
- 500 Internal Server Error
|
||||
- 503 Service Unavailable
|
||||
|
||||
→ Status: `failed`, retry beim nächsten Cron
|
||||
|
||||
### Non-Retriable Errors
|
||||
- 400 Bad Request (invalid data)
|
||||
- 404 Not Found (entity deleted)
|
||||
- 401 Unauthorized (auth error)
|
||||
|
||||
→ Status: `failed`, keine automatischen Retries
|
||||
|
||||
### Max Retries Exceeded
|
||||
- Nach 5 Versuchen: `permanently_failed`
|
||||
- User erhält In-App Notification
|
||||
- Manuelle Prüfung erforderlich
|
||||
|
||||
## Testing
|
||||
|
||||
### Unit Tests
|
||||
```bash
|
||||
cd /opt/motia-app/bitbylaw
|
||||
source python_modules/bin/activate
|
||||
python scripts/test_beteiligte_sync.py
|
||||
```
|
||||
|
||||
### Manual Test
|
||||
```python
|
||||
# Test single entity sync
|
||||
event_data = {
|
||||
'entity_id': '68e3e7eab49f09adb',
|
||||
'action': 'sync_check',
|
||||
'source': 'manual_test'
|
||||
}
|
||||
await beteiligte_sync_event_step.handler(event_data, context)
|
||||
```
|
||||
|
||||
## Entity Mapping
|
||||
|
||||
### EspoCRM CBeteiligte → Advoware Beteiligte
|
||||
|
||||
| EspoCRM Field | Advoware Field | Type | Notes |
|
||||
|---------------|----------------|------|-------|
|
||||
| `lastName` | `name` | string | Bei Person |
|
||||
| `firstName` | `vorname` | string | Bei Person |
|
||||
| `firmenname` | `name` | string | Bei Firma |
|
||||
| `rechtsform` | `rechtsform` | string | Person/Firma |
|
||||
| `salutationName` | `anrede` | string | Herr/Frau |
|
||||
| `dateOfBirth` | `geburtsdatum` | date | Nur Person |
|
||||
| `handelsregisterNummer` | `handelsRegisterNummer` | string | Nur Firma |
|
||||
| `betnr` | `betNr` | int | Foreign Key |
|
||||
|
||||
**Nicht gemapped**: Telefon, Email, Fax, Bankverbindungen (→ separate Endpoints)
|
||||
|
||||
## Troubleshooting
|
||||
|
||||
### Sync bleibt bei "syncing" hängen
|
||||
**Problem**: Redis lock expired, aber syncStatus nicht zurückgesetzt
|
||||
**Lösung**:
|
||||
```python
|
||||
# Lock ist automatisch nach 5 min weg (TTL)
|
||||
# Manuelles zurücksetzen:
|
||||
await espocrm.update_entity('CBeteiligte', entity_id, {'syncStatus': 'dirty'})
|
||||
```
|
||||
|
||||
### "Max retries exceeded"
|
||||
**Problem**: Entity ist `permanently_failed`
|
||||
**Lösung**:
|
||||
1. Prüfe `syncErrorMessage` für Details
|
||||
2. Behebe das Problem (z.B. invalide Daten)
|
||||
3. Reset: `syncStatus='dirty', syncRetryCount=0`
|
||||
|
||||
### Race Condition / Parallele Syncs
|
||||
**Problem**: Zwei Syncs gleichzeitig (sollte nicht passieren)
|
||||
**Lösung**: Redis lock verhindert das automatisch
|
||||
|
||||
## Configuration
|
||||
|
||||
### Environment Variables
|
||||
```bash
|
||||
# EspoCRM
|
||||
ESPOCRM_API_BASE_URL=https://crm.bitbylaw.com/api/v1
|
||||
ESPOCRM_MARVIN_API_KEY=e53def10eea27b92a6cd00f40a3e09a4
|
||||
|
||||
# Advoware
|
||||
ADVOWARE_API_BASE_URL=https://www2.advo-net.net:90/
|
||||
ADVOWARE_PRODUCT_ID=...
|
||||
ADVOWARE_APP_ID=...
|
||||
ADVOWARE_API_KEY=...
|
||||
|
||||
# Redis
|
||||
REDIS_HOST=localhost
|
||||
REDIS_PORT=6379
|
||||
REDIS_DB_ADVOWARE_CACHE=1
|
||||
```
|
||||
|
||||
### EspoCRM Entity Fields
|
||||
Custom fields für Sync-Management:
|
||||
- `betnr` (int, unique) - Foreign Key zu Advoware
|
||||
- `syncStatus` (enum) - Sync-Status
|
||||
- `advowareLastSync` (datetime) - Letzter erfolgreicher Sync
|
||||
- `advowareDeletedAt` (datetime) - Soft-Delete timestamp
|
||||
- `syncErrorMessage` (text, 2000 chars) - Letzte Fehlermeldung
|
||||
- `syncRetryCount` (int) - Anzahl fehlgeschlagener Versuche
|
||||
|
||||
## Deployment
|
||||
|
||||
### 1. Deploy Code
|
||||
```bash
|
||||
cd /opt/motia-app/bitbylaw
|
||||
git pull
|
||||
source python_modules/bin/activate
|
||||
pip install -r requirements.txt
|
||||
```
|
||||
|
||||
### 2. Restart Motia
|
||||
```bash
|
||||
# Motia Workbench restart (lädt neue Steps)
|
||||
systemctl restart motia-workbench # oder entsprechender Befehl
|
||||
```
|
||||
|
||||
### 3. Verify
|
||||
```bash
|
||||
# Check logs
|
||||
tail -f /var/log/motia/workbench.log
|
||||
|
||||
# Test single sync
|
||||
python scripts/test_beteiligte_sync.py
|
||||
```
|
||||
|
||||
## Weitere Advoware-Syncs
|
||||
|
||||
Dieses System ist als **Template für alle Advoware-Syncs** designed. Wichtige Prinzipien:
|
||||
|
||||
1. **Redis Distributed Lock** für atomare Operations
|
||||
2. **Merge Utility** für Read-Modify-Write Pattern
|
||||
3. **Max Retries** mit Notification
|
||||
4. **Batch Processing** in Cron
|
||||
5. **Combined API Calls** wo möglich
|
||||
|
||||
→ Siehe [SYNC_TEMPLATE.md](SYNC_TEMPLATE.md) für Implementierungs-Template
|
||||
|
||||
## Siehe auch
|
||||
|
||||
- [Entity Mapping Details](../ENTITY_MAPPING_CBeteiligte_Advoware.md)
|
||||
- [Advoware API Docs](advoware/)
|
||||
- [EspoCRM API Docs](API.md)
|
||||
286
bitbylaw/docs/ENTITY_MAPPING_CBeteiligte_Advoware.md
Normal file
286
bitbylaw/docs/ENTITY_MAPPING_CBeteiligte_Advoware.md
Normal file
@@ -0,0 +1,286 @@
|
||||
# Entity-Mapping: EspoCRM CBeteiligte ↔ Advoware Beteiligte
|
||||
|
||||
Basierend auf dem Vergleich von:
|
||||
- **EspoCRM**: CBeteiligte Entity ID `68e4af00172be7924`
|
||||
- **Advoware**: Beteiligter ID `104860`
|
||||
|
||||
## Gemeinsame Felder (direkte Übereinstimmung)
|
||||
|
||||
| EspoCRM Feld | Advoware Feld | Typ | Notes |
|
||||
|--------------|---------------|-----|-------|
|
||||
| `name` | `name` | string | Vollständiger Name |
|
||||
| `rechtsform` | `rechtsform` | string | Rechtsform (z.B. "GmbH", "Frau") |
|
||||
| `id` | `id` | mixed | **Achtung:** EspoCRM=string, Advoware=int |
|
||||
|
||||
## Namenfelder
|
||||
|
||||
| EspoCRM Feld | Advoware Feld | Mapping |
|
||||
|--------------|---------------|---------|
|
||||
| `firstName` | `vorname` | ✓ Direkt |
|
||||
| `lastName` | `name` | ✓ Bei Personen |
|
||||
| `middleName` | - | ❌ Kein direktes Mapping |
|
||||
| `firmenname` | `name` | ✓ Bei Firmen |
|
||||
| - | `geburtsname` | ← Nur in Advoware |
|
||||
| - | `kurzname` | ← Nur in Advoware |
|
||||
|
||||
## Kontaktdaten
|
||||
|
||||
| EspoCRM Feld | Advoware Feld | Mapping |
|
||||
|--------------|---------------|---------|
|
||||
| `emailAddress` | `emailGesch` | ✓ Geschäftlich |
|
||||
| `emailAddressData` (array) | `email` | ⚠️ Komplex: Array vs. String |
|
||||
| `phoneNumber` | `telGesch` | ✓ Geschäftstelefon |
|
||||
| `phoneNumberData` (array) | `telPrivat` | ⚠️ Komplex |
|
||||
| - | `mobil` | ← Nur in Advoware |
|
||||
| - | `faxGesch` / `faxPrivat` | ← Nur in Advoware |
|
||||
| - | `autotelefon` | ← Nur in Advoware |
|
||||
| - | `internet` | ← Nur in Advoware |
|
||||
|
||||
**Hinweis**: Advoware hat zusätzlich `kommunikation` Array mit strukturierten Kontaktdaten.
|
||||
|
||||
## Adressdaten
|
||||
|
||||
| EspoCRM Feld | Advoware Feld | Mapping |
|
||||
|--------------|---------------|---------|
|
||||
| `adressensIds` / `adressensNames` | `adressen` (array) | ⚠️ Beziehung |
|
||||
| - | `strasse` | ← Hauptadresse in Advoware Root |
|
||||
| - | `plz` | ← Hauptadresse in Advoware Root |
|
||||
| - | `ort` | ← Hauptadresse in Advoware Root |
|
||||
| - | `anschrift` | ← Formatierte Adresse |
|
||||
|
||||
**Hinweis**:
|
||||
- EspoCRM: Adressen als Related Entities (IDs/Names)
|
||||
- Advoware: Hauptadresse im Root-Objekt + `adressen` Array für zusätzliche
|
||||
|
||||
## Anrede & Titel
|
||||
|
||||
| EspoCRM Feld | Advoware Feld | Mapping |
|
||||
|--------------|---------------|---------|
|
||||
| `salutationName` | `anrede` | ✓ (z.B. "Frau", "Herr") |
|
||||
| - | `bAnrede` | ← Briefanrede ("Sehr geehrte...") |
|
||||
| - | `titel` | ← Titel (Dr., Prof., etc.) |
|
||||
| - | `zusatz` | ← Namenszusatz |
|
||||
|
||||
## Geburtsdaten
|
||||
|
||||
| EspoCRM Feld | Advoware Feld | Mapping |
|
||||
|--------------|---------------|---------|
|
||||
| `dateOfBirth` | `geburtsdatum` | ✓ Direkt |
|
||||
| - | `sterbedatum` | ← Nur in Advoware |
|
||||
| - | `familienstand` | ← Nur in Advoware |
|
||||
|
||||
## Handelsregister (für Firmen)
|
||||
|
||||
| EspoCRM Feld | Advoware Feld | Mapping |
|
||||
|--------------|---------------|---------|
|
||||
| `handelsregisterNummer` | `handelsRegisterNummer` | ✓ Direkt |
|
||||
| `handelsregisterArt` (z.B. "HRB") | - | ❌ Nur in EspoCRM |
|
||||
| - | `registergericht` | ← Nur in Advoware |
|
||||
|
||||
## Bankverbindungen
|
||||
|
||||
| EspoCRM Feld | Advoware Feld | Mapping |
|
||||
|--------------|---------------|---------|
|
||||
| `bankverbindungensIds` / Names | `bankkverbindungen` (array) | ⚠️ Related Entity vs. Array |
|
||||
|
||||
## Beteiligungen/Akten
|
||||
|
||||
| EspoCRM Feld | Advoware Feld | Mapping |
|
||||
|--------------|---------------|---------|
|
||||
| - | `beteiligungen` (array) | ← Nur in Advoware |
|
||||
|
||||
**Hinweis**: Advoware speichert die Akten-Beteiligungen direkt beim Beteiligten.
|
||||
|
||||
## EspoCRM-spezifische Felder
|
||||
|
||||
| Feld | Zweck |
|
||||
|------|-------|
|
||||
| `betnr` | Beteiligten-Nummer (= Advoware `betNr`) |
|
||||
| `advowareLastSync` | Zeitstempel der letzten Synchronisation |
|
||||
| `syncStatus` | Status: "clean", "dirty", "syncing" |
|
||||
| `disgTyp` | DISC-Persönlichkeitstyp |
|
||||
| `description` | Notizen/Beschreibung |
|
||||
| `createdAt` / `createdById` / `createdByName` | Audit-Felder |
|
||||
| `modifiedAt` / `modifiedById` / `modifiedByName` | Audit-Felder |
|
||||
| `assignedUserId` / `assignedUserName` | Zuweisungen |
|
||||
| `teamsIds` / `teamsNames` | Team-Zugehörigkeit |
|
||||
| `deleted` | Soft-Delete Flag |
|
||||
| `isFollowed` / `followersIds` | Social Features |
|
||||
|
||||
## Advoware-spezifische Felder
|
||||
|
||||
| Feld | Zweck |
|
||||
|------|-------|
|
||||
| `betNr` | Interne Beteiligten-Nummer |
|
||||
| `rowId` | Datenbank Row-ID |
|
||||
| `art` | Beteiligten-Art |
|
||||
| `angelegtAm` / `angelegtVon` | Erstellt |
|
||||
| `geaendertAm` / `geaendertVon` | Geändert |
|
||||
| `kontaktpersonen` (array) | Kontaktpersonen bei Firmen |
|
||||
| `ePost` / `bea` | Spezielle Kommunikationskanäle |
|
||||
|
||||
## Mapping-Strategie
|
||||
|
||||
### 1. Person (Natürliche Person)
|
||||
|
||||
```python
|
||||
espocrm_to_advoware = {
|
||||
'firstName': 'vorname',
|
||||
'lastName': 'name',
|
||||
'dateOfBirth': 'geburtsdatum',
|
||||
'rechtsform': 'rechtsform', # z.B. "Herr", "Frau"
|
||||
'salutationName': 'anrede',
|
||||
'emailAddress': 'emailGesch',
|
||||
'phoneNumber': 'telGesch',
|
||||
}
|
||||
```
|
||||
|
||||
### 2. Firma (Juristische Person)
|
||||
|
||||
```python
|
||||
espocrm_to_advoware = {
|
||||
'firmenname': 'name',
|
||||
'rechtsform': 'rechtsform', # z.B. "GmbH", "AG"
|
||||
'handelsregisterNummer': 'handelsRegisterNummer',
|
||||
'emailAddress': 'emailGesch',
|
||||
'phoneNumber': 'telGesch',
|
||||
}
|
||||
```
|
||||
|
||||
### 3. Adressen
|
||||
|
||||
**EspoCRM → Advoware**:
|
||||
- Lade Related Entity `Adressen` via `adressensIds`
|
||||
- Mappe Hauptadresse zu Root-Feldern `strasse`, `plz`, `ort`
|
||||
- Zusätzliche Adressen in `adressen` Array
|
||||
|
||||
**Advoware → EspoCRM**:
|
||||
- Hauptadresse aus Root-Feldern
|
||||
- `adressen` Array → Related Entities in EspoCRM
|
||||
|
||||
### 4. Kontaktdaten (Komplex)
|
||||
|
||||
**EspoCRM `emailAddressData`**:
|
||||
```json
|
||||
[
|
||||
{
|
||||
"emailAddress": "primary@example.com",
|
||||
"primary": true,
|
||||
"optOut": false,
|
||||
"invalid": false
|
||||
}
|
||||
]
|
||||
```
|
||||
|
||||
**Advoware `kommunikation`**:
|
||||
```json
|
||||
[
|
||||
{
|
||||
"id": 88002,
|
||||
"kommArt": 0, // 0=Telefon, 1=Email, etc.
|
||||
"tlf": "0511/12345-60",
|
||||
"online": false
|
||||
}
|
||||
]
|
||||
```
|
||||
|
||||
**Mapping**: Erfordert Transformation basierend auf `kommArt`.
|
||||
|
||||
## Sync-Richtungen
|
||||
|
||||
### EspoCRM → Advoware (Webhook-getrieben)
|
||||
|
||||
1. Webhook empfängt `CBeteiligte` create/update/delete
|
||||
2. Mappe Felder gemäß Tabelle oben
|
||||
3. `POST /api/v1/advonet/Beteiligte` (create) oder
|
||||
`PUT /api/v1/advonet/Beteiligte/{betNr}` (update)
|
||||
4. Update `advowareLastSync` und `syncStatus` in EspoCRM
|
||||
|
||||
### Advoware → EspoCRM (Polling oder Webhook)
|
||||
|
||||
1. Überwache Änderungen in Advoware
|
||||
2. Mappe Felder zurück
|
||||
3. `PUT /api/v1/CBeteiligte/{id}` in EspoCRM
|
||||
4. Setze `syncStatus = "clean"`
|
||||
|
||||
## Konflikte & Regeln
|
||||
|
||||
| Szenario | Regel |
|
||||
|----------|-------|
|
||||
| Beide Systeme geändert | Advoware als Master (führendes System) |
|
||||
| Feld nur in EspoCRM | Ignorieren beim Export, behalten |
|
||||
| Feld nur in Advoware | Null/Leer in EspoCRM setzen |
|
||||
| `betnr` vs. `betNr` | Sync-Link: Muss identisch sein |
|
||||
|
||||
## ID-Mapping
|
||||
|
||||
**Problem**: EspoCRM und Advoware haben unterschiedliche ID-Systeme.
|
||||
|
||||
**Lösung**:
|
||||
- EspoCRM `betnr` Feld = Advoware `betNr`
|
||||
- Dies ist der Sync-Link zwischen beiden Systemen
|
||||
- Bei Create in EspoCRM: `betnr` erst nach Advoware-Insert setzen
|
||||
- Bei Create in Advoware: EspoCRM ID in Custom Field speichern?
|
||||
|
||||
## Nächste Schritte
|
||||
|
||||
1. **Mapper-Modul erstellen**: `bitbylaw/services/espocrm_mapper.py`
|
||||
- `map_cbeteiligte_to_advoware(espo_data) -> advo_data`
|
||||
- `map_advoware_to_cbeteiligte(advo_data) -> espo_data`
|
||||
|
||||
2. **Sync-Event-Step implementieren**: `bitbylaw/steps/vmh/beteiligte_sync_event_step.py`
|
||||
- Subscribe to `vmh.beteiligte.create/update/delete`
|
||||
- Fetch full entity from EspoCRM
|
||||
- Transform via Mapper
|
||||
- Write to Advoware
|
||||
- Update sync metadata
|
||||
|
||||
3. **Testing**:
|
||||
- Unit Tests für Mapper
|
||||
- Integration Tests mit Sandbox-Daten
|
||||
- Konflikt-Szenarien testen
|
||||
|
||||
4. **Error Handling**:
|
||||
- Retry-Logic bei API-Fehlern
|
||||
- Validation vor dem Sync
|
||||
- Rollback bei Fehlern?
|
||||
- Logging aller Sync-Operationen
|
||||
|
||||
5. **Performance**:
|
||||
- Batch-Processing für mehrere Beteiligte
|
||||
- Rate Limiting beachten
|
||||
- Caching von Lookup-Daten
|
||||
|
||||
## Beispiel-Transformation
|
||||
|
||||
### EspoCRM CBeteiligte:
|
||||
```json
|
||||
{
|
||||
"id": "68e4af00172be7924",
|
||||
"firstName": "Angela",
|
||||
"lastName": "Mustermanns",
|
||||
"rechtsform": "Frau",
|
||||
"emailAddress": "angela@example.com",
|
||||
"phoneNumber": "0511/12345",
|
||||
"betnr": 104860,
|
||||
"handelsregisterNummer": null
|
||||
}
|
||||
```
|
||||
|
||||
### Advoware Beteiligter:
|
||||
```json
|
||||
{
|
||||
"betNr": 104860,
|
||||
"vorname": "Angela",
|
||||
"name": "Mustermanns",
|
||||
"rechtsform": "Frau",
|
||||
"anrede": "Frau",
|
||||
"emailGesch": "angela@example.com",
|
||||
"telGesch": "0511/12345"
|
||||
}
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
**Generiert am**: 2026-02-07
|
||||
**Basierend auf**: Real-Daten-Vergleich mit `scripts/compare_beteiligte.py`
|
||||
@@ -46,16 +46,35 @@
|
||||
- [calendar_sync_all_step.md](../steps/advoware_cal_sync/calendar_sync_all_step.md) - Employee cascade
|
||||
- [calendar_sync_event_step.md](../steps/advoware_cal_sync/calendar_sync_event_step.md) - Per-employee sync (complex)
|
||||
|
||||
**VMH Webhooks** ([Module README](../steps/vmh/README.md)):
|
||||
- [beteiligte_create_api_step.md](../steps/vmh/webhook/beteiligte_create_api_step.md) - Create webhook
|
||||
**VMH Webhooks & Sync** ([Module README](../steps/vmh/README.md)):
|
||||
- **Beteiligte Sync** (Bidirectional EspoCRM ↔ Advoware)
|
||||
- [BETEILIGTE_SYNC.md](BETEILIGTE_SYNC.md) - Complete documentation
|
||||
- [README_SYNC.md](../steps/vmh/README_SYNC.md) - Event handler docs
|
||||
- [beteiligte_sync_event_step.py](../steps/vmh/beteiligte_sync_event_step.py) - Event handler
|
||||
- [beteiligte_sync_cron_step.py](../steps/vmh/beteiligte_sync_cron_step.py) - Cron job
|
||||
- **Webhooks**
|
||||
- [beteiligte_create_api_step.md](../steps/vmh/webhook/beteiligte_create_api_step.md) - Create webhook
|
||||
- [beteiligte_update_api_step.md](../steps/vmh/webhook/beteiligte_update_api_step.md) - Update webhook (similar)
|
||||
- [beteiligte_delete_api_step.md](../steps/vmh/webhook/beteiligte_delete_api_step.md) - Delete webhook (similar)
|
||||
- [beteiligte_sync_event_step.md](../steps/vmh/beteiligte_sync_event_step.md) - Sync handler (placeholder)
|
||||
|
||||
### Services
|
||||
|
||||
- [Advoware Service](../services/ADVOWARE_SERVICE.md) - API Client mit HMAC-512 Auth
|
||||
- [Advoware API Swagger](advoware/advoware_api_swagger.json) - Vollständige API-Dokumentation (JSON)
|
||||
- **Advoware Service** ([ADVOWARE_SERVICE.md](../services/ADVOWARE_SERVICE.md)) - API Client mit HMAC-512 Auth
|
||||
- **Advoware API Swagger** ([advoware_api_swagger.json](advoware/advoware_api_swagger.json)) - Vollständige API-Dokumentation
|
||||
- **EspoCRM Service** ([espocrm.py](../services/espocrm.py)) - EspoCRM API Client mit X-Api-Key Auth
|
||||
- **Sync Services**
|
||||
- [beteiligte_sync_utils.py](../services/beteiligte_sync_utils.py) - Sync utilities (lock, timestamp, merge)
|
||||
- [espocrm_mapper.py](../services/espocrm_mapper.py) - Entity mapping EspoCRM ↔ Advoware
|
||||
|
||||
### Sync Documentation
|
||||
|
||||
- **[BETEILIGTE_SYNC.md](BETEILIGTE_SYNC.md)** - Complete sync documentation
|
||||
- Architecture, data flow, troubleshooting
|
||||
- **[SYNC_TEMPLATE.md](SYNC_TEMPLATE.md)** - Template für neue Advoware-Syncs
|
||||
- Best practices, code templates, architecture principles
|
||||
- **[ENTITY_MAPPING_CBeteiligte_Advoware.md](ENTITY_MAPPING_CBeteiligte_Advoware.md)** - Field mapping details
|
||||
- **[SYNC_STRATEGY_ARCHIVE.md](SYNC_STRATEGY_ARCHIVE.md)** - Original strategy analysis (archived)
|
||||
|
||||
### Utility Scripts
|
||||
|
||||
@@ -77,15 +96,21 @@ docs/
|
||||
├── DEVELOPMENT.md # Development guide
|
||||
├── GOOGLE_SETUP.md # Google Calendar setup
|
||||
├── TROUBLESHOOTING.md # Debugging guide
|
||||
├── BETEILIGTE_SYNC.md # ⭐ Beteiligte sync docs
|
||||
├── SYNC_TEMPLATE.md # ⭐ Template for new syncs
|
||||
├── ENTITY_MAPPING_CBeteiligte_Advoware.md # Field mappings
|
||||
└── advoware/
|
||||
└── advoware_api_swagger.json # Advoware API spec
|
||||
|
||||
steps/{module}/
|
||||
├── README.md # Module overview
|
||||
├── README_SYNC.md # ⭐ Sync handler docs (VMH)
|
||||
└── {step_name}.md # Step documentation
|
||||
|
||||
services/
|
||||
└── {service_name}.md # Service documentation
|
||||
├── {service_name}.md # Service documentation
|
||||
├── beteiligte_sync_utils.py # ⭐ Sync utilities
|
||||
└── espocrm_mapper.py # ⭐ Entity mapper
|
||||
|
||||
scripts/{category}/
|
||||
├── README.md # Script documentation
|
||||
|
||||
485
bitbylaw/docs/SYNC_STRATEGY_ARCHIVE.md
Normal file
485
bitbylaw/docs/SYNC_STRATEGY_ARCHIVE.md
Normal file
@@ -0,0 +1,485 @@
|
||||
# Sync-Strategie: EspoCRM-basiert (ohne PostgreSQL)
|
||||
|
||||
**Analysiert am**: 2026-02-07
|
||||
**Anpassung**: EspoCRM als primäre State-Datenbank
|
||||
|
||||
---
|
||||
|
||||
## 🎯 EspoCRM Felder (CBeteiligte Entity)
|
||||
|
||||
```json
|
||||
{
|
||||
"betnr": 1234, // Link zu Advoware betNr (int, unique)
|
||||
"syncStatus": "clean", // Sync-Status (enum)
|
||||
"advowareLastSync": null, // Letzter Sync (datetime oder null)
|
||||
"advowareDeletedAt": null, // Gelöscht in Advoware am (datetime, NEU)
|
||||
"syncErrorMessage": null, // Fehlerdetails (text, NEU)
|
||||
"syncRetryCount": 0, // Anzahl Retry-Versuche (int, NEU)
|
||||
"modifiedAt": "2026-01-23 21:58:41" // EspoCRM Änderungszeit
|
||||
}
|
||||
```
|
||||
|
||||
### syncStatus-Werte (Enum in EspoCRM):
|
||||
- `"pending_sync"` - Neu erstellt, noch nicht nach Advoware gesynct
|
||||
- `"clean"` - Synchronisiert, keine ausstehenden Änderungen
|
||||
- `"dirty"` - In EspoCRM geändert, wartet auf Sync nach Advoware
|
||||
- `"syncing"` - Sync läuft gerade (verhindert Race Conditions)
|
||||
- `"failed"` - Sync fehlgeschlagen (mit syncErrorMessage + syncRetryCount)
|
||||
- `"conflict"` - Konflikt erkannt → **EspoCRM WINS** (mit Notification)
|
||||
- `"deleted_in_advoware"` - In Advoware gelöscht (Soft-Delete Flag mit Notification)
|
||||
|
||||
---
|
||||
|
||||
## 🔄 Flow A: EspoCRM Update → Advoware (Webhook)
|
||||
|
||||
**Trigger**: EspoCRM Webhook bei Create/Update
|
||||
|
||||
```
|
||||
1. EspoCRM: User ändert CBeteiligte
|
||||
└─> Webhook: POST /vmh/webhook/beteiligte/update
|
||||
Body: [{"id": "68e4af00172be7924"}]
|
||||
|
||||
2. beteiligte_update_api_step.py:
|
||||
├─> Redis Deduplication
|
||||
└─> Emit Event: "vmh.beteiligte.update"
|
||||
|
||||
3. beteiligte_sync_event_step.py:
|
||||
├─> Fetch Entity von EspoCRM:
|
||||
│ GET /api/v1/CBeteiligte/{id}
|
||||
│ {
|
||||
│ "id": "...",
|
||||
│ "firstName": "Angela",
|
||||
│ "lastName": "Mustermann",
|
||||
│ "betnr": 104860, // Bereits vorhanden
|
||||
│ "syncStatus": "clean",
|
||||
│ "advowareLastSync": "2026-02-01T10:00:00",
|
||||
│ "modifiedAt": "2026-02-07T14:30:00"
|
||||
│ }
|
||||
│
|
||||
├─> Check syncStatus:
|
||||
│ ├─> IF syncStatus == "syncing":
|
||||
│ │ → Skip (bereits im Sync-Prozess)
|
||||
│ │
|
||||
│ ├─> IF syncStatus == "pending_sync" AND betnr == NULL:
|
||||
│ │ → NEU: Create in Advoware
|
||||
│ │ ├─> Set syncStatus = "syncing"
|
||||
│ │ ├─> Transform via Mapper
|
||||
│ │ ├─> POST /api/v1/advonet/Beteiligte
|
||||
│ │ │ Response: {betNr: 123456}
|
||||
│ │ └─> Update EspoCRM:
|
||||
│ │ PUT /api/v1/CBeteiligte/{id}
|
||||
│ │ {
|
||||
│ │ betnr: 123456,
|
||||
│ │ syncStatus: "clean",
|
||||
│ │ advowareLastSync: NOW()
|
||||
│ │ }
|
||||
│ │
|
||||
│ └─> IF betnr != NULL (bereits gesynct):
|
||||
│ → UPDATE: Vergleiche Timestamps
|
||||
│ ├─> Fetch von Advoware:
|
||||
│ │ GET /api/v1/advonet/Beteiligte/{betnr}
|
||||
│ │ {betNr: 104860, geaendertAm: "2026-02-07T12:00:00"}
|
||||
│ │
|
||||
│ ├─> Vergleiche Timestamps:
|
||||
│ │ espocrm_ts = entity.modifiedAt
|
||||
│ │ advoware_ts = advo_entity.geaendertAm
|
||||
│ │ last_sync_ts = entity.advowareLastSync
|
||||
│ │
|
||||
│ │ IF espocrm_ts > last_sync_ts AND espocrm_ts > advoware_ts:
|
||||
│ │ → EspoCRM ist neuer → Update Advoware
|
||||
│ │ ├─> Set syncStatus = "syncing"
|
||||
│ │ ├─> PUT /api/v1/advonet/Beteiligte/{betnr}
|
||||
│ │ └─> Update EspoCRM:
|
||||
│ │ syncStatus = "clean"
|
||||
│ │ advowareLastSync = NOW()
|
||||
│ │ syncErrorMessage = NULL
|
||||
│ │ syncRetryCount = 0
|
||||
│ │
|
||||
│ │ ELSE IF advoware_ts > last_sync_ts AND advoware_ts > espocrm_ts:
|
||||
│ │ → Advoware ist neuer → Update EspoCRM
|
||||
│ │ ├─> Set syncStatus = "syncing"
|
||||
│ │ ├─> Transform von Advoware
|
||||
│ │ └─> Update EspoCRM mit Advoware-Daten
|
||||
│ │ syncStatus = "clean"
|
||||
│ │ advowareLastSync = NOW()
|
||||
│ │ syncErrorMessage = NULL
|
||||
│ │ syncRetryCount = 0
|
||||
│ │
|
||||
│ │ ELSE IF espocrm_ts > last_sync_ts AND advoware_ts > last_sync_ts:
|
||||
│ │ → KONFLIKT: Beide geändert seit last_sync
|
||||
│ │
|
||||
│ │ **REGEL: EspoCRM WINS!**
|
||||
│ │
|
||||
│ │ ├─> Set syncStatus = "conflict"
|
||||
│ │ ├─> Überschreibe Advoware mit EspoCRM-Daten:
|
||||
│ │ │ PUT /api/v1/advonet/Beteiligte/{betnr}
|
||||
│ │ │
|
||||
│ │ ├─> Update EspoCRM:
|
||||
│ │ │ syncStatus = "clean" (gelöst!)
|
||||
│ │ │ advowareLastSync = NOW()
|
||||
│ │ │ syncErrorMessage = "Konflikt am {NOW}: EspoCRM={espocrm_ts}, Advoware={advoware_ts}. EspoCRM hat gewonnen."
|
||||
│ │ │
|
||||
│ │ └─> Send Notification:
|
||||
│ │ Template: "beteiligte_sync_conflict"
|
||||
│ │ To: Admin-User oder zugewiesener User
|
||||
│ │
|
||||
│ │ ELSE:
|
||||
│ │ → Keine Änderungen seit last_sync
|
||||
│ │ └─> Skip
|
||||
│ │
|
||||
│ └─> Bei Fehler:
|
||||
│ syncStatus = "failed"
|
||||
│ syncErrorMessage = Error-Details (inkl. Stack Trace)
|
||||
│ syncRetryCount += 1
|
||||
│ Log Error
|
||||
│
|
||||
└─> Handle 404 von Advoware (gelöscht):
|
||||
IF advoware.api_call returns 404:
|
||||
├─> Update EspoCRM:
|
||||
│ syncStatus = "deleted_in_advoware"
|
||||
│ advowareDeletedAt = NOW()
|
||||
│ syncErrorMessage = "Beteiligter existiert nicht mehr in Advoware"
|
||||
│
|
||||
└─> Send Notification:
|
||||
Template: "beteiligte_advoware_deleted"
|
||||
To: Admin-User oder zugewiesener User
|
||||
```
|
||||
|
||||
**Timing**: ~2-5 Sekunden nach Webhook oder Cron-Event
|
||||
|
||||
---
|
||||
|
||||
## 🔄 Flow B: Advoware → EspoCRM (Cron-basiert mit Events)
|
||||
|
||||
**Trigger**: Cron alle 15 Minuten
|
||||
|
||||
```
|
||||
1. beteiligte_sync_cron_step.py (*/15 * * * *):
|
||||
|
||||
├─> Query EspoCRM: Alle Entities die Sync benötigen
|
||||
│
|
||||
│ SELECT * FROM CBeteiligte WHERE:
|
||||
│ - syncStatus IN ('pending_sync', 'dirty', 'failed')
|
||||
│ - OR (syncStatus = 'clean' AND betnr IS NOT NULL
|
||||
│ AND advowareLastSync < NOW() - 24 HOURS)
|
||||
│
|
||||
├─> Für JEDEN Beteiligten einzeln:
|
||||
│ └─> Emit Event: "vmh.beteiligte.sync_check"
|
||||
│ payload: {
|
||||
│ entity_id: "68e4af00172be7924",
|
||||
│ source: "cron",
|
||||
│ timestamp: "2026-02-07T14:30:00Z"
|
||||
│ }
|
||||
│
|
||||
└─> Log: "Emitted {count} sync_check events"
|
||||
|
||||
2. beteiligte_sync_event_step.py (GLEICHER Handler wie Webhook!):
|
||||
|
||||
└─> Subscribe zu: "vmh.beteiligte.sync_check"
|
||||
(Dieser Event kommt von Cron oder manuellen Triggers)
|
||||
|
||||
├─> Fetch entity_id aus Event-Payload
|
||||
│
|
||||
└─> Führe GLEICHE Logik aus wie bei Webhook (siehe Flow A oben!)
|
||||
- Lock via syncStatus
|
||||
- Timestamp-Vergleich
|
||||
- Create/Update
|
||||
- Konfliktauflösung (EspoCRM wins)
|
||||
- 404 Handling (deleted_in_advoware)
|
||||
- Update syncStatus + Felder
|
||||
|
||||
**WICHTIG**: Flow B nutzt Events statt Batch-Processing!
|
||||
- Cron emittiert nur Events für zu syncende Entities
|
||||
- Der normale Sync-Handler (Flow A) verarbeitet beide Quellen gleich
|
||||
- Code-Wiederverwendung: KEIN separater Batch-Handler nötig!
|
||||
```
|
||||
|
||||
**Timing**:
|
||||
- Cron läuft alle 15 Minuten
|
||||
- Events werden sofort verarbeitet (wie Webhooks)
|
||||
|
||||
---
|
||||
|
||||
## 📊 Optimierung: Nur veraltete checken
|
||||
|
||||
### Cron-Query für zu prüfende Entities:
|
||||
|
||||
```javascript
|
||||
// In beteiligte_sync_all_event_step.py
|
||||
|
||||
// 1. Holen von Entities die Sync benötigen
|
||||
const needsSyncFilter = {
|
||||
where: [
|
||||
{
|
||||
type: 'or',
|
||||
value: [
|
||||
// Neu und noch nicht gesynct
|
||||
{
|
||||
type: 'and',
|
||||
value: [
|
||||
{type: 'equals', attribute: 'syncStatus', value: 'pending_sync'},
|
||||
{type: 'isNull', attribute: 'betnr'}
|
||||
]
|
||||
},
|
||||
// Dirty (geändert in EspoCRM)
|
||||
{type: 'equals', attribute: 'syncStatus', value: 'dirty'},
|
||||
|
||||
// Failed (Retry)
|
||||
{type: 'equals', attribute: 'syncStatus', value: 'failed'},
|
||||
|
||||
// Clean aber lange nicht gesynct (> 24h)
|
||||
{
|
||||
type: 'and',
|
||||
value: [
|
||||
{type: 'equals', attribute: 'syncStatus', value: 'clean'},
|
||||
{type: 'isNotNull', attribute: 'betnr'},
|
||||
{
|
||||
type: 'or',
|
||||
value: [
|
||||
{type: 'isNull', attribute: 'advowareLastSync'},
|
||||
{type: 'before', attribute: 'advowareLastSync', value: 'NOW() - 24 HOURS'}
|
||||
]
|
||||
}
|
||||
]
|
||||
}
|
||||
]
|
||||
}
|
||||
]
|
||||
};
|
||||
```
|
||||
|
||||
### Advoware Query-Optimierung:
|
||||
|
||||
```python
|
||||
# Nur kürzlich geänderte aus Advoware holen
|
||||
last_full_sync = get_last_full_sync_timestamp() # z.B. vor 7 Tagen
|
||||
if last_full_sync:
|
||||
# Incremental Fetch
|
||||
params = {
|
||||
'filter': f'geaendertAm gt {last_full_sync.isoformat()}',
|
||||
'orderBy': 'geaendertAm desc'
|
||||
}
|
||||
else:
|
||||
# Full Fetch (beim ersten Mal oder nach langer Zeit)
|
||||
params = {}
|
||||
|
||||
result = await advoware.api_call(
|
||||
'api/v1/advonet/Beteiligte',
|
||||
method='GET',
|
||||
params=params
|
||||
)
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## 🔐 Locking via syncStatus
|
||||
|
||||
**Verhindert Race Conditions ohne Redis Lock**:
|
||||
|
||||
```python
|
||||
# Vor Sync-Operation:
|
||||
async def acquire_sync_lock(espocrm_api, entity_id):
|
||||
"""
|
||||
Setzt syncStatus auf "syncing" wenn möglich.
|
||||
Returns: True wenn Lock erhalten, False sonst
|
||||
"""
|
||||
try:
|
||||
# Fetch current
|
||||
entity = await espocrm_api.get_entity('CBeteiligte', entity_id)
|
||||
|
||||
if entity.get('syncStatus') == 'syncing':
|
||||
# Bereits im Sync-Prozess
|
||||
return False
|
||||
|
||||
# Atomic Update (EspoCRM sollte Optimistic Locking unterstützen)
|
||||
await espocrm_api.update_entity('CBeteiligte', entity_id, {
|
||||
'syncStatus': 'syncing'
|
||||
})
|
||||
|
||||
return True
|
||||
|
||||
except Exception as e:
|
||||
logger.error(f"Failed to acquire sync lock: {e}")
|
||||
return False
|
||||
|
||||
# Nach Sync-Operation (im finally-Block):
|
||||
async def release_sync_lock(espocrm_api, entity_id, new_status='clean'):
|
||||
"""Setzt syncStatus zurück"""
|
||||
try:
|
||||
await espocrm_api.update_entity('CBeteiligte', entity_id, {
|
||||
'syncStatus': new_status,
|
||||
'advowareLastSync': datetime.now(pytz.UTC).isoformat()
|
||||
})
|
||||
except Exception as e:
|
||||
logger.error(f"Failed to release sync lock: {e}")
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## 📋 Status-Übergänge
|
||||
|
||||
```
|
||||
pending_sync → syncing → clean (erfolgreicher Create)
|
||||
pending_sync → syncing → failed (fehlgeschlagener Create)
|
||||
|
||||
clean → dirty → syncing → clean (Update nach Änderung)
|
||||
clean → dirty → syncing → conflict (Konflikt detektiert)
|
||||
clean → dirty → syncing → failed (Update fehlgeschlagen)
|
||||
|
||||
failed → syncing → clean (erfolgreicher Retry)
|
||||
failed → syncing → failed (erneuter Fehler)
|
||||
|
||||
conflict → syncing → clean (manuell aufgelöst)
|
||||
|
||||
clean → deleted_in_advoware (in Advoware gelöscht)
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## 🎯 Implementierungs-Checkliste
|
||||
|
||||
### Phase 1: Core Sync (Flow A - Webhook + Cron Events)
|
||||
|
||||
- [ ] **services/espocrm_mapper.py**
|
||||
- [ ] `map_cbeteiligte_to_advoware(espo_entity)`
|
||||
- [ ] `map_advoware_to_cbeteiligte(advo_entity)`
|
||||
|
||||
- [ ] **steps/vmh/beteiligte_sync_event_step.py** (ZENTRALER Handler!)
|
||||
- [ ] Subscribe zu: `vmh.beteiligte.create`, `vmh.beteiligte.update`, `vmh.beteiligte.delete`, `vmh.beteiligte.sync_check`
|
||||
- [ ] Fetch Entity von EspoCRM
|
||||
- [ ] Lock via syncStatus="syncing"
|
||||
- [ ] Timestamp-Vergleich
|
||||
- [ ] Create/Update in Advoware
|
||||
- [ ] **Konfliktauflösung: EspoCRM wins!**
|
||||
- [ ] **404 Handling: Soft-Delete (deleted_in_advoware)**
|
||||
- [ ] **Notifications: Bei Konflikt + Soft-Delete**
|
||||
- [ ] Update syncStatus + advowareLastSync + syncErrorMessage + syncRetryCount
|
||||
- [ ] Error Handling (→ syncStatus="failed" mit Retry-Counter)
|
||||
- [ ] Redis Cleanup (SREM pending sets)
|
||||
|
||||
### Phase 2: Cron Event Emitter (Flow B)
|
||||
|
||||
- [ ] **steps/vmh/beteiligte_sync_cron_step.py**
|
||||
- [ ] Cron: `*/15 * * * *`
|
||||
- [ ] Query EspoCRM: Entities mit Status `IN (pending_sync, dirty, failed)` ODER `clean + advowareLastSync < NOW() - 24h`
|
||||
- [ ] Für JEDEN Beteiligten: Emit `vmh.beteiligte.sync_check` Event
|
||||
- [ ] Log: Anzahl emittierter Events
|
||||
- [ ] **KEIN** Batch-Processing - Events werden einzeln vom Handler verarbeitet!
|
||||
|
||||
### Phase 3: Utilities
|
||||
|
||||
- [ ] **services/betei & Notifications
|
||||
|
||||
- [ ] **services/beteiligte_sync_utils.py**
|
||||
- [ ] `acquire_sync_lock(entity_id)` → Setzt syncStatus="syncing"
|
||||
- [ ] `release_sync_lock(entity_id, new_status)` → Setzt syncStatus + Updates
|
||||
- [ ] `compare_timestamps(espo_ts, advo_ts, last_sync)` → Returns: "espocrm_newer", "advoware_newer", "conflict", "no_change"
|
||||
- [ ] `resolve_conflict_espocrm_wins(espo_entity, advo_entity)` → Überschreibt Advoware
|
||||
- [ ] `send_notification(entity_id, template_name, extra_data=None)` → EspoCRM Notification
|
||||
- [ ] `handle_advoware_deleted(entity_id, error_msg)` → Soft-Delete + Notification
|
||||
|
||||
- [ ] Unit Tests für Mapper
|
||||
- [ ] Integration Tests für beide Flows
|
||||
- [ ] Konflikt-Szenarien testen
|
||||
- [ ] Load-Tests (Performance mit 1000+ Entities)
|
||||
- [ ] CLI Audit-Tool (analog zu calendar_sync audit)
|
||||
→ clean (Konflikt → EspoCRM wins → gelöst!)
|
||||
clean → dirty → syncing → failed (Update fehlgeschlagen)
|
||||
|
||||
dirty → syncing → deleted_in_advoware (404 von Advoware → Soft-Delete)
|
||||
|
||||
failed → syncing → clean (erfolgreicher Retry)
|
||||
failed → syncing → failed (erneuter Fehler, syncRetryCount++)
|
||||
|
||||
conflict → clean (automatisch via EspoCRM wins)
|
||||
|
||||
clean → deleted_in_advoware (Advoware hat gelöscht)
|
||||
deleted_in_advoware → clean (Re-create in Advoware via Manual-Trigger
|
||||
GET /api/v1/CBeteiligte?select=syncStatus&maxSize=1000
|
||||
→ Gruppiere und zähle
|
||||
|
||||
// Entities die Sync benötigen
|
||||
GET /api/v1/CBeteiligte?where=[
|
||||
{type: 'in', attribute: 'syncStatus', value: ['pending_sync', 'dirty', 'failed']}
|
||||
]
|
||||
|
||||
// Lange nicht gesynct (> 7 Tage)
|
||||
GET /api/v1/CBeteiligte?where=[
|
||||
{type: 'before', attribute: 'advowareLastSync', value: 'NOW() - 7 DAYS'}
|
||||
]
|
||||
|
||||
// Konflikte
|
||||
GET /api/v1/CBeteiligte?where=[
|
||||
{type: 'equals', attribute: 'syncStatus', value: 'conflict'}
|
||||
]
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## 📈 Performance-Überlegungen
|
||||
|
||||
### Batch-Größen:
|
||||
|
||||
```python
|
||||
# Cron-Job Configuration
|
||||
CRON_BATCH_SIZE = 50 # Max 50 Entities pro Cron-Run
|
||||
CRON_TIMEOUT = 300 # 5 Minuten Timeout
|
||||
|
||||
# Advoware Fetch
|
||||
ADVOWARE_PAGE_SIZE = 100 # Entities pro API-Request
|
||||
```
|
||||
|
||||
### Timing:
|
||||
|
||||
- **Webhook (Flow A)**: 2-5 Sekunden (near real-time)
|
||||
- **Cron (Flow B)**: 15 Minuten Intervall
|
||||
- **Veraltete Check**: 24 Stunden (täglich syncen)
|
||||
- **Full Sync**: 7 Tage (wöchentlich alle prüfen)
|
||||
|
||||
### Rate Limiting:
|
||||
|
||||
```python
|
||||
# Aus bestehender AdvowareAPI
|
||||
# - Bereits implementiert
|
||||
# - Token-based Rate Limiting via Redis
|
||||
|
||||
# Für EspoCRM hinzufügen:
|
||||
ESPOCRM_MAX_REQUESTS_PER_MINUTE = 100
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## 🎯 Vorteile dieser Architektur
|
||||
|
||||
✅ **Kein PostgreSQL nötig** - EspoCRM ist State-Datenbank
|
||||
✅ **Alle Daten in EspoCRM** - Single Source of Truth
|
||||
✅ **Status sichtbar** - User können syncStatus in UI sehen
|
||||
✅ **Optimiert** - Nur veraltete werden geprüft
|
||||
✅ **Robust** - Locking via syncStatus verhindert Race Conditions
|
||||
✅ **Konflikt-Tracking** - Konflikte werden explizit markiert
|
||||
✅ **Wiederverwendbar** - Lock-Pattern nutzbar für andere Syncs
|
||||
|
||||
---
|
||||
|
||||
## 🔧 Nächste Schritte
|
||||
|
||||
1. **Mapper implementieren** (services/espocrm_mapper.py)
|
||||
2. **Webhook-Handler komplettieren** (Flow A)
|
||||
3. **Cron + Polling implementieren** (Flow B)
|
||||
4. **Testing mit echten Daten**
|
||||
5. **Monitoring & Dashboard**
|
||||
|
||||
**Geschätzte Zeit**: 5-7 Tage
|
||||
|
||||
---
|
||||
Entscheidungen (vom User bestätigt)**:
|
||||
1. ✅ syncStatus als Enum in EspoCRM mit definierten Werten
|
||||
2. ✅ Soft-Delete: Nur Flag (deleted_in_advoware + advowareDeletedAt)
|
||||
3. ✅ Automatisch: **EspoCRM WINS** bei Konflikten
|
||||
4. ✅ Notifications: Ja, bei Konflikten + Soft-Deletes (EspoCRM Notifications)
|
||||
|
||||
**Architektur-Entscheidung**:
|
||||
- ✅ Cron emittiert Events (`vmh.beteiligte.sync_check`), statt Batch-Processing
|
||||
- ✅ Ein zentraler Sync-Handler für Webhooks UND Cron-Events
|
||||
- ✅ Code-Wiederverwendung maximiertdvoware wins"?
|
||||
4. Benachrichtigung bei Konflikten? (Email, Webhook, ...)
|
||||
442
bitbylaw/docs/SYNC_TEMPLATE.md
Normal file
442
bitbylaw/docs/SYNC_TEMPLATE.md
Normal file
@@ -0,0 +1,442 @@
|
||||
# Advoware Sync Template
|
||||
|
||||
Template für neue bidirektionale Syncs zwischen EspoCRM und Advoware.
|
||||
|
||||
## Quick Start
|
||||
|
||||
Für neuen Sync von Entity `XYZ`:
|
||||
|
||||
### 1. EspoCRM Custom Fields
|
||||
```sql
|
||||
-- In EspoCRM Admin → Entity Manager → XYZ
|
||||
advowareId (int, unique) -- Foreign Key
|
||||
syncStatus (enum: clean|dirty|...) -- Status
|
||||
advowareLastSync (datetime) -- Letzter Sync
|
||||
syncErrorMessage (text, 2000) -- Fehler
|
||||
syncRetryCount (int) -- Retries
|
||||
```
|
||||
|
||||
### 2. Mapper erstellen
|
||||
```python
|
||||
# services/xyz_mapper.py
|
||||
class XYZMapper:
|
||||
@staticmethod
|
||||
def map_espo_to_advoware(espo_entity: Dict) -> Dict:
|
||||
"""EspoCRM → Advoware transformation"""
|
||||
return {
|
||||
'field1': espo_entity.get('espoField1'),
|
||||
'field2': espo_entity.get('espoField2'),
|
||||
# Nur relevante Felder mappen!
|
||||
}
|
||||
|
||||
@staticmethod
|
||||
def map_advoware_to_espo(advo_entity: Dict) -> Dict:
|
||||
"""Advoware → EspoCRM transformation"""
|
||||
return {
|
||||
'espoField1': advo_entity.get('field1'),
|
||||
'espoField2': advo_entity.get('field2'),
|
||||
}
|
||||
```
|
||||
|
||||
### 3. Sync Utils erstellen
|
||||
```python
|
||||
# services/xyz_sync_utils.py
|
||||
import redis
|
||||
from typing import Dict, Any, Optional
|
||||
from datetime import datetime
|
||||
import pytz
|
||||
|
||||
MAX_SYNC_RETRIES = 5
|
||||
LOCK_TTL_SECONDS = 300
|
||||
|
||||
class XYZSync:
|
||||
def __init__(self, espocrm_api, redis_client: redis.Redis, context=None):
|
||||
self.espocrm = espocrm_api
|
||||
self.redis = redis_client
|
||||
self.context = context
|
||||
|
||||
async def acquire_sync_lock(self, entity_id: str) -> bool:
|
||||
"""Atomic distributed lock via Redis"""
|
||||
if self.redis:
|
||||
lock_key = f"sync_lock:xyz:{entity_id}"
|
||||
acquired = self.redis.set(lock_key, "locked", nx=True, ex=LOCK_TTL_SECONDS)
|
||||
if not acquired:
|
||||
return False
|
||||
|
||||
await self.espocrm.update_entity('XYZ', entity_id, {'syncStatus': 'syncing'})
|
||||
return True
|
||||
|
||||
async def release_sync_lock(
|
||||
self,
|
||||
entity_id: str,
|
||||
new_status: str = 'clean',
|
||||
error_message: Optional[str] = None,
|
||||
increment_retry: bool = False,
|
||||
extra_fields: Optional[Dict[str, Any]] = None
|
||||
):
|
||||
"""Release lock and update status (combined operation)"""
|
||||
update_data = {
|
||||
'syncStatus': new_status,
|
||||
'advowareLastSync': datetime.now(pytz.UTC).isoformat()
|
||||
}
|
||||
|
||||
if error_message:
|
||||
update_data['syncErrorMessage'] = error_message[:2000]
|
||||
else:
|
||||
update_data['syncErrorMessage'] = None
|
||||
|
||||
if increment_retry:
|
||||
entity = await self.espocrm.get_entity('XYZ', entity_id)
|
||||
retry_count = (entity.get('syncRetryCount') or 0) + 1
|
||||
update_data['syncRetryCount'] = retry_count
|
||||
|
||||
if retry_count >= MAX_SYNC_RETRIES:
|
||||
update_data['syncStatus'] = 'permanently_failed'
|
||||
await self.send_notification(
|
||||
entity_id,
|
||||
f"Sync failed after {MAX_SYNC_RETRIES} attempts"
|
||||
)
|
||||
else:
|
||||
update_data['syncRetryCount'] = 0
|
||||
|
||||
if extra_fields:
|
||||
update_data.update(extra_fields)
|
||||
|
||||
await self.espocrm.update_entity('XYZ', entity_id, update_data)
|
||||
|
||||
if self.redis:
|
||||
self.redis.delete(f"sync_lock:xyz:{entity_id}")
|
||||
|
||||
def compare_timestamps(self, espo_ts, advo_ts, last_sync_ts):
|
||||
"""Compare timestamps and determine sync direction"""
|
||||
# Parse timestamps
|
||||
espo = self._parse_ts(espo_ts)
|
||||
advo = self._parse_ts(advo_ts)
|
||||
sync = self._parse_ts(last_sync_ts)
|
||||
|
||||
if not sync:
|
||||
if not espo or not advo:
|
||||
return "no_change"
|
||||
return "espocrm_newer" if espo > advo else "advoware_newer"
|
||||
|
||||
espo_changed = espo and espo > sync
|
||||
advo_changed = advo and advo > sync
|
||||
|
||||
if espo_changed and advo_changed:
|
||||
return "conflict"
|
||||
elif espo_changed:
|
||||
return "espocrm_newer"
|
||||
elif advo_changed:
|
||||
return "advoware_newer"
|
||||
else:
|
||||
return "no_change"
|
||||
|
||||
def merge_for_advoware_put(self, advo_entity, espo_entity, mapper):
|
||||
"""Merge EspoCRM updates into Advoware entity (Read-Modify-Write)"""
|
||||
advo_updates = mapper.map_espo_to_advoware(espo_entity)
|
||||
merged = {**advo_entity, **advo_updates}
|
||||
|
||||
self._log(f"📝 Merge: {len(advo_updates)} updates → {len(merged)} total")
|
||||
return merged
|
||||
|
||||
async def send_notification(self, entity_id, message):
|
||||
"""Send in-app notification to EspoCRM"""
|
||||
# Implementation...
|
||||
pass
|
||||
|
||||
def _parse_ts(self, ts):
|
||||
"""Parse timestamp string to datetime"""
|
||||
# Implementation...
|
||||
pass
|
||||
|
||||
def _log(self, msg, level='info'):
|
||||
"""Log with context support"""
|
||||
if self.context:
|
||||
getattr(self.context.logger, level)(msg)
|
||||
```
|
||||
|
||||
### 4. Event Handler erstellen
|
||||
```python
|
||||
# steps/vmh/xyz_sync_event_step.py
|
||||
from services.advoware import AdvowareAPI
|
||||
from services.espocrm import EspoCRMAPI
|
||||
from services.xyz_mapper import XYZMapper
|
||||
from services.xyz_sync_utils import XYZSync
|
||||
import redis
|
||||
from config import Config
|
||||
|
||||
config = {
|
||||
'type': 'event',
|
||||
'name': 'VMH XYZ Sync Handler',
|
||||
'description': 'Bidirectional sync for XYZ entities',
|
||||
'subscribes': [
|
||||
'vmh.xyz.create',
|
||||
'vmh.xyz.update',
|
||||
'vmh.xyz.delete',
|
||||
'vmh.xyz.sync_check'
|
||||
],
|
||||
'flows': ['vmh']
|
||||
}
|
||||
|
||||
async def handler(event_data, context):
|
||||
entity_id = event_data.get('entity_id')
|
||||
action = event_data.get('action', 'sync_check')
|
||||
|
||||
if not entity_id:
|
||||
context.logger.error("No entity_id in event")
|
||||
return
|
||||
|
||||
# Initialize
|
||||
redis_client = redis.Redis(
|
||||
host=Config.REDIS_HOST,
|
||||
port=int(Config.REDIS_PORT),
|
||||
db=int(Config.REDIS_DB_ADVOWARE_CACHE),
|
||||
decode_responses=True
|
||||
)
|
||||
|
||||
espocrm = EspoCRMAPI()
|
||||
advoware = AdvowareAPI(context)
|
||||
sync_utils = XYZSync(espocrm, redis_client, context)
|
||||
mapper = XYZMapper()
|
||||
|
||||
try:
|
||||
# Acquire lock
|
||||
if not await sync_utils.acquire_sync_lock(entity_id):
|
||||
context.logger.warning(f"Already syncing: {entity_id}")
|
||||
return
|
||||
|
||||
# Load entity
|
||||
espo_entity = await espocrm.get_entity('XYZ', entity_id)
|
||||
advoware_id = espo_entity.get('advowareId')
|
||||
|
||||
# Route to handler
|
||||
if not advoware_id and action in ['create', 'sync_check']:
|
||||
await handle_create(entity_id, espo_entity, espocrm, advoware, sync_utils, mapper, context)
|
||||
elif advoware_id:
|
||||
await handle_update(entity_id, advoware_id, espo_entity, espocrm, advoware, sync_utils, mapper, context)
|
||||
|
||||
except Exception as e:
|
||||
context.logger.error(f"Sync failed: {e}")
|
||||
await sync_utils.release_sync_lock(entity_id, 'failed', str(e), increment_retry=True)
|
||||
|
||||
|
||||
async def handle_create(entity_id, espo_entity, espocrm, advoware, sync_utils, mapper, context):
|
||||
"""Create new entity in Advoware"""
|
||||
try:
|
||||
advo_data = mapper.map_espo_to_advoware(espo_entity)
|
||||
|
||||
result = await advoware.api_call(
|
||||
'api/v1/advonet/XYZ',
|
||||
method='POST',
|
||||
data=advo_data
|
||||
)
|
||||
|
||||
new_id = result.get('id')
|
||||
if not new_id:
|
||||
raise Exception(f"No ID in response: {result}")
|
||||
|
||||
# Combined API call: release lock + save foreign key
|
||||
await sync_utils.release_sync_lock(
|
||||
entity_id,
|
||||
'clean',
|
||||
extra_fields={'advowareId': new_id}
|
||||
)
|
||||
|
||||
context.logger.info(f"✅ Created in Advoware: {new_id}")
|
||||
|
||||
except Exception as e:
|
||||
context.logger.error(f"❌ Create failed: {e}")
|
||||
await sync_utils.release_sync_lock(entity_id, 'failed', str(e), increment_retry=True)
|
||||
|
||||
|
||||
async def handle_update(entity_id, advoware_id, espo_entity, espocrm, advoware, sync_utils, mapper, context):
|
||||
"""Sync existing entity"""
|
||||
try:
|
||||
# Fetch from Advoware
|
||||
advo_result = await advoware.api_call(f'api/v1/advonet/XYZ/{advoware_id}', method='GET')
|
||||
advo_entity = advo_result[0] if isinstance(advo_result, list) else advo_result
|
||||
|
||||
if not advo_entity:
|
||||
context.logger.error(f"Entity not found in Advoware: {advoware_id}")
|
||||
await sync_utils.release_sync_lock(entity_id, 'failed', "Not found in Advoware")
|
||||
return
|
||||
|
||||
# Compare timestamps
|
||||
comparison = sync_utils.compare_timestamps(
|
||||
espo_entity.get('modifiedAt'),
|
||||
advo_entity.get('modifiedAt'), # Advoware timestamp field
|
||||
espo_entity.get('advowareLastSync')
|
||||
)
|
||||
|
||||
# Initial sync (no last_sync)
|
||||
if not espo_entity.get('advowareLastSync'):
|
||||
merged_data = sync_utils.merge_for_advoware_put(advo_entity, espo_entity, mapper)
|
||||
await advoware.api_call(f'api/v1/advonet/XYZ/{advoware_id}', method='PUT', data=merged_data)
|
||||
await sync_utils.release_sync_lock(entity_id, 'clean')
|
||||
return
|
||||
|
||||
# No change
|
||||
if comparison == 'no_change':
|
||||
await sync_utils.release_sync_lock(entity_id, 'clean')
|
||||
return
|
||||
|
||||
# EspoCRM newer
|
||||
if comparison == 'espocrm_newer':
|
||||
merged_data = sync_utils.merge_for_advoware_put(advo_entity, espo_entity, mapper)
|
||||
await advoware.api_call(f'api/v1/advonet/XYZ/{advoware_id}', method='PUT', data=merged_data)
|
||||
await sync_utils.release_sync_lock(entity_id, 'clean')
|
||||
|
||||
# Advoware newer
|
||||
elif comparison == 'advoware_newer':
|
||||
espo_data = mapper.map_advoware_to_espo(advo_entity)
|
||||
await espocrm.update_entity('XYZ', entity_id, espo_data)
|
||||
await sync_utils.release_sync_lock(entity_id, 'clean')
|
||||
|
||||
# Conflict → EspoCRM wins
|
||||
elif comparison == 'conflict':
|
||||
merged_data = sync_utils.merge_for_advoware_put(advo_entity, espo_entity, mapper)
|
||||
await advoware.api_call(f'api/v1/advonet/XYZ/{advoware_id}', method='PUT', data=merged_data)
|
||||
await sync_utils.send_notification(entity_id, "Conflict resolved: EspoCRM won")
|
||||
await sync_utils.release_sync_lock(entity_id, 'clean')
|
||||
|
||||
except Exception as e:
|
||||
context.logger.error(f"❌ Update failed: {e}")
|
||||
await sync_utils.release_sync_lock(entity_id, 'failed', str(e), increment_retry=True)
|
||||
```
|
||||
|
||||
### 5. Cron erstellen
|
||||
```python
|
||||
# steps/vmh/xyz_sync_cron_step.py
|
||||
import asyncio
|
||||
from services.espocrm import EspoCRMAPI
|
||||
import datetime
|
||||
|
||||
config = {
|
||||
'type': 'cron',
|
||||
'name': 'VMH XYZ Sync Cron',
|
||||
'description': 'Check for XYZ entities needing sync',
|
||||
'schedule': '*/15 * * * *', # Every 15 minutes
|
||||
'flows': ['vmh'],
|
||||
'emits': ['vmh.xyz.sync_check']
|
||||
}
|
||||
|
||||
async def handler(context):
|
||||
context.logger.info("🕐 XYZ Sync Cron started")
|
||||
|
||||
espocrm = EspoCRMAPI()
|
||||
threshold = datetime.datetime.now() - datetime.timedelta(hours=24)
|
||||
|
||||
# Find entities needing sync
|
||||
unclean_filter = {
|
||||
'where': [{
|
||||
'type': 'or',
|
||||
'value': [
|
||||
{'type': 'equals', 'attribute': 'syncStatus', 'value': 'pending_sync'},
|
||||
{'type': 'equals', 'attribute': 'syncStatus', 'value': 'dirty'},
|
||||
{'type': 'equals', 'attribute': 'syncStatus', 'value': 'failed'},
|
||||
]
|
||||
}]
|
||||
}
|
||||
|
||||
result = await espocrm.search_entities('XYZ', unclean_filter, max_size=100)
|
||||
entities = result.get('list', [])
|
||||
entity_ids = [e['id'] for e in entities]
|
||||
|
||||
context.logger.info(f"Found {len(entity_ids)} entities to sync")
|
||||
|
||||
if not entity_ids:
|
||||
return
|
||||
|
||||
# Batch emit (parallel)
|
||||
tasks = [
|
||||
context.emit({
|
||||
'topic': 'vmh.xyz.sync_check',
|
||||
'data': {
|
||||
'entity_id': eid,
|
||||
'action': 'sync_check',
|
||||
'source': 'cron',
|
||||
'timestamp': datetime.datetime.now().isoformat()
|
||||
}
|
||||
})
|
||||
for eid in entity_ids
|
||||
]
|
||||
|
||||
results = await asyncio.gather(*tasks, return_exceptions=True)
|
||||
success_count = sum(1 for r in results if not isinstance(r, Exception))
|
||||
|
||||
context.logger.info(f"✅ Emitted {success_count}/{len(entity_ids)} events")
|
||||
```
|
||||
|
||||
## Best Practices
|
||||
|
||||
### ✅ DO
|
||||
- Use Redis distributed lock (atomicity)
|
||||
- Combine API calls with `extra_fields`
|
||||
- Use `merge_for_advoware_put()` utility
|
||||
- Implement max retries (5x)
|
||||
- Batch emit in cron with `asyncio.gather()`
|
||||
- Map only relevant fields (avoid overhead)
|
||||
- Add proper error logging
|
||||
|
||||
### ❌ DON'T
|
||||
- Don't use GET-then-PUT for locks (race condition)
|
||||
- Don't make unnecessary API calls
|
||||
- Don't duplicate merge logic
|
||||
- Don't retry infinitely
|
||||
- Don't emit events sequentially in cron
|
||||
- Don't map every field (performance)
|
||||
- Don't swallow exceptions silently
|
||||
|
||||
## Architecture Principles
|
||||
|
||||
1. **Atomicity**: Redis lock + TTL
|
||||
2. **Efficiency**: Combined operations
|
||||
3. **Reusability**: Utility functions
|
||||
4. **Robustness**: Max retries + notifications
|
||||
5. **Scalability**: Batch processing
|
||||
6. **Maintainability**: Clear separation of concerns
|
||||
|
||||
## Performance Targets
|
||||
|
||||
| Metric | Target |
|
||||
|--------|--------|
|
||||
| Single sync latency | < 500ms |
|
||||
| API calls per operation | ≤ 3 |
|
||||
| Cron execution (100 entities) | < 2s |
|
||||
| Lock timeout | 5 min |
|
||||
| Max retries | 5 |
|
||||
|
||||
## Testing
|
||||
|
||||
```python
|
||||
# Test script template
|
||||
async def main():
|
||||
entity_id = 'test-id'
|
||||
espo = EspoCRMAPI()
|
||||
|
||||
# Reset entity
|
||||
await espo.update_entity('XYZ', entity_id, {
|
||||
'advowareLastSync': None,
|
||||
'syncStatus': 'clean',
|
||||
'syncRetryCount': 0
|
||||
})
|
||||
|
||||
# Trigger sync
|
||||
event_data = {
|
||||
'entity_id': entity_id,
|
||||
'action': 'sync_check',
|
||||
'source': 'test'
|
||||
}
|
||||
|
||||
await xyz_sync_event_step.handler(event_data, MockContext())
|
||||
|
||||
# Verify
|
||||
entity_after = await espo.get_entity('XYZ', entity_id)
|
||||
assert entity_after['syncStatus'] == 'clean'
|
||||
```
|
||||
|
||||
## Siehe auch
|
||||
|
||||
- [Beteiligte Sync](BETEILIGTE_SYNC.md) - Reference implementation
|
||||
- [Advoware API Docs](advoware/)
|
||||
- [EspoCRM API Docs](API.md)
|
||||
Reference in New Issue
Block a user