Compare commits
2 Commits
36552903e7
...
b5abe6cf00
| Author | SHA1 | Date | |
|---|---|---|---|
| b5abe6cf00 | |||
| e6ab22d5f4 |
37
bitbylaw/.env.example
Normal file
37
bitbylaw/.env.example
Normal file
@@ -0,0 +1,37 @@
|
||||
# Redis Configuration
|
||||
REDIS_HOST=localhost
|
||||
REDIS_PORT=6379
|
||||
REDIS_DB_ADVOWARE_CACHE=1
|
||||
REDIS_DB_CALENDAR_SYNC=2
|
||||
REDIS_TIMEOUT_SECONDS=5
|
||||
|
||||
# Advoware API
|
||||
ADVOWARE_API_BASE_URL=https://www2.advo-net.net:90/
|
||||
ADVOWARE_PRODUCT_ID=64
|
||||
ADVOWARE_APP_ID=your_app_id
|
||||
ADVOWARE_API_KEY=your_api_key_base64
|
||||
ADVOWARE_KANZLEI=your_kanzlei
|
||||
ADVOWARE_DATABASE=your_database
|
||||
ADVOWARE_USER=your_user
|
||||
ADVOWARE_ROLE=2
|
||||
ADVOWARE_PASSWORD=your_password
|
||||
ADVOWARE_TOKEN_LIFETIME_MINUTES=55
|
||||
ADVOWARE_API_TIMEOUT_SECONDS=30
|
||||
|
||||
# EspoCRM API
|
||||
ESPOCRM_API_BASE_URL=https://crm.bitbylaw.com/api/v1
|
||||
ESPOCRM_MARVIN_API_KEY=your_espocrm_api_key
|
||||
ESPOCRM_API_TIMEOUT_SECONDS=30
|
||||
|
||||
# Google Calendar API (Service Account)
|
||||
GOOGLE_CALENDAR_SERVICE_ACCOUNT_PATH=service-account.json
|
||||
|
||||
# PostgreSQL (Calendar Sync Hub)
|
||||
POSTGRES_HOST=localhost
|
||||
POSTGRES_USER=calendar_sync_user
|
||||
POSTGRES_PASSWORD=default_password
|
||||
POSTGRES_DB_NAME=calendar_sync_db
|
||||
|
||||
# Calendar Sync Settings
|
||||
CALENDAR_SYNC_ANONYMIZE_GOOGLE_EVENTS=true
|
||||
CALENDAR_SYNC_DEBUG_KUERZEL=SB,AI,RO,OK,BI,ST,UR,PB,VB
|
||||
286
bitbylaw/ENTITY_MAPPING_CBeteiligte_Advoware.md
Normal file
286
bitbylaw/ENTITY_MAPPING_CBeteiligte_Advoware.md
Normal file
@@ -0,0 +1,286 @@
|
||||
# Entity-Mapping: EspoCRM CBeteiligte ↔ Advoware Beteiligte
|
||||
|
||||
Basierend auf dem Vergleich von:
|
||||
- **EspoCRM**: CBeteiligte Entity ID `68e4af00172be7924`
|
||||
- **Advoware**: Beteiligter ID `104860`
|
||||
|
||||
## Gemeinsame Felder (direkte Übereinstimmung)
|
||||
|
||||
| EspoCRM Feld | Advoware Feld | Typ | Notes |
|
||||
|--------------|---------------|-----|-------|
|
||||
| `name` | `name` | string | Vollständiger Name |
|
||||
| `rechtsform` | `rechtsform` | string | Rechtsform (z.B. "GmbH", "Frau") |
|
||||
| `id` | `id` | mixed | **Achtung:** EspoCRM=string, Advoware=int |
|
||||
|
||||
## Namenfelder
|
||||
|
||||
| EspoCRM Feld | Advoware Feld | Mapping |
|
||||
|--------------|---------------|---------|
|
||||
| `firstName` | `vorname` | ✓ Direkt |
|
||||
| `lastName` | `name` | ✓ Bei Personen |
|
||||
| `middleName` | - | ❌ Kein direktes Mapping |
|
||||
| `firmenname` | `name` | ✓ Bei Firmen |
|
||||
| - | `geburtsname` | ← Nur in Advoware |
|
||||
| - | `kurzname` | ← Nur in Advoware |
|
||||
|
||||
## Kontaktdaten
|
||||
|
||||
| EspoCRM Feld | Advoware Feld | Mapping |
|
||||
|--------------|---------------|---------|
|
||||
| `emailAddress` | `emailGesch` | ✓ Geschäftlich |
|
||||
| `emailAddressData` (array) | `email` | ⚠️ Komplex: Array vs. String |
|
||||
| `phoneNumber` | `telGesch` | ✓ Geschäftstelefon |
|
||||
| `phoneNumberData` (array) | `telPrivat` | ⚠️ Komplex |
|
||||
| - | `mobil` | ← Nur in Advoware |
|
||||
| - | `faxGesch` / `faxPrivat` | ← Nur in Advoware |
|
||||
| - | `autotelefon` | ← Nur in Advoware |
|
||||
| - | `internet` | ← Nur in Advoware |
|
||||
|
||||
**Hinweis**: Advoware hat zusätzlich `kommunikation` Array mit strukturierten Kontaktdaten.
|
||||
|
||||
## Adressdaten
|
||||
|
||||
| EspoCRM Feld | Advoware Feld | Mapping |
|
||||
|--------------|---------------|---------|
|
||||
| `adressensIds` / `adressensNames` | `adressen` (array) | ⚠️ Beziehung |
|
||||
| - | `strasse` | ← Hauptadresse in Advoware Root |
|
||||
| - | `plz` | ← Hauptadresse in Advoware Root |
|
||||
| - | `ort` | ← Hauptadresse in Advoware Root |
|
||||
| - | `anschrift` | ← Formatierte Adresse |
|
||||
|
||||
**Hinweis**:
|
||||
- EspoCRM: Adressen als Related Entities (IDs/Names)
|
||||
- Advoware: Hauptadresse im Root-Objekt + `adressen` Array für zusätzliche
|
||||
|
||||
## Anrede & Titel
|
||||
|
||||
| EspoCRM Feld | Advoware Feld | Mapping |
|
||||
|--------------|---------------|---------|
|
||||
| `salutationName` | `anrede` | ✓ (z.B. "Frau", "Herr") |
|
||||
| - | `bAnrede` | ← Briefanrede ("Sehr geehrte...") |
|
||||
| - | `titel` | ← Titel (Dr., Prof., etc.) |
|
||||
| - | `zusatz` | ← Namenszusatz |
|
||||
|
||||
## Geburtsdaten
|
||||
|
||||
| EspoCRM Feld | Advoware Feld | Mapping |
|
||||
|--------------|---------------|---------|
|
||||
| `dateOfBirth` | `geburtsdatum` | ✓ Direkt |
|
||||
| - | `sterbedatum` | ← Nur in Advoware |
|
||||
| - | `familienstand` | ← Nur in Advoware |
|
||||
|
||||
## Handelsregister (für Firmen)
|
||||
|
||||
| EspoCRM Feld | Advoware Feld | Mapping |
|
||||
|--------------|---------------|---------|
|
||||
| `handelsregisterNummer` | `handelsRegisterNummer` | ✓ Direkt |
|
||||
| `handelsregisterArt` (z.B. "HRB") | - | ❌ Nur in EspoCRM |
|
||||
| - | `registergericht` | ← Nur in Advoware |
|
||||
|
||||
## Bankverbindungen
|
||||
|
||||
| EspoCRM Feld | Advoware Feld | Mapping |
|
||||
|--------------|---------------|---------|
|
||||
| `bankverbindungensIds` / Names | `bankkverbindungen` (array) | ⚠️ Related Entity vs. Array |
|
||||
|
||||
## Beteiligungen/Akten
|
||||
|
||||
| EspoCRM Feld | Advoware Feld | Mapping |
|
||||
|--------------|---------------|---------|
|
||||
| - | `beteiligungen` (array) | ← Nur in Advoware |
|
||||
|
||||
**Hinweis**: Advoware speichert die Akten-Beteiligungen direkt beim Beteiligten.
|
||||
|
||||
## EspoCRM-spezifische Felder
|
||||
|
||||
| Feld | Zweck |
|
||||
|------|-------|
|
||||
| `betnr` | Beteiligten-Nummer (= Advoware `betNr`) |
|
||||
| `advowareLastSync` | Zeitstempel der letzten Synchronisation |
|
||||
| `syncStatus` | Status: "clean", "dirty", "syncing" |
|
||||
| `disgTyp` | DISC-Persönlichkeitstyp |
|
||||
| `description` | Notizen/Beschreibung |
|
||||
| `createdAt` / `createdById` / `createdByName` | Audit-Felder |
|
||||
| `modifiedAt` / `modifiedById` / `modifiedByName` | Audit-Felder |
|
||||
| `assignedUserId` / `assignedUserName` | Zuweisungen |
|
||||
| `teamsIds` / `teamsNames` | Team-Zugehörigkeit |
|
||||
| `deleted` | Soft-Delete Flag |
|
||||
| `isFollowed` / `followersIds` | Social Features |
|
||||
|
||||
## Advoware-spezifische Felder
|
||||
|
||||
| Feld | Zweck |
|
||||
|------|-------|
|
||||
| `betNr` | Interne Beteiligten-Nummer |
|
||||
| `rowId` | Datenbank Row-ID |
|
||||
| `art` | Beteiligten-Art |
|
||||
| `angelegtAm` / `angelegtVon` | Erstellt |
|
||||
| `geaendertAm` / `geaendertVon` | Geändert |
|
||||
| `kontaktpersonen` (array) | Kontaktpersonen bei Firmen |
|
||||
| `ePost` / `bea` | Spezielle Kommunikationskanäle |
|
||||
|
||||
## Mapping-Strategie
|
||||
|
||||
### 1. Person (Natürliche Person)
|
||||
|
||||
```python
|
||||
espocrm_to_advoware = {
|
||||
'firstName': 'vorname',
|
||||
'lastName': 'name',
|
||||
'dateOfBirth': 'geburtsdatum',
|
||||
'rechtsform': 'rechtsform', # z.B. "Herr", "Frau"
|
||||
'salutationName': 'anrede',
|
||||
'emailAddress': 'emailGesch',
|
||||
'phoneNumber': 'telGesch',
|
||||
}
|
||||
```
|
||||
|
||||
### 2. Firma (Juristische Person)
|
||||
|
||||
```python
|
||||
espocrm_to_advoware = {
|
||||
'firmenname': 'name',
|
||||
'rechtsform': 'rechtsform', # z.B. "GmbH", "AG"
|
||||
'handelsregisterNummer': 'handelsRegisterNummer',
|
||||
'emailAddress': 'emailGesch',
|
||||
'phoneNumber': 'telGesch',
|
||||
}
|
||||
```
|
||||
|
||||
### 3. Adressen
|
||||
|
||||
**EspoCRM → Advoware**:
|
||||
- Lade Related Entity `Adressen` via `adressensIds`
|
||||
- Mappe Hauptadresse zu Root-Feldern `strasse`, `plz`, `ort`
|
||||
- Zusätzliche Adressen in `adressen` Array
|
||||
|
||||
**Advoware → EspoCRM**:
|
||||
- Hauptadresse aus Root-Feldern
|
||||
- `adressen` Array → Related Entities in EspoCRM
|
||||
|
||||
### 4. Kontaktdaten (Komplex)
|
||||
|
||||
**EspoCRM `emailAddressData`**:
|
||||
```json
|
||||
[
|
||||
{
|
||||
"emailAddress": "primary@example.com",
|
||||
"primary": true,
|
||||
"optOut": false,
|
||||
"invalid": false
|
||||
}
|
||||
]
|
||||
```
|
||||
|
||||
**Advoware `kommunikation`**:
|
||||
```json
|
||||
[
|
||||
{
|
||||
"id": 88002,
|
||||
"kommArt": 0, // 0=Telefon, 1=Email, etc.
|
||||
"tlf": "0511/12345-60",
|
||||
"online": false
|
||||
}
|
||||
]
|
||||
```
|
||||
|
||||
**Mapping**: Erfordert Transformation basierend auf `kommArt`.
|
||||
|
||||
## Sync-Richtungen
|
||||
|
||||
### EspoCRM → Advoware (Webhook-getrieben)
|
||||
|
||||
1. Webhook empfängt `CBeteiligte` create/update/delete
|
||||
2. Mappe Felder gemäß Tabelle oben
|
||||
3. `POST /api/v1/advonet/Beteiligte` (create) oder
|
||||
`PUT /api/v1/advonet/Beteiligte/{betNr}` (update)
|
||||
4. Update `advowareLastSync` und `syncStatus` in EspoCRM
|
||||
|
||||
### Advoware → EspoCRM (Polling oder Webhook)
|
||||
|
||||
1. Überwache Änderungen in Advoware
|
||||
2. Mappe Felder zurück
|
||||
3. `PUT /api/v1/CBeteiligte/{id}` in EspoCRM
|
||||
4. Setze `syncStatus = "clean"`
|
||||
|
||||
## Konflikte & Regeln
|
||||
|
||||
| Szenario | Regel |
|
||||
|----------|-------|
|
||||
| Beide Systeme geändert | Advoware als Master (führendes System) |
|
||||
| Feld nur in EspoCRM | Ignorieren beim Export, behalten |
|
||||
| Feld nur in Advoware | Null/Leer in EspoCRM setzen |
|
||||
| `betnr` vs. `betNr` | Sync-Link: Muss identisch sein |
|
||||
|
||||
## ID-Mapping
|
||||
|
||||
**Problem**: EspoCRM und Advoware haben unterschiedliche ID-Systeme.
|
||||
|
||||
**Lösung**:
|
||||
- EspoCRM `betnr` Feld = Advoware `betNr`
|
||||
- Dies ist der Sync-Link zwischen beiden Systemen
|
||||
- Bei Create in EspoCRM: `betnr` erst nach Advoware-Insert setzen
|
||||
- Bei Create in Advoware: EspoCRM ID in Custom Field speichern?
|
||||
|
||||
## Nächste Schritte
|
||||
|
||||
1. **Mapper-Modul erstellen**: `bitbylaw/services/espocrm_mapper.py`
|
||||
- `map_cbeteiligte_to_advoware(espo_data) -> advo_data`
|
||||
- `map_advoware_to_cbeteiligte(advo_data) -> espo_data`
|
||||
|
||||
2. **Sync-Event-Step implementieren**: `bitbylaw/steps/vmh/beteiligte_sync_event_step.py`
|
||||
- Subscribe to `vmh.beteiligte.create/update/delete`
|
||||
- Fetch full entity from EspoCRM
|
||||
- Transform via Mapper
|
||||
- Write to Advoware
|
||||
- Update sync metadata
|
||||
|
||||
3. **Testing**:
|
||||
- Unit Tests für Mapper
|
||||
- Integration Tests mit Sandbox-Daten
|
||||
- Konflikt-Szenarien testen
|
||||
|
||||
4. **Error Handling**:
|
||||
- Retry-Logic bei API-Fehlern
|
||||
- Validation vor dem Sync
|
||||
- Rollback bei Fehlern?
|
||||
- Logging aller Sync-Operationen
|
||||
|
||||
5. **Performance**:
|
||||
- Batch-Processing für mehrere Beteiligte
|
||||
- Rate Limiting beachten
|
||||
- Caching von Lookup-Daten
|
||||
|
||||
## Beispiel-Transformation
|
||||
|
||||
### EspoCRM CBeteiligte:
|
||||
```json
|
||||
{
|
||||
"id": "68e4af00172be7924",
|
||||
"firstName": "Angela",
|
||||
"lastName": "Mustermanns",
|
||||
"rechtsform": "Frau",
|
||||
"emailAddress": "angela@example.com",
|
||||
"phoneNumber": "0511/12345",
|
||||
"betnr": 104860,
|
||||
"handelsregisterNummer": null
|
||||
}
|
||||
```
|
||||
|
||||
### Advoware Beteiligter:
|
||||
```json
|
||||
{
|
||||
"betNr": 104860,
|
||||
"vorname": "Angela",
|
||||
"name": "Mustermanns",
|
||||
"rechtsform": "Frau",
|
||||
"anrede": "Frau",
|
||||
"emailGesch": "angela@example.com",
|
||||
"telGesch": "0511/12345"
|
||||
}
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
**Generiert am**: 2026-02-07
|
||||
**Basierend auf**: Real-Daten-Vergleich mit `scripts/compare_beteiligte.py`
|
||||
114
bitbylaw/ESPOCRM_INTEGRATION_NEXT_STEPS.md
Normal file
114
bitbylaw/ESPOCRM_INTEGRATION_NEXT_STEPS.md
Normal file
@@ -0,0 +1,114 @@
|
||||
# EspoCRM Integration - Nächste Schritte
|
||||
|
||||
## ✅ Bereits erstellt:
|
||||
|
||||
### 1. EspoCRM Service (`services/espocrm.py`)
|
||||
- Vollständiger API-Client mit allen CRUD-Operationen
|
||||
- X-Api-Key Authentifizierung
|
||||
- Error Handling und Logging
|
||||
- Redis-Integration für Caching/Rate Limiting
|
||||
|
||||
### 2. Compare Script (`scripts/compare_beteiligte.py`)
|
||||
- Liest Beteiligten-Daten aus EspoCRM und Advoware
|
||||
- Zeigt Struktur-Unterschiede
|
||||
- Hilft beim Entity-Mapping
|
||||
|
||||
## 🔧 Setup
|
||||
|
||||
1. **Umgebungsvariablen setzen**:
|
||||
```bash
|
||||
cp .env.example .env
|
||||
# Dann .env editieren und echte Keys eintragen
|
||||
```
|
||||
|
||||
2. **EspoCRM API Key besorgen**:
|
||||
- In EspoCRM Admin Panel: Administration → API Users
|
||||
- Neuen API User erstellen oder bestehenden Key kopieren
|
||||
- In `.env` als `ESPOCRM_MARVIN_API_KEY` eintragen
|
||||
|
||||
## 🧪 Testing
|
||||
|
||||
### Compare Script ausführen:
|
||||
```bash
|
||||
cd /opt/motia-app/bitbylaw
|
||||
source python_modules/bin/activate
|
||||
|
||||
# Mit EspoCRM ID (sucht automatisch in Advoware nach Namen)
|
||||
python scripts/compare_beteiligte.py <espocrm_entity_id>
|
||||
|
||||
# Mit beiden IDs
|
||||
python scripts/compare_beteiligte.py <espocrm_id> <advoware_id>
|
||||
```
|
||||
|
||||
**Beispiel**:
|
||||
```bash
|
||||
python scripts/compare_beteiligte.py 507f1f77bcf86cd799439011 12345
|
||||
```
|
||||
|
||||
### Output zeigt:
|
||||
- Alle Felder aus EspoCRM
|
||||
- Alle Felder aus Advoware
|
||||
- Strukturunterschiede
|
||||
- Mapping-Vorschläge
|
||||
|
||||
## 📋 Nächste Schritte
|
||||
|
||||
### 1. Entity-Mapping definieren
|
||||
Basierend auf dem Compare-Output:
|
||||
- `bitbylaw/services/espocrm_mapper.py` erstellen
|
||||
- Mapping-Funktionen für Beteiligte ↔ Personen/Firmen
|
||||
- Feld-Transformationen
|
||||
|
||||
### 2. Sync Event Step implementieren
|
||||
`bitbylaw/steps/vmh/beteiligte_sync_event_step.py`:
|
||||
- Events von Webhooks verarbeiten
|
||||
- EspoCRM API Client nutzen
|
||||
- Mapper für Transformation
|
||||
- In Advoware schreiben (via Proxy)
|
||||
- Redis Cleanup
|
||||
|
||||
### 3. Testing & Integration
|
||||
- Unit Tests für Mapper
|
||||
- Integration Tests mit echten APIs
|
||||
- Error Handling testen
|
||||
- Rate Limiting verifizieren
|
||||
|
||||
## 📚 Dokumentation
|
||||
|
||||
- **Service**: `services/ESPOCRM_SERVICE.md`
|
||||
- **Script README**: `scripts/compare_beteiligte_README.md`
|
||||
- **API Docs**: `docs/API.md` (VMH Webhooks Sektion)
|
||||
- **Architektur**: `docs/ARCHITECTURE.md` (EspoCRM Integration)
|
||||
|
||||
## 🔍 Tipps
|
||||
|
||||
### EspoCRM Entity Types
|
||||
Häufige Entity-Types in EspoCRM:
|
||||
- `Contact` - Personen
|
||||
- `Account` - Firmen/Organisationen
|
||||
- `Lead` - Leads
|
||||
- `Opportunity` - Verkaufschancen
|
||||
- Custom Entities (z.B. `CVmhBeteiligte`, `CVmhErstgespraech`)
|
||||
|
||||
### Advoware Mapping
|
||||
- Person → `personen` Endpoint
|
||||
- Firma → `firmen` Endpoint
|
||||
- Beide sind "Beteiligte" in Advoware-Sprache
|
||||
|
||||
### API Endpoints
|
||||
```bash
|
||||
# EspoCRM
|
||||
curl -X GET "https://crm.bitbylaw.com/api/v1/Contact/ID" \
|
||||
-H "X-Api-Key: YOUR_KEY"
|
||||
|
||||
# Advoware (via Proxy)
|
||||
curl -X GET "http://localhost:3000/advoware/personen/ID" \
|
||||
-H "Authorization: Bearer YOUR_TOKEN"
|
||||
```
|
||||
|
||||
## ❓ Support
|
||||
|
||||
Bei Fragen siehe:
|
||||
- EspoCRM API Docs: https://docs.espocrm.com/development/api/
|
||||
- Advoware Integration: `docs/ADVOWARE_SERVICE.md`
|
||||
- Motia Framework: `docs/DEVELOPMENT.md`
|
||||
326
bitbylaw/IMPLEMENTATION_COMPLETE.md
Normal file
326
bitbylaw/IMPLEMENTATION_COMPLETE.md
Normal file
@@ -0,0 +1,326 @@
|
||||
# Beteiligte Sync Implementation - Fertig! ✅
|
||||
|
||||
**Stand**: 7. Februar 2026
|
||||
**Status**: Vollständig implementiert, ready for testing
|
||||
|
||||
---
|
||||
|
||||
## 📦 Implementierte Module
|
||||
|
||||
### 1. **services/espocrm_mapper.py** ✅
|
||||
**Zweck**: Entity-Transformation zwischen EspoCRM ↔ Advoware
|
||||
|
||||
**Funktionen**:
|
||||
- `map_cbeteiligte_to_advoware(espo_entity)` - EspoCRM → Advoware
|
||||
- `map_advoware_to_cbeteiligte(advo_entity)` - Advoware → EspoCRM
|
||||
- `get_changed_fields(espo, advo)` - Diff-Vergleich
|
||||
|
||||
**Features**:
|
||||
- Unterscheidet Person vs. Firma
|
||||
- Mapped Namen, Kontaktdaten, Handelsregister
|
||||
- Transformiert emailAddressData/phoneNumberData Arrays
|
||||
- Normalisiert Rechtsform und Anrede
|
||||
|
||||
---
|
||||
|
||||
### 2. **services/beteiligte_sync_utils.py** ✅
|
||||
**Zweck**: Sync-Utility-Funktionen
|
||||
|
||||
**Funktionen**:
|
||||
- `acquire_sync_lock(entity_id)` - Atomares Lock via syncStatus
|
||||
- `release_sync_lock(entity_id, status, error, retry)` - Lock freigeben + Update
|
||||
- `parse_timestamp(ts)` - Parse EspoCRM/Advoware Timestamps
|
||||
- `compare_timestamps(espo, advo, last_sync)` - Returns: espocrm_newer | advoware_newer | conflict | no_change
|
||||
- `send_notification(entity_id, type, data)` - EspoCRM In-App Notification
|
||||
- `handle_advoware_deleted(entity_id, error)` - Soft-Delete + Notification
|
||||
- `resolve_conflict_espocrm_wins(entity_id, ...)` - Konfliktauflösung
|
||||
|
||||
**Features**:
|
||||
- Race-Condition-Prevention durch syncStatus="syncing"
|
||||
- Automatische syncRetryCount Increment bei Fehlern
|
||||
- EspoCRM Notifications (🔔 Bell-Icon)
|
||||
- Timestamp-Normalisierung für beide Systeme
|
||||
|
||||
---
|
||||
|
||||
### 3. **steps/vmh/beteiligte_sync_event_step.py** ✅
|
||||
**Zweck**: Zentraler Sync-Handler (Webhooks + Cron)
|
||||
|
||||
**Config**:
|
||||
```python
|
||||
subscribes: [
|
||||
'vmh.beteiligte.create',
|
||||
'vmh.beteiligte.update',
|
||||
'vmh.beteiligte.delete',
|
||||
'vmh.beteiligte.sync_check' # Von Cron
|
||||
]
|
||||
```
|
||||
|
||||
**Ablauf**:
|
||||
1. Acquire Lock (syncStatus → syncing)
|
||||
2. Fetch Entity von EspoCRM
|
||||
3. Bestimme Aktion:
|
||||
- **Kein betnr** → `handle_create()` - Neu in Advoware
|
||||
- **Hat betnr** → `handle_update()` - Sync mit Timestamp-Vergleich
|
||||
4. Release Lock mit finalem Status
|
||||
|
||||
**handle_create()**:
|
||||
- Transform zu Advoware Format
|
||||
- POST /api/v1/advonet/Beteiligte
|
||||
- Update EspoCRM mit neuer betnr
|
||||
- Status → clean
|
||||
|
||||
**handle_update()**:
|
||||
- Fetch von Advoware (betNr)
|
||||
- 404 → `handle_advoware_deleted()` (Soft-Delete)
|
||||
- Timestamp-Vergleich:
|
||||
- `espocrm_newer` → PUT zu Advoware
|
||||
- `advoware_newer` → PUT zu EspoCRM
|
||||
- `conflict` → **EspoCRM WINS** → Überschreibe Advoware → Notification
|
||||
- `no_change` → Skip
|
||||
|
||||
**Error Handling**:
|
||||
- Try/Catch um alle Operationen
|
||||
- Bei Fehler: syncStatus=failed, syncErrorMessage, syncRetryCount++
|
||||
- Redis Queue Cleanup
|
||||
|
||||
---
|
||||
|
||||
### 4. **steps/vmh/beteiligte_sync_cron_step.py** ✅
|
||||
**Zweck**: Cron-Job der Events emittiert
|
||||
|
||||
**Config**:
|
||||
```python
|
||||
schedule: '*/15 * * * *' # Alle 15 Minuten
|
||||
emits: ['vmh.beteiligte.sync_check']
|
||||
```
|
||||
|
||||
**Ablauf**:
|
||||
1. Query 1: Entities mit Status `pending_sync`, `dirty`, `failed` (max 100)
|
||||
2. Query 2: `clean` Entities mit `advowareLastSync < NOW() - 24h` (max 50)
|
||||
3. Kombiniere + Dedupliziere
|
||||
4. Emittiere `vmh.beteiligte.sync_check` Event für JEDEN Beteiligten
|
||||
5. Log: Anzahl emittierter Events
|
||||
|
||||
**Vorteile**:
|
||||
- Kein Batch-Processing
|
||||
- Events werden einzeln vom normalen Handler verarbeitet
|
||||
- Code-Wiederverwendung (gleicher Handler wie Webhooks)
|
||||
|
||||
---
|
||||
|
||||
## 🔄 Sync-Flows
|
||||
|
||||
### Flow A: EspoCRM Create/Update → Advoware (Webhook)
|
||||
|
||||
```
|
||||
User ändert in EspoCRM
|
||||
↓
|
||||
EspoCRM Webhook → /vmh/webhook/beteiligte/update
|
||||
↓
|
||||
beteiligte_update_api_step.py → Emit 'vmh.beteiligte.update'
|
||||
↓
|
||||
beteiligte_sync_event_step.py → handler()
|
||||
↓
|
||||
Acquire Lock → Fetch EspoCRM → Timestamp-Check
|
||||
↓
|
||||
Update Advoware (oder Konflikt → EspoCRM wins)
|
||||
↓
|
||||
Release Lock → Status: clean
|
||||
```
|
||||
|
||||
**Timing**: 2-5 Sekunden
|
||||
|
||||
---
|
||||
|
||||
### Flow B: Advoware → EspoCRM Check (Cron)
|
||||
|
||||
```
|
||||
Cron (alle 15 Min)
|
||||
↓
|
||||
beteiligte_sync_cron_step.py
|
||||
↓
|
||||
Query EspoCRM: Unclean + Stale Entities
|
||||
↓
|
||||
Emit 'vmh.beteiligte.sync_check' für jeden
|
||||
↓
|
||||
beteiligte_sync_event_step.py → handler()
|
||||
↓
|
||||
GLEICHE Logik wie Flow A!
|
||||
```
|
||||
|
||||
**Timing**: Alle 15 Minuten
|
||||
|
||||
---
|
||||
|
||||
## 🎯 Status-Übergänge
|
||||
|
||||
```
|
||||
pending_sync → syncing → clean (Create erfolgreich)
|
||||
pending_sync → syncing → failed (Create fehlgeschlagen)
|
||||
|
||||
clean → dirty → syncing → clean (Update erfolgreich)
|
||||
clean → syncing → conflict → clean (Konflikt → EspoCRM wins)
|
||||
|
||||
dirty → syncing → deleted_in_advoware (404 von Advoware)
|
||||
|
||||
failed → syncing → clean (Retry erfolgreich)
|
||||
failed → syncing → failed (Retry fehlgeschlagen, retryCount++)
|
||||
|
||||
deleted_in_advoware (Soft-Delete, bleibt bis manuelle Aktion)
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## 📋 Testing Checklist
|
||||
|
||||
### Unit Tests
|
||||
- [x] Mapper Import
|
||||
- [x] Sync Utils Import
|
||||
- [x] Event Step Config Load
|
||||
- [x] Cron Step Config Load
|
||||
- [x] Mapper Transform Person
|
||||
- [x] Mapper Transform Firma
|
||||
|
||||
### Integration Tests (TODO)
|
||||
- [ ] Create: Neuer Beteiligter in EspoCRM → Advoware
|
||||
- [ ] Update: Änderung in EspoCRM → Advoware
|
||||
- [ ] Conflict: Beide geändert → EspoCRM wins
|
||||
- [ ] Advoware newer: Advoware → EspoCRM
|
||||
- [ ] 404 Handling: Soft-Delete + Notification
|
||||
- [ ] Cron: Query + Event Emission
|
||||
- [ ] Notification: In-App Notification sichtbar
|
||||
|
||||
---
|
||||
|
||||
## 🚀 Deployment
|
||||
|
||||
### Voraussetzungen
|
||||
✅ EspoCRM Felder angelegt (syncStatus, betnr, advowareLastSync, advowareDeletedAt, syncErrorMessage, syncRetryCount)
|
||||
✅ Webhooks aktiviert (Create/Update/Delete)
|
||||
⏳ Motia Workbench Restart (damit Steps geladen werden)
|
||||
|
||||
### Schritte
|
||||
1. **Motia Restart**: `systemctl restart motia` (oder wie auch immer)
|
||||
2. **Verify Steps**:
|
||||
```bash
|
||||
# Check ob Steps geladen wurden
|
||||
curl http://localhost:PORT/api/flows/vmh/steps
|
||||
```
|
||||
3. **Test Webhook**: Ändere einen Beteiligten in EspoCRM
|
||||
4. **Check Logs**: Motia Workbench Logs → Event Handler Output
|
||||
5. **Verify Advoware**: Prüfe ob betNr gesetzt wurde
|
||||
6. **Test Cron**: Warte 15 Min oder trigger manuell
|
||||
|
||||
---
|
||||
|
||||
## 🔧 Configuration
|
||||
|
||||
### Environment Variables (bereits gesetzt)
|
||||
```bash
|
||||
# EspoCRM
|
||||
ESPOCRM_API_BASE_URL=https://crm.bitbylaw.com/api/v1
|
||||
ESPOCRM_MARVIN_API_KEY=e53def10eea27b92a6cd00f40a3e09a4
|
||||
|
||||
# Advoware
|
||||
ADVOWARE_API_BASE_URL=https://www2.advo-net.net:90
|
||||
ADVOWARE_PRODUCT_ID=...
|
||||
ADVOWARE_APP_ID=...
|
||||
ADVOWARE_API_KEY=...
|
||||
|
||||
# Redis
|
||||
REDIS_HOST=localhost
|
||||
REDIS_PORT=6379
|
||||
REDIS_DB_ADVOWARE_CACHE=1
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## 📊 Monitoring
|
||||
|
||||
### EspoCRM Queries
|
||||
|
||||
**Entities die Sync benötigen**:
|
||||
```javascript
|
||||
GET /api/v1/CBeteiligte?where=[
|
||||
{type: 'in', attribute: 'syncStatus',
|
||||
value: ['pending_sync', 'dirty', 'failed']}
|
||||
]
|
||||
```
|
||||
|
||||
**Konflikte**:
|
||||
```javascript
|
||||
GET /api/v1/CBeteiligte?where=[
|
||||
{type: 'equals', attribute: 'syncStatus', value: 'conflict'}
|
||||
]
|
||||
```
|
||||
|
||||
**Soft-Deletes**:
|
||||
```javascript
|
||||
GET /api/v1/CBeteiligte?where=[
|
||||
{type: 'equals', attribute: 'syncStatus', value: 'deleted_in_advoware'}
|
||||
]
|
||||
```
|
||||
|
||||
**Sync-Fehler**:
|
||||
```javascript
|
||||
GET /api/v1/CBeteiligte?where=[
|
||||
{type: 'isNotNull', attribute: 'syncErrorMessage'}
|
||||
]
|
||||
```
|
||||
|
||||
### Motia Logs
|
||||
```bash
|
||||
# Event Handler Logs
|
||||
tail -f /path/to/motia/logs/events.log | grep "Beteiligte"
|
||||
|
||||
# Cron Logs
|
||||
tail -f /path/to/motia/logs/cron.log | grep "Sync Cron"
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## 🐛 Troubleshooting
|
||||
|
||||
### Problem: Lock bleibt auf "syncing" hängen
|
||||
**Ursache**: Handler-Crash während Sync
|
||||
**Lösung**: Manuell Status auf "failed" setzen:
|
||||
```python
|
||||
PUT /api/v1/CBeteiligte/{id}
|
||||
{"syncStatus": "failed", "syncErrorMessage": "Manual reset"}
|
||||
```
|
||||
|
||||
### Problem: Notifications werden nicht angezeigt
|
||||
**Ursache**: userId fehlt oder falsch
|
||||
**Check**:
|
||||
```python
|
||||
GET /api/v1/Notification?where=[{type: 'equals', attribute: 'relatedType', value: 'CBeteiligte'}]
|
||||
```
|
||||
|
||||
### Problem: Cron emittiert keine Events
|
||||
**Ursache**: Query findet keine Entities
|
||||
**Debug**: Führe Cron-Handler manuell aus und checke Logs
|
||||
|
||||
---
|
||||
|
||||
## 📈 Performance
|
||||
|
||||
**Erwartete Last**:
|
||||
- Webhooks: ~10-50 pro Tag (User-Änderungen)
|
||||
- Cron: Alle 15 Min → ~96 Runs/Tag
|
||||
- Events pro Cron: 0-100 (typisch 5-20)
|
||||
|
||||
**Optimization**:
|
||||
- Cron Max Entities: 150 total (100 unclean + 50 stale)
|
||||
- Event-Processing: Parallel (Motia-Standard)
|
||||
- Redis Caching: Token + Deduplication
|
||||
|
||||
---
|
||||
|
||||
## ✅ Done!
|
||||
|
||||
**Implementiert**: 4 Module, ~800 Lines of Code
|
||||
**Status**: Ready for Testing
|
||||
**Next Steps**: Deploy + Integration Testing + Monitoring Setup
|
||||
|
||||
🎉 **Viel Erfolg beim Testing!**
|
||||
676
bitbylaw/SYNC_STRATEGY_ANALYSIS.md
Normal file
676
bitbylaw/SYNC_STRATEGY_ANALYSIS.md
Normal file
@@ -0,0 +1,676 @@
|
||||
# Sync-Strategie: EspoCRM ↔ Advoware Beteiligte
|
||||
|
||||
**Analysiert am**: 2026-02-07
|
||||
**Anforderungen**:
|
||||
- a) EspoCRM Update → Advoware Update
|
||||
- b) Bi-direktionaler Sync mit Konfliktauflösung
|
||||
- c) Neue Beteiligte in EspoCRM → Neue in Advoware (Status-Feld)
|
||||
- d) Cron-basierter Sync
|
||||
|
||||
**Problem**: EspoCRM hat Webhooks, Advoware nicht.
|
||||
|
||||
---
|
||||
|
||||
## 📊 Bestehende Architektur (aus Calendar Sync)
|
||||
|
||||
Das bestehende **Calendar Sync System** bietet ein exzellentes Template:
|
||||
|
||||
### Aktuelle Komponenten:
|
||||
1. **Webhook-Receiver**: `beteiligte_create/update/delete_api_step.py` ✓ Bereits vorhanden
|
||||
2. **Event-Handler**: `beteiligte_sync_event_step.py` ⚠️ Placeholder
|
||||
3. **Cron-Job**: Analog zu `calendar_sync_cron_step.py`
|
||||
4. **PostgreSQL State DB**: Für Sync-State und Konfliktauflösung
|
||||
5. **Redis**: Deduplication + Rate Limiting
|
||||
|
||||
### Bewährte Patterns aus Calendar Sync:
|
||||
|
||||
**1. Datenbank-Schema** (PostgreSQL):
|
||||
```sql
|
||||
CREATE TABLE beteiligte_sync (
|
||||
sync_id SERIAL PRIMARY KEY,
|
||||
employee_kuerzel VARCHAR(10),
|
||||
|
||||
-- IDs beider Systeme
|
||||
espocrm_id VARCHAR(255) UNIQUE,
|
||||
advoware_betnr INTEGER UNIQUE,
|
||||
|
||||
-- Metadaten
|
||||
source_system VARCHAR(20), -- 'espocrm' oder 'advoware'
|
||||
sync_strategy VARCHAR(50) DEFAULT 'source_system_wins',
|
||||
sync_status VARCHAR(20) DEFAULT 'synced', -- 'synced', 'pending', 'failed', 'conflict'
|
||||
|
||||
-- Timestamps
|
||||
espocrm_modified_at TIMESTAMP WITH TIME ZONE,
|
||||
advoware_modified_at TIMESTAMP WITH TIME ZONE,
|
||||
last_sync TIMESTAMP WITH TIME ZONE DEFAULT NOW(),
|
||||
|
||||
-- Flags
|
||||
deleted BOOLEAN DEFAULT FALSE,
|
||||
advoware_write_allowed BOOLEAN DEFAULT TRUE,
|
||||
|
||||
-- Audit
|
||||
created_at TIMESTAMP WITH TIME ZONE DEFAULT NOW(),
|
||||
updated_at TIMESTAMP WITH TIME ZONE DEFAULT NOW()
|
||||
);
|
||||
|
||||
CREATE INDEX idx_espocrm_id ON beteiligte_sync(espocrm_id);
|
||||
CREATE INDEX idx_advoware_betnr ON beteiligte_sync(advoware_betnr);
|
||||
CREATE INDEX idx_sync_status ON beteiligte_sync(sync_status);
|
||||
CREATE INDEX idx_deleted ON beteiligte_sync(deleted) WHERE NOT deleted;
|
||||
```
|
||||
|
||||
**2. Sync-Phasen** (3-Phasen-Modell):
|
||||
- **Phase 1**: Neue aus EspoCRM → Advoware
|
||||
- **Phase 2**: Neue aus Advoware → EspoCRM
|
||||
- **Phase 3**: Updates + Konfliktauflösung
|
||||
- **Phase 4**: Deletes
|
||||
|
||||
**3. Konfliktauflösung via Timestamps**:
|
||||
```python
|
||||
# Aus calendar_sync_event_step.py, Zeile 870+
|
||||
if espo_ts and advo_ts:
|
||||
if espo_ts > advo_ts:
|
||||
# EspoCRM ist neuer → Update Advoware
|
||||
await update_advoware(...)
|
||||
elif advo_ts > espo_ts:
|
||||
# Advoware ist neuer → Update EspoCRM
|
||||
await update_espocrm(...)
|
||||
else:
|
||||
# Gleich alt → Skip
|
||||
pass
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## 🎯 Empfohlene Architektur für Beteiligte-Sync
|
||||
|
||||
### Komponenten-Übersicht
|
||||
|
||||
```
|
||||
┌─────────────────────────────────────────────────────────────────┐
|
||||
│ EspoCRM │
|
||||
│ │
|
||||
│ CBeteiligte Entity │
|
||||
│ - Webhooks: create/update/delete │
|
||||
│ - Felder: betnr, syncStatus, advowareLastSync │
|
||||
└─────────┬────────────────────────────────────────────────────────┘
|
||||
│ Webhook (HTTP POST)
|
||||
▼
|
||||
┌─────────────────────────────────────────────────────────────────┐
|
||||
│ KONG API Gateway → Motia Framework │
|
||||
│ │
|
||||
│ 1. beteiligte_create/update/delete_api_step.py │
|
||||
│ - Empfängt Webhooks │
|
||||
│ - Dedupliziert via Redis │
|
||||
│ - Emittiert Events │
|
||||
│ │
|
||||
│ 2. beteiligte_sync_event_step.py │
|
||||
│ - Subscribed zu Events │
|
||||
│ - Holt vollständige Entity aus EspoCRM │
|
||||
│ - Transformiert via Mapper │
|
||||
│ - Schreibt nach Advoware │
|
||||
│ - Updated PostgreSQL Sync-State │
|
||||
│ │
|
||||
│ 3. beteiligte_sync_cron_step.py (NEU) │
|
||||
│ - Läuft alle 15 Minuten │
|
||||
│ - Emittiert "beteiligte.sync_all" Event │
|
||||
│ │
|
||||
│ 4. beteiligte_sync_all_event_step.py (NEU) │
|
||||
│ - Fetcht alle Beteiligte aus Advoware │
|
||||
│ - Vergleicht mit PostgreSQL State │
|
||||
│ - Identifiziert Neue/Geänderte/Gelöschte in Advoware │
|
||||
│ - Sync nach EspoCRM │
|
||||
│ - 3-Phasen-Modell wie Calendar Sync │
|
||||
└─────────┬────────────────────────────────────────────────────────┘
|
||||
│
|
||||
▼
|
||||
┌─────────────────────────────────────────────────────────────────┐
|
||||
│ PostgreSQL Sync-State DB │
|
||||
│ │
|
||||
│ Tabelle: beteiligte_sync │
|
||||
│ - Mapping: espocrm_id ↔ advoware_betnr │
|
||||
│ - Timestamps beider Systeme │
|
||||
│ - Sync-Status & Konfliktflags │
|
||||
└─────────┬────────────────────────────────────────────────────────┘
|
||||
│
|
||||
▼
|
||||
┌─────────────────────────────────────────────────────────────────┐
|
||||
│ Redis (Caching & Deduplication) │
|
||||
│ │
|
||||
│ - vmh:beteiligte:create_pending (SET) │
|
||||
│ - vmh:beteiligte:update_pending (SET) │
|
||||
│ - vmh:beteiligte:delete_pending (SET) │
|
||||
│ - vmh:beteiligte:sync_lock:{espocrm_id} (Key mit TTL) │
|
||||
└──────────────────────────────────────────────────────────────────┘
|
||||
│
|
||||
▼
|
||||
┌─────────────────────────────────────────────────────────────────┐
|
||||
│ Advoware API │
|
||||
│ │
|
||||
│ /api/v1/advonet/Beteiligte │
|
||||
│ - GET: Liste + Einzelabfrage │
|
||||
│ - POST: Create │
|
||||
│ - PUT: Update │
|
||||
│ - DELETE: Delete │
|
||||
│ │
|
||||
│ ⚠️ KEIN Webhook-Support │
|
||||
│ → Polling via Cron erforderlich │
|
||||
└──────────────────────────────────────────────────────────────────┘
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## 🔄 Detaillierte Sync-Flows
|
||||
|
||||
### Flow A: EspoCRM Update → Advoware (Webhook-getrieben)
|
||||
|
||||
**Trigger**: EspoCRM sendet Webhook bei Create/Update/Delete
|
||||
|
||||
```
|
||||
1. EspoCRM: User ändert CBeteiligte Entity
|
||||
└─> Webhook: POST /vmh/webhook/beteiligte/update
|
||||
Body: [{"id": "68e4af00172be7924", ...}]
|
||||
|
||||
2. beteiligte_update_api_step.py:
|
||||
├─> Deduplizierung via Redis SET
|
||||
├─> Neue IDs → Redis: vmh:beteiligte:update_pending
|
||||
└─> Emit Event: "vmh.beteiligte.update"
|
||||
|
||||
3. beteiligte_sync_event_step.py:
|
||||
├─> Receive Event mit entity_id
|
||||
├─> Fetch full entity von EspoCRM API
|
||||
│ GET /api/v1/CBeteiligte/{entity_id}
|
||||
│
|
||||
├─> Check PostgreSQL Sync-State:
|
||||
│ SELECT * FROM beteiligte_sync WHERE espocrm_id = ?
|
||||
│
|
||||
├─> Falls NEU (nicht in DB):
|
||||
│ ├─> Check syncStatus in EspoCRM:
|
||||
│ │ - "pending_sync" → Create in Advoware
|
||||
│ │ - "clean" → Skip (bereits gesynct von anderem Flow)
|
||||
│ │
|
||||
│ ├─> Transform via Mapper:
|
||||
│ │ espocrm_mapper.map_cbeteiligte_to_advoware(entity)
|
||||
│ │
|
||||
│ ├─> POST /api/v1/advonet/Beteiligte
|
||||
│ │ → Response: {betNr: 123456, ...}
|
||||
│ │
|
||||
│ ├─> Insert in PostgreSQL:
|
||||
│ │ INSERT INTO beteiligte_sync (
|
||||
│ │ espocrm_id, advoware_betnr,
|
||||
│ │ source_system = 'espocrm',
|
||||
│ │ espocrm_modified_at = entity.modifiedAt
|
||||
│ │ )
|
||||
│ │
|
||||
│ └─> Update EspoCRM:
|
||||
│ PUT /api/v1/CBeteiligte/{entity_id}
|
||||
│ {
|
||||
│ betnr: 123456,
|
||||
│ syncStatus: "clean",
|
||||
│ advowareLastSync: NOW()
|
||||
│ }
|
||||
│
|
||||
└─> Falls EXISTIERT (in DB):
|
||||
├─> Get Advoware timestamp:
|
||||
│ Fetch /api/v1/advonet/Beteiligte/{betnr}
|
||||
│ → advoware.geaendertAm
|
||||
│
|
||||
├─> Konfliktauflösung:
|
||||
│ IF espocrm.modifiedAt > advoware.geaendertAm:
|
||||
│ → Update Advoware (EspoCRM gewinnt)
|
||||
│ PUT /api/v1/advonet/Beteiligte/{betnr}
|
||||
│ ELSE IF advoware.geaendertAm > espocrm.modifiedAt:
|
||||
│ → Update EspoCRM (Advoware gewinnt)
|
||||
│ PUT /api/v1/CBeteiligte/{entity_id}
|
||||
│ ELSE:
|
||||
│ → Skip (gleich alt)
|
||||
│
|
||||
└─> Update PostgreSQL:
|
||||
UPDATE beteiligte_sync SET
|
||||
espocrm_modified_at = ...,
|
||||
advoware_modified_at = ...,
|
||||
last_sync = NOW(),
|
||||
sync_status = 'synced'
|
||||
|
||||
4. Cleanup:
|
||||
└─> Redis: SREM vmh:beteiligte:update_pending {entity_id}
|
||||
```
|
||||
|
||||
**Timing**: ~2-5 Sekunden nach Änderung in EspoCRM
|
||||
|
||||
---
|
||||
|
||||
### Flow B: Advoware Update → EspoCRM (Polling via Cron)
|
||||
|
||||
**Trigger**: Cron-Job alle 15 Minuten
|
||||
|
||||
```
|
||||
1. beteiligte_sync_cron_step.py (Cron: */15 * * * *):
|
||||
└─> Emit Event: "beteiligte.sync_all"
|
||||
|
||||
2. beteiligte_sync_all_event_step.py:
|
||||
├─> Fetch alle Beteiligte aus PostgreSQL Sync-DB:
|
||||
│ SELECT * FROM beteiligte_sync WHERE NOT deleted
|
||||
│
|
||||
├─> Fetch alle Beteiligte aus Advoware:
|
||||
│ GET /api/v1/advonet/Beteiligte
|
||||
│ (Optional mit Filter: nur geändert seit last_sync - 1 Tag)
|
||||
│
|
||||
├─> Build Maps:
|
||||
│ db_map = {betnr: row}
|
||||
│ advo_map = {betNr: entity}
|
||||
│
|
||||
├─> PHASE 1: Neue in Advoware (nicht in DB):
|
||||
│ FOR betnr IN advo_map:
|
||||
│ IF betnr NOT IN db_map:
|
||||
│ ├─> Transform: map_advoware_to_cbeteiligte(advo_entity)
|
||||
│ │
|
||||
│ ├─> Check if exists in EspoCRM:
|
||||
│ │ Search by name/email (Fuzzy Match)
|
||||
│ │
|
||||
│ ├─> IF NOT EXISTS:
|
||||
│ │ ├─> POST /api/v1/CBeteiligte
|
||||
│ │ │ {
|
||||
│ │ │ ...fields...,
|
||||
│ │ │ betnr: {betnr},
|
||||
│ │ │ syncStatus: "clean",
|
||||
│ │ │ advowareLastSync: NOW()
|
||||
│ │ │ }
|
||||
│ │ │
|
||||
│ │ └─> INSERT INTO beteiligte_sync (
|
||||
│ │ espocrm_id = new_id,
|
||||
│ │ advoware_betnr = betnr,
|
||||
│ │ source_system = 'advoware'
|
||||
│ │ )
|
||||
│ │
|
||||
│ └─> ELSE (Match gefunden):
|
||||
│ └─> UPDATE beteiligte_sync SET
|
||||
│ advoware_betnr = betnr,
|
||||
│ source_system = 'merged'
|
||||
│
|
||||
├─> PHASE 2: Updates (beide vorhanden):
|
||||
│ FOR row IN db_map:
|
||||
│ IF row.advoware_betnr IN advo_map:
|
||||
│ advo_entity = advo_map[row.advoware_betnr]
|
||||
│ espo_entity = fetch_from_espocrm(row.espocrm_id)
|
||||
│
|
||||
│ ├─> Get Timestamps:
|
||||
│ │ advo_ts = advo_entity.geaendertAm
|
||||
│ │ espo_ts = espo_entity.modifiedAt
|
||||
│ │ last_sync_ts = row.last_sync
|
||||
│ │
|
||||
│ ├─> Konfliktauflösung:
|
||||
│ │ IF advo_ts > espo_ts AND advo_ts > last_sync_ts:
|
||||
│ │ → Advoware ist neuer
|
||||
│ │ PUT /api/v1/CBeteiligte/{espocrm_id}
|
||||
│ │ (Update EspoCRM mit Advoware-Daten)
|
||||
│ │
|
||||
│ │ ELSE IF espo_ts > advo_ts AND espo_ts > last_sync_ts:
|
||||
│ │ → EspoCRM ist neuer
|
||||
│ │ (Wurde bereits in Flow A behandelt, skip)
|
||||
│ │
|
||||
│ │ ELSE IF advo_ts == espo_ts:
|
||||
│ │ → Keine Änderung, skip
|
||||
│ │
|
||||
│ │ ELSE IF advo_ts > last_sync_ts AND espo_ts > last_sync_ts:
|
||||
│ │ → KONFLIKT: Beide seit last_sync geändert
|
||||
│ │ ├─> Strategy: "advoware_wins" (konfigurierbar)
|
||||
│ │ ├─> UPDATE mit Winner-Daten
|
||||
│ │ ├─> Log Conflict
|
||||
│ │ └─> SET sync_status = 'conflict_resolved'
|
||||
│ │
|
||||
│ └─> UPDATE beteiligte_sync SET
|
||||
│ espocrm_modified_at = espo_ts,
|
||||
│ advoware_modified_at = advo_ts,
|
||||
│ last_sync = NOW(),
|
||||
│ sync_status = 'synced'
|
||||
│
|
||||
└─> PHASE 3: Deletes (in DB, nicht in Advoware):
|
||||
FOR row IN db_map:
|
||||
IF row.advoware_betnr NOT IN advo_map:
|
||||
├─> Check if exists in EspoCRM:
|
||||
│ GET /api/v1/CBeteiligte/{espocrm_id}
|
||||
│
|
||||
├─> IF EXISTS:
|
||||
│ ├─> Soft-Delete in EspoCRM:
|
||||
│ │ PUT /api/v1/CBeteiligte/{espocrm_id}
|
||||
│ │ {deleted: true, syncStatus: "deleted"}
|
||||
│ │
|
||||
│ └─> UPDATE beteiligte_sync SET
|
||||
│ deleted = TRUE,
|
||||
│ sync_status = 'synced'
|
||||
│
|
||||
└─> ELSE (auch in EspoCRM nicht da):
|
||||
└─> UPDATE beteiligte_sync SET
|
||||
deleted = TRUE
|
||||
```
|
||||
|
||||
**Timing**: Alle 15 Minuten (konfigurierbar)
|
||||
|
||||
---
|
||||
|
||||
## 🎛️ Status-Feld in EspoCRM
|
||||
|
||||
### Feld: `syncStatus` (String, Custom Field)
|
||||
|
||||
**Werte**:
|
||||
- `"pending_sync"` → Neu erstellt, wartet auf Sync nach Advoware
|
||||
- `"clean"` → Synchronisiert, keine Änderungen
|
||||
- `"dirty"` → Geändert seit letztem Sync, wartet auf Sync
|
||||
- `"syncing"` → Sync läuft gerade
|
||||
- `"failed"` → Sync fehlgeschlagen (+ Fehlerlog)
|
||||
- `"conflict"` → Konflikt detektiert
|
||||
- `"deleted"` → In Advoware gelöscht
|
||||
|
||||
**Zusatzfeld**: `advowareLastSync` (DateTime)
|
||||
|
||||
### Alternative: PostgreSQL als Single Source of Truth
|
||||
|
||||
Statt `syncStatus` in EspoCRM:
|
||||
- Alle Status in PostgreSQL `beteiligte_sync.sync_status`
|
||||
- EspoCRM hat nur `betnr` und `advowareLastSync`
|
||||
- **Vorteil**: Keine Schema-Änderung in EspoCRM nötig
|
||||
- **Nachteil**: Status nicht direkt in EspoCRM UI sichtbar
|
||||
|
||||
**Empfehlung**: Beides nutzen
|
||||
- PostgreSQL: Master-Status für Sync-Logic
|
||||
- EspoCRM: Read-only Display für User
|
||||
|
||||
---
|
||||
|
||||
## ⚙️ Cron-Job Konfiguration
|
||||
|
||||
### Option 1: Separate Cron-Steps (empfohlen)
|
||||
|
||||
```python
|
||||
# beteiligte_sync_cron_step.py
|
||||
config = {
|
||||
'type': 'cron',
|
||||
'name': 'Beteiligte Sync Cron',
|
||||
'cron': '*/15 * * * *', # Alle 15 Minuten
|
||||
'emits': ['beteiligte.sync_all'],
|
||||
'flows': ['vmh']
|
||||
}
|
||||
```
|
||||
|
||||
### Option 2: Integriert in bestehenden Cron
|
||||
|
||||
```python
|
||||
# Kombiniert mit anderen Sync-Tasks
|
||||
config = {
|
||||
'type': 'cron',
|
||||
'name': 'All Syncs Cron',
|
||||
'cron': '*/5 * * * *', # Alle 5 Minuten
|
||||
'emits': ['calendar_sync_all', 'beteiligte.sync_all'],
|
||||
'flows': ['advoware', 'vmh']
|
||||
}
|
||||
```
|
||||
|
||||
### Timing-Überlegungen:
|
||||
|
||||
**Frequenz**:
|
||||
- **15 Minuten**: Guter Kompromiss (wie bestehende Calendar Sync)
|
||||
- **5 Minuten**: Wenn schnellere Reaktion auf Advoware-Änderungen nötig
|
||||
- **1 Stunde**: Wenn Last auf APIs minimiert werden soll
|
||||
|
||||
**Offset**:
|
||||
- Calendar Sync: 0 Minuten
|
||||
- Beteiligte Sync: +5 Minuten
|
||||
- → Verhindert API-Überlast durch gleichzeitige Requests
|
||||
|
||||
---
|
||||
|
||||
## 🔐 Sicherheit & Fehlerbehandlung
|
||||
|
||||
### 1. Rate Limiting
|
||||
|
||||
**Advoware API** (aus Calendar Sync):
|
||||
```python
|
||||
# Bereits implementiert in AdvowareAPI Service
|
||||
# - Token-basiertes Rate Limiting via Redis
|
||||
# - Backoff-Strategie bei 429 Errors
|
||||
```
|
||||
|
||||
**EspoCRM API**:
|
||||
```python
|
||||
# Zu implementieren in EspoCRMAPI Service
|
||||
ESPOCRM_RATE_LIMIT_KEY = 'espocrm_api_tokens'
|
||||
MAX_TOKENS = 100 # Basierend auf API-Limits
|
||||
REFILL_RATE = 100 / 60 # Tokens pro Sekunde
|
||||
```
|
||||
|
||||
### 2. Lock-Mechanismus (Verhindert Race Conditions)
|
||||
|
||||
```python
|
||||
# Redis Lock für Entity während Sync
|
||||
lock_key = f'vmh:beteiligte:sync_lock:{entity_id}'
|
||||
if redis.set(lock_key, '1', nx=True, ex=300): # 5 Min TTL
|
||||
try:
|
||||
await perform_sync(entity_id)
|
||||
finally:
|
||||
redis.delete(lock_key)
|
||||
else:
|
||||
logger.warning(f"Entity {entity_id} ist bereits im Sync-Prozess")
|
||||
```
|
||||
|
||||
### 3. Retry-Logic mit Exponential Backoff
|
||||
|
||||
```python
|
||||
import backoff
|
||||
|
||||
@backoff.on_exception(
|
||||
backoff.expo,
|
||||
(AdvowareAPIError, EspoCRMAPIError),
|
||||
max_tries=3,
|
||||
max_time=60
|
||||
)
|
||||
async def sync_entity_with_retry(entity_id):
|
||||
# ... Sync-Logic
|
||||
pass
|
||||
```
|
||||
|
||||
### 4. Fehler-Logging & Monitoring
|
||||
|
||||
```python
|
||||
# PostgreSQL Error-Log Tabelle
|
||||
CREATE TABLE beteiligte_sync_errors (
|
||||
error_id SERIAL PRIMARY KEY,
|
||||
sync_id INTEGER REFERENCES beteiligte_sync(sync_id),
|
||||
error_type VARCHAR(50),
|
||||
error_message TEXT,
|
||||
error_stack TEXT,
|
||||
retry_count INTEGER DEFAULT 0,
|
||||
resolved BOOLEAN DEFAULT FALSE,
|
||||
created_at TIMESTAMP WITH TIME ZONE DEFAULT NOW()
|
||||
);
|
||||
```
|
||||
|
||||
### 5. Write-Protection Flag
|
||||
|
||||
**Global** (Config):
|
||||
```python
|
||||
# config.py
|
||||
ADVOWARE_WRITE_PROTECTION = os.getenv('ADVOWARE_WRITE_PROTECTION', 'false').lower() == 'true'
|
||||
```
|
||||
|
||||
**Per-Entity** (DB):
|
||||
```sql
|
||||
ALTER TABLE beteiligte_sync ADD COLUMN advoware_write_allowed BOOLEAN DEFAULT TRUE;
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## 📦 Zu implementierende Module
|
||||
|
||||
### Priorität 1 (Core Sync):
|
||||
|
||||
1. **services/espocrm_mapper.py** ⭐⭐⭐
|
||||
- `map_cbeteiligte_to_advoware(espo_entity) -> advo_data`
|
||||
- `map_advoware_to_cbeteiligte(advo_entity) -> espo_data`
|
||||
- Feld-Mapping gemäß `ENTITY_MAPPING_CBeteiligte_Advoware.md`
|
||||
|
||||
2. **steps/vmh/beteiligte_sync_event_step.py** ⭐⭐⭐
|
||||
- Implementiert Flow A (Webhook → Advoware)
|
||||
- Subscribe to create/update/delete Events
|
||||
- PostgreSQL Integration
|
||||
- Konfliktauflösung
|
||||
|
||||
3. **PostgreSQL Migration** ⭐⭐⭐
|
||||
- `migrations/001_create_beteiligte_sync_table.sql`
|
||||
- Connection in Config
|
||||
|
||||
4. **steps/vmh/beteiligte_sync_cron_step.py** ⭐⭐
|
||||
- Emittiert Sync-All Event alle 15 Min
|
||||
|
||||
5. **steps/vmh/beteiligte_sync_all_event_step.py** ⭐⭐
|
||||
- Implementiert Flow B (Advoware Polling)
|
||||
- 3-Phasen-Sync-Modell
|
||||
|
||||
### Priorität 2 (Optimierungen):
|
||||
|
||||
6. **services/beteiligte_sync_utils.py** ⭐
|
||||
- Shared Utilities
|
||||
- Lock-Management
|
||||
- Timestamp-Handling
|
||||
- Conflict-Resolution Logic
|
||||
|
||||
7. **Testing** ⭐
|
||||
- Unit Tests für Mapper
|
||||
- Integration Tests für Sync-Flows
|
||||
- Konflikt-Szenarien
|
||||
|
||||
### Priorität 3 (Monitoring):
|
||||
|
||||
8. **steps/vmh/audit_beteiligte_sync.py**
|
||||
- Analog zu `audit_calendar_sync.py`
|
||||
- CLI-Tool für Sync-Status
|
||||
|
||||
9. **Dashboard/Metrics**
|
||||
- Prometheus Metrics
|
||||
- Grafana Dashboard
|
||||
|
||||
---
|
||||
|
||||
## 🚀 Rollout-Plan
|
||||
|
||||
### Phase 1: Setup (Tag 1-2)
|
||||
- [ ] PostgreSQL Datenbank-Schema erstellen
|
||||
- [ ] Config erweitern (DB-Connection)
|
||||
- [ ] Mapper-Modul erstellen
|
||||
- [ ] Unit Tests für Mapper
|
||||
|
||||
### Phase 2: Webhook-Flow (Tag 3-4)
|
||||
- [ ] `beteiligte_sync_event_step.py` implementieren
|
||||
- [ ] Integration mit bestehenden Webhook-Steps
|
||||
- [ ] Testing mit EspoCRM Sandbox
|
||||
|
||||
### Phase 3: Polling-Flow (Tag 5-6)
|
||||
- [ ] Cron-Step erstellen
|
||||
- [ ] `beteiligte_sync_all_event_step.py` implementieren
|
||||
- [ ] 3-Phasen-Modell
|
||||
- [ ] Integration Tests
|
||||
|
||||
### Phase 4: Konfliktauflösung (Tag 7)
|
||||
- [ ] Timestamp-Vergleich
|
||||
- [ ] Konflikt-Strategies
|
||||
- [ ] Error-Handling
|
||||
- [ ] Retry-Logic
|
||||
|
||||
### Phase 5: Monitoring & Docs (Tag 8)
|
||||
- [ ] Audit-Tool
|
||||
- [ ] Logging
|
||||
- [ ] Dokumentation
|
||||
- [ ] Runbook
|
||||
|
||||
### Phase 6: Production (Tag 9-10)
|
||||
- [ ] Staging-Tests
|
||||
- [ ] Performance-Tests
|
||||
- [ ] Production-Rollout
|
||||
- [ ] Monitoring
|
||||
|
||||
---
|
||||
|
||||
## ⚠️ Wichtige Entscheidungen
|
||||
|
||||
### 1. Conflict Resolution Strategy
|
||||
|
||||
**Option A: "Source System Wins"** (empfohlen für Start):
|
||||
```python
|
||||
sync_strategy = 'source_system_wins'
|
||||
# EspoCRM-created Entities → EspoCRM gewinnt bei Konflikt
|
||||
# Advoware-created Entities → Advoware gewinnt bei Konflikt
|
||||
```
|
||||
|
||||
**Option B: "Advoware Always Wins"**:
|
||||
```python
|
||||
sync_strategy = 'advoware_master'
|
||||
# Advoware ist Master, EspoCRM ist read-only View
|
||||
```
|
||||
|
||||
**Option C: "Last Modified Wins"**:
|
||||
```python
|
||||
sync_strategy = 'last_modified_wins'
|
||||
# Timestamp-Vergleich, neuester gewinnt
|
||||
```
|
||||
|
||||
**Empfehlung**: Start mit "Source System Wins", später auf "Last Modified Wins" mit Manual Conflict Review.
|
||||
|
||||
### 2. Advoware Polling-Frequenz
|
||||
|
||||
**Überlegungen**:
|
||||
- API-Last auf Advoware
|
||||
- Aktualitäts-Anforderungen
|
||||
- Anzahl Beteiligte (~1000? ~10.000?)
|
||||
|
||||
**Optimierung**:
|
||||
- Incremental Fetch: Nur geändert seit `last_sync - 1 Tag`
|
||||
- Delta-Detection via `geaendertAm` Timestamp
|
||||
- Pagination bei großen Datenmengen
|
||||
|
||||
### 3. EspoCRM Field: `syncStatus`
|
||||
|
||||
**Zu klären**:
|
||||
- Bestehendes Custom Field oder neu anlegen?
|
||||
- Dropdown-Werte konfigurieren
|
||||
- Permissions (nur Sync-System kann schreiben?)
|
||||
|
||||
---
|
||||
|
||||
## 📝 Zusammenfassung
|
||||
|
||||
### Was funktioniert:
|
||||
✅ EspoCRM Webhooks → Motia Events ✅ Webhook-Deduplication via Redis
|
||||
✅ EspoCRM API Client ✅ Advoware API Client
|
||||
✅ Entity-Mapping definiert
|
||||
|
||||
### Was zu implementieren ist:
|
||||
🔨 Mapper-Modul
|
||||
🔨 PostgreSQL Sync-State DB
|
||||
🔨 Event-Handler für Webhooks
|
||||
🔨 Cron-Job für Polling
|
||||
🔨 3-Phasen-Sync für Advoware → EspoCRM
|
||||
🔨 Konfliktauflösung
|
||||
🔨 Error-Handling & Monitoring
|
||||
|
||||
### Geschätzter Aufwand:
|
||||
- **Setup & Core**: 3-4 Tage
|
||||
- **Flows**: 2-3 Tage
|
||||
- **Konfliktauflösung**: 1-2 Tage
|
||||
- **Testing & Docs**: 1-2 Tage
|
||||
- **Rollout**: 1-2 Tage
|
||||
- **Total**: ~8-13 Tage
|
||||
|
||||
### Nächster Schritt:
|
||||
1. Entscheidung zu Status-Feld in EspoCRM
|
||||
2. PostgreSQL DB-Schema aufsetzen
|
||||
3. Mapper-Modul implementieren
|
||||
4. Webhook-Flow komplettieren
|
||||
|
||||
---
|
||||
|
||||
**Fragen zur Klärung**:
|
||||
1. Existiert `syncStatus` Field bereits in EspoCRM CBeteiligte?
|
||||
2. Wie viele Beteiligte gibt es ca. in Advoware?
|
||||
3. Gibt es Performance-Anforderungen? (z.B. Sync innerhalb X Sekunden)
|
||||
4. Soll es manuelle Conflict-Resolution geben oder automatisch?
|
||||
5. PostgreSQL Server bereits vorhanden? (Wie Calendar Sync)
|
||||
485
bitbylaw/SYNC_STRATEGY_ESPOCRM_BASED.md
Normal file
485
bitbylaw/SYNC_STRATEGY_ESPOCRM_BASED.md
Normal file
@@ -0,0 +1,485 @@
|
||||
# Sync-Strategie: EspoCRM-basiert (ohne PostgreSQL)
|
||||
|
||||
**Analysiert am**: 2026-02-07
|
||||
**Anpassung**: EspoCRM als primäre State-Datenbank
|
||||
|
||||
---
|
||||
|
||||
## 🎯 EspoCRM Felder (CBeteiligte Entity)
|
||||
|
||||
```json
|
||||
{
|
||||
"betnr": 1234, // Link zu Advoware betNr (int, unique)
|
||||
"syncStatus": "clean", // Sync-Status (enum)
|
||||
"advowareLastSync": null, // Letzter Sync (datetime oder null)
|
||||
"advowareDeletedAt": null, // Gelöscht in Advoware am (datetime, NEU)
|
||||
"syncErrorMessage": null, // Fehlerdetails (text, NEU)
|
||||
"syncRetryCount": 0, // Anzahl Retry-Versuche (int, NEU)
|
||||
"modifiedAt": "2026-01-23 21:58:41" // EspoCRM Änderungszeit
|
||||
}
|
||||
```
|
||||
|
||||
### syncStatus-Werte (Enum in EspoCRM):
|
||||
- `"pending_sync"` - Neu erstellt, noch nicht nach Advoware gesynct
|
||||
- `"clean"` - Synchronisiert, keine ausstehenden Änderungen
|
||||
- `"dirty"` - In EspoCRM geändert, wartet auf Sync nach Advoware
|
||||
- `"syncing"` - Sync läuft gerade (verhindert Race Conditions)
|
||||
- `"failed"` - Sync fehlgeschlagen (mit syncErrorMessage + syncRetryCount)
|
||||
- `"conflict"` - Konflikt erkannt → **EspoCRM WINS** (mit Notification)
|
||||
- `"deleted_in_advoware"` - In Advoware gelöscht (Soft-Delete Flag mit Notification)
|
||||
|
||||
---
|
||||
|
||||
## 🔄 Flow A: EspoCRM Update → Advoware (Webhook)
|
||||
|
||||
**Trigger**: EspoCRM Webhook bei Create/Update
|
||||
|
||||
```
|
||||
1. EspoCRM: User ändert CBeteiligte
|
||||
└─> Webhook: POST /vmh/webhook/beteiligte/update
|
||||
Body: [{"id": "68e4af00172be7924"}]
|
||||
|
||||
2. beteiligte_update_api_step.py:
|
||||
├─> Redis Deduplication
|
||||
└─> Emit Event: "vmh.beteiligte.update"
|
||||
|
||||
3. beteiligte_sync_event_step.py:
|
||||
├─> Fetch Entity von EspoCRM:
|
||||
│ GET /api/v1/CBeteiligte/{id}
|
||||
│ {
|
||||
│ "id": "...",
|
||||
│ "firstName": "Angela",
|
||||
│ "lastName": "Mustermann",
|
||||
│ "betnr": 104860, // Bereits vorhanden
|
||||
│ "syncStatus": "clean",
|
||||
│ "advowareLastSync": "2026-02-01T10:00:00",
|
||||
│ "modifiedAt": "2026-02-07T14:30:00"
|
||||
│ }
|
||||
│
|
||||
├─> Check syncStatus:
|
||||
│ ├─> IF syncStatus == "syncing":
|
||||
│ │ → Skip (bereits im Sync-Prozess)
|
||||
│ │
|
||||
│ ├─> IF syncStatus == "pending_sync" AND betnr == NULL:
|
||||
│ │ → NEU: Create in Advoware
|
||||
│ │ ├─> Set syncStatus = "syncing"
|
||||
│ │ ├─> Transform via Mapper
|
||||
│ │ ├─> POST /api/v1/advonet/Beteiligte
|
||||
│ │ │ Response: {betNr: 123456}
|
||||
│ │ └─> Update EspoCRM:
|
||||
│ │ PUT /api/v1/CBeteiligte/{id}
|
||||
│ │ {
|
||||
│ │ betnr: 123456,
|
||||
│ │ syncStatus: "clean",
|
||||
│ │ advowareLastSync: NOW()
|
||||
│ │ }
|
||||
│ │
|
||||
│ └─> IF betnr != NULL (bereits gesynct):
|
||||
│ → UPDATE: Vergleiche Timestamps
|
||||
│ ├─> Fetch von Advoware:
|
||||
│ │ GET /api/v1/advonet/Beteiligte/{betnr}
|
||||
│ │ {betNr: 104860, geaendertAm: "2026-02-07T12:00:00"}
|
||||
│ │
|
||||
│ ├─> Vergleiche Timestamps:
|
||||
│ │ espocrm_ts = entity.modifiedAt
|
||||
│ │ advoware_ts = advo_entity.geaendertAm
|
||||
│ │ last_sync_ts = entity.advowareLastSync
|
||||
│ │
|
||||
│ │ IF espocrm_ts > last_sync_ts AND espocrm_ts > advoware_ts:
|
||||
│ │ → EspoCRM ist neuer → Update Advoware
|
||||
│ │ ├─> Set syncStatus = "syncing"
|
||||
│ │ ├─> PUT /api/v1/advonet/Beteiligte/{betnr}
|
||||
│ │ └─> Update EspoCRM:
|
||||
│ │ syncStatus = "clean"
|
||||
│ │ advowareLastSync = NOW()
|
||||
│ │ syncErrorMessage = NULL
|
||||
│ │ syncRetryCount = 0
|
||||
│ │
|
||||
│ │ ELSE IF advoware_ts > last_sync_ts AND advoware_ts > espocrm_ts:
|
||||
│ │ → Advoware ist neuer → Update EspoCRM
|
||||
│ │ ├─> Set syncStatus = "syncing"
|
||||
│ │ ├─> Transform von Advoware
|
||||
│ │ └─> Update EspoCRM mit Advoware-Daten
|
||||
│ │ syncStatus = "clean"
|
||||
│ │ advowareLastSync = NOW()
|
||||
│ │ syncErrorMessage = NULL
|
||||
│ │ syncRetryCount = 0
|
||||
│ │
|
||||
│ │ ELSE IF espocrm_ts > last_sync_ts AND advoware_ts > last_sync_ts:
|
||||
│ │ → KONFLIKT: Beide geändert seit last_sync
|
||||
│ │
|
||||
│ │ **REGEL: EspoCRM WINS!**
|
||||
│ │
|
||||
│ │ ├─> Set syncStatus = "conflict"
|
||||
│ │ ├─> Überschreibe Advoware mit EspoCRM-Daten:
|
||||
│ │ │ PUT /api/v1/advonet/Beteiligte/{betnr}
|
||||
│ │ │
|
||||
│ │ ├─> Update EspoCRM:
|
||||
│ │ │ syncStatus = "clean" (gelöst!)
|
||||
│ │ │ advowareLastSync = NOW()
|
||||
│ │ │ syncErrorMessage = "Konflikt am {NOW}: EspoCRM={espocrm_ts}, Advoware={advoware_ts}. EspoCRM hat gewonnen."
|
||||
│ │ │
|
||||
│ │ └─> Send Notification:
|
||||
│ │ Template: "beteiligte_sync_conflict"
|
||||
│ │ To: Admin-User oder zugewiesener User
|
||||
│ │
|
||||
│ │ ELSE:
|
||||
│ │ → Keine Änderungen seit last_sync
|
||||
│ │ └─> Skip
|
||||
│ │
|
||||
│ └─> Bei Fehler:
|
||||
│ syncStatus = "failed"
|
||||
│ syncErrorMessage = Error-Details (inkl. Stack Trace)
|
||||
│ syncRetryCount += 1
|
||||
│ Log Error
|
||||
│
|
||||
└─> Handle 404 von Advoware (gelöscht):
|
||||
IF advoware.api_call returns 404:
|
||||
├─> Update EspoCRM:
|
||||
│ syncStatus = "deleted_in_advoware"
|
||||
│ advowareDeletedAt = NOW()
|
||||
│ syncErrorMessage = "Beteiligter existiert nicht mehr in Advoware"
|
||||
│
|
||||
└─> Send Notification:
|
||||
Template: "beteiligte_advoware_deleted"
|
||||
To: Admin-User oder zugewiesener User
|
||||
```
|
||||
|
||||
**Timing**: ~2-5 Sekunden nach Webhook oder Cron-Event
|
||||
|
||||
---
|
||||
|
||||
## 🔄 Flow B: Advoware → EspoCRM (Cron-basiert mit Events)
|
||||
|
||||
**Trigger**: Cron alle 15 Minuten
|
||||
|
||||
```
|
||||
1. beteiligte_sync_cron_step.py (*/15 * * * *):
|
||||
|
||||
├─> Query EspoCRM: Alle Entities die Sync benötigen
|
||||
│
|
||||
│ SELECT * FROM CBeteiligte WHERE:
|
||||
│ - syncStatus IN ('pending_sync', 'dirty', 'failed')
|
||||
│ - OR (syncStatus = 'clean' AND betnr IS NOT NULL
|
||||
│ AND advowareLastSync < NOW() - 24 HOURS)
|
||||
│
|
||||
├─> Für JEDEN Beteiligten einzeln:
|
||||
│ └─> Emit Event: "vmh.beteiligte.sync_check"
|
||||
│ payload: {
|
||||
│ entity_id: "68e4af00172be7924",
|
||||
│ source: "cron",
|
||||
│ timestamp: "2026-02-07T14:30:00Z"
|
||||
│ }
|
||||
│
|
||||
└─> Log: "Emitted {count} sync_check events"
|
||||
|
||||
2. beteiligte_sync_event_step.py (GLEICHER Handler wie Webhook!):
|
||||
|
||||
└─> Subscribe zu: "vmh.beteiligte.sync_check"
|
||||
(Dieser Event kommt von Cron oder manuellen Triggers)
|
||||
|
||||
├─> Fetch entity_id aus Event-Payload
|
||||
│
|
||||
└─> Führe GLEICHE Logik aus wie bei Webhook (siehe Flow A oben!)
|
||||
- Lock via syncStatus
|
||||
- Timestamp-Vergleich
|
||||
- Create/Update
|
||||
- Konfliktauflösung (EspoCRM wins)
|
||||
- 404 Handling (deleted_in_advoware)
|
||||
- Update syncStatus + Felder
|
||||
|
||||
**WICHTIG**: Flow B nutzt Events statt Batch-Processing!
|
||||
- Cron emittiert nur Events für zu syncende Entities
|
||||
- Der normale Sync-Handler (Flow A) verarbeitet beide Quellen gleich
|
||||
- Code-Wiederverwendung: KEIN separater Batch-Handler nötig!
|
||||
```
|
||||
|
||||
**Timing**:
|
||||
- Cron läuft alle 15 Minuten
|
||||
- Events werden sofort verarbeitet (wie Webhooks)
|
||||
|
||||
---
|
||||
|
||||
## 📊 Optimierung: Nur veraltete checken
|
||||
|
||||
### Cron-Query für zu prüfende Entities:
|
||||
|
||||
```javascript
|
||||
// In beteiligte_sync_all_event_step.py
|
||||
|
||||
// 1. Holen von Entities die Sync benötigen
|
||||
const needsSyncFilter = {
|
||||
where: [
|
||||
{
|
||||
type: 'or',
|
||||
value: [
|
||||
// Neu und noch nicht gesynct
|
||||
{
|
||||
type: 'and',
|
||||
value: [
|
||||
{type: 'equals', attribute: 'syncStatus', value: 'pending_sync'},
|
||||
{type: 'isNull', attribute: 'betnr'}
|
||||
]
|
||||
},
|
||||
// Dirty (geändert in EspoCRM)
|
||||
{type: 'equals', attribute: 'syncStatus', value: 'dirty'},
|
||||
|
||||
// Failed (Retry)
|
||||
{type: 'equals', attribute: 'syncStatus', value: 'failed'},
|
||||
|
||||
// Clean aber lange nicht gesynct (> 24h)
|
||||
{
|
||||
type: 'and',
|
||||
value: [
|
||||
{type: 'equals', attribute: 'syncStatus', value: 'clean'},
|
||||
{type: 'isNotNull', attribute: 'betnr'},
|
||||
{
|
||||
type: 'or',
|
||||
value: [
|
||||
{type: 'isNull', attribute: 'advowareLastSync'},
|
||||
{type: 'before', attribute: 'advowareLastSync', value: 'NOW() - 24 HOURS'}
|
||||
]
|
||||
}
|
||||
]
|
||||
}
|
||||
]
|
||||
}
|
||||
]
|
||||
};
|
||||
```
|
||||
|
||||
### Advoware Query-Optimierung:
|
||||
|
||||
```python
|
||||
# Nur kürzlich geänderte aus Advoware holen
|
||||
last_full_sync = get_last_full_sync_timestamp() # z.B. vor 7 Tagen
|
||||
if last_full_sync:
|
||||
# Incremental Fetch
|
||||
params = {
|
||||
'filter': f'geaendertAm gt {last_full_sync.isoformat()}',
|
||||
'orderBy': 'geaendertAm desc'
|
||||
}
|
||||
else:
|
||||
# Full Fetch (beim ersten Mal oder nach langer Zeit)
|
||||
params = {}
|
||||
|
||||
result = await advoware.api_call(
|
||||
'api/v1/advonet/Beteiligte',
|
||||
method='GET',
|
||||
params=params
|
||||
)
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## 🔐 Locking via syncStatus
|
||||
|
||||
**Verhindert Race Conditions ohne Redis Lock**:
|
||||
|
||||
```python
|
||||
# Vor Sync-Operation:
|
||||
async def acquire_sync_lock(espocrm_api, entity_id):
|
||||
"""
|
||||
Setzt syncStatus auf "syncing" wenn möglich.
|
||||
Returns: True wenn Lock erhalten, False sonst
|
||||
"""
|
||||
try:
|
||||
# Fetch current
|
||||
entity = await espocrm_api.get_entity('CBeteiligte', entity_id)
|
||||
|
||||
if entity.get('syncStatus') == 'syncing':
|
||||
# Bereits im Sync-Prozess
|
||||
return False
|
||||
|
||||
# Atomic Update (EspoCRM sollte Optimistic Locking unterstützen)
|
||||
await espocrm_api.update_entity('CBeteiligte', entity_id, {
|
||||
'syncStatus': 'syncing'
|
||||
})
|
||||
|
||||
return True
|
||||
|
||||
except Exception as e:
|
||||
logger.error(f"Failed to acquire sync lock: {e}")
|
||||
return False
|
||||
|
||||
# Nach Sync-Operation (im finally-Block):
|
||||
async def release_sync_lock(espocrm_api, entity_id, new_status='clean'):
|
||||
"""Setzt syncStatus zurück"""
|
||||
try:
|
||||
await espocrm_api.update_entity('CBeteiligte', entity_id, {
|
||||
'syncStatus': new_status,
|
||||
'advowareLastSync': datetime.now(pytz.UTC).isoformat()
|
||||
})
|
||||
except Exception as e:
|
||||
logger.error(f"Failed to release sync lock: {e}")
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## 📋 Status-Übergänge
|
||||
|
||||
```
|
||||
pending_sync → syncing → clean (erfolgreicher Create)
|
||||
pending_sync → syncing → failed (fehlgeschlagener Create)
|
||||
|
||||
clean → dirty → syncing → clean (Update nach Änderung)
|
||||
clean → dirty → syncing → conflict (Konflikt detektiert)
|
||||
clean → dirty → syncing → failed (Update fehlgeschlagen)
|
||||
|
||||
failed → syncing → clean (erfolgreicher Retry)
|
||||
failed → syncing → failed (erneuter Fehler)
|
||||
|
||||
conflict → syncing → clean (manuell aufgelöst)
|
||||
|
||||
clean → deleted_in_advoware (in Advoware gelöscht)
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## 🎯 Implementierungs-Checkliste
|
||||
|
||||
### Phase 1: Core Sync (Flow A - Webhook + Cron Events)
|
||||
|
||||
- [ ] **services/espocrm_mapper.py**
|
||||
- [ ] `map_cbeteiligte_to_advoware(espo_entity)`
|
||||
- [ ] `map_advoware_to_cbeteiligte(advo_entity)`
|
||||
|
||||
- [ ] **steps/vmh/beteiligte_sync_event_step.py** (ZENTRALER Handler!)
|
||||
- [ ] Subscribe zu: `vmh.beteiligte.create`, `vmh.beteiligte.update`, `vmh.beteiligte.delete`, `vmh.beteiligte.sync_check`
|
||||
- [ ] Fetch Entity von EspoCRM
|
||||
- [ ] Lock via syncStatus="syncing"
|
||||
- [ ] Timestamp-Vergleich
|
||||
- [ ] Create/Update in Advoware
|
||||
- [ ] **Konfliktauflösung: EspoCRM wins!**
|
||||
- [ ] **404 Handling: Soft-Delete (deleted_in_advoware)**
|
||||
- [ ] **Notifications: Bei Konflikt + Soft-Delete**
|
||||
- [ ] Update syncStatus + advowareLastSync + syncErrorMessage + syncRetryCount
|
||||
- [ ] Error Handling (→ syncStatus="failed" mit Retry-Counter)
|
||||
- [ ] Redis Cleanup (SREM pending sets)
|
||||
|
||||
### Phase 2: Cron Event Emitter (Flow B)
|
||||
|
||||
- [ ] **steps/vmh/beteiligte_sync_cron_step.py**
|
||||
- [ ] Cron: `*/15 * * * *`
|
||||
- [ ] Query EspoCRM: Entities mit Status `IN (pending_sync, dirty, failed)` ODER `clean + advowareLastSync < NOW() - 24h`
|
||||
- [ ] Für JEDEN Beteiligten: Emit `vmh.beteiligte.sync_check` Event
|
||||
- [ ] Log: Anzahl emittierter Events
|
||||
- [ ] **KEIN** Batch-Processing - Events werden einzeln vom Handler verarbeitet!
|
||||
|
||||
### Phase 3: Utilities
|
||||
|
||||
- [ ] **services/betei & Notifications
|
||||
|
||||
- [ ] **services/beteiligte_sync_utils.py**
|
||||
- [ ] `acquire_sync_lock(entity_id)` → Setzt syncStatus="syncing"
|
||||
- [ ] `release_sync_lock(entity_id, new_status)` → Setzt syncStatus + Updates
|
||||
- [ ] `compare_timestamps(espo_ts, advo_ts, last_sync)` → Returns: "espocrm_newer", "advoware_newer", "conflict", "no_change"
|
||||
- [ ] `resolve_conflict_espocrm_wins(espo_entity, advo_entity)` → Überschreibt Advoware
|
||||
- [ ] `send_notification(entity_id, template_name, extra_data=None)` → EspoCRM Notification
|
||||
- [ ] `handle_advoware_deleted(entity_id, error_msg)` → Soft-Delete + Notification
|
||||
|
||||
- [ ] Unit Tests für Mapper
|
||||
- [ ] Integration Tests für beide Flows
|
||||
- [ ] Konflikt-Szenarien testen
|
||||
- [ ] Load-Tests (Performance mit 1000+ Entities)
|
||||
- [ ] CLI Audit-Tool (analog zu calendar_sync audit)
|
||||
→ clean (Konflikt → EspoCRM wins → gelöst!)
|
||||
clean → dirty → syncing → failed (Update fehlgeschlagen)
|
||||
|
||||
dirty → syncing → deleted_in_advoware (404 von Advoware → Soft-Delete)
|
||||
|
||||
failed → syncing → clean (erfolgreicher Retry)
|
||||
failed → syncing → failed (erneuter Fehler, syncRetryCount++)
|
||||
|
||||
conflict → clean (automatisch via EspoCRM wins)
|
||||
|
||||
clean → deleted_in_advoware (Advoware hat gelöscht)
|
||||
deleted_in_advoware → clean (Re-create in Advoware via Manual-Trigger
|
||||
GET /api/v1/CBeteiligte?select=syncStatus&maxSize=1000
|
||||
→ Gruppiere und zähle
|
||||
|
||||
// Entities die Sync benötigen
|
||||
GET /api/v1/CBeteiligte?where=[
|
||||
{type: 'in', attribute: 'syncStatus', value: ['pending_sync', 'dirty', 'failed']}
|
||||
]
|
||||
|
||||
// Lange nicht gesynct (> 7 Tage)
|
||||
GET /api/v1/CBeteiligte?where=[
|
||||
{type: 'before', attribute: 'advowareLastSync', value: 'NOW() - 7 DAYS'}
|
||||
]
|
||||
|
||||
// Konflikte
|
||||
GET /api/v1/CBeteiligte?where=[
|
||||
{type: 'equals', attribute: 'syncStatus', value: 'conflict'}
|
||||
]
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## 📈 Performance-Überlegungen
|
||||
|
||||
### Batch-Größen:
|
||||
|
||||
```python
|
||||
# Cron-Job Configuration
|
||||
CRON_BATCH_SIZE = 50 # Max 50 Entities pro Cron-Run
|
||||
CRON_TIMEOUT = 300 # 5 Minuten Timeout
|
||||
|
||||
# Advoware Fetch
|
||||
ADVOWARE_PAGE_SIZE = 100 # Entities pro API-Request
|
||||
```
|
||||
|
||||
### Timing:
|
||||
|
||||
- **Webhook (Flow A)**: 2-5 Sekunden (near real-time)
|
||||
- **Cron (Flow B)**: 15 Minuten Intervall
|
||||
- **Veraltete Check**: 24 Stunden (täglich syncen)
|
||||
- **Full Sync**: 7 Tage (wöchentlich alle prüfen)
|
||||
|
||||
### Rate Limiting:
|
||||
|
||||
```python
|
||||
# Aus bestehender AdvowareAPI
|
||||
# - Bereits implementiert
|
||||
# - Token-based Rate Limiting via Redis
|
||||
|
||||
# Für EspoCRM hinzufügen:
|
||||
ESPOCRM_MAX_REQUESTS_PER_MINUTE = 100
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## 🎯 Vorteile dieser Architektur
|
||||
|
||||
✅ **Kein PostgreSQL nötig** - EspoCRM ist State-Datenbank
|
||||
✅ **Alle Daten in EspoCRM** - Single Source of Truth
|
||||
✅ **Status sichtbar** - User können syncStatus in UI sehen
|
||||
✅ **Optimiert** - Nur veraltete werden geprüft
|
||||
✅ **Robust** - Locking via syncStatus verhindert Race Conditions
|
||||
✅ **Konflikt-Tracking** - Konflikte werden explizit markiert
|
||||
✅ **Wiederverwendbar** - Lock-Pattern nutzbar für andere Syncs
|
||||
|
||||
---
|
||||
|
||||
## 🔧 Nächste Schritte
|
||||
|
||||
1. **Mapper implementieren** (services/espocrm_mapper.py)
|
||||
2. **Webhook-Handler komplettieren** (Flow A)
|
||||
3. **Cron + Polling implementieren** (Flow B)
|
||||
4. **Testing mit echten Daten**
|
||||
5. **Monitoring & Dashboard**
|
||||
|
||||
**Geschätzte Zeit**: 5-7 Tage
|
||||
|
||||
---
|
||||
Entscheidungen (vom User bestätigt)**:
|
||||
1. ✅ syncStatus als Enum in EspoCRM mit definierten Werten
|
||||
2. ✅ Soft-Delete: Nur Flag (deleted_in_advoware + advowareDeletedAt)
|
||||
3. ✅ Automatisch: **EspoCRM WINS** bei Konflikten
|
||||
4. ✅ Notifications: Ja, bei Konflikten + Soft-Deletes (EspoCRM Notifications)
|
||||
|
||||
**Architektur-Entscheidung**:
|
||||
- ✅ Cron emittiert Events (`vmh.beteiligte.sync_check`), statt Batch-Processing
|
||||
- ✅ Ein zentraler Sync-Handler für Webhooks UND Cron-Events
|
||||
- ✅ Code-Wiederverwendung maximiertdvoware wins"?
|
||||
4. Benachrichtigung bei Konflikten? (Email, Webhook, ...)
|
||||
@@ -39,3 +39,8 @@ class Config:
|
||||
CALENDAR_SYNC_ANONYMIZE_GOOGLE_EVENTS = os.getenv('CALENDAR_SYNC_ANONYMIZE_GOOGLE_EVENTS', 'true').lower() == 'true'
|
||||
CALENDAR_SYNC_DEBUG_KUERZEL = [k.strip().upper() for k in os.getenv('CALENDAR_SYNC_DEBUG_KUERZEL', 'SB,AI,RO,OK,BI,ST,UR,PB,VB').split(',')]
|
||||
ADVOWARE_WRITE_PROTECTION = True
|
||||
|
||||
# EspoCRM API settings
|
||||
ESPOCRM_API_BASE_URL = os.getenv('ESPOCRM_API_BASE_URL', 'https://crm.bitbylaw.com/api/v1')
|
||||
ESPOCRM_API_KEY = os.getenv('ESPOCRM_MARVIN_API_KEY', '')
|
||||
ESPOCRM_API_TIMEOUT_SECONDS = int(os.getenv('ESPOCRM_API_TIMEOUT_SECONDS', '30'))
|
||||
@@ -1,624 +0,0 @@
|
||||
# Deployment Guide
|
||||
|
||||
## Production Deployment
|
||||
|
||||
### Prerequisites
|
||||
|
||||
- Root/sudo access zum Server
|
||||
- Ubuntu/Debian Linux (tested on Ubuntu 22.04+)
|
||||
- Internet-Zugang für Package-Installation
|
||||
|
||||
### Installation Steps
|
||||
|
||||
#### 1. System Dependencies
|
||||
|
||||
```bash
|
||||
# Update system
|
||||
sudo apt-get update
|
||||
sudo apt-get upgrade -y
|
||||
|
||||
# Install Node.js 18.x
|
||||
curl -fsSL https://deb.nodesource.com/setup_18.x | sudo -E bash -
|
||||
sudo apt-get install -y nodejs
|
||||
|
||||
# Install Python 3.13
|
||||
sudo apt-get install -y python3.13 python3.13-venv python3.13-dev
|
||||
|
||||
# Install Redis
|
||||
sudo apt-get install -y redis-server
|
||||
|
||||
# Install Git
|
||||
sudo apt-get install -y git
|
||||
|
||||
# Start Redis
|
||||
sudo systemctl enable redis-server
|
||||
sudo systemctl start redis-server
|
||||
```
|
||||
|
||||
#### 2. Application Setup
|
||||
|
||||
```bash
|
||||
# Create application directory
|
||||
sudo mkdir -p /opt/motia-app
|
||||
cd /opt/motia-app
|
||||
|
||||
# Clone repository (oder rsync von Development)
|
||||
git clone <repository-url> bitbylaw
|
||||
cd bitbylaw
|
||||
|
||||
# Create www-data user if not exists
|
||||
sudo useradd -r -s /bin/bash www-data || true
|
||||
|
||||
# Set ownership
|
||||
sudo chown -R www-data:www-data /opt/motia-app
|
||||
```
|
||||
|
||||
#### 3. Node.js Dependencies
|
||||
|
||||
```bash
|
||||
# Als www-data user
|
||||
sudo -u www-data bash
|
||||
cd /opt/motia-app/bitbylaw
|
||||
|
||||
# Install Node.js packages
|
||||
npm install
|
||||
|
||||
# Build TypeScript (falls nötig)
|
||||
npm run build
|
||||
```
|
||||
|
||||
#### 4. Python Dependencies
|
||||
|
||||
```bash
|
||||
# Als www-data user
|
||||
cd /opt/motia-app/bitbylaw
|
||||
|
||||
# Create virtual environment
|
||||
python3.13 -m venv python_modules
|
||||
|
||||
# Activate
|
||||
source python_modules/bin/activate
|
||||
|
||||
# Install dependencies
|
||||
pip install -r requirements.txt
|
||||
|
||||
# Deactivate
|
||||
deactivate
|
||||
```
|
||||
|
||||
#### 5. Service Account Setup
|
||||
|
||||
```bash
|
||||
# Copy service account JSON
|
||||
sudo cp service-account.json /opt/motia-app/service-account.json
|
||||
|
||||
# Set secure permissions
|
||||
sudo chmod 600 /opt/motia-app/service-account.json
|
||||
sudo chown www-data:www-data /opt/motia-app/service-account.json
|
||||
```
|
||||
|
||||
Siehe auch: [GOOGLE_SETUP_README.md](../GOOGLE_SETUP_README.md)
|
||||
|
||||
#### 6. systemd Service
|
||||
|
||||
Erstellen Sie `/etc/systemd/system/motia.service`:
|
||||
|
||||
```ini
|
||||
[Unit]
|
||||
Description=Motia Backend Framework
|
||||
After=network.target redis-server.service
|
||||
|
||||
[Service]
|
||||
Type=simple
|
||||
User=www-data
|
||||
WorkingDirectory=/opt/motia-app/bitbylaw
|
||||
|
||||
# Environment Variables
|
||||
Environment=NODE_ENV=production
|
||||
Environment=NODE_OPTIONS=--max-old-space-size=8192 --inspect --heapsnapshot-signal=SIGUSR2
|
||||
Environment=HOST=0.0.0.0
|
||||
Environment=MOTIA_LOG_LEVEL=info
|
||||
Environment=NPM_CONFIG_CACHE=/opt/motia-app/.npm-cache
|
||||
|
||||
# Advoware Configuration (ADJUST VALUES!)
|
||||
Environment=ADVOWARE_API_BASE_URL=https://www2.advo-net.net:90/
|
||||
Environment=ADVOWARE_PRODUCT_ID=64
|
||||
Environment=ADVOWARE_APP_ID=your_app_id
|
||||
Environment=ADVOWARE_API_KEY=your_api_key_base64
|
||||
Environment=ADVOWARE_KANZLEI=your_kanzlei
|
||||
Environment=ADVOWARE_DATABASE=your_database
|
||||
Environment=ADVOWARE_USER=your_user
|
||||
Environment=ADVOWARE_ROLE=2
|
||||
Environment=ADVOWARE_PASSWORD=your_password
|
||||
Environment=ADVOWARE_WRITE_PROTECTION=false
|
||||
|
||||
# Redis Configuration
|
||||
Environment=REDIS_HOST=localhost
|
||||
Environment=REDIS_PORT=6379
|
||||
Environment=REDIS_DB_ADVOWARE_CACHE=1
|
||||
Environment=REDIS_DB_CALENDAR_SYNC=2
|
||||
|
||||
# Google Calendar
|
||||
Environment=GOOGLE_CALENDAR_SERVICE_ACCOUNT_PATH=/opt/motia-app/service-account.json
|
||||
|
||||
# EspoCRM (if used)
|
||||
Environment=ESPOCRM_MARVIN_API_KEY=your_webhook_key
|
||||
|
||||
# Start Command
|
||||
ExecStart=/bin/bash -c 'source /opt/motia-app/python_modules/bin/activate && /usr/bin/npm start'
|
||||
|
||||
# Restart Policy
|
||||
Restart=always
|
||||
RestartSec=10
|
||||
|
||||
# Security
|
||||
NoNewPrivileges=true
|
||||
PrivateTmp=true
|
||||
|
||||
[Install]
|
||||
WantedBy=multi-user.target
|
||||
```
|
||||
|
||||
**WICHTIG**: Passen Sie alle `your_*` Werte an!
|
||||
|
||||
#### 7. Enable and Start Service
|
||||
|
||||
```bash
|
||||
# Reload systemd
|
||||
sudo systemctl daemon-reload
|
||||
|
||||
# Enable service (autostart)
|
||||
sudo systemctl enable motia.service
|
||||
|
||||
# Start service
|
||||
sudo systemctl start motia.service
|
||||
|
||||
# Check status
|
||||
sudo systemctl status motia.service
|
||||
```
|
||||
|
||||
#### 8. Verify Installation
|
||||
|
||||
```bash
|
||||
# Check logs
|
||||
sudo journalctl -u motia.service -f
|
||||
|
||||
# Test API
|
||||
curl http://localhost:3000/health # (wenn implementiert)
|
||||
|
||||
# Test Advoware Proxy
|
||||
curl "http://localhost:3000/advoware/proxy?endpoint=employees"
|
||||
```
|
||||
|
||||
## Reverse Proxy Setup (nginx)
|
||||
|
||||
### Install nginx
|
||||
|
||||
```bash
|
||||
sudo apt-get install -y nginx
|
||||
```
|
||||
|
||||
### Configure
|
||||
|
||||
`/etc/nginx/sites-available/motia`:
|
||||
|
||||
```nginx
|
||||
upstream motia_backend {
|
||||
server 127.0.0.1:3000;
|
||||
}
|
||||
|
||||
server {
|
||||
listen 80;
|
||||
server_name your-domain.com;
|
||||
|
||||
# Redirect to HTTPS
|
||||
return 301 https://$server_name$request_uri;
|
||||
}
|
||||
|
||||
server {
|
||||
listen 443 ssl http2;
|
||||
server_name your-domain.com;
|
||||
|
||||
# SSL Configuration (Let's Encrypt)
|
||||
ssl_certificate /etc/letsencrypt/live/your-domain.com/fullchain.pem;
|
||||
ssl_certificate_key /etc/letsencrypt/live/your-domain.com/privkey.pem;
|
||||
|
||||
# Security Headers
|
||||
add_header X-Frame-Options "SAMEORIGIN" always;
|
||||
add_header X-Content-Type-Options "nosniff" always;
|
||||
add_header X-XSS-Protection "1; mode=block" always;
|
||||
|
||||
# Proxy Settings
|
||||
location / {
|
||||
proxy_pass http://motia_backend;
|
||||
proxy_http_version 1.1;
|
||||
proxy_set_header Upgrade $http_upgrade;
|
||||
proxy_set_header Connection 'upgrade';
|
||||
proxy_set_header Host $host;
|
||||
proxy_cache_bypass $http_upgrade;
|
||||
proxy_set_header X-Real-IP $remote_addr;
|
||||
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
|
||||
proxy_set_header X-Forwarded-Proto $scheme;
|
||||
|
||||
# Timeouts
|
||||
proxy_connect_timeout 60s;
|
||||
proxy_send_timeout 60s;
|
||||
proxy_read_timeout 60s;
|
||||
}
|
||||
|
||||
# Access Log
|
||||
access_log /var/log/nginx/motia-access.log;
|
||||
error_log /var/log/nginx/motia-error.log;
|
||||
}
|
||||
```
|
||||
|
||||
### Enable and Restart
|
||||
|
||||
```bash
|
||||
# Enable site
|
||||
sudo ln -s /etc/nginx/sites-available/motia /etc/nginx/sites-enabled/
|
||||
|
||||
# Test configuration
|
||||
sudo nginx -t
|
||||
|
||||
# Restart nginx
|
||||
sudo systemctl restart nginx
|
||||
```
|
||||
|
||||
### SSL Certificate (Let's Encrypt)
|
||||
|
||||
```bash
|
||||
# Install certbot
|
||||
sudo apt-get install -y certbot python3-certbot-nginx
|
||||
|
||||
# Obtain certificate
|
||||
sudo certbot --nginx -d your-domain.com
|
||||
|
||||
# Auto-renewal is configured automatically
|
||||
```
|
||||
|
||||
## Firewall Configuration
|
||||
|
||||
```bash
|
||||
# Allow SSH
|
||||
sudo ufw allow 22/tcp
|
||||
|
||||
# Allow HTTP/HTTPS (if using nginx)
|
||||
sudo ufw allow 80/tcp
|
||||
sudo ufw allow 443/tcp
|
||||
|
||||
# Enable firewall
|
||||
sudo ufw enable
|
||||
```
|
||||
|
||||
**Wichtig**: Port 3000 NICHT öffentlich öffnen (nur via nginx reverse proxy)
|
||||
|
||||
## Monitoring
|
||||
|
||||
### systemd Service Status
|
||||
|
||||
```bash
|
||||
# Status anzeigen
|
||||
sudo systemctl status motia.service
|
||||
|
||||
# Ist enabled?
|
||||
sudo systemctl is-enabled motia.service
|
||||
|
||||
# Ist aktiv?
|
||||
sudo systemctl is-active motia.service
|
||||
```
|
||||
|
||||
### Logs
|
||||
|
||||
```bash
|
||||
# Live logs
|
||||
sudo journalctl -u motia.service -f
|
||||
|
||||
# Last 100 lines
|
||||
sudo journalctl -u motia.service -n 100
|
||||
|
||||
# Since today
|
||||
sudo journalctl -u motia.service --since today
|
||||
|
||||
# Filter by priority (error only)
|
||||
sudo journalctl -u motia.service -p err
|
||||
```
|
||||
|
||||
### Resource Usage
|
||||
|
||||
```bash
|
||||
# CPU and Memory
|
||||
sudo systemctl status motia.service
|
||||
|
||||
# Detailed process info
|
||||
ps aux | grep motia
|
||||
|
||||
# Memory usage
|
||||
sudo pmap $(pgrep -f "motia start") | tail -n 1
|
||||
```
|
||||
|
||||
### Redis Monitoring
|
||||
|
||||
```bash
|
||||
# Connect to Redis
|
||||
redis-cli
|
||||
|
||||
# Show info
|
||||
INFO
|
||||
|
||||
# Show database sizes
|
||||
INFO keyspace
|
||||
|
||||
# Monitor commands (real-time)
|
||||
MONITOR
|
||||
|
||||
# Show memory usage
|
||||
MEMORY USAGE <key>
|
||||
```
|
||||
|
||||
## Backup Strategy
|
||||
|
||||
### Application Code
|
||||
|
||||
```bash
|
||||
# Git-based backup
|
||||
cd /opt/motia-app/bitbylaw
|
||||
git pull origin main
|
||||
|
||||
# Or: rsync backup
|
||||
rsync -av /opt/motia-app/bitbylaw/ /backup/motia-app/
|
||||
```
|
||||
|
||||
### Redis Data
|
||||
|
||||
```bash
|
||||
# RDB snapshot (automatic by Redis)
|
||||
# Location: /var/lib/redis/dump.rdb
|
||||
|
||||
# Manual backup
|
||||
sudo cp /var/lib/redis/dump.rdb /backup/redis-dump-$(date +%Y%m%d).rdb
|
||||
|
||||
# Restore
|
||||
sudo systemctl stop redis-server
|
||||
sudo cp /backup/redis-dump-20260207.rdb /var/lib/redis/dump.rdb
|
||||
sudo chown redis:redis /var/lib/redis/dump.rdb
|
||||
sudo systemctl start redis-server
|
||||
```
|
||||
|
||||
### Configuration
|
||||
|
||||
```bash
|
||||
# Backup systemd service
|
||||
sudo cp /etc/systemd/system/motia.service /backup/motia.service
|
||||
|
||||
# Backup nginx config
|
||||
sudo cp /etc/nginx/sites-available/motia /backup/nginx-motia.conf
|
||||
|
||||
# Backup service account
|
||||
sudo cp /opt/motia-app/service-account.json /backup/service-account.json.backup
|
||||
```
|
||||
|
||||
## Updates & Maintenance
|
||||
|
||||
### Application Update
|
||||
|
||||
```bash
|
||||
# 1. Pull latest code
|
||||
cd /opt/motia-app/bitbylaw
|
||||
sudo -u www-data git pull origin main
|
||||
|
||||
# 2. Update dependencies
|
||||
sudo -u www-data npm install
|
||||
sudo -u www-data bash -c 'source python_modules/bin/activate && pip install -r requirements.txt'
|
||||
|
||||
# 3. Restart service
|
||||
sudo systemctl restart motia.service
|
||||
|
||||
# 4. Verify
|
||||
sudo journalctl -u motia.service -f
|
||||
```
|
||||
|
||||
### Zero-Downtime Deployment
|
||||
|
||||
Für zukünftige Implementierung mit Blue-Green Deployment:
|
||||
|
||||
```bash
|
||||
# 1. Deploy to staging directory
|
||||
# 2. Run health checks
|
||||
# 3. Switch symlink
|
||||
# 4. Reload service
|
||||
# 5. Rollback if issues
|
||||
```
|
||||
|
||||
### Database Migrations
|
||||
|
||||
**Aktuell**: Keine Datenbank-Migrationen (nur Redis)
|
||||
|
||||
**Zukünftig** (PostgreSQL):
|
||||
```bash
|
||||
# Run migrations
|
||||
python manage.py migrate
|
||||
```
|
||||
|
||||
## Security Hardening
|
||||
|
||||
### File Permissions
|
||||
|
||||
```bash
|
||||
# Application files
|
||||
sudo chown -R www-data:www-data /opt/motia-app
|
||||
sudo chmod 755 /opt/motia-app
|
||||
sudo chmod 755 /opt/motia-app/bitbylaw
|
||||
|
||||
# Service account
|
||||
sudo chmod 600 /opt/motia-app/service-account.json
|
||||
sudo chown www-data:www-data /opt/motia-app/service-account.json
|
||||
|
||||
# No world-readable secrets
|
||||
sudo find /opt/motia-app -type f -name "*.json" -exec chmod 600 {} \;
|
||||
```
|
||||
|
||||
### Redis Security
|
||||
|
||||
```bash
|
||||
# Edit Redis config
|
||||
sudo nano /etc/redis/redis.conf
|
||||
|
||||
# Bind to localhost only
|
||||
bind 127.0.0.1 ::1
|
||||
|
||||
# Disable dangerous commands (optional)
|
||||
rename-command FLUSHDB ""
|
||||
rename-command FLUSHALL ""
|
||||
rename-command CONFIG ""
|
||||
|
||||
# Restart Redis
|
||||
sudo systemctl restart redis-server
|
||||
```
|
||||
|
||||
### systemd Hardening
|
||||
|
||||
Bereits in Service-Datei enthalten:
|
||||
- `NoNewPrivileges=true` - Verhindert Privilege-Escalation
|
||||
- `PrivateTmp=true` - Isoliertes /tmp
|
||||
- User: `www-data` (non-root)
|
||||
|
||||
Weitere Optionen:
|
||||
```ini
|
||||
[Service]
|
||||
ProtectSystem=strict
|
||||
ProtectHome=true
|
||||
ReadWritePaths=/opt/motia-app
|
||||
```
|
||||
|
||||
## Disaster Recovery
|
||||
|
||||
### Service Crashed
|
||||
|
||||
```bash
|
||||
# Check status
|
||||
sudo systemctl status motia.service
|
||||
|
||||
# View logs
|
||||
sudo journalctl -u motia.service -n 100
|
||||
|
||||
# Restart
|
||||
sudo systemctl restart motia.service
|
||||
|
||||
# If still failing, check:
|
||||
# - Redis is running
|
||||
# - Service account file exists
|
||||
# - Environment variables are set
|
||||
```
|
||||
|
||||
### Redis Data Loss
|
||||
|
||||
```bash
|
||||
# Restore from backup
|
||||
sudo systemctl stop redis-server
|
||||
sudo cp /backup/redis-dump-latest.rdb /var/lib/redis/dump.rdb
|
||||
sudo chown redis:redis /var/lib/redis/dump.rdb
|
||||
sudo systemctl start redis-server
|
||||
|
||||
# Clear specific data if corrupted
|
||||
redis-cli -n 1 FLUSHDB # Advoware cache
|
||||
redis-cli -n 2 FLUSHDB # Calendar sync
|
||||
```
|
||||
|
||||
### Complete System Failure
|
||||
|
||||
```bash
|
||||
# 1. Fresh server setup (siehe Installation Steps)
|
||||
# 2. Restore application code from Git/Backup
|
||||
# 3. Restore configuration (systemd, nginx)
|
||||
# 4. Restore service-account.json
|
||||
# 5. Restore Redis data (optional, will rebuild)
|
||||
# 6. Start services
|
||||
```
|
||||
|
||||
## Performance Tuning
|
||||
|
||||
### Node.js Memory
|
||||
|
||||
In systemd service:
|
||||
```ini
|
||||
Environment=NODE_OPTIONS=--max-old-space-size=8192 # 8GB
|
||||
```
|
||||
|
||||
### Redis Memory
|
||||
|
||||
In `/etc/redis/redis.conf`:
|
||||
```
|
||||
maxmemory 2gb
|
||||
maxmemory-policy allkeys-lru
|
||||
```
|
||||
|
||||
### Linux Kernel
|
||||
|
||||
```bash
|
||||
# Increase file descriptors
|
||||
echo "fs.file-max = 65536" | sudo tee -a /etc/sysctl.conf
|
||||
sudo sysctl -p
|
||||
|
||||
# For www-data user
|
||||
sudo nano /etc/security/limits.conf
|
||||
# Add:
|
||||
www-data soft nofile 65536
|
||||
www-data hard nofile 65536
|
||||
```
|
||||
|
||||
## Health Checks
|
||||
|
||||
### Automated Monitoring
|
||||
|
||||
Cron job für Health Checks:
|
||||
|
||||
```bash
|
||||
# /usr/local/bin/motia-health-check.sh
|
||||
#!/bin/bash
|
||||
if ! systemctl is-active --quiet motia.service; then
|
||||
echo "Motia service is down!" | mail -s "ALERT: Motia Down" admin@example.com
|
||||
systemctl start motia.service
|
||||
fi
|
||||
```
|
||||
|
||||
```bash
|
||||
# Add to crontab
|
||||
sudo crontab -e
|
||||
# Add line:
|
||||
*/5 * * * * /usr/local/bin/motia-health-check.sh
|
||||
```
|
||||
|
||||
### External Monitoring
|
||||
|
||||
Services wie Uptime Robot, Pingdom, etc. können verwendet werden:
|
||||
- HTTP Endpoint: `https://your-domain.com/health`
|
||||
- Check-Interval: 5 Minuten
|
||||
- Alert via Email/SMS
|
||||
|
||||
## Rollback Procedure
|
||||
|
||||
```bash
|
||||
# 1. Stop current service
|
||||
sudo systemctl stop motia.service
|
||||
|
||||
# 2. Revert to previous version
|
||||
cd /opt/motia-app/bitbylaw
|
||||
sudo -u www-data git log # Find previous commit
|
||||
sudo -u www-data git reset --hard <commit-hash>
|
||||
|
||||
# 3. Restore dependencies (if needed)
|
||||
sudo -u www-data npm install
|
||||
|
||||
# 4. Start service
|
||||
sudo systemctl start motia.service
|
||||
|
||||
# 5. Verify
|
||||
sudo journalctl -u motia.service -f
|
||||
```
|
||||
|
||||
## Related Documentation
|
||||
|
||||
- [Architecture](ARCHITECTURE.md)
|
||||
- [Configuration](CONFIGURATION.md)
|
||||
- [Troubleshooting](TROUBLESHOOTING.md)
|
||||
|
||||
@@ -1,800 +0,0 @@
|
||||
# Troubleshooting Guide
|
||||
|
||||
## Service Issues
|
||||
|
||||
### Service Won't Start
|
||||
|
||||
**Symptoms**: `systemctl start motia.service` schlägt fehl
|
||||
|
||||
**Diagnose**:
|
||||
```bash
|
||||
# Check service status
|
||||
sudo systemctl status motia.service
|
||||
|
||||
# View detailed logs
|
||||
sudo journalctl -u motia.service -n 100 --no-pager
|
||||
|
||||
# Check for port conflicts
|
||||
sudo netstat -tlnp | grep 3000
|
||||
```
|
||||
|
||||
**Häufige Ursachen**:
|
||||
|
||||
1. **Port 3000 bereits belegt**:
|
||||
```bash
|
||||
# Find process
|
||||
sudo lsof -i :3000
|
||||
|
||||
# Kill process
|
||||
sudo kill -9 <PID>
|
||||
```
|
||||
|
||||
2. **Fehlende Dependencies**:
|
||||
```bash
|
||||
cd /opt/motia-app/bitbylaw
|
||||
sudo -u www-data npm install
|
||||
sudo -u www-data bash -c 'source python_modules/bin/activate && pip install -r requirements.txt'
|
||||
```
|
||||
|
||||
3. **Falsche Permissions**:
|
||||
```bash
|
||||
sudo chown -R www-data:www-data /opt/motia-app
|
||||
sudo chmod 600 /opt/motia-app/service-account.json
|
||||
```
|
||||
|
||||
4. **Environment Variables fehlen**:
|
||||
```bash
|
||||
# Check systemd environment
|
||||
sudo systemctl show motia.service -p Environment
|
||||
|
||||
# Verify required vars
|
||||
sudo systemctl cat motia.service | grep Environment
|
||||
```
|
||||
|
||||
### Service Keeps Crashing
|
||||
|
||||
**Symptoms**: Service startet, crashed aber nach kurzer Zeit
|
||||
|
||||
**Diagnose**:
|
||||
```bash
|
||||
# Watch logs in real-time
|
||||
sudo journalctl -u motia.service -f
|
||||
|
||||
# Check for OOM (Out of Memory)
|
||||
dmesg | grep -i "out of memory"
|
||||
sudo grep -i "killed process" /var/log/syslog
|
||||
```
|
||||
|
||||
**Solutions**:
|
||||
|
||||
1. **Memory Limit erhöhen**:
|
||||
```ini
|
||||
# In /etc/systemd/system/motia.service
|
||||
Environment=NODE_OPTIONS=--max-old-space-size=8192
|
||||
```
|
||||
|
||||
2. **Python Memory Leak**:
|
||||
```bash
|
||||
# Check memory usage
|
||||
ps aux | grep python
|
||||
|
||||
# Restart service periodically (workaround)
|
||||
# Add to crontab:
|
||||
0 3 * * * systemctl restart motia.service
|
||||
```
|
||||
|
||||
3. **Unhandled Exception**:
|
||||
```bash
|
||||
# Check error logs
|
||||
sudo journalctl -u motia.service -p err
|
||||
|
||||
# Add try-catch in problematic step
|
||||
```
|
||||
|
||||
## Redis Issues
|
||||
|
||||
### Redis Connection Failed
|
||||
|
||||
**Symptoms**: "Redis connection failed" in logs
|
||||
|
||||
**Diagnose**:
|
||||
```bash
|
||||
# Check Redis status
|
||||
sudo systemctl status redis-server
|
||||
|
||||
# Test connection
|
||||
redis-cli ping
|
||||
|
||||
# Check config
|
||||
redis-cli CONFIG GET bind
|
||||
redis-cli CONFIG GET port
|
||||
```
|
||||
|
||||
**Solutions**:
|
||||
|
||||
1. **Redis not running**:
|
||||
```bash
|
||||
sudo systemctl start redis-server
|
||||
sudo systemctl enable redis-server
|
||||
```
|
||||
|
||||
2. **Wrong host/port**:
|
||||
```bash
|
||||
# Check environment
|
||||
echo $REDIS_HOST
|
||||
echo $REDIS_PORT
|
||||
|
||||
# Test connection
|
||||
redis-cli -h $REDIS_HOST -p $REDIS_PORT ping
|
||||
```
|
||||
|
||||
3. **Permission denied**:
|
||||
```bash
|
||||
# Check Redis log
|
||||
sudo tail -f /var/log/redis/redis-server.log
|
||||
|
||||
# Fix permissions
|
||||
sudo chown redis:redis /var/lib/redis
|
||||
sudo chmod 750 /var/lib/redis
|
||||
```
|
||||
|
||||
### Redis Out of Memory
|
||||
|
||||
**Symptoms**: "OOM command not allowed" errors
|
||||
|
||||
**Diagnose**:
|
||||
```bash
|
||||
# Check memory usage
|
||||
redis-cli INFO memory
|
||||
|
||||
# Check maxmemory setting
|
||||
redis-cli CONFIG GET maxmemory
|
||||
```
|
||||
|
||||
**Solutions**:
|
||||
|
||||
1. **Increase maxmemory**:
|
||||
```bash
|
||||
# In /etc/redis/redis.conf
|
||||
maxmemory 2gb
|
||||
maxmemory-policy allkeys-lru
|
||||
|
||||
sudo systemctl restart redis-server
|
||||
```
|
||||
|
||||
2. **Clear old data**:
|
||||
```bash
|
||||
# Clear cache (safe for Advoware tokens)
|
||||
redis-cli -n 1 FLUSHDB
|
||||
|
||||
# Clear calendar sync state
|
||||
redis-cli -n 2 FLUSHDB
|
||||
```
|
||||
|
||||
3. **Check for memory leaks**:
|
||||
```bash
|
||||
# Find large keys
|
||||
redis-cli --bigkeys
|
||||
|
||||
# Check specific key size
|
||||
redis-cli MEMORY USAGE <key>
|
||||
```
|
||||
|
||||
## Advoware API Issues
|
||||
|
||||
### Authentication Failed
|
||||
|
||||
**Symptoms**: "401 Unauthorized" oder "HMAC signature invalid"
|
||||
|
||||
**Diagnose**:
|
||||
```bash
|
||||
# Check logs for auth errors
|
||||
sudo journalctl -u motia.service | grep -i "auth\|token\|401"
|
||||
|
||||
# Test token fetch manually
|
||||
python3 << 'EOF'
|
||||
from services.advoware import AdvowareAPI
|
||||
api = AdvowareAPI()
|
||||
token = api.get_access_token(force_refresh=True)
|
||||
print(f"Token: {token[:20]}...")
|
||||
EOF
|
||||
```
|
||||
|
||||
**Solutions**:
|
||||
|
||||
1. **Invalid API Key**:
|
||||
```bash
|
||||
# Verify API Key is Base64
|
||||
echo $ADVOWARE_API_KEY | base64 -d
|
||||
|
||||
# Re-encode if needed
|
||||
echo -n "raw_key" | base64
|
||||
```
|
||||
|
||||
2. **Wrong credentials**:
|
||||
```bash
|
||||
# Verify environment variables
|
||||
sudo systemctl show motia.service -p Environment | grep ADVOWARE
|
||||
|
||||
# Update in systemd service
|
||||
sudo nano /etc/systemd/system/motia.service
|
||||
sudo systemctl daemon-reload
|
||||
sudo systemctl restart motia.service
|
||||
```
|
||||
|
||||
3. **Token expired**:
|
||||
```bash
|
||||
# Clear cached token
|
||||
redis-cli -n 1 DEL advoware_access_token advoware_token_timestamp
|
||||
|
||||
# Retry request (will fetch new token)
|
||||
```
|
||||
|
||||
### API Timeout
|
||||
|
||||
**Symptoms**: "Request timeout" oder "API call failed"
|
||||
|
||||
**Diagnose**:
|
||||
```bash
|
||||
# Check API response time
|
||||
time curl "http://localhost:3000/advoware/proxy?endpoint=employees"
|
||||
|
||||
# Check network connectivity
|
||||
ping www2.advo-net.net
|
||||
curl -I https://www2.advo-net.net:90/
|
||||
```
|
||||
|
||||
**Solutions**:
|
||||
|
||||
1. **Increase timeout**:
|
||||
```bash
|
||||
# In environment
|
||||
export ADVOWARE_API_TIMEOUT_SECONDS=60
|
||||
|
||||
# Or in systemd service
|
||||
Environment=ADVOWARE_API_TIMEOUT_SECONDS=60
|
||||
```
|
||||
|
||||
2. **Network issues**:
|
||||
```bash
|
||||
# Check firewall
|
||||
sudo ufw status
|
||||
|
||||
# Test direct connection
|
||||
curl -v https://www2.advo-net.net:90/
|
||||
```
|
||||
|
||||
3. **Advoware API down**:
|
||||
```bash
|
||||
# Wait and retry
|
||||
# Implement exponential backoff in code
|
||||
```
|
||||
|
||||
## Google Calendar Issues
|
||||
|
||||
### Service Account Not Found
|
||||
|
||||
**Symptoms**: "service-account.json not found"
|
||||
|
||||
**Diagnose**:
|
||||
```bash
|
||||
# Check file exists
|
||||
ls -la /opt/motia-app/service-account.json
|
||||
|
||||
# Check permissions
|
||||
ls -la /opt/motia-app/service-account.json
|
||||
|
||||
# Check environment variable
|
||||
echo $GOOGLE_CALENDAR_SERVICE_ACCOUNT_PATH
|
||||
```
|
||||
|
||||
**Solutions**:
|
||||
|
||||
1. **File missing**:
|
||||
```bash
|
||||
# Copy from backup
|
||||
sudo cp /backup/service-account.json /opt/motia-app/
|
||||
|
||||
# Set permissions
|
||||
sudo chmod 600 /opt/motia-app/service-account.json
|
||||
sudo chown www-data:www-data /opt/motia-app/service-account.json
|
||||
```
|
||||
|
||||
2. **Wrong path**:
|
||||
```bash
|
||||
# Update environment
|
||||
# In /etc/systemd/system/motia.service:
|
||||
Environment=GOOGLE_CALENDAR_SERVICE_ACCOUNT_PATH=/opt/motia-app/service-account.json
|
||||
|
||||
sudo systemctl daemon-reload
|
||||
sudo systemctl restart motia.service
|
||||
```
|
||||
|
||||
### Calendar API Rate Limit
|
||||
|
||||
**Symptoms**: "403 Rate limit exceeded" oder "429 Too Many Requests"
|
||||
|
||||
**Diagnose**:
|
||||
```bash
|
||||
# Check rate limiting in logs
|
||||
sudo journalctl -u motia.service | grep -i "rate\|403\|429"
|
||||
|
||||
# Check Redis rate limit tokens
|
||||
redis-cli -n 2 GET google_calendar_api_tokens
|
||||
```
|
||||
|
||||
**Solutions**:
|
||||
|
||||
1. **Wait for rate limit reset**:
|
||||
```bash
|
||||
# Rate limit resets every minute
|
||||
# Wait 60 seconds and retry
|
||||
```
|
||||
|
||||
2. **Adjust rate limit settings**:
|
||||
```python
|
||||
# In calendar_sync_event_step.py
|
||||
MAX_TOKENS = 7 # Decrease if hitting limits
|
||||
REFILL_RATE_PER_MS = 7 / 1000
|
||||
```
|
||||
|
||||
3. **Request quota increase**:
|
||||
- Go to Google Cloud Console
|
||||
- Navigate to "APIs & Services" → "Quotas"
|
||||
- Request increase for Calendar API
|
||||
|
||||
### Calendar Access Denied
|
||||
|
||||
**Symptoms**: "Access denied" oder "Insufficient permissions"
|
||||
|
||||
**Diagnose**:
|
||||
```bash
|
||||
# Check service account email
|
||||
python3 << 'EOF'
|
||||
import json
|
||||
with open('/opt/motia-app/service-account.json') as f:
|
||||
data = json.load(f)
|
||||
print(f"Service Account: {data['client_email']}")
|
||||
EOF
|
||||
|
||||
# Test API access
|
||||
python3 << 'EOF'
|
||||
from google.oauth2 import service_account
|
||||
from googleapiclient.discovery import build
|
||||
|
||||
creds = service_account.Credentials.from_service_account_file(
|
||||
'/opt/motia-app/service-account.json',
|
||||
scopes=['https://www.googleapis.com/auth/calendar']
|
||||
)
|
||||
service = build('calendar', 'v3', credentials=creds)
|
||||
result = service.calendarList().list().execute()
|
||||
print(f"Calendars: {len(result.get('items', []))}")
|
||||
EOF
|
||||
```
|
||||
|
||||
**Solutions**:
|
||||
|
||||
1. **Calendar not shared**:
|
||||
```bash
|
||||
# Share calendar with service account email
|
||||
# In Google Calendar UI: Settings → Share → Add service account email
|
||||
```
|
||||
|
||||
2. **Wrong scopes**:
|
||||
```bash
|
||||
# Verify scopes in code
|
||||
# Should be: https://www.googleapis.com/auth/calendar
|
||||
```
|
||||
|
||||
3. **Domain-wide delegation**:
|
||||
```bash
|
||||
# For G Suite, enable domain-wide delegation
|
||||
# See GOOGLE_SETUP_README.md
|
||||
```
|
||||
|
||||
## Calendar Sync Issues
|
||||
|
||||
### Sync Not Running
|
||||
|
||||
**Symptoms**: Keine Calendar-Updates, keine Sync-Logs
|
||||
|
||||
**Diagnose**:
|
||||
```bash
|
||||
# Check if cron is triggering
|
||||
sudo journalctl -u motia.service | grep -i "calendar_sync_cron"
|
||||
|
||||
# Manually trigger sync
|
||||
curl -X POST "http://localhost:3000/advoware/calendar/sync" \
|
||||
-H "Content-Type: application/json" \
|
||||
-d '{"full_content": true}'
|
||||
|
||||
# Check for locks
|
||||
redis-cli -n 1 KEYS "calendar_sync:lock:*"
|
||||
```
|
||||
|
||||
**Solutions**:
|
||||
|
||||
1. **Cron not configured**:
|
||||
```python
|
||||
# Verify calendar_sync_cron_step.py has correct schedule
|
||||
config = {
|
||||
'schedule': '0 2 * * *', # Daily at 2 AM
|
||||
}
|
||||
```
|
||||
|
||||
2. **Lock stuck**:
|
||||
```bash
|
||||
# Clear all locks
|
||||
python /opt/motia-app/bitbylaw/delete_employee_locks.py
|
||||
|
||||
# Or manually
|
||||
redis-cli -n 1 DEL calendar_sync:lock:SB
|
||||
```
|
||||
|
||||
3. **Errors in sync**:
|
||||
```bash
|
||||
# Check error logs
|
||||
sudo journalctl -u motia.service -p err | grep calendar
|
||||
```
|
||||
|
||||
### Duplicate Events
|
||||
|
||||
**Symptoms**: Events erscheinen mehrfach in Google Calendar
|
||||
|
||||
**Diagnose**:
|
||||
```bash
|
||||
# Check for concurrent syncs
|
||||
redis-cli -n 1 KEYS "calendar_sync:lock:*"
|
||||
|
||||
# Check logs for duplicate processing
|
||||
sudo journalctl -u motia.service | grep -i "duplicate\|already exists"
|
||||
```
|
||||
|
||||
**Solutions**:
|
||||
|
||||
1. **Locking not working**:
|
||||
```bash
|
||||
# Verify Redis lock TTL
|
||||
redis-cli -n 1 TTL calendar_sync:lock:SB
|
||||
|
||||
# Should return positive number if locked
|
||||
```
|
||||
|
||||
2. **Manual cleanup**:
|
||||
```bash
|
||||
# Delete duplicates in Google Calendar UI
|
||||
# Or use cleanup script (if available)
|
||||
```
|
||||
|
||||
3. **Improve deduplication logic**:
|
||||
```python
|
||||
# In calendar_sync_event_step.py
|
||||
# Add better event matching logic
|
||||
```
|
||||
|
||||
### Events Not Syncing
|
||||
|
||||
**Symptoms**: Advoware events nicht in Google Calendar
|
||||
|
||||
**Diagnose**:
|
||||
```bash
|
||||
# Check specific employee
|
||||
curl -X POST "http://localhost:3000/advoware/calendar/sync" \
|
||||
-H "Content-Type: application/json" \
|
||||
-d '{"kuerzel": "SB", "full_content": true}'
|
||||
|
||||
# Check logs for that employee
|
||||
sudo journalctl -u motia.service | grep "SB"
|
||||
|
||||
# Check if calendar exists
|
||||
python3 << 'EOF'
|
||||
from google.oauth2 import service_account
|
||||
from googleapiclient.discovery import build
|
||||
|
||||
creds = service_account.Credentials.from_service_account_file(
|
||||
'/opt/motia-app/service-account.json',
|
||||
scopes=['https://www.googleapis.com/auth/calendar']
|
||||
)
|
||||
service = build('calendar', 'v3', credentials=creds)
|
||||
result = service.calendarList().list().execute()
|
||||
for cal in result.get('items', []):
|
||||
if 'AW-SB' in cal['summary']:
|
||||
print(f"Found: {cal['summary']} - {cal['id']}")
|
||||
EOF
|
||||
```
|
||||
|
||||
**Solutions**:
|
||||
|
||||
1. **Calendar doesn't exist**:
|
||||
```bash
|
||||
# Will be auto-created on first sync
|
||||
# Force sync to trigger creation
|
||||
```
|
||||
|
||||
2. **Date range mismatch**:
|
||||
```python
|
||||
# Check FETCH_FROM and FETCH_TO in calendar_sync_event_step.py
|
||||
# Default: Previous year to 9 years ahead
|
||||
```
|
||||
|
||||
3. **Write protection enabled**:
|
||||
```bash
|
||||
# Check environment
|
||||
echo $ADVOWARE_WRITE_PROTECTION
|
||||
|
||||
# Should be "false" for two-way sync
|
||||
```
|
||||
|
||||
## Webhook Issues
|
||||
|
||||
### Webhooks Not Received
|
||||
|
||||
**Symptoms**: EspoCRM sendet Webhooks, aber keine Verarbeitung
|
||||
|
||||
**Diagnose**:
|
||||
```bash
|
||||
# Check if endpoint reachable
|
||||
curl -X POST "http://localhost:3000/vmh/webhook/beteiligte/create" \
|
||||
-H "Content-Type: application/json" \
|
||||
-d '[{"id": "test-123"}]'
|
||||
|
||||
# Check firewall
|
||||
sudo ufw status
|
||||
|
||||
# Check nginx logs (if using reverse proxy)
|
||||
sudo tail -f /var/log/nginx/motia-access.log
|
||||
sudo tail -f /var/log/nginx/motia-error.log
|
||||
```
|
||||
|
||||
**Solutions**:
|
||||
|
||||
1. **Firewall blocking**:
|
||||
```bash
|
||||
# Allow port (if direct access)
|
||||
sudo ufw allow 3000/tcp
|
||||
|
||||
# Or use reverse proxy (recommended)
|
||||
```
|
||||
|
||||
2. **Wrong URL in EspoCRM**:
|
||||
```bash
|
||||
# Verify URL in EspoCRM webhook configuration
|
||||
# Should be: https://your-domain.com/vmh/webhook/beteiligte/create
|
||||
```
|
||||
|
||||
3. **SSL certificate issues**:
|
||||
```bash
|
||||
# Check certificate
|
||||
openssl s_client -connect your-domain.com:443
|
||||
|
||||
# Renew certificate
|
||||
sudo certbot renew
|
||||
```
|
||||
|
||||
### Webhook Deduplication Not Working
|
||||
|
||||
**Symptoms**: Mehrfache Verarbeitung derselben Webhooks
|
||||
|
||||
**Diagnose**:
|
||||
```bash
|
||||
# Check Redis dedup sets
|
||||
redis-cli -n 1 SMEMBERS vmh:beteiligte:create_pending
|
||||
redis-cli -n 1 SMEMBERS vmh:beteiligte:update_pending
|
||||
redis-cli -n 1 SMEMBERS vmh:beteiligte:delete_pending
|
||||
|
||||
# Check for concurrent webhook processing
|
||||
sudo journalctl -u motia.service | grep "Webhook.*received"
|
||||
```
|
||||
|
||||
**Solutions**:
|
||||
|
||||
1. **Redis SET not working**:
|
||||
```bash
|
||||
# Test Redis SET operations
|
||||
redis-cli -n 1 SADD test_set "value1"
|
||||
redis-cli -n 1 SMEMBERS test_set
|
||||
redis-cli -n 1 DEL test_set
|
||||
```
|
||||
|
||||
2. **Clear dedup sets**:
|
||||
```bash
|
||||
# If corrupted
|
||||
redis-cli -n 1 DEL vmh:beteiligte:create_pending
|
||||
redis-cli -n 1 DEL vmh:beteiligte:update_pending
|
||||
redis-cli -n 1 DEL vmh:beteiligte:delete_pending
|
||||
```
|
||||
|
||||
## Performance Issues
|
||||
|
||||
### High CPU Usage
|
||||
|
||||
**Diagnose**:
|
||||
```bash
|
||||
# Check CPU usage
|
||||
top -p $(pgrep -f "motia start")
|
||||
|
||||
# Profile with Node.js
|
||||
# Already enabled with --inspect flag
|
||||
# Connect to chrome://inspect
|
||||
```
|
||||
|
||||
**Solutions**:
|
||||
|
||||
1. **Too many parallel syncs**:
|
||||
```bash
|
||||
# Reduce concurrent syncs
|
||||
# Adjust DEBUG_KUERZEL to process fewer employees
|
||||
```
|
||||
|
||||
2. **Infinite loop**:
|
||||
```bash
|
||||
# Check logs for repeated patterns
|
||||
sudo journalctl -u motia.service | tail -n 1000 | sort | uniq -c | sort -rn
|
||||
```
|
||||
|
||||
### High Memory Usage
|
||||
|
||||
**Diagnose**:
|
||||
```bash
|
||||
# Check memory
|
||||
ps aux | grep motia | awk '{print $6}'
|
||||
|
||||
# Heap snapshot (if enabled)
|
||||
kill -SIGUSR2 $(pgrep -f "motia start")
|
||||
# Snapshot saved to current directory
|
||||
```
|
||||
|
||||
**Solutions**:
|
||||
|
||||
1. **Increase memory limit**:
|
||||
```ini
|
||||
# In systemd service
|
||||
Environment=NODE_OPTIONS=--max-old-space-size=16384
|
||||
```
|
||||
|
||||
2. **Memory leak**:
|
||||
```bash
|
||||
# Restart service periodically
|
||||
# Add to crontab:
|
||||
0 3 * * * systemctl restart motia.service
|
||||
```
|
||||
|
||||
### Slow API Responses
|
||||
|
||||
**Diagnose**:
|
||||
```bash
|
||||
# Measure response time
|
||||
time curl "http://localhost:3000/advoware/proxy?endpoint=employees"
|
||||
|
||||
# Check for database/Redis latency
|
||||
redis-cli --latency
|
||||
```
|
||||
|
||||
**Solutions**:
|
||||
|
||||
1. **Redis slow**:
|
||||
```bash
|
||||
# Check slow log
|
||||
redis-cli SLOWLOG GET 10
|
||||
|
||||
# Optimize Redis
|
||||
redis-cli CONFIG SET tcp-backlog 511
|
||||
```
|
||||
|
||||
2. **Advoware API slow**:
|
||||
```bash
|
||||
# Increase timeout
|
||||
export ADVOWARE_API_TIMEOUT_SECONDS=60
|
||||
|
||||
# Add caching layer
|
||||
```
|
||||
|
||||
## Debugging Tools
|
||||
|
||||
### Enable Debug Logging
|
||||
|
||||
```bash
|
||||
# Set in systemd service
|
||||
Environment=MOTIA_LOG_LEVEL=debug
|
||||
|
||||
sudo systemctl daemon-reload
|
||||
sudo systemctl restart motia.service
|
||||
```
|
||||
|
||||
### Redis Debugging
|
||||
|
||||
```bash
|
||||
# Connect to Redis
|
||||
redis-cli
|
||||
|
||||
# Monitor all commands
|
||||
MONITOR
|
||||
|
||||
# Slow log
|
||||
SLOWLOG GET 10
|
||||
|
||||
# Info
|
||||
INFO all
|
||||
```
|
||||
|
||||
### Python Debugging
|
||||
|
||||
```python
|
||||
# Add to step code
|
||||
import pdb; pdb.set_trace()
|
||||
|
||||
# Or use logging
|
||||
context.logger.debug(f"Variable value: {variable}")
|
||||
```
|
||||
|
||||
### Node.js Debugging
|
||||
|
||||
```bash
|
||||
# Connect to inspector
|
||||
# Chrome DevTools: chrome://inspect
|
||||
# VSCode: Attach to Process
|
||||
```
|
||||
|
||||
## Getting Help
|
||||
|
||||
### Check Logs First
|
||||
|
||||
```bash
|
||||
# Last 100 lines
|
||||
sudo journalctl -u motia.service -n 100
|
||||
|
||||
# Errors only
|
||||
sudo journalctl -u motia.service -p err
|
||||
|
||||
# Specific time range
|
||||
sudo journalctl -u motia.service --since "1 hour ago"
|
||||
```
|
||||
|
||||
### Common Log Patterns
|
||||
|
||||
**Success**:
|
||||
```
|
||||
[INFO] Calendar sync completed for SB
|
||||
[INFO] VMH Webhook received
|
||||
```
|
||||
|
||||
**Warning**:
|
||||
```
|
||||
[WARNING] Rate limit approaching
|
||||
[WARNING] Lock already exists for SB
|
||||
```
|
||||
|
||||
**Error**:
|
||||
```
|
||||
[ERROR] Redis connection failed
|
||||
[ERROR] API call failed: 401 Unauthorized
|
||||
[ERROR] Unexpected error: ...
|
||||
```
|
||||
|
||||
### Collect Debug Information
|
||||
|
||||
```bash
|
||||
# System info
|
||||
uname -a
|
||||
node --version
|
||||
python3 --version
|
||||
|
||||
# Service status
|
||||
sudo systemctl status motia.service
|
||||
|
||||
# Recent logs
|
||||
sudo journalctl -u motia.service -n 200 > motia-logs.txt
|
||||
|
||||
# Redis info
|
||||
redis-cli INFO > redis-info.txt
|
||||
|
||||
# Configuration (redact secrets!)
|
||||
sudo systemctl show motia.service -p Environment > env.txt
|
||||
```
|
||||
|
||||
## Related Documentation
|
||||
|
||||
- [Architecture](ARCHITECTURE.md)
|
||||
- [Configuration](CONFIGURATION.md)
|
||||
- [Deployment](DEPLOYMENT.md)
|
||||
- [Development Guide](DEVELOPMENT.md)
|
||||
|
||||
296
bitbylaw/scripts/README.md
Normal file
296
bitbylaw/scripts/README.md
Normal file
@@ -0,0 +1,296 @@
|
||||
# Beteiligte Structure Comparison Tool
|
||||
|
||||
## Purpose
|
||||
|
||||
This helper script fetches entity data from both **EspoCRM** and **Advoware** to compare their data structures. This helps understand:
|
||||
|
||||
- What fields exist in each system
|
||||
- How field names differ
|
||||
- Potential field mappings for synchronization
|
||||
- Data type differences
|
||||
|
||||
## Usage
|
||||
|
||||
```bash
|
||||
cd /opt/motia-app
|
||||
|
||||
# Basic usage: Compare by EspoCRM ID (will auto-search in Advoware)
|
||||
python bitbylaw/scripts/compare_beteiligte.py <espocrm_entity_id>
|
||||
|
||||
# Advanced: Specify both IDs
|
||||
python bitbylaw/scripts/compare_beteiligte.py <espocrm_entity_id> <advoware_id>
|
||||
```
|
||||
|
||||
## Examples
|
||||
|
||||
```bash
|
||||
# Example 1: Fetch from EspoCRM and search in Advoware by name
|
||||
python bitbylaw/scripts/compare_beteiligte.py 64a3f2b8c9e1234567890abc
|
||||
|
||||
# Example 2: Fetch from both systems by ID
|
||||
python bitbylaw/scripts/compare_beteiligte.py 64a3f2b8c9e1234567890abc 12345
|
||||
|
||||
# Example 3: Using the virtual environment
|
||||
source python_modules/bin/activate
|
||||
python bitbylaw/scripts/compare_beteiligte.py 64a3f2b8c9e1234567890abc
|
||||
```
|
||||
|
||||
## Requirements
|
||||
|
||||
### Environment Variables
|
||||
|
||||
Make sure these are set in `.env` or environment:
|
||||
|
||||
```bash
|
||||
# EspoCRM
|
||||
ESPOCRM_API_BASE_URL=https://crm.bitbylaw.com/api/v1
|
||||
ESPOCRM_MARVIN_API_KEY=your_api_key_here
|
||||
|
||||
# Advoware
|
||||
ADVOWARE_API_BASE_URL=https://www2.advo-net.net:90/
|
||||
ADVOWARE_API_KEY=your_base64_encoded_key
|
||||
ADVOWARE_USER=your_user
|
||||
ADVOWARE_PASSWORD=your_password
|
||||
ADVOWARE_KANZLEI=your_kanzlei
|
||||
ADVOWARE_DATABASE=your_database
|
||||
# ... (see config.py for all required vars)
|
||||
```
|
||||
|
||||
### Dependencies
|
||||
|
||||
```bash
|
||||
pip install aiohttp redis python-dotenv requests
|
||||
```
|
||||
|
||||
## Output
|
||||
|
||||
The script produces:
|
||||
|
||||
### 1. Console Output
|
||||
|
||||
```
|
||||
================================================================================
|
||||
BETEILIGTE STRUCTURE COMPARISON TOOL
|
||||
================================================================================
|
||||
|
||||
EspoCRM Entity ID: 64a3f2b8c9e1234567890abc
|
||||
|
||||
Environment Check:
|
||||
----------------------------------------
|
||||
ESPOCRM_API_BASE_URL: https://crm.bitbylaw.com/api/v1
|
||||
ESPOCRM_API_KEY: ✓ Set
|
||||
ADVOWARE_API_BASE_URL: https://www2.advo-net.net:90/
|
||||
ADVOWARE_API_KEY: ✓ Set
|
||||
|
||||
================================================================================
|
||||
ESPOCRM - Fetching Beteiligter
|
||||
================================================================================
|
||||
|
||||
Trying entity type: Beteiligte
|
||||
|
||||
✓ Success! Found in Beteiligte
|
||||
|
||||
Entity Structure:
|
||||
--------------------------------------------------------------------------------
|
||||
{
|
||||
"id": "64a3f2b8c9e1234567890abc",
|
||||
"name": "Max Mustermann",
|
||||
"firstName": "Max",
|
||||
"lastName": "Mustermann",
|
||||
"email": "max@example.com",
|
||||
"phone": "+49123456789",
|
||||
...
|
||||
}
|
||||
|
||||
================================================================================
|
||||
ADVOWARE - Fetching Beteiligter
|
||||
================================================================================
|
||||
|
||||
Searching by name: Max Mustermann
|
||||
Trying endpoint: /contacts
|
||||
|
||||
✓ Found 2 results
|
||||
|
||||
Search Results:
|
||||
--------------------------------------------------------------------------------
|
||||
[
|
||||
{
|
||||
"id": 12345,
|
||||
"full_name": "Max Mustermann",
|
||||
"email": "max@example.com",
|
||||
...
|
||||
}
|
||||
]
|
||||
|
||||
================================================================================
|
||||
STRUCTURE COMPARISON
|
||||
================================================================================
|
||||
|
||||
EspoCRM Fields (25):
|
||||
----------------------------------------
|
||||
id (str)
|
||||
name (str)
|
||||
firstName (str)
|
||||
lastName (str)
|
||||
email (str)
|
||||
phone (str)
|
||||
...
|
||||
|
||||
Advoware Fields (30):
|
||||
----------------------------------------
|
||||
id (int)
|
||||
full_name (str)
|
||||
email (str)
|
||||
phone_number (str)
|
||||
...
|
||||
|
||||
Common Fields (5):
|
||||
----------------------------------------
|
||||
✓ id
|
||||
✓ email
|
||||
✗ phone
|
||||
EspoCRM: +49123456789
|
||||
Advoware: 0123456789
|
||||
|
||||
EspoCRM Only (20):
|
||||
----------------------------------------
|
||||
firstName
|
||||
lastName
|
||||
...
|
||||
|
||||
Advoware Only (25):
|
||||
----------------------------------------
|
||||
full_name
|
||||
phone_number
|
||||
...
|
||||
|
||||
Potential Field Mappings:
|
||||
----------------------------------------
|
||||
firstName → first_name
|
||||
lastName → last_name
|
||||
email → email
|
||||
phone → phone_number
|
||||
...
|
||||
|
||||
================================================================================
|
||||
Comparison saved to: /opt/motia-app/bitbylaw/scripts/beteiligte_comparison_result.json
|
||||
================================================================================
|
||||
```
|
||||
|
||||
### 2. JSON Output File
|
||||
|
||||
Saved to `bitbylaw/scripts/beteiligte_comparison_result.json`:
|
||||
|
||||
```json
|
||||
{
|
||||
"espocrm_data": {
|
||||
"id": "64a3f2b8c9e1234567890abc",
|
||||
"name": "Max Mustermann",
|
||||
...
|
||||
},
|
||||
"advoware_data": {
|
||||
"id": 12345,
|
||||
"full_name": "Max Mustermann",
|
||||
...
|
||||
},
|
||||
"comparison": {
|
||||
"espo_fields": ["id", "name", "firstName", ...],
|
||||
"advo_fields": ["id", "full_name", "email", ...],
|
||||
"common": ["id", "email"],
|
||||
"espo_only": ["firstName", "lastName", ...],
|
||||
"advo_only": ["full_name", "phone_number", ...],
|
||||
"suggested_mappings": [
|
||||
["firstName", "first_name"],
|
||||
["lastName", "last_name"],
|
||||
...
|
||||
]
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
## How It Works
|
||||
|
||||
### 1. EspoCRM Fetch
|
||||
|
||||
The script tries multiple entity types to find the data:
|
||||
- `Beteiligte` (custom VMH entity)
|
||||
- `Contact` (standard)
|
||||
- `Account` (standard)
|
||||
- `Lead` (standard)
|
||||
|
||||
### 2. Advoware Fetch
|
||||
|
||||
**By ID (if provided):**
|
||||
- Tries: `/contacts/{id}`, `/parties/{id}`, `/clients/{id}`
|
||||
|
||||
**By Name (if EspoCRM data available):**
|
||||
- Searches: `/contacts?search=...`, `/parties?search=...`, `/clients?search=...`
|
||||
|
||||
### 3. Comparison
|
||||
|
||||
- Lists all fields from both systems
|
||||
- Identifies common fields (same name)
|
||||
- Shows values for common fields
|
||||
- Suggests mappings based on naming patterns
|
||||
- Exports full comparison to JSON
|
||||
|
||||
## Troubleshooting
|
||||
|
||||
### "ESPOCRM_API_KEY not set"
|
||||
|
||||
```bash
|
||||
# Check if .env exists and has the key
|
||||
cat .env | grep ESPOCRM_MARVIN_API_KEY
|
||||
|
||||
# Or set it manually
|
||||
export ESPOCRM_MARVIN_API_KEY=your_key_here
|
||||
```
|
||||
|
||||
### "Authentication failed - check API key"
|
||||
|
||||
1. Verify API key in EspoCRM admin panel
|
||||
2. Check API User permissions
|
||||
3. Ensure API User has access to entity type
|
||||
|
||||
### "Entity not found"
|
||||
|
||||
- Check if entity ID is correct
|
||||
- Verify entity type exists in EspoCRM
|
||||
- Check API User permissions for that entity
|
||||
|
||||
### "Advoware token error"
|
||||
|
||||
- Verify all Advoware credentials in `.env`
|
||||
- Check HMAC signature generation
|
||||
- Ensure API key is base64 encoded
|
||||
- Test token generation separately
|
||||
|
||||
## Next Steps
|
||||
|
||||
After running this script:
|
||||
|
||||
1. **Review JSON output** - Check `beteiligte_comparison_result.json`
|
||||
2. **Define mappings** - Create mapping table based on suggestions
|
||||
3. **Implement mapper** - Create transformation functions
|
||||
4. **Test sync** - Use mappings in sync event step
|
||||
|
||||
Example mapping implementation:
|
||||
|
||||
```python
|
||||
def map_espocrm_to_advoware(espo_entity: dict) -> dict:
|
||||
"""Transform EspoCRM Beteiligter to Advoware format"""
|
||||
return {
|
||||
'first_name': espo_entity.get('firstName'),
|
||||
'last_name': espo_entity.get('lastName'),
|
||||
'email': espo_entity.get('email'),
|
||||
'phone_number': espo_entity.get('phone'),
|
||||
# Add more mappings based on comparison...
|
||||
}
|
||||
```
|
||||
|
||||
## Related Files
|
||||
|
||||
- [services/espocrm.py](../services/espocrm.py) - EspoCRM API client
|
||||
- [services/advoware.py](../services/advoware.py) - Advoware API client
|
||||
- [services/ESPOCRM_SERVICE.md](../services/ESPOCRM_SERVICE.md) - EspoCRM docs
|
||||
- [config.py](../config.py) - Configuration
|
||||
399
bitbylaw/scripts/beteiligte_comparison_result.json
Normal file
399
bitbylaw/scripts/beteiligte_comparison_result.json
Normal file
@@ -0,0 +1,399 @@
|
||||
{
|
||||
"espocrm_data": {
|
||||
"id": "68e4af00172be7924",
|
||||
"name": "dasdas dasdasdas dasdasdas",
|
||||
"deleted": false,
|
||||
"salutationName": null,
|
||||
"rechtsform": "GmbH",
|
||||
"firmenname": "Filli llu GmbH",
|
||||
"firstName": "dasdasdas",
|
||||
"lastName": "dasdas",
|
||||
"dateOfBirth": null,
|
||||
"description": null,
|
||||
"emailAddress": "meier@meier.de",
|
||||
"phoneNumber": null,
|
||||
"createdAt": "2025-10-07 06:11:12",
|
||||
"modifiedAt": "2026-01-23 21:58:41",
|
||||
"betnr": 1234,
|
||||
"advowareLastSync": null,
|
||||
"syncStatus": "clean",
|
||||
"handelsregisterNummer": "12244546",
|
||||
"handelsregisterArt": "HRB",
|
||||
"disgTyp": "Unbekannt",
|
||||
"middleName": "dasdasdas",
|
||||
"emailAddressIsOptedOut": false,
|
||||
"emailAddressIsInvalid": false,
|
||||
"phoneNumberIsOptedOut": null,
|
||||
"phoneNumberIsInvalid": null,
|
||||
"streamUpdatedAt": null,
|
||||
"emailAddressData": [
|
||||
{
|
||||
"emailAddress": "meier@meier.de",
|
||||
"lower": "meier@meier.de",
|
||||
"primary": true,
|
||||
"optOut": false,
|
||||
"invalid": false
|
||||
},
|
||||
{
|
||||
"emailAddress": "a@r028tuj08wefj0w8efjw0d.de",
|
||||
"lower": "a@r028tuj08wefj0w8efjw0d.de",
|
||||
"primary": false,
|
||||
"optOut": false,
|
||||
"invalid": false
|
||||
}
|
||||
],
|
||||
"phoneNumberData": [],
|
||||
"createdById": "68d65929f18c2afef",
|
||||
"createdByName": "Admin",
|
||||
"modifiedById": "68d65929f18c2afef",
|
||||
"modifiedByName": "Admin",
|
||||
"assignedUserId": null,
|
||||
"assignedUserName": null,
|
||||
"teamsIds": [],
|
||||
"teamsNames": {},
|
||||
"adressensIds": [],
|
||||
"adressensNames": {},
|
||||
"calls1Ids": [],
|
||||
"calls1Names": {},
|
||||
"bankverbindungensIds": [],
|
||||
"bankverbindungensNames": {},
|
||||
"isFollowed": false,
|
||||
"followersIds": [],
|
||||
"followersNames": {}
|
||||
},
|
||||
"advoware_data": {
|
||||
"betNr": 104860,
|
||||
"kommunikation": [
|
||||
{
|
||||
"rowId": "FBABAAAANJFGABAAGJDOAEAPAAAAAPGFPDAFAAAA",
|
||||
"id": 88002,
|
||||
"betNr": 104860,
|
||||
"kommArt": 0,
|
||||
"tlf": "0511/12345-60",
|
||||
"bemerkung": null,
|
||||
"kommKz": 0,
|
||||
"online": false
|
||||
},
|
||||
{
|
||||
"rowId": "FBABAAAABBLIABAAGIDOAEAPAAAAAPHBEOAEAAAA",
|
||||
"id": 114914,
|
||||
"betNr": 104860,
|
||||
"kommArt": 0,
|
||||
"tlf": "kanzlei@ralup.de",
|
||||
"bemerkung": null,
|
||||
"kommKz": 0,
|
||||
"online": true
|
||||
}
|
||||
],
|
||||
"kontaktpersonen": [],
|
||||
"beteiligungen": [
|
||||
{
|
||||
"rowId": "LAADAAAAAHMDABAAGAAEIPBAAAAADGKEMPAFAAAA",
|
||||
"beteiligtenArt": "Sachverständiger",
|
||||
"akte": {
|
||||
"rowId": "",
|
||||
"nr": 2020001684,
|
||||
"az": "1684/20",
|
||||
"rubrum": "Siggel / Siggel",
|
||||
"referat": "SON",
|
||||
"wegen": "Bruderzwist II",
|
||||
"ablage": 1,
|
||||
"abgelegt": null
|
||||
}
|
||||
},
|
||||
{
|
||||
"rowId": "LAADAAAAPGKFABAAGAAEIPBAAAAADGJOMBABAAAA",
|
||||
"beteiligtenArt": "Sachverständiger",
|
||||
"akte": {
|
||||
"rowId": "",
|
||||
"nr": 2020000203,
|
||||
"az": "203/20",
|
||||
"rubrum": "Siggel / Siggel",
|
||||
"referat": "SON",
|
||||
"wegen": "Bruderzwist",
|
||||
"ablage": 1,
|
||||
"abgelegt": null
|
||||
}
|
||||
},
|
||||
{
|
||||
"rowId": "LAADAAAAPJAGACAAGAAEIPBAAAAADGLDFGADAAAA",
|
||||
"beteiligtenArt": "Mandant",
|
||||
"akte": {
|
||||
"rowId": "",
|
||||
"nr": 2019001145,
|
||||
"az": "1145/19",
|
||||
"rubrum": "Siggel / Siggel LALA",
|
||||
"referat": "VMH",
|
||||
"wegen": null,
|
||||
"ablage": 0,
|
||||
"abgelegt": null
|
||||
}
|
||||
}
|
||||
],
|
||||
"adressen": [
|
||||
{
|
||||
"rowId": "KOADAAAAALNFAAAAFPAEIPBAAAAADGGPGAAJAAAA",
|
||||
"id": 0,
|
||||
"beteiligterId": 104860,
|
||||
"reihenfolgeIndex": 1,
|
||||
"strasse": "Musterstraße 12",
|
||||
"plz": "12345",
|
||||
"ort": "Musterort",
|
||||
"land": "D",
|
||||
"postfach": null,
|
||||
"postfachPLZ": null,
|
||||
"anschrift": "Frau\r\nAngela Mustermanns\r\nVorzimmer\r\nMusterstraße 12\r\n12345 Musterort",
|
||||
"standardAnschrift": false,
|
||||
"bemerkung": null,
|
||||
"gueltigVon": null,
|
||||
"gueltigBis": null
|
||||
}
|
||||
],
|
||||
"bankkverbindungen": [
|
||||
{
|
||||
"rowId": "EPABAAAAHBNFAAAAFPNBCGAAAAAAAPDIJDAJAAAA",
|
||||
"id": 54665,
|
||||
"bank": null,
|
||||
"ktoNr": null,
|
||||
"blz": null,
|
||||
"iban": null,
|
||||
"bic": null,
|
||||
"kontoinhaber": null,
|
||||
"mandatsreferenz": null,
|
||||
"mandatVom": null
|
||||
}
|
||||
],
|
||||
"rowId": "EMABAAAAFBNFAAAAFOAEIPBAAAAAAOMNKPAHAAAA",
|
||||
"id": 104860,
|
||||
"anschrift": "Frau\r\nAngela Mustermanns\r\nVorzimmer\r\nMusterstraße 12\r\n12345 Musterort",
|
||||
"strasse": "Musterstraße 12",
|
||||
"plz": "12345",
|
||||
"ort": "Musterort",
|
||||
"email": null,
|
||||
"emailGesch": "kanzlei@ralup.de",
|
||||
"mobil": null,
|
||||
"internet": null,
|
||||
"telGesch": "0511/12345-60",
|
||||
"telPrivat": null,
|
||||
"faxGesch": null,
|
||||
"faxPrivat": null,
|
||||
"autotelefon": null,
|
||||
"sonstige": null,
|
||||
"ePost": null,
|
||||
"bea": null,
|
||||
"art": null,
|
||||
"vorname": "Angela",
|
||||
"name": "Mustermanns",
|
||||
"kurzname": null,
|
||||
"geburtsname": null,
|
||||
"familienstand": null,
|
||||
"titel": null,
|
||||
"anrede": "Frau",
|
||||
"bAnrede": "Sehr geehrte Frau Mustermanns,",
|
||||
"geburtsdatum": null,
|
||||
"sterbedatum": null,
|
||||
"zusatz": "Vorzimmer",
|
||||
"rechtsform": "Frau",
|
||||
"geaendertAm": null,
|
||||
"geaendertVon": null,
|
||||
"angelegtAm": null,
|
||||
"angelegtVon": null,
|
||||
"handelsRegisterNummer": null,
|
||||
"registergericht": null
|
||||
},
|
||||
"comparison": {
|
||||
"espo_fields": [
|
||||
"emailAddressIsInvalid",
|
||||
"followersNames",
|
||||
"id",
|
||||
"handelsregisterNummer",
|
||||
"teamsNames",
|
||||
"assignedUserName",
|
||||
"modifiedAt",
|
||||
"modifiedByName",
|
||||
"betnr",
|
||||
"middleName",
|
||||
"disgTyp",
|
||||
"bankverbindungensNames",
|
||||
"phoneNumberIsOptedOut",
|
||||
"adressensIds",
|
||||
"emailAddressData",
|
||||
"deleted",
|
||||
"teamsIds",
|
||||
"phoneNumber",
|
||||
"isFollowed",
|
||||
"advowareLastSync",
|
||||
"createdById",
|
||||
"createdAt",
|
||||
"calls1Ids",
|
||||
"handelsregisterArt",
|
||||
"name",
|
||||
"phoneNumberIsInvalid",
|
||||
"rechtsform",
|
||||
"emailAddress",
|
||||
"emailAddressIsOptedOut",
|
||||
"firmenname",
|
||||
"description",
|
||||
"adressensNames",
|
||||
"createdByName",
|
||||
"lastName",
|
||||
"assignedUserId",
|
||||
"salutationName",
|
||||
"bankverbindungensIds",
|
||||
"phoneNumberData",
|
||||
"dateOfBirth",
|
||||
"modifiedById",
|
||||
"firstName",
|
||||
"followersIds",
|
||||
"streamUpdatedAt",
|
||||
"syncStatus",
|
||||
"calls1Names"
|
||||
],
|
||||
"advo_fields": [
|
||||
"kontaktpersonen",
|
||||
"rowId",
|
||||
"id",
|
||||
"angelegtVon",
|
||||
"zusatz",
|
||||
"bAnrede",
|
||||
"faxGesch",
|
||||
"bankkverbindungen",
|
||||
"geburtsname",
|
||||
"plz",
|
||||
"adressen",
|
||||
"kurzname",
|
||||
"telPrivat",
|
||||
"anrede",
|
||||
"sonstige",
|
||||
"email",
|
||||
"titel",
|
||||
"sterbedatum",
|
||||
"faxPrivat",
|
||||
"autotelefon",
|
||||
"name",
|
||||
"kommunikation",
|
||||
"rechtsform",
|
||||
"art",
|
||||
"geaendertAm",
|
||||
"anschrift",
|
||||
"beteiligungen",
|
||||
"bea",
|
||||
"handelsRegisterNummer",
|
||||
"registergericht",
|
||||
"internet",
|
||||
"ort",
|
||||
"geburtsdatum",
|
||||
"angelegtAm",
|
||||
"mobil",
|
||||
"emailGesch",
|
||||
"ePost",
|
||||
"strasse",
|
||||
"vorname",
|
||||
"familienstand",
|
||||
"betNr",
|
||||
"geaendertVon",
|
||||
"telGesch"
|
||||
],
|
||||
"common": [
|
||||
"name",
|
||||
"id",
|
||||
"rechtsform"
|
||||
],
|
||||
"espo_only": [
|
||||
"emailAddressIsInvalid",
|
||||
"followersNames",
|
||||
"handelsregisterNummer",
|
||||
"teamsNames",
|
||||
"assignedUserName",
|
||||
"modifiedAt",
|
||||
"modifiedByName",
|
||||
"betnr",
|
||||
"middleName",
|
||||
"disgTyp",
|
||||
"bankverbindungensNames",
|
||||
"phoneNumberIsOptedOut",
|
||||
"adressensIds",
|
||||
"emailAddressData",
|
||||
"deleted",
|
||||
"teamsIds",
|
||||
"phoneNumber",
|
||||
"isFollowed",
|
||||
"advowareLastSync",
|
||||
"createdById",
|
||||
"createdAt",
|
||||
"calls1Ids",
|
||||
"handelsregisterArt",
|
||||
"phoneNumberIsInvalid",
|
||||
"emailAddress",
|
||||
"emailAddressIsOptedOut",
|
||||
"firmenname",
|
||||
"description",
|
||||
"adressensNames",
|
||||
"createdByName",
|
||||
"lastName",
|
||||
"assignedUserId",
|
||||
"salutationName",
|
||||
"bankverbindungensIds",
|
||||
"phoneNumberData",
|
||||
"dateOfBirth",
|
||||
"modifiedById",
|
||||
"firstName",
|
||||
"followersIds",
|
||||
"streamUpdatedAt",
|
||||
"syncStatus",
|
||||
"calls1Names"
|
||||
],
|
||||
"advo_only": [
|
||||
"kontaktpersonen",
|
||||
"rowId",
|
||||
"angelegtVon",
|
||||
"zusatz",
|
||||
"bAnrede",
|
||||
"faxGesch",
|
||||
"bankkverbindungen",
|
||||
"geburtsname",
|
||||
"plz",
|
||||
"adressen",
|
||||
"kurzname",
|
||||
"telPrivat",
|
||||
"anrede",
|
||||
"sonstige",
|
||||
"email",
|
||||
"titel",
|
||||
"sterbedatum",
|
||||
"autotelefon",
|
||||
"faxPrivat",
|
||||
"kommunikation",
|
||||
"art",
|
||||
"geaendertAm",
|
||||
"anschrift",
|
||||
"beteiligungen",
|
||||
"bea",
|
||||
"handelsRegisterNummer",
|
||||
"registergericht",
|
||||
"internet",
|
||||
"ort",
|
||||
"geburtsdatum",
|
||||
"angelegtAm",
|
||||
"mobil",
|
||||
"emailGesch",
|
||||
"ePost",
|
||||
"strasse",
|
||||
"vorname",
|
||||
"familienstand",
|
||||
"betNr",
|
||||
"geaendertVon",
|
||||
"telGesch"
|
||||
],
|
||||
"suggested_mappings": [
|
||||
[
|
||||
"name",
|
||||
"name"
|
||||
],
|
||||
[
|
||||
"emailAddress",
|
||||
"email"
|
||||
]
|
||||
]
|
||||
}
|
||||
}
|
||||
323
bitbylaw/scripts/compare_beteiligte.py
Executable file
323
bitbylaw/scripts/compare_beteiligte.py
Executable file
@@ -0,0 +1,323 @@
|
||||
#!/usr/bin/env python3
|
||||
"""
|
||||
Helper-Script zum Vergleichen der Beteiligten-Strukturen zwischen Advoware und EspoCRM.
|
||||
|
||||
Usage:
|
||||
python scripts/compare_beteiligte.py <entity_id_espocrm> [advoware_id]
|
||||
|
||||
Examples:
|
||||
# Vergleiche EspoCRM Beteiligten (automatische Suche in Advoware)
|
||||
python scripts/compare_beteiligte.py 64a3f2b8c9e1234567890abc
|
||||
|
||||
# Vergleiche mit spezifischer Advoware ID
|
||||
python scripts/compare_beteiligte.py 64a3f2b8c9e1234567890abc 12345
|
||||
"""
|
||||
|
||||
import sys
|
||||
import asyncio
|
||||
import json
|
||||
import os
|
||||
from pathlib import Path
|
||||
|
||||
# Add bitbylaw directory to path for imports
|
||||
bitbylaw_dir = Path(__file__).parent.parent
|
||||
sys.path.insert(0, str(bitbylaw_dir))
|
||||
|
||||
from services.espocrm import EspoCRMAPI
|
||||
from services.advoware import AdvowareAPI
|
||||
from config import Config
|
||||
|
||||
|
||||
class SimpleContext:
|
||||
"""Simple context for logging"""
|
||||
class Logger:
|
||||
def info(self, msg):
|
||||
print(f"[INFO] {msg}")
|
||||
|
||||
def error(self, msg):
|
||||
print(f"[ERROR] {msg}")
|
||||
|
||||
def debug(self, msg):
|
||||
print(f"[DEBUG] {msg}")
|
||||
|
||||
def warning(self, msg):
|
||||
print(f"[WARNING] {msg}")
|
||||
|
||||
def __init__(self):
|
||||
self.logger = self.Logger()
|
||||
|
||||
|
||||
async def fetch_from_espocrm(entity_id: str):
|
||||
"""Fetch Beteiligter from EspoCRM"""
|
||||
print("\n" + "="*80)
|
||||
print("ESPOCRM - Fetching Beteiligter")
|
||||
print("="*80)
|
||||
|
||||
context = SimpleContext()
|
||||
espo = EspoCRMAPI(context=context)
|
||||
|
||||
try:
|
||||
# Try different entity types that might contain Beteiligte
|
||||
entity_types = ['CBeteiligte', 'Beteiligte', 'Contact', 'Account', 'Lead', 'CVmhErstgespraech', 'CVmhBeteiligte']
|
||||
|
||||
for entity_type in entity_types:
|
||||
try:
|
||||
print(f"\nTrying entity type: {entity_type}")
|
||||
result = await espo.get_entity(entity_type, entity_id)
|
||||
|
||||
print(f"\n✓ Success! Found in {entity_type}")
|
||||
print(f"\nEntity Structure:")
|
||||
print("-" * 80)
|
||||
print(json.dumps(result, indent=2, ensure_ascii=False))
|
||||
|
||||
return result
|
||||
|
||||
except Exception as e:
|
||||
print(f" ✗ Not found in {entity_type}: {e}")
|
||||
continue
|
||||
|
||||
print("\n✗ Entity not found in any known entity type")
|
||||
return None
|
||||
|
||||
except Exception as e:
|
||||
print(f"\n✗ Error fetching from EspoCRM: {e}")
|
||||
return None
|
||||
|
||||
|
||||
async def fetch_from_advoware(advoware_id: str = None, search_name: str = None):
|
||||
"""Fetch Beteiligter from Advoware"""
|
||||
print("\n" + "="*80)
|
||||
print("ADVOWARE - Fetching Beteiligter")
|
||||
print("="*80)
|
||||
|
||||
context = SimpleContext()
|
||||
advo = AdvowareAPI(context=context)
|
||||
|
||||
try:
|
||||
# Try to fetch by ID if provided
|
||||
if advoware_id:
|
||||
print(f"\nFetching by ID: {advoware_id}")
|
||||
# Try correct Advoware endpoint
|
||||
endpoints = [
|
||||
f'/api/v1/advonet/Beteiligte/{advoware_id}',
|
||||
]
|
||||
|
||||
for endpoint in endpoints:
|
||||
try:
|
||||
print(f" Trying endpoint: {endpoint}")
|
||||
result = await advo.api_call(endpoint, method='GET')
|
||||
|
||||
if result:
|
||||
# Advoware gibt oft Listen zurück, nehme erstes Element
|
||||
if isinstance(result, list) and len(result) > 0:
|
||||
result = result[0]
|
||||
|
||||
print(f"\n✓ Success! Found at {endpoint}")
|
||||
print(f"\nEntity Structure:")
|
||||
print("-" * 80)
|
||||
print(json.dumps(result, indent=2, ensure_ascii=False))
|
||||
return result
|
||||
|
||||
except Exception as e:
|
||||
print(f" ✗ Not found at {endpoint}: {e}")
|
||||
continue
|
||||
|
||||
# Try to search by name if EspoCRM data available
|
||||
if search_name:
|
||||
print(f"\nSearching by name: {search_name}")
|
||||
search_endpoints = [
|
||||
'/api/v1/advonet/Beteiligte',
|
||||
]
|
||||
|
||||
for endpoint in search_endpoints:
|
||||
try:
|
||||
print(f" Trying endpoint: {endpoint}")
|
||||
result = await advo.api_call(
|
||||
endpoint,
|
||||
method='GET',
|
||||
params={'search': search_name, 'limit': 5}
|
||||
)
|
||||
|
||||
if result and (isinstance(result, list) and len(result) > 0 or
|
||||
isinstance(result, dict) and result.get('data')):
|
||||
print(f"\n✓ Found {len(result) if isinstance(result, list) else len(result.get('data', []))} results")
|
||||
print(f"\nSearch Results:")
|
||||
print("-" * 80)
|
||||
print(json.dumps(result, indent=2, ensure_ascii=False))
|
||||
return result
|
||||
|
||||
except Exception as e:
|
||||
print(f" ✗ Search failed at {endpoint}: {e}")
|
||||
continue
|
||||
|
||||
print("\n✗ Entity not found in Advoware")
|
||||
return None
|
||||
|
||||
except Exception as e:
|
||||
print(f"\n✗ Error fetching from Advoware: {e}")
|
||||
import traceback
|
||||
traceback.print_exc()
|
||||
return None
|
||||
|
||||
|
||||
async def compare_structures(espo_data: dict, advo_data: dict):
|
||||
"""Compare field structures between EspoCRM and Advoware"""
|
||||
print("\n" + "="*80)
|
||||
print("STRUCTURE COMPARISON")
|
||||
print("="*80)
|
||||
|
||||
if not espo_data or not advo_data:
|
||||
print("\n⚠ Cannot compare - missing data from one or both systems")
|
||||
return
|
||||
|
||||
# Extract fields
|
||||
espo_fields = set(espo_data.keys()) if isinstance(espo_data, dict) else set()
|
||||
|
||||
# Handle Advoware data structure (might be nested)
|
||||
if isinstance(advo_data, dict):
|
||||
if 'data' in advo_data:
|
||||
advo_data = advo_data['data']
|
||||
if isinstance(advo_data, list) and len(advo_data) > 0:
|
||||
advo_data = advo_data[0]
|
||||
|
||||
advo_fields = set(advo_data.keys()) if isinstance(advo_data, dict) else set()
|
||||
|
||||
print(f"\nEspoCRM Fields ({len(espo_fields)}):")
|
||||
print("-" * 40)
|
||||
for field in sorted(espo_fields):
|
||||
value = espo_data.get(field)
|
||||
value_type = type(value).__name__
|
||||
print(f" {field:<30} ({value_type})")
|
||||
|
||||
print(f"\nAdvoware Fields ({len(advo_fields)}):")
|
||||
print("-" * 40)
|
||||
for field in sorted(advo_fields):
|
||||
value = advo_data.get(field)
|
||||
value_type = type(value).__name__
|
||||
print(f" {field:<30} ({value_type})")
|
||||
|
||||
# Find common fields (potential mappings)
|
||||
common = espo_fields & advo_fields
|
||||
espo_only = espo_fields - advo_fields
|
||||
advo_only = advo_fields - espo_fields
|
||||
|
||||
print(f"\nCommon Fields ({len(common)}):")
|
||||
print("-" * 40)
|
||||
for field in sorted(common):
|
||||
espo_val = espo_data.get(field)
|
||||
advo_val = advo_data.get(field)
|
||||
match = "✓" if espo_val == advo_val else "✗"
|
||||
print(f" {match} {field}")
|
||||
if espo_val != advo_val:
|
||||
print(f" EspoCRM: {espo_val}")
|
||||
print(f" Advoware: {advo_val}")
|
||||
|
||||
print(f"\nEspoCRM Only ({len(espo_only)}):")
|
||||
print("-" * 40)
|
||||
for field in sorted(espo_only):
|
||||
print(f" {field}")
|
||||
|
||||
print(f"\nAdvoware Only ({len(advo_only)}):")
|
||||
print("-" * 40)
|
||||
for field in sorted(advo_only):
|
||||
print(f" {field}")
|
||||
|
||||
# Suggest potential mappings based on field names
|
||||
print(f"\nPotential Field Mappings:")
|
||||
print("-" * 40)
|
||||
|
||||
mapping_suggestions = []
|
||||
|
||||
# Common name patterns
|
||||
name_patterns = [
|
||||
('name', 'name'),
|
||||
('firstName', 'first_name'),
|
||||
('lastName', 'last_name'),
|
||||
('email', 'email'),
|
||||
('emailAddress', 'email'),
|
||||
('phone', 'phone'),
|
||||
('phoneNumber', 'phone_number'),
|
||||
('address', 'address'),
|
||||
('street', 'street'),
|
||||
('city', 'city'),
|
||||
('postalCode', 'postal_code'),
|
||||
('zipCode', 'postal_code'),
|
||||
('country', 'country'),
|
||||
]
|
||||
|
||||
for espo_field, advo_field in name_patterns:
|
||||
if espo_field in espo_fields and advo_field in advo_fields:
|
||||
mapping_suggestions.append((espo_field, advo_field))
|
||||
print(f" {espo_field:<30} → {advo_field}")
|
||||
|
||||
return {
|
||||
'espo_fields': list(espo_fields),
|
||||
'advo_fields': list(advo_fields),
|
||||
'common': list(common),
|
||||
'espo_only': list(espo_only),
|
||||
'advo_only': list(advo_only),
|
||||
'suggested_mappings': mapping_suggestions
|
||||
}
|
||||
|
||||
|
||||
async def main():
|
||||
"""Main function"""
|
||||
if len(sys.argv) < 2:
|
||||
print(__doc__)
|
||||
sys.exit(1)
|
||||
|
||||
espocrm_id = sys.argv[1]
|
||||
advoware_id = sys.argv[2] if len(sys.argv) > 2 else None
|
||||
|
||||
print("\n" + "="*80)
|
||||
print("BETEILIGTE STRUCTURE COMPARISON TOOL")
|
||||
print("="*80)
|
||||
print(f"\nEspoCRM Entity ID: {espocrm_id}")
|
||||
if advoware_id:
|
||||
print(f"Advoware ID: {advoware_id}")
|
||||
|
||||
# Check environment variables
|
||||
print("\nEnvironment Check:")
|
||||
print("-" * 40)
|
||||
print(f"ESPOCRM_API_BASE_URL: {Config.ESPOCRM_API_BASE_URL}")
|
||||
print(f"ESPOCRM_API_KEY: {'✓ Set' if Config.ESPOCRM_API_KEY else '✗ Missing'}")
|
||||
print(f"ADVOWARE_API_BASE_URL: {Config.ADVOWARE_API_BASE_URL}")
|
||||
print(f"ADVOWARE_API_KEY: {'✓ Set' if Config.ADVOWARE_API_KEY else '✗ Missing'}")
|
||||
|
||||
# Fetch from EspoCRM
|
||||
espo_data = await fetch_from_espocrm(espocrm_id)
|
||||
|
||||
# Extract name for Advoware search
|
||||
search_name = None
|
||||
if espo_data:
|
||||
search_name = (
|
||||
espo_data.get('name') or
|
||||
f"{espo_data.get('firstName', '')} {espo_data.get('lastName', '')}".strip() or
|
||||
None
|
||||
)
|
||||
|
||||
# Fetch from Advoware
|
||||
advo_data = await fetch_from_advoware(advoware_id, search_name)
|
||||
|
||||
# Compare structures
|
||||
if espo_data or advo_data:
|
||||
comparison = await compare_structures(espo_data, advo_data)
|
||||
|
||||
# Save comparison to file
|
||||
output_file = Path(__file__).parent / 'beteiligte_comparison_result.json'
|
||||
with open(output_file, 'w', encoding='utf-8') as f:
|
||||
json.dump({
|
||||
'espocrm_data': espo_data,
|
||||
'advoware_data': advo_data,
|
||||
'comparison': comparison
|
||||
}, f, indent=2, ensure_ascii=False)
|
||||
|
||||
print(f"\n\n{'='*80}")
|
||||
print(f"Comparison saved to: {output_file}")
|
||||
print(f"{'='*80}\n")
|
||||
else:
|
||||
print("\n⚠ No data available for comparison")
|
||||
|
||||
|
||||
if __name__ == '__main__':
|
||||
asyncio.run(main())
|
||||
403
bitbylaw/services/ESPOCRM_SERVICE.md
Normal file
403
bitbylaw/services/ESPOCRM_SERVICE.md
Normal file
@@ -0,0 +1,403 @@
|
||||
# EspoCRM API Service
|
||||
|
||||
## Overview
|
||||
|
||||
Python client for EspoCRM REST API integration. Provides type-safe, async operations for managing entities in EspoCRM.
|
||||
|
||||
## Features
|
||||
|
||||
- ✅ API Key authentication
|
||||
- ✅ Async/await support (aiohttp)
|
||||
- ✅ Full CRUD operations
|
||||
- ✅ Entity search and filtering
|
||||
- ✅ Error handling with custom exceptions
|
||||
- ✅ Optional Redis integration for caching
|
||||
- ✅ Logging via Motia context
|
||||
|
||||
## Installation
|
||||
|
||||
```python
|
||||
from services.espocrm import EspoCRMAPI
|
||||
|
||||
# Initialize with optional context for logging
|
||||
espo = EspoCRMAPI(context=context)
|
||||
```
|
||||
|
||||
## Configuration
|
||||
|
||||
Add to `.env` or environment:
|
||||
|
||||
```bash
|
||||
# EspoCRM API Configuration
|
||||
ESPOCRM_API_BASE_URL=https://crm.bitbylaw.com/api/v1
|
||||
ESPOCRM_MARVIN_API_KEY=your_api_key_here
|
||||
ESPOCRM_API_TIMEOUT_SECONDS=30
|
||||
```
|
||||
|
||||
Required in `config.py`:
|
||||
|
||||
```python
|
||||
class Config:
|
||||
ESPOCRM_API_BASE_URL = os.getenv('ESPOCRM_API_BASE_URL', 'https://crm.bitbylaw.com/api/v1')
|
||||
ESPOCRM_API_KEY = os.getenv('ESPOCRM_MARVIN_API_KEY', '')
|
||||
ESPOCRM_API_TIMEOUT_SECONDS = int(os.getenv('ESPOCRM_API_TIMEOUT_SECONDS', '30'))
|
||||
```
|
||||
|
||||
## API Methods
|
||||
|
||||
### Get Single Entity
|
||||
|
||||
```python
|
||||
async def get_entity(entity_type: str, entity_id: str) -> Dict[str, Any]
|
||||
```
|
||||
|
||||
**Usage:**
|
||||
```python
|
||||
# Get Beteiligter by ID
|
||||
result = await espo.get_entity('Beteiligte', '64a3f2b8c9e1234567890abc')
|
||||
print(result['name'])
|
||||
```
|
||||
|
||||
### List Entities
|
||||
|
||||
```python
|
||||
async def list_entities(
|
||||
entity_type: str,
|
||||
where: Optional[List[Dict]] = None,
|
||||
select: Optional[str] = None,
|
||||
order_by: Optional[str] = None,
|
||||
offset: int = 0,
|
||||
max_size: int = 50
|
||||
) -> Dict[str, Any]
|
||||
```
|
||||
|
||||
**Usage:**
|
||||
```python
|
||||
# List all Beteiligte with status "Active"
|
||||
result = await espo.list_entities(
|
||||
'Beteiligte',
|
||||
where=[{
|
||||
'type': 'equals',
|
||||
'attribute': 'status',
|
||||
'value': 'Active'
|
||||
}],
|
||||
select='id,name,email',
|
||||
max_size=100
|
||||
)
|
||||
|
||||
for entity in result['list']:
|
||||
print(entity['name'])
|
||||
print(f"Total: {result['total']}")
|
||||
```
|
||||
|
||||
**Complex Filters:**
|
||||
```python
|
||||
# OR condition
|
||||
where=[{
|
||||
'type': 'or',
|
||||
'value': [
|
||||
{'type': 'equals', 'attribute': 'status', 'value': 'Zurückgestellt'},
|
||||
{'type': 'equals', 'attribute': 'status', 'value': 'Warte auf neuen Anruf'}
|
||||
]
|
||||
}]
|
||||
|
||||
# AND condition
|
||||
where=[
|
||||
{'type': 'equals', 'attribute': 'status', 'value': 'Active'},
|
||||
{'type': 'greaterThan', 'attribute': 'createdAt', 'value': '2026-01-01'}
|
||||
]
|
||||
```
|
||||
|
||||
### Create Entity
|
||||
|
||||
```python
|
||||
async def create_entity(entity_type: str, data: Dict[str, Any]) -> Dict[str, Any]
|
||||
```
|
||||
|
||||
**Usage:**
|
||||
```python
|
||||
# Create new Beteiligter
|
||||
result = await espo.create_entity('Beteiligte', {
|
||||
'name': 'Max Mustermann',
|
||||
'email': 'max@example.com',
|
||||
'phone': '+49123456789',
|
||||
'status': 'New'
|
||||
})
|
||||
print(f"Created with ID: {result['id']}")
|
||||
```
|
||||
|
||||
### Update Entity
|
||||
|
||||
```python
|
||||
async def update_entity(
|
||||
entity_type: str,
|
||||
entity_id: str,
|
||||
data: Dict[str, Any]
|
||||
) -> Dict[str, Any]
|
||||
```
|
||||
|
||||
**Usage:**
|
||||
```python
|
||||
# Update Beteiligter status
|
||||
result = await espo.update_entity(
|
||||
'Beteiligte',
|
||||
'64a3f2b8c9e1234567890abc',
|
||||
{'status': 'Converted'}
|
||||
)
|
||||
```
|
||||
|
||||
### Delete Entity
|
||||
|
||||
```python
|
||||
async def delete_entity(entity_type: str, entity_id: str) -> bool
|
||||
```
|
||||
|
||||
**Usage:**
|
||||
```python
|
||||
# Delete Beteiligter
|
||||
success = await espo.delete_entity('Beteiligte', '64a3f2b8c9e1234567890abc')
|
||||
```
|
||||
|
||||
### Search Entities
|
||||
|
||||
```python
|
||||
async def search_entities(
|
||||
entity_type: str,
|
||||
query: str,
|
||||
fields: Optional[List[str]] = None
|
||||
) -> List[Dict[str, Any]]
|
||||
```
|
||||
|
||||
**Usage:**
|
||||
```python
|
||||
# Full-text search
|
||||
results = await espo.search_entities('Beteiligte', 'Mustermann')
|
||||
for entity in results:
|
||||
print(entity['name'])
|
||||
```
|
||||
|
||||
## Common Entity Types
|
||||
|
||||
Based on EspoCRM standard and VMH customization:
|
||||
|
||||
- `Beteiligte` - Custom entity for VMH participants
|
||||
- `CVmhErstgespraech` - Custom entity for VMH initial consultations
|
||||
- `Contact` - Standard contacts
|
||||
- `Account` - Companies/Organizations
|
||||
- `Lead` - Sales leads
|
||||
- `Opportunity` - Sales opportunities
|
||||
- `Case` - Support cases
|
||||
- `Meeting` - Calendar meetings
|
||||
- `Call` - Phone calls
|
||||
- `Email` - Email records
|
||||
|
||||
## Error Handling
|
||||
|
||||
```python
|
||||
from services.espocrm import EspoCRMError, EspoCRMAuthError
|
||||
|
||||
try:
|
||||
result = await espo.get_entity('Beteiligte', entity_id)
|
||||
except EspoCRMAuthError as e:
|
||||
# Invalid API key
|
||||
context.logger.error(f"Authentication failed: {e}")
|
||||
except EspoCRMError as e:
|
||||
# General API error (404, 403, etc.)
|
||||
context.logger.error(f"API error: {e}")
|
||||
```
|
||||
|
||||
## Authentication
|
||||
|
||||
EspoCRM uses **API Key authentication** via `X-Api-Key` header.
|
||||
|
||||
**Create API Key in EspoCRM:**
|
||||
1. Login as admin
|
||||
2. Go to Administration → API Users
|
||||
3. Create new API User
|
||||
4. Copy API Key
|
||||
5. Set permissions for API User
|
||||
|
||||
**Headers sent automatically:**
|
||||
```
|
||||
X-Api-Key: your_api_key_here
|
||||
Content-Type: application/json
|
||||
Accept: application/json
|
||||
```
|
||||
|
||||
## Integration Examples
|
||||
|
||||
### In Motia Step
|
||||
|
||||
```python
|
||||
from services.espocrm import EspoCRMAPI
|
||||
|
||||
config = {
|
||||
'type': 'event',
|
||||
'name': 'Sync Beteiligter to Advoware',
|
||||
'subscribes': ['vmh.beteiligte.create']
|
||||
}
|
||||
|
||||
async def handler(event, context):
|
||||
entity_id = event['data']['entity_id']
|
||||
|
||||
# Fetch from EspoCRM
|
||||
espo = EspoCRMAPI(context=context)
|
||||
beteiligter = await espo.get_entity('Beteiligte', entity_id)
|
||||
|
||||
context.logger.info(f"Processing: {beteiligter['name']}")
|
||||
|
||||
# Transform and sync to Advoware...
|
||||
# ...
|
||||
```
|
||||
|
||||
### In Cron Step
|
||||
|
||||
```python
|
||||
from services.espocrm import EspoCRMAPI
|
||||
from datetime import datetime, timedelta
|
||||
|
||||
config = {
|
||||
'type': 'cron',
|
||||
'cron': '*/5 * * * *',
|
||||
'name': 'Check Expired Callbacks'
|
||||
}
|
||||
|
||||
async def handler(input, context):
|
||||
espo = EspoCRMAPI(context=context)
|
||||
|
||||
# Find expired callbacks
|
||||
now = datetime.utcnow().isoformat() + 'Z'
|
||||
|
||||
result = await espo.list_entities(
|
||||
'CVmhErstgespraech',
|
||||
where=[
|
||||
{'type': 'lessThan', 'attribute': 'nchsterAnruf', 'value': now},
|
||||
{'type': 'equals', 'attribute': 'status', 'value': 'Warte auf neuen Anruf'}
|
||||
]
|
||||
)
|
||||
|
||||
# Update status for expired entries
|
||||
for entry in result['list']:
|
||||
await espo.update_entity(
|
||||
'CVmhErstgespraech',
|
||||
entry['id'],
|
||||
{'status': 'Neu'}
|
||||
)
|
||||
context.logger.info(f"Reset status for {entry['id']}")
|
||||
```
|
||||
|
||||
## Helper Script: Compare Structures
|
||||
|
||||
Compare entity structures between EspoCRM and Advoware:
|
||||
|
||||
```bash
|
||||
# Compare by EspoCRM ID (auto-search in Advoware)
|
||||
python bitbylaw/scripts/compare_beteiligte.py 64a3f2b8c9e1234567890abc
|
||||
|
||||
# Compare with specific Advoware ID
|
||||
python bitbylaw/scripts/compare_beteiligte.py 64a3f2b8c9e1234567890abc 12345
|
||||
```
|
||||
|
||||
**Output:**
|
||||
- Entity data from both systems
|
||||
- Field structure comparison
|
||||
- Suggested field mappings
|
||||
- JSON output saved to `scripts/beteiligte_comparison_result.json`
|
||||
|
||||
## Performance
|
||||
|
||||
### Timeout
|
||||
|
||||
Default: 30 seconds (configurable via `ESPOCRM_API_TIMEOUT_SECONDS`)
|
||||
|
||||
```python
|
||||
# Custom timeout for specific call
|
||||
result = await espo.api_call('/Beteiligte', timeout_seconds=60)
|
||||
```
|
||||
|
||||
### Pagination
|
||||
|
||||
```python
|
||||
# Fetch in pages
|
||||
offset = 0
|
||||
max_size = 50
|
||||
|
||||
while True:
|
||||
result = await espo.list_entities(
|
||||
'Beteiligte',
|
||||
offset=offset,
|
||||
max_size=max_size
|
||||
)
|
||||
|
||||
entities = result['list']
|
||||
if not entities:
|
||||
break
|
||||
|
||||
# Process entities...
|
||||
|
||||
offset += len(entities)
|
||||
|
||||
if len(entities) < max_size:
|
||||
break # Last page
|
||||
```
|
||||
|
||||
### Rate Limiting
|
||||
|
||||
Optional Redis-based rate limiting can be implemented:
|
||||
|
||||
```python
|
||||
# Check rate limit before API call
|
||||
rate_limit_key = f'espocrm:rate_limit:{entity_type}'
|
||||
if espo.redis_client:
|
||||
count = espo.redis_client.incr(rate_limit_key)
|
||||
espo.redis_client.expire(rate_limit_key, 60) # 1 minute window
|
||||
|
||||
if count > 100: # Max 100 requests per minute
|
||||
raise Exception("Rate limit exceeded")
|
||||
```
|
||||
|
||||
## Testing
|
||||
|
||||
```python
|
||||
import pytest
|
||||
from services.espocrm import EspoCRMAPI
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_get_entity():
|
||||
espo = EspoCRMAPI()
|
||||
|
||||
# Mock or use test entity ID
|
||||
result = await espo.get_entity('Contact', 'test-id-123')
|
||||
|
||||
assert 'id' in result
|
||||
assert result['id'] == 'test-id-123'
|
||||
```
|
||||
|
||||
## Logging
|
||||
|
||||
All operations are logged via context.logger:
|
||||
|
||||
```
|
||||
[INFO] [EspoCRM] EspoCRM API initialized with base URL: https://crm.bitbylaw.com/api/v1
|
||||
[DEBUG] [EspoCRM] API call: GET https://crm.bitbylaw.com/api/v1/Beteiligte/123
|
||||
[DEBUG] [EspoCRM] Response status: 200
|
||||
[INFO] [EspoCRM] Getting Beteiligte with ID: 123
|
||||
```
|
||||
|
||||
## Related Files
|
||||
|
||||
- [services/espocrm.py](./espocrm.py) - Implementation
|
||||
- [scripts/compare_beteiligte.py](../scripts/compare_beteiligte.py) - Comparison tool
|
||||
- [steps/crm-bbl-vmh-reset-nextcall_step.py](../../steps/crm-bbl-vmh-reset-nextcall_step.py) - Example usage
|
||||
- [config.py](../config.py) - Configuration
|
||||
|
||||
## EspoCRM API Documentation
|
||||
|
||||
Official docs: https://docs.espocrm.com/development/api/
|
||||
|
||||
**Key Concepts:**
|
||||
- RESTful API with JSON
|
||||
- Entity-based operations
|
||||
- Filter operators: `equals`, `notEquals`, `greaterThan`, `lessThan`, `like`, `contains`, `in`, `isNull`, `isNotNull`
|
||||
- Boolean operators: `and` (default), `or`
|
||||
- Metadata API: `/Metadata` (for entity definitions)
|
||||
@@ -122,7 +122,9 @@ class AdvowareAPI:
|
||||
params: Optional[Dict] = None, json_data: Optional[Dict] = None,
|
||||
files: Optional[Any] = None, data: Optional[Any] = None,
|
||||
timeout_seconds: Optional[int] = None) -> Any:
|
||||
url = self.API_BASE_URL + endpoint
|
||||
# Bereinige doppelte Slashes
|
||||
endpoint = endpoint.lstrip('/')
|
||||
url = self.API_BASE_URL.rstrip('/') + '/' + endpoint
|
||||
effective_timeout = aiohttp.ClientTimeout(total=timeout_seconds or Config.ADVOWARE_API_TIMEOUT_SECONDS)
|
||||
token = self.get_access_token() # Sync call
|
||||
effective_headers = headers.copy() if headers else {}
|
||||
|
||||
341
bitbylaw/services/beteiligte_sync_utils.py
Normal file
341
bitbylaw/services/beteiligte_sync_utils.py
Normal file
@@ -0,0 +1,341 @@
|
||||
"""
|
||||
Beteiligte Sync Utilities
|
||||
|
||||
Hilfsfunktionen für Sync-Operationen:
|
||||
- Locking via syncStatus
|
||||
- Timestamp-Vergleich
|
||||
- Konfliktauflösung (EspoCRM wins)
|
||||
- EspoCRM In-App Notifications
|
||||
- Soft-Delete Handling
|
||||
"""
|
||||
|
||||
from typing import Dict, Any, Optional, Tuple, Literal
|
||||
from datetime import datetime
|
||||
import pytz
|
||||
import logging
|
||||
from services.espocrm import EspoCRMAPI
|
||||
|
||||
logger = logging.getLogger(__name__)
|
||||
|
||||
# Timestamp-Vergleich Ergebnis-Typen
|
||||
TimestampResult = Literal["espocrm_newer", "advoware_newer", "conflict", "no_change"]
|
||||
|
||||
|
||||
class BeteiligteSync:
|
||||
"""Utility-Klasse für Beteiligte-Synchronisation"""
|
||||
|
||||
def __init__(self, espocrm_api: EspoCRMAPI, context=None):
|
||||
self.espocrm = espocrm_api
|
||||
self.context = context
|
||||
|
||||
def _log(self, message: str, level: str = 'info'):
|
||||
"""Logging mit Context-Support"""
|
||||
if self.context and hasattr(self.context, 'logger'):
|
||||
getattr(self.context.logger, level)(message)
|
||||
else:
|
||||
getattr(logger, level)(message)
|
||||
|
||||
async def acquire_sync_lock(self, entity_id: str) -> bool:
|
||||
"""
|
||||
Setzt syncStatus auf "syncing" (atomares Lock)
|
||||
|
||||
Args:
|
||||
entity_id: EspoCRM CBeteiligte ID
|
||||
|
||||
Returns:
|
||||
True wenn Lock erfolgreich, False wenn bereits im Sync
|
||||
"""
|
||||
try:
|
||||
entity = await self.espocrm.get_entity('CBeteiligte', entity_id)
|
||||
|
||||
current_status = entity.get('syncStatus')
|
||||
|
||||
if current_status == 'syncing':
|
||||
self._log(f"Entity {entity_id} bereits im Sync-Prozess", level='warning')
|
||||
return False
|
||||
|
||||
# Setze Lock
|
||||
await self.espocrm.update_entity('CBeteiligte', entity_id, {
|
||||
'syncStatus': 'syncing'
|
||||
})
|
||||
|
||||
self._log(f"Sync-Lock für {entity_id} erworben (vorher: {current_status})")
|
||||
return True
|
||||
|
||||
except Exception as e:
|
||||
self._log(f"Fehler beim Acquire Lock: {e}", level='error')
|
||||
return False
|
||||
|
||||
async def release_sync_lock(
|
||||
self,
|
||||
entity_id: str,
|
||||
new_status: str = 'clean',
|
||||
error_message: Optional[str] = None,
|
||||
increment_retry: bool = False
|
||||
) -> None:
|
||||
"""
|
||||
Gibt Sync-Lock frei und setzt finalen Status
|
||||
|
||||
Args:
|
||||
entity_id: EspoCRM CBeteiligte ID
|
||||
new_status: Neuer syncStatus (clean, failed, conflict, etc.)
|
||||
error_message: Optional: Fehlermeldung für syncErrorMessage
|
||||
increment_retry: Ob syncRetryCount erhöht werden soll
|
||||
"""
|
||||
try:
|
||||
update_data = {
|
||||
'syncStatus': new_status,
|
||||
'advowareLastSync': datetime.now(pytz.UTC).isoformat()
|
||||
}
|
||||
|
||||
if error_message:
|
||||
update_data['syncErrorMessage'] = error_message[:2000] # Max. 2000 chars
|
||||
else:
|
||||
update_data['syncErrorMessage'] = None
|
||||
|
||||
if increment_retry:
|
||||
# Hole aktuellen Retry-Count
|
||||
entity = await self.espocrm.get_entity('CBeteiligte', entity_id)
|
||||
current_retry = entity.get('syncRetryCount') or 0
|
||||
update_data['syncRetryCount'] = current_retry + 1
|
||||
else:
|
||||
update_data['syncRetryCount'] = 0
|
||||
|
||||
await self.espocrm.update_entity('CBeteiligte', entity_id, update_data)
|
||||
|
||||
self._log(f"Sync-Lock released: {entity_id} → {new_status}")
|
||||
|
||||
except Exception as e:
|
||||
self._log(f"Fehler beim Release Lock: {e}", level='error')
|
||||
|
||||
@staticmethod
|
||||
def parse_timestamp(ts: Any) -> Optional[datetime]:
|
||||
"""
|
||||
Parse verschiedene Timestamp-Formate zu datetime
|
||||
|
||||
Args:
|
||||
ts: String, datetime oder None
|
||||
|
||||
Returns:
|
||||
datetime-Objekt oder None
|
||||
"""
|
||||
if not ts:
|
||||
return None
|
||||
|
||||
if isinstance(ts, datetime):
|
||||
return ts
|
||||
|
||||
if isinstance(ts, str):
|
||||
# EspoCRM Format: "2026-02-07 14:30:00"
|
||||
# Advoware Format: "2026-02-07T14:30:00" oder "2026-02-07T14:30:00Z"
|
||||
try:
|
||||
# Entferne trailing Z falls vorhanden
|
||||
ts = ts.rstrip('Z')
|
||||
|
||||
# Versuche verschiedene Formate
|
||||
for fmt in [
|
||||
'%Y-%m-%d %H:%M:%S',
|
||||
'%Y-%m-%dT%H:%M:%S',
|
||||
'%Y-%m-%d',
|
||||
]:
|
||||
try:
|
||||
return datetime.strptime(ts, fmt)
|
||||
except ValueError:
|
||||
continue
|
||||
|
||||
# Fallback: ISO-Format
|
||||
return datetime.fromisoformat(ts)
|
||||
|
||||
except Exception as e:
|
||||
logger.warning(f"Konnte Timestamp nicht parsen: {ts} - {e}")
|
||||
return None
|
||||
|
||||
return None
|
||||
|
||||
def compare_timestamps(
|
||||
self,
|
||||
espo_modified_at: Any,
|
||||
advo_geaendert_am: Any,
|
||||
last_sync_ts: Any
|
||||
) -> TimestampResult:
|
||||
"""
|
||||
Vergleicht Timestamps und bestimmt Sync-Richtung
|
||||
|
||||
Args:
|
||||
espo_modified_at: EspoCRM modifiedAt
|
||||
advo_geaendert_am: Advoware geaendertAm
|
||||
last_sync_ts: Letzter Sync (advowareLastSync)
|
||||
|
||||
Returns:
|
||||
"espocrm_newer": EspoCRM wurde nach last_sync geändert und ist neuer
|
||||
"advoware_newer": Advoware wurde nach last_sync geändert und ist neuer
|
||||
"conflict": Beide wurden nach last_sync geändert
|
||||
"no_change": Keine Änderungen seit last_sync
|
||||
"""
|
||||
espo_ts = self.parse_timestamp(espo_modified_at)
|
||||
advo_ts = self.parse_timestamp(advo_geaendert_am)
|
||||
sync_ts = self.parse_timestamp(last_sync_ts)
|
||||
|
||||
# Logging
|
||||
self._log(
|
||||
f"Timestamp-Vergleich: EspoCRM={espo_ts}, Advoware={advo_ts}, LastSync={sync_ts}",
|
||||
level='debug'
|
||||
)
|
||||
|
||||
# Falls kein last_sync → erster Sync, vergleiche direkt
|
||||
if not sync_ts:
|
||||
if not espo_ts or not advo_ts:
|
||||
return "no_change"
|
||||
|
||||
if espo_ts > advo_ts:
|
||||
return "espocrm_newer"
|
||||
elif advo_ts > espo_ts:
|
||||
return "advoware_newer"
|
||||
else:
|
||||
return "no_change"
|
||||
|
||||
# Check ob seit last_sync Änderungen
|
||||
espo_changed = espo_ts and espo_ts > sync_ts
|
||||
advo_changed = advo_ts and advo_ts > sync_ts
|
||||
|
||||
if espo_changed and advo_changed:
|
||||
# Beide geändert seit last_sync → Konflikt
|
||||
return "conflict"
|
||||
elif espo_changed:
|
||||
# Nur EspoCRM geändert
|
||||
return "espocrm_newer" if (not advo_ts or espo_ts > advo_ts) else "conflict"
|
||||
elif advo_changed:
|
||||
# Nur Advoware geändert
|
||||
return "advoware_newer"
|
||||
else:
|
||||
# Keine Änderungen
|
||||
return "no_change"
|
||||
|
||||
async def send_notification(
|
||||
self,
|
||||
entity_id: str,
|
||||
notification_type: Literal["conflict", "deleted"],
|
||||
extra_data: Optional[Dict[str, Any]] = None
|
||||
) -> None:
|
||||
"""
|
||||
Sendet EspoCRM In-App Notification
|
||||
|
||||
Args:
|
||||
entity_id: CBeteiligte Entity ID
|
||||
notification_type: "conflict" oder "deleted"
|
||||
extra_data: Zusätzliche Daten für Nachricht
|
||||
"""
|
||||
try:
|
||||
# Hole Entity-Daten
|
||||
entity = await self.espocrm.get_entity('CBeteiligte', entity_id)
|
||||
name = entity.get('name', 'Unbekannt')
|
||||
betnr = entity.get('betnr')
|
||||
assigned_user = entity.get('assignedUserId')
|
||||
|
||||
# Erstelle Nachricht basierend auf Typ
|
||||
if notification_type == "conflict":
|
||||
message = (
|
||||
f"⚠️ Sync-Konflikt bei Beteiligten '{name}' (betNr: {betnr}). "
|
||||
f"EspoCRM hat Vorrang - Änderungen wurden nach Advoware übertragen. "
|
||||
f"Bitte prüfen Sie die Details."
|
||||
)
|
||||
elif notification_type == "deleted":
|
||||
deleted_at = entity.get('advowareDeletedAt', 'unbekannt')
|
||||
message = (
|
||||
f"🗑️ Beteiligter '{name}' (betNr: {betnr}) wurde in Advoware gelöscht "
|
||||
f"(am {deleted_at}). Der Datensatz wurde in EspoCRM markiert, aber nicht gelöscht. "
|
||||
f"Bitte prüfen Sie, ob dies beabsichtigt war."
|
||||
)
|
||||
else:
|
||||
message = f"Benachrichtigung für Beteiligten '{name}'"
|
||||
|
||||
# Erstelle Notification in EspoCRM
|
||||
notification_data = {
|
||||
'type': 'message',
|
||||
'message': message,
|
||||
'relatedType': 'CBeteiligte',
|
||||
'relatedId': entity_id,
|
||||
}
|
||||
|
||||
# Wenn assigned user vorhanden, sende an diesen
|
||||
if assigned_user:
|
||||
notification_data['userId'] = assigned_user
|
||||
|
||||
# Sende via API
|
||||
result = await self.espocrm.api_call(
|
||||
'Notification',
|
||||
method='POST',
|
||||
data=notification_data
|
||||
)
|
||||
|
||||
self._log(f"Notification gesendet für {entity_id}: {notification_type}")
|
||||
|
||||
except Exception as e:
|
||||
self._log(f"Fehler beim Senden der Notification: {e}", level='error')
|
||||
|
||||
async def handle_advoware_deleted(
|
||||
self,
|
||||
entity_id: str,
|
||||
error_details: str
|
||||
) -> None:
|
||||
"""
|
||||
Behandelt Fall dass Beteiligter in Advoware gelöscht wurde (404)
|
||||
|
||||
Args:
|
||||
entity_id: CBeteiligte Entity ID
|
||||
error_details: Fehlerdetails von Advoware API
|
||||
"""
|
||||
try:
|
||||
now = datetime.now(pytz.UTC).isoformat()
|
||||
|
||||
# Update Entity: Soft-Delete Flag
|
||||
await self.espocrm.update_entity('CBeteiligte', entity_id, {
|
||||
'syncStatus': 'deleted_in_advoware',
|
||||
'advowareDeletedAt': now,
|
||||
'syncErrorMessage': f"Beteiligter existiert nicht mehr in Advoware. {error_details}"
|
||||
})
|
||||
|
||||
self._log(f"Entity {entity_id} als deleted_in_advoware markiert")
|
||||
|
||||
# Sende Notification
|
||||
await self.send_notification(entity_id, 'deleted')
|
||||
|
||||
except Exception as e:
|
||||
self._log(f"Fehler beim Handle Deleted: {e}", level='error')
|
||||
|
||||
async def resolve_conflict_espocrm_wins(
|
||||
self,
|
||||
entity_id: str,
|
||||
espo_entity: Dict[str, Any],
|
||||
advo_entity: Dict[str, Any],
|
||||
conflict_details: str
|
||||
) -> None:
|
||||
"""
|
||||
Löst Konflikt auf: EspoCRM wins (überschreibt Advoware)
|
||||
|
||||
Args:
|
||||
entity_id: CBeteiligte Entity ID
|
||||
espo_entity: EspoCRM Entity-Daten
|
||||
advo_entity: Advoware Entity-Daten
|
||||
conflict_details: Details zum Konflikt
|
||||
"""
|
||||
try:
|
||||
now = datetime.now(pytz.UTC).isoformat()
|
||||
|
||||
# Markiere als gelöst mit Konflikt-Info
|
||||
await self.espocrm.update_entity('CBeteiligte', entity_id, {
|
||||
'syncStatus': 'clean', # Gelöst!
|
||||
'advowareLastSync': now,
|
||||
'syncErrorMessage': f"Konflikt am {now}: {conflict_details}. EspoCRM hat gewonnen.",
|
||||
'syncRetryCount': 0
|
||||
})
|
||||
|
||||
self._log(f"Konflikt gelöst für {entity_id}: EspoCRM wins")
|
||||
|
||||
# Sende Notification
|
||||
await self.send_notification(entity_id, 'conflict', {
|
||||
'details': conflict_details
|
||||
})
|
||||
|
||||
except Exception as e:
|
||||
self._log(f"Fehler beim Resolve Conflict: {e}", level='error')
|
||||
276
bitbylaw/services/espocrm.py
Normal file
276
bitbylaw/services/espocrm.py
Normal file
@@ -0,0 +1,276 @@
|
||||
import aiohttp
|
||||
import asyncio
|
||||
import logging
|
||||
import redis
|
||||
from typing import Optional, Dict, Any, List
|
||||
from config import Config
|
||||
|
||||
logger = logging.getLogger(__name__)
|
||||
|
||||
class EspoCRMError(Exception):
|
||||
"""Base exception for EspoCRM API errors"""
|
||||
pass
|
||||
|
||||
class EspoCRMAuthError(EspoCRMError):
|
||||
"""Authentication error"""
|
||||
pass
|
||||
|
||||
class EspoCRMAPI:
|
||||
"""
|
||||
EspoCRM API Client for bitbylaw integration.
|
||||
|
||||
Supports:
|
||||
- API Key authentication (X-Api-Key header)
|
||||
- Standard REST operations (GET, POST, PUT, DELETE)
|
||||
- Entity management (Beteiligte, CVmhErstgespraech, etc.)
|
||||
"""
|
||||
|
||||
def __init__(self, context=None):
|
||||
self.context = context
|
||||
self._log("EspoCRMAPI __init__ started", level='debug')
|
||||
|
||||
# Configuration
|
||||
self.api_base_url = Config.ESPOCRM_API_BASE_URL
|
||||
self.api_key = Config.ESPOCRM_API_KEY
|
||||
|
||||
if not self.api_key:
|
||||
raise EspoCRMAuthError("ESPOCRM_MARVIN_API_KEY not configured in environment")
|
||||
|
||||
self._log(f"EspoCRM API initialized with base URL: {self.api_base_url}")
|
||||
|
||||
# Optional Redis for caching/rate limiting
|
||||
try:
|
||||
self.redis_client = redis.Redis(
|
||||
host=Config.REDIS_HOST,
|
||||
port=int(Config.REDIS_PORT),
|
||||
db=int(Config.REDIS_DB_ADVOWARE_CACHE),
|
||||
socket_timeout=Config.REDIS_TIMEOUT_SECONDS,
|
||||
socket_connect_timeout=Config.REDIS_TIMEOUT_SECONDS,
|
||||
decode_responses=True
|
||||
)
|
||||
self.redis_client.ping()
|
||||
self._log("Connected to Redis for EspoCRM operations")
|
||||
except Exception as e:
|
||||
self._log(f"Could not connect to Redis: {e}. Continuing without caching.", level='warning')
|
||||
self.redis_client = None
|
||||
|
||||
def _log(self, message: str, level: str = 'info'):
|
||||
"""Log message via context.logger if available, otherwise use module logger"""
|
||||
log_func = getattr(logger, level, logger.info)
|
||||
if self.context and hasattr(self.context, 'logger'):
|
||||
ctx_log_func = getattr(self.context.logger, level, self.context.logger.info)
|
||||
ctx_log_func(f"[EspoCRM] {message}")
|
||||
else:
|
||||
log_func(f"[EspoCRM] {message}")
|
||||
|
||||
def _get_headers(self) -> Dict[str, str]:
|
||||
"""Generate request headers with API key"""
|
||||
return {
|
||||
'X-Api-Key': self.api_key,
|
||||
'Content-Type': 'application/json',
|
||||
'Accept': 'application/json'
|
||||
}
|
||||
|
||||
async def api_call(
|
||||
self,
|
||||
endpoint: str,
|
||||
method: str = 'GET',
|
||||
params: Optional[Dict] = None,
|
||||
json_data: Optional[Dict] = None,
|
||||
timeout_seconds: Optional[int] = None
|
||||
) -> Any:
|
||||
"""
|
||||
Make an API call to EspoCRM.
|
||||
|
||||
Args:
|
||||
endpoint: API endpoint (e.g., '/Beteiligte/123' or '/CVmhErstgespraech')
|
||||
method: HTTP method (GET, POST, PUT, DELETE)
|
||||
params: Query parameters
|
||||
json_data: JSON body for POST/PUT
|
||||
timeout_seconds: Request timeout
|
||||
|
||||
Returns:
|
||||
Parsed JSON response or None
|
||||
|
||||
Raises:
|
||||
EspoCRMError: On API errors
|
||||
"""
|
||||
# Ensure endpoint starts with /
|
||||
if not endpoint.startswith('/'):
|
||||
endpoint = '/' + endpoint
|
||||
|
||||
url = self.api_base_url.rstrip('/') + endpoint
|
||||
headers = self._get_headers()
|
||||
effective_timeout = aiohttp.ClientTimeout(
|
||||
total=timeout_seconds or Config.ESPOCRM_API_TIMEOUT_SECONDS
|
||||
)
|
||||
|
||||
self._log(f"API call: {method} {url}", level='debug')
|
||||
if params:
|
||||
self._log(f"Params: {params}", level='debug')
|
||||
|
||||
async with aiohttp.ClientSession(timeout=effective_timeout) as session:
|
||||
try:
|
||||
async with session.request(
|
||||
method,
|
||||
url,
|
||||
headers=headers,
|
||||
params=params,
|
||||
json=json_data
|
||||
) as response:
|
||||
# Log response status
|
||||
self._log(f"Response status: {response.status}", level='debug')
|
||||
|
||||
# Handle errors
|
||||
if response.status == 401:
|
||||
raise EspoCRMAuthError("Authentication failed - check API key")
|
||||
elif response.status == 403:
|
||||
raise EspoCRMError("Access forbidden")
|
||||
elif response.status == 404:
|
||||
raise EspoCRMError(f"Resource not found: {endpoint}")
|
||||
elif response.status >= 400:
|
||||
error_text = await response.text()
|
||||
raise EspoCRMError(f"API error {response.status}: {error_text}")
|
||||
|
||||
# Parse response
|
||||
if response.content_type == 'application/json':
|
||||
result = await response.json()
|
||||
self._log(f"Response received", level='debug')
|
||||
return result
|
||||
else:
|
||||
# For DELETE or other non-JSON responses
|
||||
return None
|
||||
|
||||
except aiohttp.ClientError as e:
|
||||
self._log(f"API call failed: {e}", level='error')
|
||||
raise EspoCRMError(f"Request failed: {e}") from e
|
||||
|
||||
async def get_entity(self, entity_type: str, entity_id: str) -> Dict[str, Any]:
|
||||
"""
|
||||
Get a single entity by ID.
|
||||
|
||||
Args:
|
||||
entity_type: Entity type (e.g., 'Beteiligte', 'CVmhErstgespraech')
|
||||
entity_id: Entity ID
|
||||
|
||||
Returns:
|
||||
Entity data as dict
|
||||
"""
|
||||
self._log(f"Getting {entity_type} with ID: {entity_id}")
|
||||
return await self.api_call(f"/{entity_type}/{entity_id}", method='GET')
|
||||
|
||||
async def list_entities(
|
||||
self,
|
||||
entity_type: str,
|
||||
where: Optional[List[Dict]] = None,
|
||||
select: Optional[str] = None,
|
||||
order_by: Optional[str] = None,
|
||||
offset: int = 0,
|
||||
max_size: int = 50
|
||||
) -> Dict[str, Any]:
|
||||
"""
|
||||
List entities with filtering and pagination.
|
||||
|
||||
Args:
|
||||
entity_type: Entity type
|
||||
where: Filter conditions (EspoCRM format)
|
||||
select: Comma-separated field list
|
||||
order_by: Sort field
|
||||
offset: Pagination offset
|
||||
max_size: Max results per page
|
||||
|
||||
Returns:
|
||||
Dict with 'list' and 'total' keys
|
||||
"""
|
||||
params = {
|
||||
'offset': offset,
|
||||
'maxSize': max_size
|
||||
}
|
||||
|
||||
if where:
|
||||
params['where'] = where
|
||||
if select:
|
||||
params['select'] = select
|
||||
if order_by:
|
||||
params['orderBy'] = order_by
|
||||
|
||||
self._log(f"Listing {entity_type} entities")
|
||||
return await self.api_call(f"/{entity_type}", method='GET', params=params)
|
||||
|
||||
async def create_entity(
|
||||
self,
|
||||
entity_type: str,
|
||||
data: Dict[str, Any]
|
||||
) -> Dict[str, Any]:
|
||||
"""
|
||||
Create a new entity.
|
||||
|
||||
Args:
|
||||
entity_type: Entity type
|
||||
data: Entity data
|
||||
|
||||
Returns:
|
||||
Created entity with ID
|
||||
"""
|
||||
self._log(f"Creating {entity_type} entity")
|
||||
return await self.api_call(f"/{entity_type}", method='POST', json_data=data)
|
||||
|
||||
async def update_entity(
|
||||
self,
|
||||
entity_type: str,
|
||||
entity_id: str,
|
||||
data: Dict[str, Any]
|
||||
) -> Dict[str, Any]:
|
||||
"""
|
||||
Update an existing entity.
|
||||
|
||||
Args:
|
||||
entity_type: Entity type
|
||||
entity_id: Entity ID
|
||||
data: Updated fields
|
||||
|
||||
Returns:
|
||||
Updated entity
|
||||
"""
|
||||
self._log(f"Updating {entity_type} with ID: {entity_id}")
|
||||
return await self.api_call(f"/{entity_type}/{entity_id}", method='PUT', json_data=data)
|
||||
|
||||
async def delete_entity(self, entity_type: str, entity_id: str) -> bool:
|
||||
"""
|
||||
Delete an entity.
|
||||
|
||||
Args:
|
||||
entity_type: Entity type
|
||||
entity_id: Entity ID
|
||||
|
||||
Returns:
|
||||
True if successful
|
||||
"""
|
||||
self._log(f"Deleting {entity_type} with ID: {entity_id}")
|
||||
await self.api_call(f"/{entity_type}/{entity_id}", method='DELETE')
|
||||
return True
|
||||
|
||||
async def search_entities(
|
||||
self,
|
||||
entity_type: str,
|
||||
query: str,
|
||||
fields: Optional[List[str]] = None
|
||||
) -> List[Dict[str, Any]]:
|
||||
"""
|
||||
Search entities by text query.
|
||||
|
||||
Args:
|
||||
entity_type: Entity type
|
||||
query: Search query
|
||||
fields: Fields to search in
|
||||
|
||||
Returns:
|
||||
List of matching entities
|
||||
"""
|
||||
where = [{
|
||||
'type': 'textFilter',
|
||||
'value': query
|
||||
}]
|
||||
|
||||
result = await self.list_entities(entity_type, where=where)
|
||||
return result.get('list', [])
|
||||
243
bitbylaw/services/espocrm_mapper.py
Normal file
243
bitbylaw/services/espocrm_mapper.py
Normal file
@@ -0,0 +1,243 @@
|
||||
"""
|
||||
EspoCRM ↔ Advoware Entity Mapper
|
||||
|
||||
Transformiert Beteiligte zwischen den beiden Systemen basierend auf ENTITY_MAPPING_CBeteiligte_Advoware.md
|
||||
"""
|
||||
|
||||
from typing import Dict, Any, Optional, List
|
||||
from datetime import datetime
|
||||
import logging
|
||||
|
||||
logger = logging.getLogger(__name__)
|
||||
|
||||
|
||||
class BeteiligteMapper:
|
||||
"""Mapper für CBeteiligte (EspoCRM) ↔ Beteiligte (Advoware)"""
|
||||
|
||||
@staticmethod
|
||||
def map_cbeteiligte_to_advoware(espo_entity: Dict[str, Any]) -> Dict[str, Any]:
|
||||
"""
|
||||
Transformiert EspoCRM CBeteiligte → Advoware Beteiligte Format
|
||||
|
||||
Args:
|
||||
espo_entity: CBeteiligte Entity von EspoCRM
|
||||
|
||||
Returns:
|
||||
Dict für Advoware API (POST/PUT /api/v1/advonet/Beteiligte)
|
||||
"""
|
||||
logger.debug(f"Mapping EspoCRM → Advoware: {espo_entity.get('id')}")
|
||||
|
||||
# Bestimme ob Person oder Firma
|
||||
is_firma = bool(espo_entity.get('firmenname'))
|
||||
rechtsform = espo_entity.get('rechtsform', '')
|
||||
|
||||
# Basis-Struktur
|
||||
advo_data = {
|
||||
'rechtsform': rechtsform,
|
||||
}
|
||||
|
||||
# NAME: Person vs. Firma
|
||||
if is_firma:
|
||||
# Firma: name = firmenname
|
||||
advo_data['name'] = espo_entity.get('firmenname', '')
|
||||
advo_data['vorname'] = None
|
||||
else:
|
||||
# Person: name = lastName, vorname = firstName
|
||||
advo_data['name'] = espo_entity.get('lastName', '')
|
||||
advo_data['vorname'] = espo_entity.get('firstName', '')
|
||||
|
||||
# ANREDE
|
||||
salutation = espo_entity.get('salutationName', '')
|
||||
if salutation:
|
||||
advo_data['anrede'] = salutation
|
||||
|
||||
# GEBURTSDATUM
|
||||
date_of_birth = espo_entity.get('dateOfBirth')
|
||||
if date_of_birth:
|
||||
advo_data['geburtsdatum'] = date_of_birth
|
||||
|
||||
# KONTAKTDATEN
|
||||
# E-Mail (emailAddressData ist Array, wir nehmen Primary)
|
||||
email_data = espo_entity.get('emailAddressData')
|
||||
if email_data and isinstance(email_data, list):
|
||||
primary_email = next((e for e in email_data if e.get('primary')), None)
|
||||
if primary_email:
|
||||
advo_data['emailGesch'] = primary_email.get('emailAddress')
|
||||
elif espo_entity.get('emailAddress'):
|
||||
advo_data['emailGesch'] = espo_entity.get('emailAddress')
|
||||
|
||||
# Telefon (phoneNumberData ist Array, wir nehmen Primary)
|
||||
phone_data = espo_entity.get('phoneNumberData')
|
||||
if phone_data and isinstance(phone_data, list):
|
||||
primary_phone = next((p for p in phone_data if p.get('primary')), None)
|
||||
if primary_phone:
|
||||
phone_num = primary_phone.get('phoneNumber')
|
||||
phone_type = primary_phone.get('type', '').lower()
|
||||
|
||||
if 'mobile' in phone_type or 'mobil' in phone_type:
|
||||
advo_data['mobil'] = phone_num
|
||||
else:
|
||||
advo_data['telGesch'] = phone_num
|
||||
elif espo_entity.get('phoneNumber'):
|
||||
advo_data['telGesch'] = espo_entity.get('phoneNumber')
|
||||
|
||||
# HANDELSREGISTER (nur für Firmen)
|
||||
if is_firma:
|
||||
hr_nummer = espo_entity.get('handelsregisterNummer')
|
||||
if hr_nummer:
|
||||
advo_data['handelsRegisterNummer'] = hr_nummer
|
||||
|
||||
# DISGTYP (EspoCRM spezifisch - falls vorhanden)
|
||||
disgtyp = espo_entity.get('disgTyp')
|
||||
if disgtyp:
|
||||
advo_data['disgTyp'] = disgtyp
|
||||
|
||||
logger.debug(f"Mapped to Advoware: name={advo_data.get('name')}, vorname={advo_data.get('vorname')}")
|
||||
|
||||
return advo_data
|
||||
|
||||
@staticmethod
|
||||
def map_advoware_to_cbeteiligte(advo_entity: Dict[str, Any]) -> Dict[str, Any]:
|
||||
"""
|
||||
Transformiert Advoware Beteiligte → EspoCRM CBeteiligte Format
|
||||
|
||||
Args:
|
||||
advo_entity: Beteiligter von Advoware API
|
||||
|
||||
Returns:
|
||||
Dict für EspoCRM API (POST/PUT /api/v1/CBeteiligte)
|
||||
"""
|
||||
logger.debug(f"Mapping Advoware → EspoCRM: betNr={advo_entity.get('betNr')}")
|
||||
|
||||
# Bestimme ob Person oder Firma
|
||||
vorname = advo_entity.get('vorname')
|
||||
is_person = bool(vorname)
|
||||
|
||||
# Basis-Struktur
|
||||
espo_data = {
|
||||
'rechtsform': advo_entity.get('rechtsform', ''),
|
||||
'betnr': advo_entity.get('betNr'), # Link zu Advoware
|
||||
}
|
||||
|
||||
# NAME: Person vs. Firma
|
||||
if is_person:
|
||||
# Person
|
||||
espo_data['firstName'] = vorname
|
||||
espo_data['lastName'] = advo_entity.get('name', '')
|
||||
espo_data['name'] = f"{vorname} {advo_entity.get('name', '')}".strip()
|
||||
espo_data['firmenname'] = None
|
||||
else:
|
||||
# Firma
|
||||
espo_data['firmenname'] = advo_entity.get('name', '')
|
||||
espo_data['name'] = advo_entity.get('name', '')
|
||||
espo_data['firstName'] = None
|
||||
espo_data['lastName'] = None
|
||||
|
||||
# ANREDE
|
||||
anrede = advo_entity.get('anrede')
|
||||
if anrede:
|
||||
espo_data['salutationName'] = anrede
|
||||
|
||||
# GEBURTSDATUM
|
||||
geburtsdatum = advo_entity.get('geburtsdatum')
|
||||
if geburtsdatum:
|
||||
espo_data['dateOfBirth'] = geburtsdatum
|
||||
|
||||
# KONTAKTDATEN
|
||||
# E-Mail (emailGesch ist primary)
|
||||
email_gesch = advo_entity.get('emailGesch')
|
||||
email = advo_entity.get('email')
|
||||
|
||||
primary_email = email_gesch or email
|
||||
if primary_email:
|
||||
espo_data['emailAddress'] = primary_email
|
||||
espo_data['emailAddressData'] = [
|
||||
{
|
||||
'emailAddress': primary_email,
|
||||
'primary': True,
|
||||
'optOut': False,
|
||||
'invalid': False
|
||||
}
|
||||
]
|
||||
|
||||
# Telefon (telGesch ist primary, mobil als secondary)
|
||||
tel_gesch = advo_entity.get('telGesch')
|
||||
tel_privat = advo_entity.get('telPrivat')
|
||||
mobil = advo_entity.get('mobil')
|
||||
|
||||
phone_data = []
|
||||
|
||||
# Primary: telGesch oder telPrivat
|
||||
primary_tel = tel_gesch or tel_privat
|
||||
if primary_tel:
|
||||
espo_data['phoneNumber'] = primary_tel
|
||||
phone_data.append({
|
||||
'phoneNumber': primary_tel,
|
||||
'primary': True,
|
||||
'type': 'Office' if tel_gesch else 'Home'
|
||||
})
|
||||
|
||||
# Secondary: mobil
|
||||
if mobil and mobil != primary_tel:
|
||||
phone_data.append({
|
||||
'phoneNumber': mobil,
|
||||
'primary': False,
|
||||
'type': 'Mobile'
|
||||
})
|
||||
|
||||
if phone_data:
|
||||
espo_data['phoneNumberData'] = phone_data
|
||||
|
||||
# HANDELSREGISTER (nur für Firmen)
|
||||
if not is_person:
|
||||
hr_nummer = advo_entity.get('handelsRegisterNummer')
|
||||
if hr_nummer:
|
||||
espo_data['handelsregisterNummer'] = hr_nummer
|
||||
|
||||
# DISGTYP
|
||||
disgtyp = advo_entity.get('disgTyp')
|
||||
if disgtyp:
|
||||
espo_data['disgTyp'] = disgtyp
|
||||
|
||||
logger.debug(f"Mapped to EspoCRM: name={espo_data.get('name')}")
|
||||
|
||||
return espo_data
|
||||
|
||||
@staticmethod
|
||||
def get_changed_fields(espo_entity: Dict[str, Any], advo_entity: Dict[str, Any]) -> List[str]:
|
||||
"""
|
||||
Vergleicht zwei Entities und gibt Liste der geänderten Felder zurück
|
||||
|
||||
Args:
|
||||
espo_entity: EspoCRM CBeteiligte
|
||||
advo_entity: Advoware Beteiligte
|
||||
|
||||
Returns:
|
||||
Liste von Feldnamen die unterschiedlich sind
|
||||
"""
|
||||
# Mappe Advoware zu EspoCRM Format für Vergleich
|
||||
mapped_advo = BeteiligteMapper.map_advoware_to_cbeteiligte(advo_entity)
|
||||
|
||||
changed = []
|
||||
|
||||
# Vergleiche wichtige Felder
|
||||
compare_fields = [
|
||||
'name', 'firstName', 'lastName', 'firmenname',
|
||||
'emailAddress', 'phoneNumber',
|
||||
'dateOfBirth', 'rechtsform',
|
||||
'handelsregisterNummer'
|
||||
]
|
||||
|
||||
for field in compare_fields:
|
||||
espo_val = espo_entity.get(field)
|
||||
advo_val = mapped_advo.get(field)
|
||||
|
||||
# Normalisiere None und leere Strings
|
||||
espo_val = espo_val if espo_val else None
|
||||
advo_val = advo_val if advo_val else None
|
||||
|
||||
if espo_val != advo_val:
|
||||
changed.append(field)
|
||||
logger.debug(f"Field '{field}' changed: EspoCRM='{espo_val}' vs Advoware='{advo_val}'")
|
||||
|
||||
return changed
|
||||
117
bitbylaw/steps/vmh/beteiligte_sync_cron_step.py
Normal file
117
bitbylaw/steps/vmh/beteiligte_sync_cron_step.py
Normal file
@@ -0,0 +1,117 @@
|
||||
"""
|
||||
Beteiligte Sync Cron Job
|
||||
|
||||
Läuft alle 15 Minuten und emittiert Sync-Events für Beteiligte die:
|
||||
- Neu sind (pending_sync)
|
||||
- Geändert wurden (dirty)
|
||||
- Fehlgeschlagen sind (failed → Retry)
|
||||
- Lange nicht gesynct wurden (clean aber > 24h alt)
|
||||
"""
|
||||
|
||||
import asyncio
|
||||
from services.espocrm import EspoCRMAPI
|
||||
import datetime
|
||||
|
||||
config = {
|
||||
'type': 'cron',
|
||||
'name': 'VMH Beteiligte Sync Cron',
|
||||
'description': 'Prüft alle 15 Minuten welche Beteiligte synchronisiert werden müssen',
|
||||
'schedule': '*/15 * * * *', # Alle 15 Minuten
|
||||
'flows': ['vmh'],
|
||||
'emits': ['vmh.beteiligte.sync_check']
|
||||
}
|
||||
|
||||
async def handler(context):
|
||||
"""
|
||||
Cron-Handler: Findet alle Beteiligte die Sync benötigen und emittiert Events
|
||||
"""
|
||||
context.logger.info("🕐 Beteiligte Sync Cron gestartet")
|
||||
|
||||
try:
|
||||
espocrm = EspoCRMAPI()
|
||||
|
||||
# Berechne Threshold für "veraltete" Syncs (24 Stunden)
|
||||
threshold = datetime.datetime.now() - datetime.timedelta(hours=24)
|
||||
threshold_str = threshold.strftime('%Y-%m-%d %H:%M:%S')
|
||||
|
||||
context.logger.info(f"📅 Suche Entities mit Sync-Bedarf (älter als {threshold_str})")
|
||||
|
||||
# QUERY 1: Entities mit Status pending_sync, dirty oder failed
|
||||
unclean_filter = {
|
||||
'where': [
|
||||
{
|
||||
'type': 'or',
|
||||
'value': [
|
||||
{'type': 'equals', 'attribute': 'syncStatus', 'value': 'pending_sync'},
|
||||
{'type': 'equals', 'attribute': 'syncStatus', 'value': 'dirty'},
|
||||
{'type': 'equals', 'attribute': 'syncStatus', 'value': 'failed'},
|
||||
]
|
||||
}
|
||||
]
|
||||
}
|
||||
|
||||
unclean_result = await espocrm.search_entities('CBeteiligte', unclean_filter, max_size=100)
|
||||
unclean_entities = unclean_result.get('list', [])
|
||||
|
||||
context.logger.info(f"📊 Gefunden: {len(unclean_entities)} Entities mit Status pending/dirty/failed")
|
||||
|
||||
# QUERY 2: Clean Entities die > 24h nicht gesynct wurden
|
||||
stale_filter = {
|
||||
'where': [
|
||||
{
|
||||
'type': 'and',
|
||||
'value': [
|
||||
{'type': 'equals', 'attribute': 'syncStatus', 'value': 'clean'},
|
||||
{'type': 'isNotNull', 'attribute': 'betnr'},
|
||||
{
|
||||
'type': 'or',
|
||||
'value': [
|
||||
{'type': 'isNull', 'attribute': 'advowareLastSync'},
|
||||
{'type': 'before', 'attribute': 'advowareLastSync', 'value': threshold_str}
|
||||
]
|
||||
}
|
||||
]
|
||||
}
|
||||
]
|
||||
}
|
||||
|
||||
stale_result = await espocrm.search_entities('CBeteiligte', stale_filter, max_size=50)
|
||||
stale_entities = stale_result.get('list', [])
|
||||
|
||||
context.logger.info(f"📊 Gefunden: {len(stale_entities)} Entities mit veraltetem Sync (> 24h)")
|
||||
|
||||
# KOMBINIERE ALLE
|
||||
all_entities = unclean_entities + stale_entities
|
||||
entity_ids = list(set([e['id'] for e in all_entities])) # Dedupliziere
|
||||
|
||||
context.logger.info(f"🎯 Total: {len(entity_ids)} eindeutige Entities zum Sync")
|
||||
|
||||
if not entity_ids:
|
||||
context.logger.info("✅ Keine Entities benötigen Sync")
|
||||
return
|
||||
|
||||
# EMITTIERE EVENT FÜR JEDEN BETEILIGTEN
|
||||
emitted_count = 0
|
||||
|
||||
for entity_id in entity_ids:
|
||||
try:
|
||||
await context.emit({
|
||||
'topic': 'vmh.beteiligte.sync_check',
|
||||
'data': {
|
||||
'entity_id': entity_id,
|
||||
'action': 'sync_check',
|
||||
'source': 'cron',
|
||||
'timestamp': datetime.datetime.now().isoformat()
|
||||
}
|
||||
})
|
||||
emitted_count += 1
|
||||
|
||||
except Exception as e:
|
||||
context.logger.error(f"❌ Fehler beim Emittieren für {entity_id}: {e}")
|
||||
|
||||
context.logger.info(f"✅ Cron fertig: {emitted_count}/{len(entity_ids)} Events emittiert")
|
||||
|
||||
except Exception as e:
|
||||
context.logger.error(f"❌ Fehler im Sync Cron: {e}")
|
||||
import traceback
|
||||
context.logger.error(traceback.format_exc())
|
||||
@@ -1,41 +1,46 @@
|
||||
from services.advoware import AdvowareAPI
|
||||
from services.espocrm import EspoCRMAPI
|
||||
from services.espocrm_mapper import BeteiligteMapper
|
||||
from services.beteiligte_sync_utils import BeteiligteSync
|
||||
import json
|
||||
import redis
|
||||
from config import Config
|
||||
|
||||
config = {
|
||||
'type': 'event',
|
||||
'name': 'VMH Beteiligte Sync',
|
||||
'description': 'Synchronisiert Beteiligte Entities von Advoware nach Änderungen (Create/Update/Delete)',
|
||||
'subscribes': ['vmh.beteiligte.create', 'vmh.beteiligte.update', 'vmh.beteiligte.delete'],
|
||||
'name': 'VMH Beteiligte Sync Handler',
|
||||
'description': 'Zentraler Sync-Handler für Beteiligte (Webhooks + Cron Events)',
|
||||
'subscribes': [
|
||||
'vmh.beteiligte.create',
|
||||
'vmh.beteiligte.update',
|
||||
'vmh.beteiligte.delete',
|
||||
'vmh.beteiligte.sync_check' # Von Cron
|
||||
],
|
||||
'flows': ['vmh'],
|
||||
'emits': []
|
||||
}
|
||||
|
||||
async def handler(event_data, context):
|
||||
try:
|
||||
"""
|
||||
Zentraler Sync-Handler für Beteiligte
|
||||
|
||||
Verarbeitet:
|
||||
- vmh.beteiligte.create: Neu in EspoCRM → Create in Advoware
|
||||
- vmh.beteiligte.update: Geändert in EspoCRM → Update in Advoware
|
||||
- vmh.beteiligte.delete: Gelöscht in EspoCRM → Delete in Advoware
|
||||
- vmh.beteiligte.sync_check: Cron-Check → Sync wenn nötig
|
||||
"""
|
||||
entity_id = event_data.get('entity_id')
|
||||
action = event_data.get('action', 'unknown')
|
||||
action = event_data.get('action', 'sync_check')
|
||||
source = event_data.get('source', 'unknown')
|
||||
|
||||
if not entity_id:
|
||||
context.logger.error("Keine entity_id im Event gefunden")
|
||||
return
|
||||
|
||||
context.logger.info(f"Starte {action.upper()} Sync für Beteiligte Entity: {entity_id}")
|
||||
context.logger.info(f"🔄 Sync-Handler gestartet: {action.upper()} | Entity: {entity_id} | Source: {source}")
|
||||
|
||||
# Advoware API initialisieren (für später)
|
||||
# advoware = AdvowareAPI(context)
|
||||
|
||||
# PLATZHALTER: Für jetzt nur loggen, keine API-Anfrage
|
||||
context.logger.info(f"PLATZHALTER: {action.upper()} Sync für Entity {entity_id} würde hier Advoware API aufrufen")
|
||||
context.logger.info(f"PLATZHALTER: Entity-Daten würden hier verarbeitet werden")
|
||||
|
||||
# TODO: Hier die Entity in das Zielsystem syncen (EspoCRM?)
|
||||
# Für Create: Neu anlegen
|
||||
# Für Update: Aktualisieren
|
||||
# Für Delete: Löschen
|
||||
|
||||
# Entferne die ID aus der entsprechenden Pending-Queue
|
||||
# Redis für Queue-Management
|
||||
redis_client = redis.Redis(
|
||||
host=Config.REDIS_HOST,
|
||||
port=int(Config.REDIS_PORT),
|
||||
@@ -43,10 +48,214 @@ async def handler(event_data, context):
|
||||
decode_responses=True
|
||||
)
|
||||
|
||||
# APIs initialisieren
|
||||
espocrm = EspoCRMAPI()
|
||||
advoware = AdvowareAPI(context)
|
||||
sync_utils = BeteiligteSync(espocrm, context)
|
||||
mapper = BeteiligteMapper()
|
||||
|
||||
try:
|
||||
# 1. ACQUIRE LOCK (verhindert parallele Syncs)
|
||||
lock_acquired = await sync_utils.acquire_sync_lock(entity_id)
|
||||
|
||||
if not lock_acquired:
|
||||
context.logger.warning(f"⏸️ Sync bereits aktiv für {entity_id}, überspringe")
|
||||
return
|
||||
|
||||
# 2. FETCH ENTITY VON ESPOCRM
|
||||
try:
|
||||
espo_entity = await espocrm.get_entity('CBeteiligte', entity_id)
|
||||
except Exception as e:
|
||||
context.logger.error(f"❌ Fehler beim Laden von EspoCRM Entity: {e}")
|
||||
await sync_utils.release_sync_lock(entity_id, 'failed', str(e), increment_retry=True)
|
||||
return
|
||||
|
||||
context.logger.info(f"📋 Entity geladen: {espo_entity.get('name')} (betnr: {espo_entity.get('betnr')})")
|
||||
|
||||
betnr = espo_entity.get('betnr')
|
||||
sync_status = espo_entity.get('syncStatus', 'pending_sync')
|
||||
|
||||
# 3. BESTIMME SYNC-AKTION
|
||||
|
||||
# FALL A: Neu (kein betnr) → CREATE in Advoware
|
||||
if not betnr and action in ['create', 'sync_check']:
|
||||
context.logger.info(f"🆕 Neuer Beteiligter → CREATE in Advoware")
|
||||
await handle_create(entity_id, espo_entity, espocrm, advoware, sync_utils, mapper, context)
|
||||
|
||||
# FALL B: Existiert (hat betnr) → UPDATE oder CHECK
|
||||
elif betnr:
|
||||
context.logger.info(f"♻️ Existierender Beteiligter (betNr: {betnr}) → UPDATE/CHECK")
|
||||
await handle_update(entity_id, betnr, espo_entity, espocrm, advoware, sync_utils, mapper, context)
|
||||
|
||||
# FALL C: DELETE (TODO: Implementierung später)
|
||||
elif action == 'delete':
|
||||
context.logger.warning(f"🗑️ DELETE noch nicht implementiert für {entity_id}")
|
||||
await sync_utils.release_sync_lock(entity_id, 'failed', 'Delete-Operation nicht implementiert')
|
||||
|
||||
else:
|
||||
context.logger.warning(f"⚠️ Unbekannte Kombination: action={action}, betnr={betnr}")
|
||||
await sync_utils.release_sync_lock(entity_id, 'failed', f'Unbekannte Aktion: {action}')
|
||||
|
||||
# Redis Queue Cleanup
|
||||
pending_key = f'vmh:beteiligte:{action}_pending'
|
||||
redis_client.srem(pending_key, entity_id)
|
||||
context.logger.info(f"Entity {entity_id} aus {action.upper()}-Pending-Queue entfernt")
|
||||
|
||||
except Exception as e:
|
||||
context.logger.error(f"Fehler beim {event_data.get('action', 'unknown').upper()} Sync von Beteiligte Entity: {e}")
|
||||
context.logger.error(f"Event Data: {event_data}")
|
||||
context.logger.error(f"❌ Unerwarteter Fehler im Sync-Handler: {e}")
|
||||
import traceback
|
||||
context.logger.error(traceback.format_exc())
|
||||
|
||||
try:
|
||||
await sync_utils.release_sync_lock(
|
||||
entity_id,
|
||||
'failed',
|
||||
f'Unerwarteter Fehler: {str(e)[:1900]}',
|
||||
increment_retry=True
|
||||
)
|
||||
except:
|
||||
pass
|
||||
|
||||
|
||||
async def handle_create(entity_id, espo_entity, espocrm, advoware, sync_utils, mapper, context):
|
||||
"""Erstellt neuen Beteiligten in Advoware"""
|
||||
try:
|
||||
context.logger.info(f"🔨 CREATE in Advoware...")
|
||||
|
||||
# Transform zu Advoware Format
|
||||
advo_data = mapper.map_cbeteiligte_to_advoware(espo_entity)
|
||||
|
||||
context.logger.info(f"📤 Sende an Advoware: {json.dumps(advo_data, ensure_ascii=False)[:200]}...")
|
||||
|
||||
# POST zu Advoware
|
||||
result = await advoware.api_call(
|
||||
'api/v1/advonet/Beteiligte',
|
||||
method='POST',
|
||||
data=advo_data
|
||||
)
|
||||
|
||||
# Extrahiere betNr aus Response
|
||||
new_betnr = result.get('betNr') if isinstance(result, dict) else None
|
||||
|
||||
if not new_betnr:
|
||||
raise Exception(f"Keine betNr in Advoware Response: {result}")
|
||||
|
||||
context.logger.info(f"✅ In Advoware erstellt: betNr={new_betnr}")
|
||||
|
||||
# Update EspoCRM mit neuer betNr
|
||||
await sync_utils.release_sync_lock(entity_id, 'clean', error_message=None)
|
||||
await espocrm.update_entity('CBeteiligte', entity_id, {
|
||||
'betnr': new_betnr
|
||||
})
|
||||
|
||||
context.logger.info(f"✅ CREATE erfolgreich: {entity_id} → betNr {new_betnr}")
|
||||
|
||||
except Exception as e:
|
||||
context.logger.error(f"❌ CREATE fehlgeschlagen: {e}")
|
||||
await sync_utils.release_sync_lock(entity_id, 'failed', str(e), increment_retry=True)
|
||||
|
||||
|
||||
async def handle_update(entity_id, betnr, espo_entity, espocrm, advoware, sync_utils, mapper, context):
|
||||
"""Synchronisiert existierenden Beteiligten"""
|
||||
try:
|
||||
context.logger.info(f"🔍 Fetch von Advoware betNr={betnr}...")
|
||||
|
||||
# Fetch von Advoware
|
||||
try:
|
||||
advo_result = await advoware.api_call(
|
||||
f'api/v1/advonet/Beteiligte/{betnr}',
|
||||
method='GET'
|
||||
)
|
||||
|
||||
# Advoware gibt manchmal Listen zurück
|
||||
if isinstance(advo_result, list):
|
||||
advo_entity = advo_result[0] if advo_result else None
|
||||
else:
|
||||
advo_entity = advo_result
|
||||
|
||||
if not advo_entity:
|
||||
raise Exception(f"Beteiligter betNr={betnr} nicht gefunden")
|
||||
|
||||
except Exception as e:
|
||||
# 404 oder anderer Fehler → Beteiligter wurde in Advoware gelöscht
|
||||
if '404' in str(e) or 'nicht gefunden' in str(e).lower():
|
||||
context.logger.warning(f"🗑️ Beteiligter in Advoware gelöscht: betNr={betnr}")
|
||||
await sync_utils.handle_advoware_deleted(entity_id, str(e))
|
||||
return
|
||||
else:
|
||||
raise
|
||||
|
||||
context.logger.info(f"📥 Von Advoware geladen: {advo_entity.get('name')}")
|
||||
|
||||
# TIMESTAMP-VERGLEICH
|
||||
comparison = sync_utils.compare_timestamps(
|
||||
espo_entity.get('modifiedAt'),
|
||||
advo_entity.get('geaendertAm'),
|
||||
espo_entity.get('advowareLastSync')
|
||||
)
|
||||
|
||||
context.logger.info(f"⏱️ Timestamp-Vergleich: {comparison}")
|
||||
|
||||
# KEIN SYNC NÖTIG
|
||||
if comparison == 'no_change':
|
||||
context.logger.info(f"✅ Keine Änderungen, Sync übersprungen")
|
||||
await sync_utils.release_sync_lock(entity_id, 'clean')
|
||||
return
|
||||
|
||||
# ESPOCRM NEUER → Update Advoware
|
||||
if comparison == 'espocrm_newer':
|
||||
context.logger.info(f"📤 EspoCRM ist neuer → Update Advoware")
|
||||
|
||||
advo_data = mapper.map_cbeteiligte_to_advoware(espo_entity)
|
||||
|
||||
await advoware.api_call(
|
||||
f'api/v1/advonet/Beteiligte/{betnr}',
|
||||
method='PUT',
|
||||
data=advo_data
|
||||
)
|
||||
|
||||
await sync_utils.release_sync_lock(entity_id, 'clean')
|
||||
context.logger.info(f"✅ Advoware aktualisiert")
|
||||
|
||||
# ADVOWARE NEUER → Update EspoCRM
|
||||
elif comparison == 'advoware_newer':
|
||||
context.logger.info(f"📥 Advoware ist neuer → Update EspoCRM")
|
||||
|
||||
espo_data = mapper.map_advoware_to_cbeteiligte(advo_entity)
|
||||
|
||||
await espocrm.update_entity('CBeteiligte', entity_id, espo_data)
|
||||
await sync_utils.release_sync_lock(entity_id, 'clean')
|
||||
context.logger.info(f"✅ EspoCRM aktualisiert")
|
||||
|
||||
# KONFLIKT → EspoCRM WINS
|
||||
elif comparison == 'conflict':
|
||||
context.logger.warning(f"⚠️ KONFLIKT erkannt → EspoCRM WINS")
|
||||
|
||||
# Überschreibe Advoware mit EspoCRM
|
||||
advo_data = mapper.map_cbeteiligte_to_advoware(espo_entity)
|
||||
|
||||
await advoware.api_call(
|
||||
f'api/v1/advonet/Beteiligte/{betnr}',
|
||||
method='PUT',
|
||||
data=advo_data
|
||||
)
|
||||
|
||||
conflict_msg = (
|
||||
f"EspoCRM: {espo_entity.get('modifiedAt')}, "
|
||||
f"Advoware: {advo_entity.get('geaendertAm')}. "
|
||||
f"EspoCRM hat gewonnen."
|
||||
)
|
||||
|
||||
await sync_utils.resolve_conflict_espocrm_wins(
|
||||
entity_id,
|
||||
espo_entity,
|
||||
advo_entity,
|
||||
conflict_msg
|
||||
)
|
||||
|
||||
context.logger.info(f"✅ Konflikt gelöst: EspoCRM → Advoware")
|
||||
|
||||
except Exception as e:
|
||||
context.logger.error(f"❌ UPDATE fehlgeschlagen: {e}")
|
||||
import traceback
|
||||
context.logger.error(traceback.format_exc())
|
||||
await sync_utils.release_sync_lock(entity_id, 'failed', str(e), increment_retry=True)
|
||||
Reference in New Issue
Block a user