Compare commits
2 Commits
93d4d89531
...
d69801ed97
| Author | SHA1 | Date | |
|---|---|---|---|
|
|
d69801ed97 | ||
|
|
6e2303c5eb |
131
docs/INDEX.md
131
docs/INDEX.md
@@ -4,6 +4,7 @@
|
|||||||
|
|
||||||
**Quick Navigation:**
|
**Quick Navigation:**
|
||||||
- [Core Concepts](#core-concepts) - System architecture and patterns
|
- [Core Concepts](#core-concepts) - System architecture and patterns
|
||||||
|
- [Design Principles](#design-principles) - Event Storm & Bidirectional References
|
||||||
- [Step Development](#step-development-best-practices) - How to create new steps
|
- [Step Development](#step-development-best-practices) - How to create new steps
|
||||||
- [Services](#service-layer-patterns) - Reusable business logic
|
- [Services](#service-layer-patterns) - Reusable business logic
|
||||||
- [Integrations](#external-integrations) - xAI, EspoCRM, Advoware
|
- [Integrations](#external-integrations) - xAI, EspoCRM, Advoware
|
||||||
@@ -47,6 +48,136 @@ Webhook (HTTP) → Queue Event → Event Handler → External APIs
|
|||||||
|
|
||||||
---
|
---
|
||||||
|
|
||||||
|
## Design Principles
|
||||||
|
|
||||||
|
### Event Storm over Lock Coordination
|
||||||
|
|
||||||
|
**Core Philosophy: "Idempotent Chaos - Check Cheap, Sync Once"**
|
||||||
|
|
||||||
|
```
|
||||||
|
╔═══════════════════════════════════════════════════════╗
|
||||||
|
║ Prefer: ║
|
||||||
|
║ ✅ Multiple redundant events over coordination ║
|
||||||
|
║ ✅ Idempotent handlers over distributed locks ║
|
||||||
|
║ ✅ Eventually consistent over perfectly synced ║
|
||||||
|
║ ✅ Simple duplication over complex orchestration ║
|
||||||
|
╚═══════════════════════════════════════════════════════╝
|
||||||
|
```
|
||||||
|
|
||||||
|
**Guidelines:**
|
||||||
|
|
||||||
|
1. **Fire events liberally** - 10 redundant events are cheaper than complex coordination
|
||||||
|
2. **Make handlers idempotent** - Early returns when nothing to do
|
||||||
|
3. **Sequential per entity, parallel across entities** - Lock prevents collisions, not updates
|
||||||
|
4. **Accept event storms** - Handlers queue up, process one by one
|
||||||
|
|
||||||
|
**Lock Strategy: Sequential Processing per Entity**
|
||||||
|
|
||||||
|
```
|
||||||
|
Event Storm für Document A:
|
||||||
|
├─ Event 1: update → Handler starts → Lock acquired
|
||||||
|
├─ Event 2: update → Queued (waits for lock)
|
||||||
|
├─ Event 3: update → Queued (after Event 2)
|
||||||
|
└─ Event 4: update → Queued (after Event 3)
|
||||||
|
|
||||||
|
Document B (parallel):
|
||||||
|
└─ Event 1: update → Own lock, processes in parallel!
|
||||||
|
|
||||||
|
Result:
|
||||||
|
- Same entity: Sequential (prevents file upload collisions)
|
||||||
|
- Different entities: Parallel (independent locks)
|
||||||
|
- Lost events: Zero (all queued and processed)
|
||||||
|
- Duplicate work: Prevented by idempotency checks
|
||||||
|
```
|
||||||
|
|
||||||
|
**Example Flow:**
|
||||||
|
```
|
||||||
|
t=0: User ändert Document A (fileStatus → "changed")
|
||||||
|
t=1: Event 1 fired → Lock acquired → Sync starts
|
||||||
|
t=2: User ändert Document A again (fileStatus → "changed")
|
||||||
|
t=3: Event 2 fired → Queued (lock busy)
|
||||||
|
t=4: Event 1 completes → fileStatus="unchanged", xaiSyncStatus="clean"
|
||||||
|
t=5: Event 2 starts → Lock acquired
|
||||||
|
Check: fileStatus="unchanged", xaiSyncStatus="clean"
|
||||||
|
→ Early return (nothing to do) ✅
|
||||||
|
|
||||||
|
Result: Second event processed but no duplicate work!
|
||||||
|
```
|
||||||
|
|
||||||
|
**Why This Works:**
|
||||||
|
- **Lock prevents chaos**: No parallel file uploads for same entity
|
||||||
|
- **Queue enables updates**: New changes processed sequentially
|
||||||
|
- **Idempotency prevents waste**: Redundant events → cheap early returns
|
||||||
|
- **Parallel scaling**: Different entities process simultaneously
|
||||||
|
|
||||||
|
**Practical Example: Entity Link Event**
|
||||||
|
```
|
||||||
|
User links Document ↔ Räumungsklage
|
||||||
|
|
||||||
|
Webhooks fire:
|
||||||
|
├─ POST /vmh/webhook/entity/link
|
||||||
|
└─ Emits: raeumungsklage.update, cdokumente.update
|
||||||
|
|
||||||
|
Handlers (parallel, different entities):
|
||||||
|
├─ Räumungsklage Handler
|
||||||
|
│ ├─ Lock: raeumungsklage:abc123
|
||||||
|
│ ├─ Creates xAI Collection (if missing)
|
||||||
|
│ └─ Fires: cdokumente.update (for all linked docs) ← Event Storm!
|
||||||
|
│
|
||||||
|
└─ Document Handler (may run 2-3x on same doc)
|
||||||
|
├─ Lock: document:doc456 (sequential processing)
|
||||||
|
├─ Run 1: Collections not ready → Skip (cheap return)
|
||||||
|
├─ Run 2: Collection ready → Upload & sync
|
||||||
|
└─ Run 3: Already synced → Early return (idempotent!)
|
||||||
|
```
|
||||||
|
|
||||||
|
### Bidirectional Reference Pattern
|
||||||
|
|
||||||
|
**Principle: Every sync maintains references on both sides**
|
||||||
|
|
||||||
|
**EspoCRM as Central Hub:**
|
||||||
|
```
|
||||||
|
┌─────────────┐ ┌──────────────┐ ┌─────────────┐
|
||||||
|
│ Advoware │◄────────┤ EspoCRM ├────────►│ xAI │
|
||||||
|
│ │ │ │ │ │
|
||||||
|
│ Akte 12345 │ │ Entity │ │ Collection │
|
||||||
|
└─────────────┘ │ - advowareId │ └─────────────┘
|
||||||
|
│ - xaiColId │
|
||||||
|
└──────────────┘
|
||||||
|
```
|
||||||
|
|
||||||
|
**Implementation:**
|
||||||
|
- **xAI Collection** → stores `entityType` + `entityId` in metadata
|
||||||
|
- **EspoCRM Entity** → stores `xaiCollectionId` field
|
||||||
|
- **EspoCRM Document** → stores `xaiFileId` + `xaiCollections[]` fields
|
||||||
|
- **Advoware Integration** → stores `advowareAkteId` in EspoCRM
|
||||||
|
|
||||||
|
**Benefits:**
|
||||||
|
- Bidirectional navigation without complex queries
|
||||||
|
- Easy relationship building (e.g., Advoware Akte ↔ xAI Collection via EspoCRM)
|
||||||
|
- Idempotent lookups (can verify both directions)
|
||||||
|
- Debugging: Always know where things came from
|
||||||
|
|
||||||
|
**Example: Collection Creation**
|
||||||
|
```python
|
||||||
|
# xAI Collection
|
||||||
|
collection = await xai.create_collection(
|
||||||
|
name="CBeteiligte - Max Mustermann",
|
||||||
|
metadata={
|
||||||
|
"espocrm_entity_type": "CBeteiligte",
|
||||||
|
"espocrm_entity_id": "abc123",
|
||||||
|
"created_at": "2026-03-08T19:00:00Z"
|
||||||
|
}
|
||||||
|
)
|
||||||
|
|
||||||
|
# EspoCRM Entity
|
||||||
|
await espocrm.update_entity("CBeteiligte", "abc123", {
|
||||||
|
"xaiCollectionId": collection.id
|
||||||
|
})
|
||||||
|
```
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
## Step Development Best Practices
|
## Step Development Best Practices
|
||||||
|
|
||||||
### File Naming Convention
|
### File Naming Convention
|
||||||
|
|||||||
Reference in New Issue
Block a user