update to iii 0.90 and change directory structure

This commit is contained in:
bsiggel
2026-03-19 20:33:49 +00:00
parent 2ac83df1e0
commit 46085bd8dd
38 changed files with 0 additions and 49 deletions

View File

@@ -0,0 +1,268 @@
# Advoware Calendar Sync - Event-Driven Design
Dieser Abschnitt implementiert die bidirektionale Synchronisation zwischen Advoware-Terminen und Google Calendar. Das System nutzt einen event-driven Ansatz mit **Motia III v1.0-RC**, der auf direkten API-Calls basiert, mit Redis für Locking und Deduplikation. Es stellt sicher, dass Termine konsistent gehalten werden, mit Fokus auf Robustheit, Fehlerbehandlung und korrekte Handhabung von mehrtägigen Terminen.
## Übersicht
Das System synchronisiert Termine zwischen:
- **Advoware**: Zentrale Terminverwaltung mit detaillierten Informationen.
- **Google Calendar**: Benutzerfreundliche Kalenderansicht für jeden Mitarbeiter.
## Architektur
### Event-Driven Design
- **Direkte API-Synchronisation**: Kein zentraler Hub; Sync läuft direkt zwischen APIs.
- **Redis Locking**: Per-Employee Locking verhindert Race-Conditions.
- **Event Emission**: Cron → All-Step → Employee-Step für skalierbare Verarbeitung.
- **Fehlerresistenz**: Einzelne Fehler stoppen nicht den gesamten Sync.
- **Logging**: Alle Logs erscheinen in der iii Console via `ctx.logger`.
### Sync-Phasen
1. **Cron-Step**: Automatische Auslösung alle 15 Minuten.
2. **All-Step**: Fetcht alle Mitarbeiter und emittiert Events pro Employee.
3. **Employee-Step**: Synchronisiert Termine für einen einzelnen Mitarbeiter.
### Datenmapping und Standardisierung
Beide Systeme werden auf gemeinsames Format normalisiert (Berlin TZ):
```python
{
'start': datetime, # Berlin TZ
'end': datetime,
'text': str,
'notiz': str,
'ort': str,
'dauertermin': int, # 0/1
'turnus': int, # 0/1
'turnusArt': int,
'recurrence': str # RRULE oder None
}
```
#### Advoware → Standard
- Start: `datum` + `uhrzeitVon` (Fallback 09:00), oder `datum` als datetime.
- End: `datumBis` + `uhrzeitBis` (Fallback 10:00), oder `datum` + 1h.
- All-Day: `dauertermin=1` oder Dauer >1 Tag.
- Recurring: `turnus`/`turnusArt` (vereinfacht, keine RRULE).
#### Google → Standard
- Start/End: `dateTime` oder `date` (All-Day).
- All-Day: `dauertermin=1` wenn All-Day oder Dauer >1 Tag.
- Recurring: RRULE aus `recurrence`.
#### Standard → Advoware
- POST/PUT: `datum`/`uhrzeitBis`/`datumBis` aus start/end.
- Defaults: `vorbereitungsDauer='00:00:00'`, `sb`/`anwalt`=employee_kuerzel.
#### Standard → Google
- All-Day: `date` statt `dateTime`, end +1 Tag.
- Recurring: RRULE aus `recurrence`.
## Funktionalität
### Automatische Kalender-Erstellung
- Für jeden Advoware-Mitarbeiter wird ein Google Calendar mit dem Namen `AW-{Kuerzel}` erstellt.
- Beispiel: Mitarbeiter mit Kürzel "SB" → Calendar "AW-SB".
- Kalender wird mit dem Haupt-Google-Account (`lehmannundpartner@gmail.com`) als Owner geteilt.
### Sync-Details
#### Cron-Step (calendar_sync_cron_step.py)
- Läuft alle 15 Minuten und emittiert "calendar_sync_all".
- **Trigger**: `cron("0 */15 * * * *")` (6-field: Sekunde Minute Stunde Tag Monat Wochentag)
#### All-Step (calendar_sync_all_step.py)
- Fetcht alle Mitarbeiter aus Advoware.
- Filtert Debug-Liste (falls konfiguriert).
- Setzt Redis-Lock pro Employee.
- Emittiert "calendar_sync" Event pro Employee.
- **Trigger**: `queue('calendar_sync_all')`
#### Employee-Step (calendar_sync_event_step.py)
- Fetcht Advoware-Termine für den Employee.
- Fetcht Google-Events für den Employee.
- Synchronisiert: Neue erstellen, Updates anwenden, Deletes handhaben.
- Verwendet Locking, um parallele Syncs zu verhindern.
- **Trigger**: `queue('calendar_sync')`
#### API-Step (calendar_sync_api_step.py)
- Manueller Trigger für einzelnen Employee oder "ALL".
- Bei "ALL": Emittiert "calendar_sync_all".
- Bei Employee: Setzt Lock und emittiert "calendar_sync".
- **Trigger**: `http('POST', '/advoware/calendar/sync')`
## API-Schwächen und Fixes
### Advoware API
- **Mehrtägige Termine**: `datumBis` wird korrekt für Enddatum verwendet; '00:00:00' als '23:59:59' interpretiert.
- **Zeitformate**: Robuste Parsing mit Fallbacks.
- **Keine 24h-Limit**: Termine können länger als 24h sein; Google Calendar unterstützt das.
### Google Calendar API
- **Zeitbereiche**: Akzeptiert Events >24h ohne Probleme.
- **Rate Limits**: Backoff-Retry implementiert.
## Step-Konfiguration (Motia III)
### calendar_sync_cron_step.py
```python
config = {
'name': 'Calendar Sync Cron Job',
'flows': ['advoware'],
'triggers': [
cron("0 */15 * * * *") # Alle 15 Minuten (6-field format)
],
'enqueues': ['calendar_sync_all']
}
```
### calendar_sync_all_step.py
```python
config = {
'name': 'Calendar Sync All Step',
'flows': ['advoware'],
'triggers': [
queue('calendar_sync_all')
],
'enqueues': ['calendar_sync']
}
```
### calendar_sync_event_step.py
```python
config = {
'name': 'Calendar Sync Event Step',
'flows': ['advoware'],
'triggers': [
queue('calendar_sync')
],
'enqueues': []
}
```
### calendar_sync_api_step.py
```python
config = {
'name': 'Calendar Sync API Trigger',
'flows': ['advoware'],
'triggers': [
http('POST', '/advoware/calendar/sync')
],
'enqueues': ['calendar_sync', 'calendar_sync_all']
}
```
## Setup
### Umgebungsvariablen
```env
# Google Calendar
GOOGLE_CALENDAR_SERVICE_ACCOUNT_PATH=service-account.json
# Advoware API
ADVOWARE_API_BASE_URL=https://www2.advo-net.net:90/
ADVOWARE_PRODUCT_ID=64
ADVOWARE_APP_ID=your_app_id
ADVOWARE_API_KEY=your_api_key
ADVOWARE_KANZLEI=your_kanzlei
ADVOWARE_DATABASE=your_database
ADVOWARE_USER=your_user
ADVOWARE_ROLE=2
ADVOWARE_PASSWORD=your_password
ADVOWARE_TOKEN_LIFETIME_MINUTES=55
ADVOWARE_API_TIMEOUT_SECONDS=30
# Redis
REDIS_HOST=localhost
REDIS_PORT=6379
REDIS_DB_CALENDAR_SYNC=1
REDIS_TIMEOUT_SECONDS=5
# Debug
CALENDAR_SYNC_DEBUG_EMPLOYEES=PB,AI # Optional, filter employees
```
## Verwendung
### Manueller Sync
```bash
# Sync für einen bestimmten Mitarbeiter
curl -X POST "http://localhost:3111/advoware/calendar/sync" \
-H "Content-Type: application/json" \
-d '{"kuerzel": "PB"}'
# Sync für alle Mitarbeiter
curl -X POST "http://localhost:3111/advoware/calendar/sync" \
-H "Content-Type: application/json" \
-d '{"kuerzel": "ALL"}'
```
### Automatischer Sync
Cron-Step läuft automatisch alle 15 Minuten.
## Fehlerbehandlung und Logging
- **Locking**: Redis NX/EX verhindert parallele Syncs.
- **Logging**: `ctx.logger` für iii Console-Sichtbarkeit.
- **API-Fehler**: Retry mit Backoff.
- **Parsing-Fehler**: Robuste Fallbacks.
## Sicherheit
- Service Account für Google Calendar API.
- HMAC-512 Authentifizierung für Advoware API.
- Redis für Concurrency-Control.
## Bekannte Probleme
- **Recurring-Events**: Begrenzte Unterstützung für komplexe Wiederholungen.
- **Performance**: Bei vielen Terminen Paginierung beachten.
- **Timezone-Handling**: Alle Operationen in Europe/Berlin TZ.
## Datenfluss
```
Cron (alle 15min)
→ calendar_sync_cron_step
→ ctx.enqueue(topic: "calendar_sync_all")
→ calendar_sync_all_step
→ Fetch Employees from Advoware
→ For each Employee:
→ Set Redis Lock (key: calendar_sync:employee:{kuerzel})
→ ctx.enqueue(topic: "calendar_sync", data: {kuerzel, ...})
→ calendar_sync_event_step
→ Fetch Advoware Termine (frNr, datum, text, etc.)
→ Fetch Google Calendar Events
→ 4-Phase Sync:
1. New from Advoware → Google
2. New from Google → Advoware
3. Process Deletes
4. Process Updates
→ Clear Redis Lock
```
## Weitere Dokumentation
- **Individual Step Docs**: Siehe `docs/` Ordner in diesem Verzeichnis
- **Architecture Overview**: [../../docs/ARCHITECTURE.md](../../docs/ARCHITECTURE.md)
- **Google Setup Guide**: [../../docs/GOOGLE_SETUP.md](../../docs/GOOGLE_SETUP.md)
- **Troubleshooting**: [../../docs/TROUBLESHOOTING.md](../../docs/TROUBLESHOOTING.md)
## Migration Notes
Dieses System wurde von **Motia v0.17** nach **Motia III v1.0-RC** migriert:
### Wichtige Änderungen:
-`type: 'event'``triggers: [queue('topic')]`
-`type: 'cron'``triggers: [cron('expression')]` (6-field format)
-`type: 'api'``triggers: [http('METHOD', 'path')]`
-`context.emit()``ctx.enqueue()`
-`emits: [...]``enqueues: [...]`
- ✅ Relative Imports → Absolute Imports mit `sys.path.insert()`
- ✅ Motia Workbench → iii Console
### Kompatibilität:
- ✅ Alle 4 Steps vollständig migriert
- ✅ Google Calendar API Integration unverändert
- ✅ Advoware API Integration unverändert
- ✅ Redis Locking-Mechanismus unverändert
- ✅ Datenbank-Schema kompatibel

View File

@@ -0,0 +1,5 @@
"""
Advoware Calendar Sync Module
Bidirectional synchronization between Google Calendar and Advoware appointments.
"""

View File

@@ -0,0 +1,113 @@
"""
Calendar Sync All Step
Handles calendar_sync_all event and emits individual sync events for oldest employees.
Uses Redis to track last sync times and distribute work.
"""
import sys
from pathlib import Path
sys.path.insert(0, str(Path(__file__).parent))
from calendar_sync_utils import (
get_redis_client,
get_advoware_employees,
set_employee_lock,
log_operation
)
import math
import time
from datetime import datetime
from typing import Any, Dict
from motia import queue, FlowContext
from pydantic import BaseModel, Field
from services.advoware_service import AdvowareService
config = {
'name': 'Calendar Sync All Step',
'description': 'Receives sync-all event and emits individual events for oldest employees',
'flows': ['advoware-calendar-sync'],
'triggers': [
queue('calendar_sync_all')
],
'enqueues': ['calendar_sync_employee']
}
async def handler(input_data: Dict[str, Any], ctx: FlowContext) -> None:
"""
Handler that fetches all employees, sorts by last sync time,
and emits calendar_sync_employee events for the oldest ones.
"""
try:
triggered_by = input_data.get('triggered_by', 'unknown')
log_operation('info', f"Calendar Sync All: Starting to emit events for oldest employees, triggered by {triggered_by}", context=ctx)
# Initialize Advoware service
advoware = AdvowareService(ctx)
# Fetch employees
employees = await get_advoware_employees(advoware, ctx)
if not employees:
log_operation('error', "Keine Mitarbeiter gefunden. All-Sync abgebrochen.", context=ctx)
return {'status': 500, 'body': {'error': 'Keine Mitarbeiter gefunden'}}
redis_client = get_redis_client(ctx)
# Collect last_synced timestamps
employee_timestamps = {}
for employee in employees:
kuerzel = employee.get('kuerzel')
if not kuerzel:
continue
employee_last_synced_key = f'calendar_sync_last_synced_{kuerzel}'
timestamp_str = redis_client.get(employee_last_synced_key)
timestamp = int(timestamp_str) if timestamp_str else 0 # 0 if no timestamp (very old)
employee_timestamps[kuerzel] = timestamp
# Sort employees by last_synced (ascending, oldest first), then by kuerzel alphabetically
sorted_kuerzel = sorted(employee_timestamps.keys(), key=lambda k: (employee_timestamps[k], k))
# Log the sorted list with timestamps
def format_timestamp(ts):
if ts == 0:
return "never"
return datetime.fromtimestamp(ts).strftime('%Y-%m-%d %H:%M:%S')
sorted_list_str = ", ".join(f"{k} ({format_timestamp(employee_timestamps[k])})" for k in sorted_kuerzel)
log_operation('info', f"Calendar Sync All: Sorted employees by last synced: {sorted_list_str}", context=ctx)
# Calculate number to sync: ceil(N / 10)
num_to_sync = math.ceil(len(sorted_kuerzel) / 1)
log_operation('info', f"Calendar Sync All: Total employees {len(sorted_kuerzel)}, syncing {num_to_sync} oldest", context=ctx)
# Emit for the oldest num_to_sync employees, if not locked
emitted_count = 0
for kuerzel in sorted_kuerzel[:num_to_sync]:
if not set_employee_lock(redis_client, kuerzel, triggered_by, ctx):
log_operation('info', f"Calendar Sync All: Sync already active for {kuerzel}, skipping", context=ctx)
continue
# Emit event for this employee
await ctx.enqueue({
"topic": "calendar_sync_employee",
"data": {
"kuerzel": kuerzel,
"triggered_by": triggered_by
}
})
log_operation('info', f"Calendar Sync All: Emitted event for employee {kuerzel} (last synced: {format_timestamp(employee_timestamps[kuerzel])})", context=ctx)
emitted_count += 1
log_operation('info', f"Calendar Sync All: Completed, emitted {emitted_count} events", context=ctx)
return {
'status': 'completed',
'triggered_by': triggered_by,
'emitted_count': emitted_count
}
except Exception as e:
log_operation('error', f"Fehler beim All-Sync: {e}", context=ctx)
return {
'status': 'error',
'error': str(e)
}

View File

@@ -0,0 +1,112 @@
"""
Calendar Sync API Step
HTTP API endpoint for manual calendar sync triggering.
Supports syncing a single employee or all employees.
"""
import sys
from pathlib import Path
sys.path.insert(0, str(Path(__file__).parent))
from calendar_sync_utils import get_redis_client, set_employee_lock, get_logger
from motia import http, ApiRequest, ApiResponse, FlowContext
config = {
'name': 'Calendar Sync API Trigger',
'description': 'API endpoint for manual calendar sync triggering',
'flows': ['advoware-calendar-sync'],
'triggers': [
http('POST', '/advoware/calendar/sync')
],
'enqueues': ['calendar_sync_employee', 'calendar_sync_all']
}
async def handler(request: ApiRequest, ctx: FlowContext) -> ApiResponse:
"""
HTTP handler for manual calendar sync triggering.
Request body:
{
"kuerzel": "SB" // or "ALL" for all employees
}
"""
try:
# Get kuerzel from request body
body = request.body
kuerzel = body.get('kuerzel')
if not kuerzel:
return ApiResponse(
status=400,
body={
'error': 'kuerzel required',
'message': 'Please provide kuerzel in body'
}
)
kuerzel_upper = kuerzel.upper()
if kuerzel_upper == 'ALL':
# Emit sync-all event
ctx.logger.info("Calendar Sync API: Emitting sync-all event")
await ctx.enqueue({
"topic": "calendar_sync_all",
"data": {
"triggered_by": "api"
}
})
return ApiResponse(
status=200,
body={
'status': 'triggered',
'message': 'Calendar sync triggered for all employees',
'triggered_by': 'api'
}
)
else:
# Single employee sync
redis_client = get_redis_client(ctx)
if not set_employee_lock(redis_client, kuerzel_upper, 'api', ctx):
ctx.logger.info(f"Calendar Sync API: Sync already active for {kuerzel_upper}, skipping")
return ApiResponse(
status=409,
body={
'status': 'conflict',
'message': f'Calendar sync already active for {kuerzel_upper}',
'kuerzel': kuerzel_upper,
'triggered_by': 'api'
}
)
ctx.logger.info(f"Calendar Sync API called for {kuerzel_upper}")
# Lock successfully set, now emit event
await ctx.enqueue({
"topic": "calendar_sync_employee",
"data": {
"kuerzel": kuerzel_upper,
"triggered_by": "api"
}
})
return ApiResponse(
status=200,
body={
'status': 'triggered',
'message': f'Calendar sync triggered for {kuerzel_upper}',
'kuerzel': kuerzel_upper,
'triggered_by': 'api'
}
)
except Exception as e:
ctx.logger.error(f"Error in API trigger: {e}")
return ApiResponse(
status=500,
body={
'error': 'Internal server error',
'details': str(e)
}
)

View File

@@ -0,0 +1,50 @@
"""
Calendar Sync Cron Step
Cron trigger for automatic calendar synchronization.
Emits calendar_sync_all event to start sync cascade.
"""
import sys
from pathlib import Path
sys.path.insert(0, str(Path(__file__).parent))
from calendar_sync_utils import log_operation
from typing import Dict, Any
from motia import cron, FlowContext
config = {
'name': 'Calendar Sync Cron Job',
'description': 'Runs calendar sync automatically every 15 minutes',
'flows': ['advoware-calendar-sync'],
'triggers': [
cron("0 15 1 * * *") # Every 15 minutes at second 0 (6-field: sec min hour day month weekday)
],
'enqueues': ['calendar_sync_all']
}
async def handler(input_data: None, ctx: FlowContext) -> None:
"""Cron handler that triggers the calendar sync cascade."""
try:
ctx.logger.info("=" * 80)
ctx.logger.info("🕐 CALENDAR SYNC CRON: STARTING")
ctx.logger.info("=" * 80)
ctx.logger.info("Emitting sync-all event")
# Enqueue sync-all event
await ctx.enqueue({
"topic": "calendar_sync_all",
"data": {
"triggered_by": "cron"
}
})
ctx.logger.info("✅ Calendar sync-all event emitted successfully")
ctx.logger.info("=" * 80)
except Exception as e:
ctx.logger.error("=" * 80)
ctx.logger.error("❌ ERROR: CALENDAR SYNC CRON")
ctx.logger.error(f"Error: {e}")
ctx.logger.error("=" * 80)

File diff suppressed because it is too large Load Diff

View File

@@ -0,0 +1,133 @@
"""
Calendar Sync Utilities
Shared utility functions for calendar synchronization between Google Calendar and Advoware.
"""
import asyncpg
import os
import redis
import time
from typing import Optional, Any, List
from googleapiclient.discovery import build
from google.oauth2 import service_account
from services.logging_utils import get_service_logger
def get_logger(context=None):
"""Get logger for calendar sync operations"""
return get_service_logger('calendar_sync', context)
def log_operation(level: str, message: str, context=None, **extra):
"""
Log calendar sync operations with structured context.
Args:
level: Log level ('debug', 'info', 'warning', 'error')
message: Log message
context: FlowContext if available
**extra: Additional key-value pairs to log
"""
logger = get_logger(context)
log_func = getattr(logger, level.lower(), logger.info)
if extra:
extra_str = " | " + " | ".join(f"{k}={v}" for k, v in extra.items())
log_func(message + extra_str)
else:
log_func(message)
async def connect_db(context=None):
"""Connect to Postgres DB from environment variables."""
logger = get_logger(context)
try:
conn = await asyncpg.connect(
host=os.getenv('POSTGRES_HOST', 'localhost'),
user=os.getenv('POSTGRES_USER', 'calendar_sync_user'),
password=os.getenv('POSTGRES_PASSWORD', 'default_password'),
database=os.getenv('POSTGRES_DB_NAME', 'calendar_sync_db'),
timeout=10
)
return conn
except Exception as e:
logger.error(f"Failed to connect to DB: {e}")
raise
async def get_google_service(context=None):
"""Initialize Google Calendar service."""
logger = get_logger(context)
try:
service_account_path = os.getenv('GOOGLE_CALENDAR_SERVICE_ACCOUNT_PATH', 'service-account.json')
if not os.path.exists(service_account_path):
raise FileNotFoundError(f"Service account file not found: {service_account_path}")
scopes = ['https://www.googleapis.com/auth/calendar']
creds = service_account.Credentials.from_service_account_file(
service_account_path, scopes=scopes
)
service = build('calendar', 'v3', credentials=creds)
return service
except Exception as e:
logger.error(f"Failed to initialize Google service: {e}")
raise
def get_redis_client(context=None) -> redis.Redis:
"""Initialize Redis client for calendar sync operations."""
logger = get_logger(context)
try:
redis_client = redis.Redis(
host=os.getenv('REDIS_HOST', 'localhost'),
port=int(os.getenv('REDIS_PORT', '6379')),
db=int(os.getenv('REDIS_DB_CALENDAR_SYNC', '2')),
socket_timeout=int(os.getenv('REDIS_TIMEOUT_SECONDS', '5')),
decode_responses=True
)
return redis_client
except Exception as e:
logger.error(f"Failed to initialize Redis client: {e}")
raise
async def get_advoware_employees(advoware, context=None) -> List[Any]:
"""Fetch list of employees from Advoware."""
logger = get_logger(context)
try:
result = await advoware.api_call('api/v1/advonet/Mitarbeiter', method='GET', params={'aktiv': 'true'})
employees = result if isinstance(result, list) else []
logger.info(f"Fetched {len(employees)} Advoware employees")
return employees
except Exception as e:
logger.error(f"Failed to fetch Advoware employees: {e}")
raise
def set_employee_lock(redis_client: redis.Redis, kuerzel: str, triggered_by: str, context=None) -> bool:
"""Set lock for employee sync operation."""
logger = get_logger(context)
employee_lock_key = f'calendar_sync_lock_{kuerzel}'
if redis_client.set(employee_lock_key, triggered_by, ex=1800, nx=True) is None:
logger.info(f"Sync already active for {kuerzel}, skipping")
return False
return True
def clear_employee_lock(redis_client: redis.Redis, kuerzel: str, context=None) -> None:
"""Clear lock for employee sync operation and update last-synced timestamp."""
logger = get_logger(context)
try:
employee_lock_key = f'calendar_sync_lock_{kuerzel}'
employee_last_synced_key = f'calendar_sync_last_synced_{kuerzel}'
# Update last-synced timestamp (no TTL, persistent)
current_time = int(time.time())
redis_client.set(employee_last_synced_key, current_time)
# Delete the lock
redis_client.delete(employee_lock_key)
logger.debug(f"Cleared lock and updated last-synced for {kuerzel} to {current_time}")
except Exception as e:
logger.warning(f"Failed to clear lock and update last-synced for {kuerzel}: {e}")