update to iii 0.90 and change directory structure
This commit is contained in:
268
src/steps/advoware_cal_sync/README.md
Normal file
268
src/steps/advoware_cal_sync/README.md
Normal file
@@ -0,0 +1,268 @@
|
||||
# Advoware Calendar Sync - Event-Driven Design
|
||||
|
||||
Dieser Abschnitt implementiert die bidirektionale Synchronisation zwischen Advoware-Terminen und Google Calendar. Das System nutzt einen event-driven Ansatz mit **Motia III v1.0-RC**, der auf direkten API-Calls basiert, mit Redis für Locking und Deduplikation. Es stellt sicher, dass Termine konsistent gehalten werden, mit Fokus auf Robustheit, Fehlerbehandlung und korrekte Handhabung von mehrtägigen Terminen.
|
||||
|
||||
## Übersicht
|
||||
|
||||
Das System synchronisiert Termine zwischen:
|
||||
- **Advoware**: Zentrale Terminverwaltung mit detaillierten Informationen.
|
||||
- **Google Calendar**: Benutzerfreundliche Kalenderansicht für jeden Mitarbeiter.
|
||||
|
||||
## Architektur
|
||||
|
||||
### Event-Driven Design
|
||||
- **Direkte API-Synchronisation**: Kein zentraler Hub; Sync läuft direkt zwischen APIs.
|
||||
- **Redis Locking**: Per-Employee Locking verhindert Race-Conditions.
|
||||
- **Event Emission**: Cron → All-Step → Employee-Step für skalierbare Verarbeitung.
|
||||
- **Fehlerresistenz**: Einzelne Fehler stoppen nicht den gesamten Sync.
|
||||
- **Logging**: Alle Logs erscheinen in der iii Console via `ctx.logger`.
|
||||
|
||||
### Sync-Phasen
|
||||
1. **Cron-Step**: Automatische Auslösung alle 15 Minuten.
|
||||
2. **All-Step**: Fetcht alle Mitarbeiter und emittiert Events pro Employee.
|
||||
3. **Employee-Step**: Synchronisiert Termine für einen einzelnen Mitarbeiter.
|
||||
|
||||
### Datenmapping und Standardisierung
|
||||
Beide Systeme werden auf gemeinsames Format normalisiert (Berlin TZ):
|
||||
```python
|
||||
{
|
||||
'start': datetime, # Berlin TZ
|
||||
'end': datetime,
|
||||
'text': str,
|
||||
'notiz': str,
|
||||
'ort': str,
|
||||
'dauertermin': int, # 0/1
|
||||
'turnus': int, # 0/1
|
||||
'turnusArt': int,
|
||||
'recurrence': str # RRULE oder None
|
||||
}
|
||||
```
|
||||
|
||||
#### Advoware → Standard
|
||||
- Start: `datum` + `uhrzeitVon` (Fallback 09:00), oder `datum` als datetime.
|
||||
- End: `datumBis` + `uhrzeitBis` (Fallback 10:00), oder `datum` + 1h.
|
||||
- All-Day: `dauertermin=1` oder Dauer >1 Tag.
|
||||
- Recurring: `turnus`/`turnusArt` (vereinfacht, keine RRULE).
|
||||
|
||||
#### Google → Standard
|
||||
- Start/End: `dateTime` oder `date` (All-Day).
|
||||
- All-Day: `dauertermin=1` wenn All-Day oder Dauer >1 Tag.
|
||||
- Recurring: RRULE aus `recurrence`.
|
||||
|
||||
#### Standard → Advoware
|
||||
- POST/PUT: `datum`/`uhrzeitBis`/`datumBis` aus start/end.
|
||||
- Defaults: `vorbereitungsDauer='00:00:00'`, `sb`/`anwalt`=employee_kuerzel.
|
||||
|
||||
#### Standard → Google
|
||||
- All-Day: `date` statt `dateTime`, end +1 Tag.
|
||||
- Recurring: RRULE aus `recurrence`.
|
||||
|
||||
## Funktionalität
|
||||
|
||||
### Automatische Kalender-Erstellung
|
||||
- Für jeden Advoware-Mitarbeiter wird ein Google Calendar mit dem Namen `AW-{Kuerzel}` erstellt.
|
||||
- Beispiel: Mitarbeiter mit Kürzel "SB" → Calendar "AW-SB".
|
||||
- Kalender wird mit dem Haupt-Google-Account (`lehmannundpartner@gmail.com`) als Owner geteilt.
|
||||
|
||||
### Sync-Details
|
||||
|
||||
#### Cron-Step (calendar_sync_cron_step.py)
|
||||
- Läuft alle 15 Minuten und emittiert "calendar_sync_all".
|
||||
- **Trigger**: `cron("0 */15 * * * *")` (6-field: Sekunde Minute Stunde Tag Monat Wochentag)
|
||||
|
||||
#### All-Step (calendar_sync_all_step.py)
|
||||
- Fetcht alle Mitarbeiter aus Advoware.
|
||||
- Filtert Debug-Liste (falls konfiguriert).
|
||||
- Setzt Redis-Lock pro Employee.
|
||||
- Emittiert "calendar_sync" Event pro Employee.
|
||||
- **Trigger**: `queue('calendar_sync_all')`
|
||||
|
||||
#### Employee-Step (calendar_sync_event_step.py)
|
||||
- Fetcht Advoware-Termine für den Employee.
|
||||
- Fetcht Google-Events für den Employee.
|
||||
- Synchronisiert: Neue erstellen, Updates anwenden, Deletes handhaben.
|
||||
- Verwendet Locking, um parallele Syncs zu verhindern.
|
||||
- **Trigger**: `queue('calendar_sync')`
|
||||
|
||||
#### API-Step (calendar_sync_api_step.py)
|
||||
- Manueller Trigger für einzelnen Employee oder "ALL".
|
||||
- Bei "ALL": Emittiert "calendar_sync_all".
|
||||
- Bei Employee: Setzt Lock und emittiert "calendar_sync".
|
||||
- **Trigger**: `http('POST', '/advoware/calendar/sync')`
|
||||
|
||||
## API-Schwächen und Fixes
|
||||
|
||||
### Advoware API
|
||||
- **Mehrtägige Termine**: `datumBis` wird korrekt für Enddatum verwendet; '00:00:00' als '23:59:59' interpretiert.
|
||||
- **Zeitformate**: Robuste Parsing mit Fallbacks.
|
||||
- **Keine 24h-Limit**: Termine können länger als 24h sein; Google Calendar unterstützt das.
|
||||
|
||||
### Google Calendar API
|
||||
- **Zeitbereiche**: Akzeptiert Events >24h ohne Probleme.
|
||||
- **Rate Limits**: Backoff-Retry implementiert.
|
||||
|
||||
## Step-Konfiguration (Motia III)
|
||||
|
||||
### calendar_sync_cron_step.py
|
||||
```python
|
||||
config = {
|
||||
'name': 'Calendar Sync Cron Job',
|
||||
'flows': ['advoware'],
|
||||
'triggers': [
|
||||
cron("0 */15 * * * *") # Alle 15 Minuten (6-field format)
|
||||
],
|
||||
'enqueues': ['calendar_sync_all']
|
||||
}
|
||||
```
|
||||
|
||||
### calendar_sync_all_step.py
|
||||
```python
|
||||
config = {
|
||||
'name': 'Calendar Sync All Step',
|
||||
'flows': ['advoware'],
|
||||
'triggers': [
|
||||
queue('calendar_sync_all')
|
||||
],
|
||||
'enqueues': ['calendar_sync']
|
||||
}
|
||||
```
|
||||
|
||||
### calendar_sync_event_step.py
|
||||
```python
|
||||
config = {
|
||||
'name': 'Calendar Sync Event Step',
|
||||
'flows': ['advoware'],
|
||||
'triggers': [
|
||||
queue('calendar_sync')
|
||||
],
|
||||
'enqueues': []
|
||||
}
|
||||
```
|
||||
|
||||
### calendar_sync_api_step.py
|
||||
```python
|
||||
config = {
|
||||
'name': 'Calendar Sync API Trigger',
|
||||
'flows': ['advoware'],
|
||||
'triggers': [
|
||||
http('POST', '/advoware/calendar/sync')
|
||||
],
|
||||
'enqueues': ['calendar_sync', 'calendar_sync_all']
|
||||
}
|
||||
```
|
||||
|
||||
## Setup
|
||||
|
||||
### Umgebungsvariablen
|
||||
```env
|
||||
# Google Calendar
|
||||
GOOGLE_CALENDAR_SERVICE_ACCOUNT_PATH=service-account.json
|
||||
|
||||
# Advoware API
|
||||
ADVOWARE_API_BASE_URL=https://www2.advo-net.net:90/
|
||||
ADVOWARE_PRODUCT_ID=64
|
||||
ADVOWARE_APP_ID=your_app_id
|
||||
ADVOWARE_API_KEY=your_api_key
|
||||
ADVOWARE_KANZLEI=your_kanzlei
|
||||
ADVOWARE_DATABASE=your_database
|
||||
ADVOWARE_USER=your_user
|
||||
ADVOWARE_ROLE=2
|
||||
ADVOWARE_PASSWORD=your_password
|
||||
ADVOWARE_TOKEN_LIFETIME_MINUTES=55
|
||||
ADVOWARE_API_TIMEOUT_SECONDS=30
|
||||
|
||||
# Redis
|
||||
REDIS_HOST=localhost
|
||||
REDIS_PORT=6379
|
||||
REDIS_DB_CALENDAR_SYNC=1
|
||||
REDIS_TIMEOUT_SECONDS=5
|
||||
|
||||
# Debug
|
||||
CALENDAR_SYNC_DEBUG_EMPLOYEES=PB,AI # Optional, filter employees
|
||||
```
|
||||
|
||||
## Verwendung
|
||||
|
||||
### Manueller Sync
|
||||
```bash
|
||||
# Sync für einen bestimmten Mitarbeiter
|
||||
curl -X POST "http://localhost:3111/advoware/calendar/sync" \
|
||||
-H "Content-Type: application/json" \
|
||||
-d '{"kuerzel": "PB"}'
|
||||
|
||||
# Sync für alle Mitarbeiter
|
||||
curl -X POST "http://localhost:3111/advoware/calendar/sync" \
|
||||
-H "Content-Type: application/json" \
|
||||
-d '{"kuerzel": "ALL"}'
|
||||
```
|
||||
|
||||
### Automatischer Sync
|
||||
Cron-Step läuft automatisch alle 15 Minuten.
|
||||
|
||||
## Fehlerbehandlung und Logging
|
||||
|
||||
- **Locking**: Redis NX/EX verhindert parallele Syncs.
|
||||
- **Logging**: `ctx.logger` für iii Console-Sichtbarkeit.
|
||||
- **API-Fehler**: Retry mit Backoff.
|
||||
- **Parsing-Fehler**: Robuste Fallbacks.
|
||||
|
||||
## Sicherheit
|
||||
|
||||
- Service Account für Google Calendar API.
|
||||
- HMAC-512 Authentifizierung für Advoware API.
|
||||
- Redis für Concurrency-Control.
|
||||
|
||||
## Bekannte Probleme
|
||||
|
||||
- **Recurring-Events**: Begrenzte Unterstützung für komplexe Wiederholungen.
|
||||
- **Performance**: Bei vielen Terminen Paginierung beachten.
|
||||
- **Timezone-Handling**: Alle Operationen in Europe/Berlin TZ.
|
||||
|
||||
## Datenfluss
|
||||
|
||||
```
|
||||
Cron (alle 15min)
|
||||
→ calendar_sync_cron_step
|
||||
→ ctx.enqueue(topic: "calendar_sync_all")
|
||||
→ calendar_sync_all_step
|
||||
→ Fetch Employees from Advoware
|
||||
→ For each Employee:
|
||||
→ Set Redis Lock (key: calendar_sync:employee:{kuerzel})
|
||||
→ ctx.enqueue(topic: "calendar_sync", data: {kuerzel, ...})
|
||||
→ calendar_sync_event_step
|
||||
→ Fetch Advoware Termine (frNr, datum, text, etc.)
|
||||
→ Fetch Google Calendar Events
|
||||
→ 4-Phase Sync:
|
||||
1. New from Advoware → Google
|
||||
2. New from Google → Advoware
|
||||
3. Process Deletes
|
||||
4. Process Updates
|
||||
→ Clear Redis Lock
|
||||
```
|
||||
|
||||
## Weitere Dokumentation
|
||||
|
||||
- **Individual Step Docs**: Siehe `docs/` Ordner in diesem Verzeichnis
|
||||
- **Architecture Overview**: [../../docs/ARCHITECTURE.md](../../docs/ARCHITECTURE.md)
|
||||
- **Google Setup Guide**: [../../docs/GOOGLE_SETUP.md](../../docs/GOOGLE_SETUP.md)
|
||||
- **Troubleshooting**: [../../docs/TROUBLESHOOTING.md](../../docs/TROUBLESHOOTING.md)
|
||||
|
||||
## Migration Notes
|
||||
|
||||
Dieses System wurde von **Motia v0.17** nach **Motia III v1.0-RC** migriert:
|
||||
|
||||
### Wichtige Änderungen:
|
||||
- ✅ `type: 'event'` → `triggers: [queue('topic')]`
|
||||
- ✅ `type: 'cron'` → `triggers: [cron('expression')]` (6-field format)
|
||||
- ✅ `type: 'api'` → `triggers: [http('METHOD', 'path')]`
|
||||
- ✅ `context.emit()` → `ctx.enqueue()`
|
||||
- ✅ `emits: [...]` → `enqueues: [...]`
|
||||
- ✅ Relative Imports → Absolute Imports mit `sys.path.insert()`
|
||||
- ✅ Motia Workbench → iii Console
|
||||
|
||||
### Kompatibilität:
|
||||
- ✅ Alle 4 Steps vollständig migriert
|
||||
- ✅ Google Calendar API Integration unverändert
|
||||
- ✅ Advoware API Integration unverändert
|
||||
- ✅ Redis Locking-Mechanismus unverändert
|
||||
- ✅ Datenbank-Schema kompatibel
|
||||
5
src/steps/advoware_cal_sync/__init__.py
Normal file
5
src/steps/advoware_cal_sync/__init__.py
Normal file
@@ -0,0 +1,5 @@
|
||||
"""
|
||||
Advoware Calendar Sync Module
|
||||
|
||||
Bidirectional synchronization between Google Calendar and Advoware appointments.
|
||||
"""
|
||||
113
src/steps/advoware_cal_sync/calendar_sync_all_step.py
Normal file
113
src/steps/advoware_cal_sync/calendar_sync_all_step.py
Normal file
@@ -0,0 +1,113 @@
|
||||
"""
|
||||
Calendar Sync All Step
|
||||
|
||||
Handles calendar_sync_all event and emits individual sync events for oldest employees.
|
||||
Uses Redis to track last sync times and distribute work.
|
||||
"""
|
||||
import sys
|
||||
from pathlib import Path
|
||||
sys.path.insert(0, str(Path(__file__).parent))
|
||||
from calendar_sync_utils import (
|
||||
get_redis_client,
|
||||
get_advoware_employees,
|
||||
set_employee_lock,
|
||||
log_operation
|
||||
)
|
||||
|
||||
import math
|
||||
import time
|
||||
from datetime import datetime
|
||||
from typing import Any, Dict
|
||||
from motia import queue, FlowContext
|
||||
from pydantic import BaseModel, Field
|
||||
from services.advoware_service import AdvowareService
|
||||
|
||||
config = {
|
||||
'name': 'Calendar Sync All Step',
|
||||
'description': 'Receives sync-all event and emits individual events for oldest employees',
|
||||
'flows': ['advoware-calendar-sync'],
|
||||
'triggers': [
|
||||
queue('calendar_sync_all')
|
||||
],
|
||||
'enqueues': ['calendar_sync_employee']
|
||||
}
|
||||
|
||||
|
||||
async def handler(input_data: Dict[str, Any], ctx: FlowContext) -> None:
|
||||
"""
|
||||
Handler that fetches all employees, sorts by last sync time,
|
||||
and emits calendar_sync_employee events for the oldest ones.
|
||||
"""
|
||||
try:
|
||||
triggered_by = input_data.get('triggered_by', 'unknown')
|
||||
log_operation('info', f"Calendar Sync All: Starting to emit events for oldest employees, triggered by {triggered_by}", context=ctx)
|
||||
|
||||
# Initialize Advoware service
|
||||
advoware = AdvowareService(ctx)
|
||||
|
||||
# Fetch employees
|
||||
employees = await get_advoware_employees(advoware, ctx)
|
||||
if not employees:
|
||||
log_operation('error', "Keine Mitarbeiter gefunden. All-Sync abgebrochen.", context=ctx)
|
||||
return {'status': 500, 'body': {'error': 'Keine Mitarbeiter gefunden'}}
|
||||
|
||||
redis_client = get_redis_client(ctx)
|
||||
|
||||
# Collect last_synced timestamps
|
||||
employee_timestamps = {}
|
||||
for employee in employees:
|
||||
kuerzel = employee.get('kuerzel')
|
||||
if not kuerzel:
|
||||
continue
|
||||
employee_last_synced_key = f'calendar_sync_last_synced_{kuerzel}'
|
||||
timestamp_str = redis_client.get(employee_last_synced_key)
|
||||
timestamp = int(timestamp_str) if timestamp_str else 0 # 0 if no timestamp (very old)
|
||||
employee_timestamps[kuerzel] = timestamp
|
||||
|
||||
# Sort employees by last_synced (ascending, oldest first), then by kuerzel alphabetically
|
||||
sorted_kuerzel = sorted(employee_timestamps.keys(), key=lambda k: (employee_timestamps[k], k))
|
||||
|
||||
# Log the sorted list with timestamps
|
||||
def format_timestamp(ts):
|
||||
if ts == 0:
|
||||
return "never"
|
||||
return datetime.fromtimestamp(ts).strftime('%Y-%m-%d %H:%M:%S')
|
||||
|
||||
sorted_list_str = ", ".join(f"{k} ({format_timestamp(employee_timestamps[k])})" for k in sorted_kuerzel)
|
||||
log_operation('info', f"Calendar Sync All: Sorted employees by last synced: {sorted_list_str}", context=ctx)
|
||||
|
||||
# Calculate number to sync: ceil(N / 10)
|
||||
num_to_sync = math.ceil(len(sorted_kuerzel) / 1)
|
||||
log_operation('info', f"Calendar Sync All: Total employees {len(sorted_kuerzel)}, syncing {num_to_sync} oldest", context=ctx)
|
||||
|
||||
# Emit for the oldest num_to_sync employees, if not locked
|
||||
emitted_count = 0
|
||||
for kuerzel in sorted_kuerzel[:num_to_sync]:
|
||||
if not set_employee_lock(redis_client, kuerzel, triggered_by, ctx):
|
||||
log_operation('info', f"Calendar Sync All: Sync already active for {kuerzel}, skipping", context=ctx)
|
||||
continue
|
||||
|
||||
# Emit event for this employee
|
||||
await ctx.enqueue({
|
||||
"topic": "calendar_sync_employee",
|
||||
"data": {
|
||||
"kuerzel": kuerzel,
|
||||
"triggered_by": triggered_by
|
||||
}
|
||||
})
|
||||
log_operation('info', f"Calendar Sync All: Emitted event for employee {kuerzel} (last synced: {format_timestamp(employee_timestamps[kuerzel])})", context=ctx)
|
||||
emitted_count += 1
|
||||
|
||||
log_operation('info', f"Calendar Sync All: Completed, emitted {emitted_count} events", context=ctx)
|
||||
return {
|
||||
'status': 'completed',
|
||||
'triggered_by': triggered_by,
|
||||
'emitted_count': emitted_count
|
||||
}
|
||||
|
||||
except Exception as e:
|
||||
log_operation('error', f"Fehler beim All-Sync: {e}", context=ctx)
|
||||
return {
|
||||
'status': 'error',
|
||||
'error': str(e)
|
||||
}
|
||||
112
src/steps/advoware_cal_sync/calendar_sync_api_step.py
Normal file
112
src/steps/advoware_cal_sync/calendar_sync_api_step.py
Normal file
@@ -0,0 +1,112 @@
|
||||
"""
|
||||
Calendar Sync API Step
|
||||
|
||||
HTTP API endpoint for manual calendar sync triggering.
|
||||
Supports syncing a single employee or all employees.
|
||||
"""
|
||||
import sys
|
||||
from pathlib import Path
|
||||
sys.path.insert(0, str(Path(__file__).parent))
|
||||
from calendar_sync_utils import get_redis_client, set_employee_lock, get_logger
|
||||
|
||||
from motia import http, ApiRequest, ApiResponse, FlowContext
|
||||
|
||||
|
||||
config = {
|
||||
'name': 'Calendar Sync API Trigger',
|
||||
'description': 'API endpoint for manual calendar sync triggering',
|
||||
'flows': ['advoware-calendar-sync'],
|
||||
'triggers': [
|
||||
http('POST', '/advoware/calendar/sync')
|
||||
],
|
||||
'enqueues': ['calendar_sync_employee', 'calendar_sync_all']
|
||||
}
|
||||
|
||||
|
||||
async def handler(request: ApiRequest, ctx: FlowContext) -> ApiResponse:
|
||||
"""
|
||||
HTTP handler for manual calendar sync triggering.
|
||||
|
||||
Request body:
|
||||
{
|
||||
"kuerzel": "SB" // or "ALL" for all employees
|
||||
}
|
||||
"""
|
||||
try:
|
||||
# Get kuerzel from request body
|
||||
body = request.body
|
||||
kuerzel = body.get('kuerzel')
|
||||
if not kuerzel:
|
||||
return ApiResponse(
|
||||
status=400,
|
||||
body={
|
||||
'error': 'kuerzel required',
|
||||
'message': 'Please provide kuerzel in body'
|
||||
}
|
||||
)
|
||||
|
||||
kuerzel_upper = kuerzel.upper()
|
||||
|
||||
if kuerzel_upper == 'ALL':
|
||||
# Emit sync-all event
|
||||
ctx.logger.info("Calendar Sync API: Emitting sync-all event")
|
||||
await ctx.enqueue({
|
||||
"topic": "calendar_sync_all",
|
||||
"data": {
|
||||
"triggered_by": "api"
|
||||
}
|
||||
})
|
||||
return ApiResponse(
|
||||
status=200,
|
||||
body={
|
||||
'status': 'triggered',
|
||||
'message': 'Calendar sync triggered for all employees',
|
||||
'triggered_by': 'api'
|
||||
}
|
||||
)
|
||||
else:
|
||||
# Single employee sync
|
||||
redis_client = get_redis_client(ctx)
|
||||
|
||||
if not set_employee_lock(redis_client, kuerzel_upper, 'api', ctx):
|
||||
ctx.logger.info(f"Calendar Sync API: Sync already active for {kuerzel_upper}, skipping")
|
||||
return ApiResponse(
|
||||
status=409,
|
||||
body={
|
||||
'status': 'conflict',
|
||||
'message': f'Calendar sync already active for {kuerzel_upper}',
|
||||
'kuerzel': kuerzel_upper,
|
||||
'triggered_by': 'api'
|
||||
}
|
||||
)
|
||||
|
||||
ctx.logger.info(f"Calendar Sync API called for {kuerzel_upper}")
|
||||
|
||||
# Lock successfully set, now emit event
|
||||
await ctx.enqueue({
|
||||
"topic": "calendar_sync_employee",
|
||||
"data": {
|
||||
"kuerzel": kuerzel_upper,
|
||||
"triggered_by": "api"
|
||||
}
|
||||
})
|
||||
|
||||
return ApiResponse(
|
||||
status=200,
|
||||
body={
|
||||
'status': 'triggered',
|
||||
'message': f'Calendar sync triggered for {kuerzel_upper}',
|
||||
'kuerzel': kuerzel_upper,
|
||||
'triggered_by': 'api'
|
||||
}
|
||||
)
|
||||
|
||||
except Exception as e:
|
||||
ctx.logger.error(f"Error in API trigger: {e}")
|
||||
return ApiResponse(
|
||||
status=500,
|
||||
body={
|
||||
'error': 'Internal server error',
|
||||
'details': str(e)
|
||||
}
|
||||
)
|
||||
50
src/steps/advoware_cal_sync/calendar_sync_cron_step.py
Normal file
50
src/steps/advoware_cal_sync/calendar_sync_cron_step.py
Normal file
@@ -0,0 +1,50 @@
|
||||
"""
|
||||
Calendar Sync Cron Step
|
||||
|
||||
Cron trigger for automatic calendar synchronization.
|
||||
Emits calendar_sync_all event to start sync cascade.
|
||||
"""
|
||||
import sys
|
||||
from pathlib import Path
|
||||
sys.path.insert(0, str(Path(__file__).parent))
|
||||
from calendar_sync_utils import log_operation
|
||||
|
||||
from typing import Dict, Any
|
||||
from motia import cron, FlowContext
|
||||
|
||||
|
||||
config = {
|
||||
'name': 'Calendar Sync Cron Job',
|
||||
'description': 'Runs calendar sync automatically every 15 minutes',
|
||||
'flows': ['advoware-calendar-sync'],
|
||||
'triggers': [
|
||||
cron("0 15 1 * * *") # Every 15 minutes at second 0 (6-field: sec min hour day month weekday)
|
||||
],
|
||||
'enqueues': ['calendar_sync_all']
|
||||
}
|
||||
|
||||
|
||||
async def handler(input_data: None, ctx: FlowContext) -> None:
|
||||
"""Cron handler that triggers the calendar sync cascade."""
|
||||
try:
|
||||
ctx.logger.info("=" * 80)
|
||||
ctx.logger.info("🕐 CALENDAR SYNC CRON: STARTING")
|
||||
ctx.logger.info("=" * 80)
|
||||
ctx.logger.info("Emitting sync-all event")
|
||||
|
||||
# Enqueue sync-all event
|
||||
await ctx.enqueue({
|
||||
"topic": "calendar_sync_all",
|
||||
"data": {
|
||||
"triggered_by": "cron"
|
||||
}
|
||||
})
|
||||
|
||||
ctx.logger.info("✅ Calendar sync-all event emitted successfully")
|
||||
ctx.logger.info("=" * 80)
|
||||
|
||||
except Exception as e:
|
||||
ctx.logger.error("=" * 80)
|
||||
ctx.logger.error("❌ ERROR: CALENDAR SYNC CRON")
|
||||
ctx.logger.error(f"Error: {e}")
|
||||
ctx.logger.error("=" * 80)
|
||||
1078
src/steps/advoware_cal_sync/calendar_sync_event_step.py
Normal file
1078
src/steps/advoware_cal_sync/calendar_sync_event_step.py
Normal file
File diff suppressed because it is too large
Load Diff
133
src/steps/advoware_cal_sync/calendar_sync_utils.py
Normal file
133
src/steps/advoware_cal_sync/calendar_sync_utils.py
Normal file
@@ -0,0 +1,133 @@
|
||||
"""
|
||||
Calendar Sync Utilities
|
||||
|
||||
Shared utility functions for calendar synchronization between Google Calendar and Advoware.
|
||||
"""
|
||||
import asyncpg
|
||||
import os
|
||||
import redis
|
||||
import time
|
||||
from typing import Optional, Any, List
|
||||
from googleapiclient.discovery import build
|
||||
from google.oauth2 import service_account
|
||||
from services.logging_utils import get_service_logger
|
||||
|
||||
|
||||
def get_logger(context=None):
|
||||
"""Get logger for calendar sync operations"""
|
||||
return get_service_logger('calendar_sync', context)
|
||||
|
||||
|
||||
def log_operation(level: str, message: str, context=None, **extra):
|
||||
"""
|
||||
Log calendar sync operations with structured context.
|
||||
|
||||
Args:
|
||||
level: Log level ('debug', 'info', 'warning', 'error')
|
||||
message: Log message
|
||||
context: FlowContext if available
|
||||
**extra: Additional key-value pairs to log
|
||||
"""
|
||||
logger = get_logger(context)
|
||||
log_func = getattr(logger, level.lower(), logger.info)
|
||||
|
||||
if extra:
|
||||
extra_str = " | " + " | ".join(f"{k}={v}" for k, v in extra.items())
|
||||
log_func(message + extra_str)
|
||||
else:
|
||||
log_func(message)
|
||||
|
||||
|
||||
async def connect_db(context=None):
|
||||
"""Connect to Postgres DB from environment variables."""
|
||||
logger = get_logger(context)
|
||||
try:
|
||||
conn = await asyncpg.connect(
|
||||
host=os.getenv('POSTGRES_HOST', 'localhost'),
|
||||
user=os.getenv('POSTGRES_USER', 'calendar_sync_user'),
|
||||
password=os.getenv('POSTGRES_PASSWORD', 'default_password'),
|
||||
database=os.getenv('POSTGRES_DB_NAME', 'calendar_sync_db'),
|
||||
timeout=10
|
||||
)
|
||||
return conn
|
||||
except Exception as e:
|
||||
logger.error(f"Failed to connect to DB: {e}")
|
||||
raise
|
||||
|
||||
|
||||
async def get_google_service(context=None):
|
||||
"""Initialize Google Calendar service."""
|
||||
logger = get_logger(context)
|
||||
try:
|
||||
service_account_path = os.getenv('GOOGLE_CALENDAR_SERVICE_ACCOUNT_PATH', 'service-account.json')
|
||||
if not os.path.exists(service_account_path):
|
||||
raise FileNotFoundError(f"Service account file not found: {service_account_path}")
|
||||
|
||||
scopes = ['https://www.googleapis.com/auth/calendar']
|
||||
creds = service_account.Credentials.from_service_account_file(
|
||||
service_account_path, scopes=scopes
|
||||
)
|
||||
service = build('calendar', 'v3', credentials=creds)
|
||||
return service
|
||||
except Exception as e:
|
||||
logger.error(f"Failed to initialize Google service: {e}")
|
||||
raise
|
||||
|
||||
|
||||
def get_redis_client(context=None) -> redis.Redis:
|
||||
"""Initialize Redis client for calendar sync operations."""
|
||||
logger = get_logger(context)
|
||||
try:
|
||||
redis_client = redis.Redis(
|
||||
host=os.getenv('REDIS_HOST', 'localhost'),
|
||||
port=int(os.getenv('REDIS_PORT', '6379')),
|
||||
db=int(os.getenv('REDIS_DB_CALENDAR_SYNC', '2')),
|
||||
socket_timeout=int(os.getenv('REDIS_TIMEOUT_SECONDS', '5')),
|
||||
decode_responses=True
|
||||
)
|
||||
return redis_client
|
||||
except Exception as e:
|
||||
logger.error(f"Failed to initialize Redis client: {e}")
|
||||
raise
|
||||
|
||||
|
||||
async def get_advoware_employees(advoware, context=None) -> List[Any]:
|
||||
"""Fetch list of employees from Advoware."""
|
||||
logger = get_logger(context)
|
||||
try:
|
||||
result = await advoware.api_call('api/v1/advonet/Mitarbeiter', method='GET', params={'aktiv': 'true'})
|
||||
employees = result if isinstance(result, list) else []
|
||||
logger.info(f"Fetched {len(employees)} Advoware employees")
|
||||
return employees
|
||||
except Exception as e:
|
||||
logger.error(f"Failed to fetch Advoware employees: {e}")
|
||||
raise
|
||||
|
||||
|
||||
def set_employee_lock(redis_client: redis.Redis, kuerzel: str, triggered_by: str, context=None) -> bool:
|
||||
"""Set lock for employee sync operation."""
|
||||
logger = get_logger(context)
|
||||
employee_lock_key = f'calendar_sync_lock_{kuerzel}'
|
||||
if redis_client.set(employee_lock_key, triggered_by, ex=1800, nx=True) is None:
|
||||
logger.info(f"Sync already active for {kuerzel}, skipping")
|
||||
return False
|
||||
return True
|
||||
|
||||
|
||||
def clear_employee_lock(redis_client: redis.Redis, kuerzel: str, context=None) -> None:
|
||||
"""Clear lock for employee sync operation and update last-synced timestamp."""
|
||||
logger = get_logger(context)
|
||||
try:
|
||||
employee_lock_key = f'calendar_sync_lock_{kuerzel}'
|
||||
employee_last_synced_key = f'calendar_sync_last_synced_{kuerzel}'
|
||||
|
||||
# Update last-synced timestamp (no TTL, persistent)
|
||||
current_time = int(time.time())
|
||||
redis_client.set(employee_last_synced_key, current_time)
|
||||
|
||||
# Delete the lock
|
||||
redis_client.delete(employee_lock_key)
|
||||
|
||||
logger.debug(f"Cleared lock and updated last-synced for {kuerzel} to {current_time}")
|
||||
except Exception as e:
|
||||
logger.warning(f"Failed to clear lock and update last-synced for {kuerzel}: {e}")
|
||||
Reference in New Issue
Block a user