bsiggel/motia
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity
72 Commits 1 Branch 0 Tags
b5abe6cf00e332fe7215398f012cfaeba9d92e7f
Commit Graph

66 Commits

Author SHA1 Message Date
bitbylaw
b5abe6cf00 Implement EspoCRM-based sync strategy for Beteiligte entities 
- Add SYNC_STRATEGY_ESPOCRM_BASED.md detailing the sync flows and status management.
- Create utilities for sync operations in services/beteiligte_sync_utils.py, including locking, timestamp comparison, conflict resolution, and notification handling.
- Implement entity mapping between EspoCRM and Advoware in services/espocrm_mapper.py.
- Develop a cron job for periodic sync checks in steps/vmh/beteiligte_sync_cron_step.py, emitting events for entities needing synchronization.
 2026-02-07 15:21:16 +00:00
bitbylaw
36552903e7 cleanup 2026-02-07 09:23:49 +00:00
root
96eabe3db6 Refactor calendar sync to prioritize oldest synced employees with human-readable timestamps 2025-10-26 08:58:48 +00:00
root
b18e770f12 refactor: extract common functions to utils 
- Add get_redis_client() to calendar_sync_utils.py
- Add get_advoware_employees() to calendar_sync_utils.py
- Add set_employee_lock() and clear_employee_lock() to calendar_sync_utils.py
- Update all step files to use shared utility functions
- Remove duplicate code across calendar_sync_*.py files
 2025-10-25 09:21:45 +00:00
root
e4bf21e676 Refactor: Extract common utilities to shared module 
- Created calendar_sync_utils.py with shared functions:
  - log_operation: Centralized logging with context support
  - connect_db: Database connection wrapper
  - get_google_service: Google Calendar API initialization
- Updated imports in calendar_sync_event_step.py and audit_calendar_sync.py
- Removed duplicated function definitions
- Maintained function logic without changes
- Improved code maintainability and reduced duplications
 2025-10-25 08:54:54 +00:00
root
c5600b42ec feat: Enhance audit_calendar_sync tool with comprehensive management features 
- Add detailed documentation in README.md for all 10 audit commands
- Fix Advoware API parameter from 'frNr' to 'frnr' for proper filtering
- Fix subject field mapping from 'betreff' to 'text' in API responses
- Add verify-sync command for bidirectional sync verification
- Add query-frnr and query-event commands for individual record lookup
- Add management commands: find-duplicates, delete-duplicates, find-orphaned, cleanup-orphaned
- Improve error handling and output formatting
- Remove temporary test script
 2025-10-25 08:18:48 +00:00
root
c5ddd02307 Add query and verification features for individual appointments by frNr or Google Event ID 2025-10-25 07:53:43 +00:00
root
1539c26be6 Expand audit_calendar_sync.py to comprehensive management tool with pagination, duplicate detection/deletion, and orphaned calendar cleanup 2025-10-25 07:48:35 +00:00
root
62f57bb035 Fix ensure_google_calendar to fetch all calendars with pagination to avoid missing calendars on later pages 2025-10-25 07:42:05 +00:00
root
b4c4bf0a9e Improve calendar sync: Add Token Bucket rate limiting, security check to prevent syncing Advoware-sourced events back, fix random import 2025-10-24 23:56:00 +00:00
root
6ab7b4a376 Rate Limiting: Atomisches Lua Script zur Vermeidung von Race Conditions 2025-10-24 21:54:47 +00:00
root
3c6d7e13c6 Globales Rate Limiting: 600 Anfragen/Minute über gleitendes Fenster mit Redis 2025-10-24 20:43:46 +00:00
root
dcb2dba50f API-Schutz: 100ms Verzögerung bei allen Google API Calls (create, update, delete) 2025-10-24 19:58:18 +00:00
root
2bf37b8616 Fix: Korrekte Endzeiten für recurring Events 
- parse_times erkennt jetzt recurring Events (dauertermin=1 + turnus/turnusArt > 0)
- Für recurring Events wird das Ende aus dem gleichen Tag wie Start berechnet
- datumBis wird nur noch für die RRULE-Serie verwendet, nicht für jeden Termin
- Fix für Termin 84389: Statt 15-Tage-Terminen jetzt korrekte 30-Minuten-Termine
 2025-10-24 19:48:12 +00:00
root
d154ba8172 Fix: ACL-Prüfung bei existierenden Google-Kalendern 
- ensure_google_calendar prüft jetzt auch bei existierenden Kalendern die ACL-Regel
- Fügt fehlende ACL-Regel hinzu, falls sie nicht vorhanden ist
- Verhindert Sync-Abbruch bei unvollständigen Kalendern aus früheren Läufen
 2025-10-24 19:27:59 +00:00
root
9d40f47e19 Refaktorierung Calendar Sync: Event-driven Design, Fixes für mehrtägige Termine, Logging und Locking 
- Refaktorierung zu event-driven Ansatz ohne PostgreSQL Hub
- Fixes für mehrtägige Termine: korrekte Verwendung von datumBis, Entfernung 24h-Limit
- Per-Employee Locking mit Redis
- Logging via context.logger für Motia Workbench
- Neue Schritte: calendar_sync_all_step.py, calendar_sync_cron_step.py
- Aktualisiertes README.md mit aktueller Architektur
- Workbench-Gruppierung: advoware-calendar-sync
 2025-10-24 19:13:41 +00:00
root
f4490f21cb Fix logging to appear in Motia workbench 
- Updated log_operation to use context.logger.warn for warnings
- Added context parameter to all functions with logging
- Replaced all direct logger calls with log_operation calls
- Ensured all logging goes through context.logger for workbench visibility
- Adjusted backoff base from 4 to 3 for faster retries
- Added debug kuerzel list support in cron step
 2025-10-24 07:04:57 +00:00
root
72ee01b74b Add recreated count to sync statistics 2025-10-24 06:32:33 +00:00
root
858d3d4fb3 Add final statistics logging to event step showing operations performed 2025-10-24 06:16:02 +00:00
root
c14273cac9 Integrate delete calendar functionality into audit script 2025-10-24 01:11:06 +00:00
root
25429edd76 Extend debug mode in cron step to accept list of kuerzel (SB, AI, RO) 2025-10-24 01:08:09 +00:00
root
2eb8330b1d Implement atomic locking in API and cron steps for per-employee parallel sync 2025-10-24 00:48:25 +00:00
root
9a1eb5bf0b Remove lock checking and setting from event step, keep only deletion in finally 2025-10-24 00:44:31 +00:00
root
f6bcbe664c Fix event handler signature to async def handler(event_data, context) 2025-10-24 00:38:12 +00:00
root
d18187f3aa Change event topic to calendar_sync_employee without dots to fix INVALID EMIT 2025-10-24 00:29:09 +00:00
root
6d2089ec69 Ensure Redis lock is always released with finally block 2025-10-24 00:26:18 +00:00
root
9ca3191542 Add flows to API and cron steps to fix INVALID EMIT error 2025-10-24 00:22:51 +00:00
root
b70619ab2d Update API step to trigger sync for specific employee with kuerzel 2025-10-24 00:20:41 +00:00
root
8e9dc87b2a Refactor calendar sync for parallel processing: cron emits per-employee events, event-step processes one employee with per-employee locks 2025-10-24 00:19:34 +00:00
root
409bea3615 Update logging to use Motia workbench context for visibility 2025-10-24 00:12:53 +00:00
root
ddad58faa3 Refactor calendar sync phases into separate functions for better modularity 2025-10-24 00:10:09 +00:00
root
c91d3fc76d Fix 'adv_map' not defined error by correcting state dict references 2025-10-24 00:08:18 +00:00
root
11ca7a5e75 Add missing phase functions to event step 2025-10-23 23:58:53 +00:00
root
3a1ed1ee55 Refactor calendar sync: move employee loop to cron step, parallel sync per employee 2025-10-23 23:57:51 +00:00
root
d15bd04167 Fix variable references in calendar sync after refactoring 2025-10-23 23:53:38 +00:00
root
a6d061a708 Refactor calendar sync: reduce redundancies, remove dead code, improve structure 
- Remove unused share_google_calendar function
- Add centralized log_operation for consistent logging with context
- Add get_timestamps helper for Phase 4 timestamp handling
- Consolidate safe_* functions into generic safe_advoware_operation
- Optimize reload functions to handle initial fetches
- Split standardize_appointment_data into parse_times, adjust_times, build_notiz helpers
- Use state dict for better encapsulation in handler
 2025-10-23 23:48:16 +00:00
root
42949c6871 Fix systemd service for automatic Motia startup 2025-10-23 23:37:52 +00:00
root
5ef77c91d5 Fix Advoware time filtering pitfall: extend ranges (Advoware -1 to +9 years, Google -2 to +10 years) and document issue in README 2025-10-23 23:28:49 +00:00
root
db1206f91c Fix recurring event duplication - handle recurringEventId properly in all phases 2025-10-23 18:46:49 +00:00
root
da2b9960b0 Fix recurring event duplication - handle recurringEventId properly in all phases 2025-10-23 18:30:46 +00:00
root
ddd64a5e0f Fix critical bug: End date for recurring events should use 'datum' not 'datumBis' 
- Fixed: For recurring appointments (dauertermin=1), end date now correctly uses 'datum' instead of 'datumBis'
- 'datumBis' is only for recurrence series end, not individual event duration
- Events now have correct duration instead of being capped at 24h unnecessarily
- Updated README with detailed explanation of the bug and fix
- Time breakdowns in descriptions are now accurate
 2025-10-23 18:12:02 +00:00
root
08b21ef268 Fix cron handler signature: remove event parameter from cron step 
- Fix calendar_sync_cron_step.py handler signature from (event, context) to (context)
- Cron steps only receive context parameter, not event
 2025-10-23 17:33:58 +00:00
root
080eebd5ea Fix calendar sync issues: correct True/False, limit recurrence to 2 years, add share_google_calendar function 
- Fix Python syntax error: change 'true' to 'True' in config.py
- Limit recurring events to max 2 years to avoid Google Calendar API limits
- Add share_google_calendar function for manual calendar sharing
- Update README with calendar sync documentation
- Debug mode: limit sync to user AI only
 2025-10-23 13:53:51 +00:00
root
974116c024 Sync all employees in loop, remove employee_kuerzel requirement, update README 2025-10-23 13:02:52 +00:00
root
7b6e287b66 Remove AI limit, update README, fix update loops by setting last_sync to post-update time 2025-10-23 12:57:33 +00:00
root
48ce8d8e73 Add exponential backoff for Google API calls 
- Use backoff library with 5s base, max 5 tries for HttpError 429/5xx
- Decorate Google functions: ensure_calendar, fetch_events, create/update/delete events
- Add backoff to requirements.txt
 2025-10-23 12:49:45 +00:00
root
0d758b07ea Fix KeyError in Phase 3 recreate: use str(frnr) for adv_map key 
- adv_map keys are str, frnr is int, so adv_map[str(frnr)] fixes KeyError
 2025-10-23 12:44:54 +00:00
root
9312586f18 Fix Advoware timestamp parsing bug causing sync loops 
- Change .replace(tzinfo=BERLIN_TZ) to BERLIN_TZ.localize() for correct pytz TZ handling
- Update Phase 4 to use proper TZ localization for adv_ts
- Add documentation section on correct Advoware timestamp handling
- Add .gitignore, GOOGLE_SETUP_README.md, and check_db.py utility
 2025-10-23 12:43:22 +00:00
root
0d54243e9d Fix Phase 3 logic for symmetric delete handling: remove 'sb' field, add employee_kuerzel to updates, correct missing event recreation/deletion based on source_system 2025-10-23 11:53:24 +00:00
root
9f50f201df Fix Phase 4 in calendar sync to reset unauthorized changes in target system based on timestamps for source_system_wins strategy 2025-10-23 11:26:25 +00:00