Commit Graph

18 Commits

Author SHA1 Message Date
bsiggel
6c3cf3ca91 refactor(logging): remove unused logger instances and enhance error logging in webhook steps 2026-03-08 22:21:08 +00:00
bsiggel
1c765d1eec refactor(logging): standardize status code handling and enhance logging in webhook and cron handlers 2026-03-08 22:09:22 +00:00
bsiggel
a0cf845877 Refactor and enhance logging in webhook handlers and Redis client
- Translated comments and docstrings from German to English for better clarity.
- Improved logging consistency across various webhook handlers for create, delete, and update operations.
- Centralized logging functionality by utilizing a dedicated logger utility.
- Added new enums for file and XAI sync statuses in models.
- Updated Redis client factory to use a centralized logger and improved error handling.
- Enhanced API responses to include more descriptive messages and status codes.
2026-03-08 21:50:34 +00:00
bsiggel
f392ec0f06 refactor(typing): update handler signatures to use Dict and Any for improved type hinting 2026-03-08 21:24:12 +00:00
bsiggel
2532bd89ee refactor(logging): standardize logging approach across services and steps 2026-03-08 21:20:49 +00:00
bsiggel
4ed752b19e feat(document-sync): enhance metadata update to reset file status after preview generation 2026-03-08 18:59:56 +00:00
bsiggel
82b48eee8e fix(api-steps): update response status field in document create, delete, and update handlers 2026-03-08 18:48:56 +00:00
bsiggel
91ae2947e5 feat(xai-service): implement xAI Files & Collections service for document synchronization 2026-03-08 18:31:29 +00:00
bsiggel
69a48f5f9a Implement central configuration, custom exceptions, logging utilities, Pydantic models, and Redis client for BitByLaw integration
- Added `config.py` for centralized configuration management including Sync, API, Advoware, EspoCRM, Redis, Logging, Calendar Sync, and Feature Flags.
- Created `exceptions.py` with a hierarchy of custom exceptions for integration errors, API errors, sync errors, and Redis errors.
- Developed `logging_utils.py` for a unified logging wrapper supporting structured logging and performance tracking.
- Defined Pydantic models in `models.py` for data validation of Advoware and EspoCRM entities, including sync operation models.
- Introduced `redis_client.py` for a centralized Redis client factory with connection pooling, automatic reconnection, and health checks.
2026-03-03 17:18:49 +00:00
bsiggel
bcb6454b2a Add comprehensive test scripts for thumbnail generation and xAI collections API
- Implemented `test_thumbnail_generation.py` to validate the complete flow of document thumbnail generation in EspoCRM, including document creation, file upload, webhook triggering, and preview verification.
- Created `test_xai_collections_api.py` to test critical operations of the xAI Collections API, covering file uploads, collection CRUD operations, document management, and response validation.
- Both scripts include detailed logging for success and error states, ensuring robust testing and easier debugging.
2026-03-03 17:03:08 +00:00
bsiggel
c45bfb7233 Enhance EspoCRM API and Webhook Handling
- Improved logging for file uploads in EspoCRMAPI to include upload parameters and error details.
- Updated cron job configurations for calendar sync and participant sync to trigger every 15 minutes on the first minute of the hour.
- Enhanced document create, delete, and update webhook handlers to determine and log the entity type.
- Refactored document sync event handler to include entity type in sync operations and logging.
- Added a new test script for uploading preview images to EspoCRM and verifying the upload process.
- Created a test script for document thumbnail generation, including document creation, file upload, webhook triggering, and preview verification.
2026-03-03 16:53:55 +00:00
bsiggel
0e521f22f8 feat(preview-generation): implement thumbnail generation for documents; add preview upload to EspoCRM 2026-03-03 09:28:49 +00:00
bsiggel
ee9aab049f feat(document-sync): add Document Sync Utilities and VMH Document Sync Handler for xAI integration 2026-03-03 06:55:54 +00:00
bsiggel
0282149613 Fix cron expression in Calendar Sync Cron Job to trigger at second 0 and update entity retrieval methods in Beteiligte Sync steps for consistency 2026-03-02 09:36:09 +00:00
bsiggel
721339ca9b Update flow names for Advoware and VMH steps to improve clarity and organization 2026-03-01 23:28:29 +00:00
bsiggel
52356e634e Add calendar sync utilities and Beteiligte sync cron job
- Implemented calendar_sync_utils.py for shared utility functions
  including DB connection, Google Calendar service initialization,
  Redis client setup, and employee sync operations.

- Created beteiligte_sync_cron_step.py to handle periodic sync
  of Beteiligte entities, checking for new, modified, failed,
  and stale records, and emitting sync events accordingly.
2026-03-01 22:55:17 +00:00
bsiggel
014947e9e0 Migrate VMH Integration - Phase 3: Core sync handlers & utilities
- Added 5 core service modules:
  * services/notification_utils.py: NotificationManager for manual actions (412 lines)
  * services/advoware_service.py: Extended Advoware operations wrapper
  * services/espocrm_mapper.py: BeteiligteMapper for data transformation (198 lines)
  * services/bankverbindungen_mapper.py: BankverbindungenMapper (174 lines)
  * services/beteiligte_sync_utils.py: BeteiligteSync with complex logic (663 lines)
    - Distributed locking via Redis + syncStatus
    - rowId-based change detection (primary) + timestamp fallback
    - Retry with exponential backoff (1min, 5min, 15min, 1h, 4h)
    - Conflict resolution (EspoCRM wins)
    - Soft-delete handling & round-trip validation

- Added 2 event handler steps:
  * steps/vmh/beteiligte_sync_event_step.py: Handles vmh.beteiligte.* queue events
    - CREATE: New Beteiligte → POST to Advoware
    - UPDATE: Bi-directional sync with conflict resolution
    - DELETE: Not yet implemented
    - SYNC_CHECK: Periodic sync check via cron
  * steps/vmh/bankverbindungen_sync_event_step.py: Handles vmh.bankverbindungen.* events
    - CREATE: New Bankverbindung → POST to Advoware
    - UPDATE/DELETE: Send notification (API limitation - no PUT/DELETE support)

- Added pytz dependency to pyproject.toml (required for timezone handling)

System Status:
- 27 steps registered (14 VMH-specific)
- 8 queue subscriptions active (4 Beteiligte + 4 Bankverbindungen)
- All webhook endpoints operational and emitting queue events
- Sync handlers ready for processing

Note: Kommunikation sync (kommunikation_sync_utils.py) not yet migrated.
Complex bi-directional sync for Telefon/Email/Fax will be added in Phase 4.

Updated MIGRATION_STATUS.md
2026-03-01 22:19:36 +00:00
bsiggel
0216c4c3ae Migrate VMH Integration - Phase 2: Webhook endpoints
- Added services/espocrm.py: EspoCRM API client with Redis support
- Added 6 VMH webhook steps for EspoCRM integration:

  Beteiligte webhooks:
  - POST /vmh/webhook/beteiligte/create
  - POST /vmh/webhook/beteiligte/update
  - POST /vmh/webhook/beteiligte/delete

  Bankverbindungen webhooks:
  - POST /vmh/webhook/bankverbindungen/create
  - POST /vmh/webhook/bankverbindungen/update
  - POST /vmh/webhook/bankverbindungen/delete

All webhook endpoints receive batch/single entity notifications from
EspoCRM and emit queue events for downstream processing.

Note: Complex sync handlers (event processors) not yet migrated -
they require additional utility modules (beteiligte_sync_utils.py,
mappers, notification_utils) which will be migrated in Phase 3.

Updated MIGRATION_STATUS.md with Phase 2 completion.
2026-03-01 21:52:19 +00:00