Commit Graph

23 Commits

Author SHA1 Message Date
bsiggel
9bbfa61b3b feat: Implement AI Knowledge Sync Utilities and Event Handlers
- Added AIKnowledgeActivationStatus and AIKnowledgeSyncStatus enums to models.py for managing activation and sync states.
- Introduced AIKnowledgeSync class in aiknowledge_sync_utils.py for synchronizing CAIKnowledge entities with XAI Collections, including collection lifecycle management, document synchronization, and metadata updates.
- Created a daily cron job (aiknowledge_full_sync_cron_step.py) to perform a full sync of CAIKnowledge entities.
- Developed an event handler (aiknowledge_sync_event_step.py) to synchronize CAIKnowledge entities with XAI Collections triggered by webhooks and cron jobs.
- Implemented a webhook handler (aiknowledge_update_api_step.py) to receive updates from EspoCRM for CAIKnowledge entities and enqueue sync events.
- Enhanced xai_service.py with methods for collection management, document listing, and metadata updates.
2026-03-11 21:14:52 +00:00
bsiggel
a5a122b688 refactor(logging): enhance error handling and resource management in rate limiting and sync operations 2026-03-08 22:47:05 +00:00
bsiggel
6c3cf3ca91 refactor(logging): remove unused logger instances and enhance error logging in webhook steps 2026-03-08 22:21:08 +00:00
bsiggel
a0cf845877 Refactor and enhance logging in webhook handlers and Redis client
- Translated comments and docstrings from German to English for better clarity.
- Improved logging consistency across various webhook handlers for create, delete, and update operations.
- Centralized logging functionality by utilizing a dedicated logger utility.
- Added new enums for file and XAI sync statuses in models.
- Updated Redis client factory to use a centralized logger and improved error handling.
- Enhanced API responses to include more descriptive messages and status codes.
2026-03-08 21:50:34 +00:00
bsiggel
2532bd89ee refactor(logging): standardize logging approach across services and steps 2026-03-08 21:20:49 +00:00
bsiggel
93d4d89531 feat(document-sync): update xaiSyncStatus handling and logging for document synchronization 2026-03-08 19:21:17 +00:00
bsiggel
4ed752b19e feat(document-sync): enhance metadata update to reset file status after preview generation 2026-03-08 18:59:56 +00:00
bsiggel
7fd6eed86d feat(sync-utils): add logging delegation method to BaseSyncUtils 2026-03-08 18:38:48 +00:00
bsiggel
91ae2947e5 feat(xai-service): implement xAI Files & Collections service for document synchronization 2026-03-08 18:31:29 +00:00
bsiggel
6f7d62293e feat(espocrm): add logging method to EspoCRMAPI for improved message handling
fix(calendar_sync): correct cron expression for calendar sync job
2026-03-08 17:58:10 +00:00
bsiggel
d7b2b5543f feat(espocrm): add caching for entity definitions and implement related entity listing 2026-03-08 17:51:36 +00:00
bsiggel
a53051ea8e feat(api-client): implement session management for AdvowareAPI and EspoCRMAPI 2026-03-03 17:24:35 +00:00
bsiggel
69a48f5f9a Implement central configuration, custom exceptions, logging utilities, Pydantic models, and Redis client for BitByLaw integration
- Added `config.py` for centralized configuration management including Sync, API, Advoware, EspoCRM, Redis, Logging, Calendar Sync, and Feature Flags.
- Created `exceptions.py` with a hierarchy of custom exceptions for integration errors, API errors, sync errors, and Redis errors.
- Developed `logging_utils.py` for a unified logging wrapper supporting structured logging and performance tracking.
- Defined Pydantic models in `models.py` for data validation of Advoware and EspoCRM entities, including sync operation models.
- Introduced `redis_client.py` for a centralized Redis client factory with connection pooling, automatic reconnection, and health checks.
2026-03-03 17:18:49 +00:00
bsiggel
bcb6454b2a Add comprehensive test scripts for thumbnail generation and xAI collections API
- Implemented `test_thumbnail_generation.py` to validate the complete flow of document thumbnail generation in EspoCRM, including document creation, file upload, webhook triggering, and preview verification.
- Created `test_xai_collections_api.py` to test critical operations of the xAI Collections API, covering file uploads, collection CRUD operations, document management, and response validation.
- Both scripts include detailed logging for success and error states, ensuring robust testing and easier debugging.
2026-03-03 17:03:08 +00:00
bsiggel
c45bfb7233 Enhance EspoCRM API and Webhook Handling
- Improved logging for file uploads in EspoCRMAPI to include upload parameters and error details.
- Updated cron job configurations for calendar sync and participant sync to trigger every 15 minutes on the first minute of the hour.
- Enhanced document create, delete, and update webhook handlers to determine and log the entity type.
- Refactored document sync event handler to include entity type in sync operations and logging.
- Added a new test script for uploading preview images to EspoCRM and verifying the upload process.
- Created a test script for document thumbnail generation, including document creation, file upload, webhook triggering, and preview verification.
2026-03-03 16:53:55 +00:00
bsiggel
0e521f22f8 feat(preview-generation): implement thumbnail generation for documents; add preview upload to EspoCRM 2026-03-03 09:28:49 +00:00
bsiggel
70265c9adf feat(document-sync): enhance DocumentSync with file status checks and hash-based change detection; add thumbnail generation and metadata update methods 2026-03-03 09:15:02 +00:00
bsiggel
ee9aab049f feat(document-sync): add Document Sync Utilities and VMH Document Sync Handler for xAI integration 2026-03-03 06:55:54 +00:00
bsiggel
5c204ba16c Add api_call method to AdvowareService for delegating API calls 2026-03-01 23:40:50 +00:00
bsiggel
17f908d036 Add requests dependency to project
- Included 'requests' in the dependencies list of uv.lock.
- Specified minimum version of 'requests' as 2.32.0 in requires-dist.
2026-03-01 22:39:33 +00:00
bsiggel
014947e9e0 Migrate VMH Integration - Phase 3: Core sync handlers & utilities
- Added 5 core service modules:
  * services/notification_utils.py: NotificationManager for manual actions (412 lines)
  * services/advoware_service.py: Extended Advoware operations wrapper
  * services/espocrm_mapper.py: BeteiligteMapper for data transformation (198 lines)
  * services/bankverbindungen_mapper.py: BankverbindungenMapper (174 lines)
  * services/beteiligte_sync_utils.py: BeteiligteSync with complex logic (663 lines)
    - Distributed locking via Redis + syncStatus
    - rowId-based change detection (primary) + timestamp fallback
    - Retry with exponential backoff (1min, 5min, 15min, 1h, 4h)
    - Conflict resolution (EspoCRM wins)
    - Soft-delete handling & round-trip validation

- Added 2 event handler steps:
  * steps/vmh/beteiligte_sync_event_step.py: Handles vmh.beteiligte.* queue events
    - CREATE: New Beteiligte → POST to Advoware
    - UPDATE: Bi-directional sync with conflict resolution
    - DELETE: Not yet implemented
    - SYNC_CHECK: Periodic sync check via cron
  * steps/vmh/bankverbindungen_sync_event_step.py: Handles vmh.bankverbindungen.* events
    - CREATE: New Bankverbindung → POST to Advoware
    - UPDATE/DELETE: Send notification (API limitation - no PUT/DELETE support)

- Added pytz dependency to pyproject.toml (required for timezone handling)

System Status:
- 27 steps registered (14 VMH-specific)
- 8 queue subscriptions active (4 Beteiligte + 4 Bankverbindungen)
- All webhook endpoints operational and emitting queue events
- Sync handlers ready for processing

Note: Kommunikation sync (kommunikation_sync_utils.py) not yet migrated.
Complex bi-directional sync for Telefon/Email/Fax will be added in Phase 4.

Updated MIGRATION_STATUS.md
2026-03-01 22:19:36 +00:00
bsiggel
0216c4c3ae Migrate VMH Integration - Phase 2: Webhook endpoints
- Added services/espocrm.py: EspoCRM API client with Redis support
- Added 6 VMH webhook steps for EspoCRM integration:

  Beteiligte webhooks:
  - POST /vmh/webhook/beteiligte/create
  - POST /vmh/webhook/beteiligte/update
  - POST /vmh/webhook/beteiligte/delete

  Bankverbindungen webhooks:
  - POST /vmh/webhook/bankverbindungen/create
  - POST /vmh/webhook/bankverbindungen/update
  - POST /vmh/webhook/bankverbindungen/delete

All webhook endpoints receive batch/single entity notifications from
EspoCRM and emit queue events for downstream processing.

Note: Complex sync handlers (event processors) not yet migrated -
they require additional utility modules (beteiligte_sync_utils.py,
mappers, notification_utils) which will be migrated in Phase 3.

Updated MIGRATION_STATUS.md with Phase 2 completion.
2026-03-01 21:52:19 +00:00
bsiggel
a24282c346 Migrate Advoware Proxy steps from old-motia
- Added services/advoware.py: Advoware API client with Redis token caching
- Added advoware_proxy steps (GET, POST, PUT, DELETE)
- Updated pyproject.toml: added aiohttp, redis, python-dotenv
- Updated iii-config.yaml: fixed ExecModule command to use full path
- Created MIGRATION_STATUS.md: documentation of migration progress

All 4 proxy endpoints registered successfully:
- GET /advoware/proxy
- POST /advoware/proxy
- PUT /advoware/proxy
- DELETE /advoware/proxy
2026-03-01 21:47:43 +00:00