bsiggel/motia
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

feat: Add EspoCRM and Advoware integration for Beteiligte comparison

Browse Source 
- Implemented `compare_beteiligte.py` script for comparing Beteiligte structures between EspoCRM and Advoware.
- Created `beteiligte_comparison_result.json` to store comparison results.
- Developed `EspoCRMAPI` service for handling API interactions with EspoCRM.
- Added comprehensive documentation for the EspoCRM API service.
- Included error handling and logging for API operations.
- Enhanced entity management with CRUD operations and search capabilities.
This commit is contained in:
bitbylaw
2026-02-07 14:42:58 +00:00
parent 36552903e7
commit e6ab22d5f4
12 changed files with 2143 additions and 1426 deletions
Download Patch File Download Diff File Expand all files Collapse all files

403
bitbylaw/services/ESPOCRM_SERVICE.md Normal file
View File

@@ -0,0 +1,403 @@
# EspoCRM API Service
## Overview
Python client for EspoCRM REST API integration. Provides type-safe, async operations for managing entities in EspoCRM.
## Features
- ✅ API Key authentication
- ✅ Async/await support (aiohttp)
- ✅ Full CRUD operations
- ✅ Entity search and filtering
- ✅ Error handling with custom exceptions
- ✅ Optional Redis integration for caching
- ✅ Logging via Motia context
## Installation
```python
from services.espocrm import EspoCRMAPI
# Initialize with optional context for logging
espo = EspoCRMAPI(context=context)
```
## Configuration
Add to `.env` or environment:
```bash
# EspoCRM API Configuration
ESPOCRM_API_BASE_URL=https://crm.bitbylaw.com/api/v1
ESPOCRM_MARVIN_API_KEY=your_api_key_here
ESPOCRM_API_TIMEOUT_SECONDS=30
```
Required in `config.py`:
```python
class Config:
ESPOCRM_API_BASE_URL = os.getenv('ESPOCRM_API_BASE_URL', 'https://crm.bitbylaw.com/api/v1')
ESPOCRM_API_KEY = os.getenv('ESPOCRM_MARVIN_API_KEY', '')
ESPOCRM_API_TIMEOUT_SECONDS = int(os.getenv('ESPOCRM_API_TIMEOUT_SECONDS', '30'))
```
## API Methods
### Get Single Entity
```python
async def get_entity(entity_type: str, entity_id: str) -> Dict[str, Any]
```
**Usage:**
```python
# Get Beteiligter by ID
result = await espo.get_entity('Beteiligte', '64a3f2b8c9e1234567890abc')
print(result['name'])
```
### List Entities
```python
async def list_entities(
entity_type: str,
where: Optional[List[Dict]] = None,
select: Optional[str] = None,
order_by: Optional[str] = None,
offset: int = 0,
max_size: int = 50
) -> Dict[str, Any]
```
**Usage:**
```python
# List all Beteiligte with status "Active"
result = await espo.list_entities(
'Beteiligte',
where=[{
'type': 'equals',
'attribute': 'status',
'value': 'Active'
}],
select='id,name,email',
max_size=100
)
for entity in result['list']:
print(entity['name'])
print(f"Total: {result['total']}")
```
**Complex Filters:**
```python
# OR condition
where=[{
'type': 'or',
'value': [
{'type': 'equals', 'attribute': 'status', 'value': 'Zurückgestellt'},
{'type': 'equals', 'attribute': 'status', 'value': 'Warte auf neuen Anruf'}
]
}]
# AND condition
where=[
{'type': 'equals', 'attribute': 'status', 'value': 'Active'},
{'type': 'greaterThan', 'attribute': 'createdAt', 'value': '2026-01-01'}
]
```
### Create Entity
```python
async def create_entity(entity_type: str, data: Dict[str, Any]) -> Dict[str, Any]
```
**Usage:**
```python
# Create new Beteiligter
result = await espo.create_entity('Beteiligte', {
'name': 'Max Mustermann',
'email': 'max@example.com',
'phone': '+49123456789',
'status': 'New'
})
print(f"Created with ID: {result['id']}")
```
### Update Entity
```python
async def update_entity(
entity_type: str,
entity_id: str,
data: Dict[str, Any]
) -> Dict[str, Any]
```
**Usage:**
```python
# Update Beteiligter status
result = await espo.update_entity(
'Beteiligte',
'64a3f2b8c9e1234567890abc',
{'status': 'Converted'}
)
```
### Delete Entity
```python
async def delete_entity(entity_type: str, entity_id: str) -> bool
```
**Usage:**
```python
# Delete Beteiligter
success = await espo.delete_entity('Beteiligte', '64a3f2b8c9e1234567890abc')
```
### Search Entities
```python
async def search_entities(
entity_type: str,
query: str,
fields: Optional[List[str]] = None
) -> List[Dict[str, Any]]
```
**Usage:**
```python
# Full-text search
results = await espo.search_entities('Beteiligte', 'Mustermann')
for entity in results:
print(entity['name'])
```
## Common Entity Types
Based on EspoCRM standard and VMH customization:
- `Beteiligte` - Custom entity for VMH participants
- `CVmhErstgespraech` - Custom entity for VMH initial consultations
- `Contact` - Standard contacts
- `Account` - Companies/Organizations
- `Lead` - Sales leads
- `Opportunity` - Sales opportunities
- `Case` - Support cases
- `Meeting` - Calendar meetings
- `Call` - Phone calls
- `Email` - Email records
## Error Handling
```python
from services.espocrm import EspoCRMError, EspoCRMAuthError
try:
result = await espo.get_entity('Beteiligte', entity_id)
except EspoCRMAuthError as e:
# Invalid API key
context.logger.error(f"Authentication failed: {e}")
except EspoCRMError as e:
# General API error (404, 403, etc.)
context.logger.error(f"API error: {e}")
```
## Authentication
EspoCRM uses **API Key authentication** via `X-Api-Key` header.
**Create API Key in EspoCRM:**
1. Login as admin
2. Go to Administration → API Users
3. Create new API User
4. Copy API Key
5. Set permissions for API User
**Headers sent automatically:**
```
X-Api-Key: your_api_key_here
Content-Type: application/json
Accept: application/json
```
## Integration Examples
### In Motia Step
```python
from services.espocrm import EspoCRMAPI
config = {
'type': 'event',
'name': 'Sync Beteiligter to Advoware',
'subscribes': ['vmh.beteiligte.create']
}
async def handler(event, context):
entity_id = event['data']['entity_id']
# Fetch from EspoCRM
espo = EspoCRMAPI(context=context)
beteiligter = await espo.get_entity('Beteiligte', entity_id)
context.logger.info(f"Processing: {beteiligter['name']}")
# Transform and sync to Advoware...
# ...
```
### In Cron Step
```python
from services.espocrm import EspoCRMAPI
from datetime import datetime, timedelta
config = {
'type': 'cron',
'cron': '*/5 * * * *',
'name': 'Check Expired Callbacks'
}
async def handler(input, context):
espo = EspoCRMAPI(context=context)
# Find expired callbacks
now = datetime.utcnow().isoformat() + 'Z'
result = await espo.list_entities(
'CVmhErstgespraech',
where=[
{'type': 'lessThan', 'attribute': 'nchsterAnruf', 'value': now},
{'type': 'equals', 'attribute': 'status', 'value': 'Warte auf neuen Anruf'}
]
)
# Update status for expired entries
for entry in result['list']:
await espo.update_entity(
'CVmhErstgespraech',
entry['id'],
{'status': 'Neu'}
)
context.logger.info(f"Reset status for {entry['id']}")
```
## Helper Script: Compare Structures
Compare entity structures between EspoCRM and Advoware:
```bash
# Compare by EspoCRM ID (auto-search in Advoware)
python bitbylaw/scripts/compare_beteiligte.py 64a3f2b8c9e1234567890abc
# Compare with specific Advoware ID
python bitbylaw/scripts/compare_beteiligte.py 64a3f2b8c9e1234567890abc 12345
```
**Output:**
- Entity data from both systems
- Field structure comparison
- Suggested field mappings
- JSON output saved to `scripts/beteiligte_comparison_result.json`
## Performance
### Timeout
Default: 30 seconds (configurable via `ESPOCRM_API_TIMEOUT_SECONDS`)
```python
# Custom timeout for specific call
result = await espo.api_call('/Beteiligte', timeout_seconds=60)
```
### Pagination
```python
# Fetch in pages
offset = 0
max_size = 50
while True:
result = await espo.list_entities(
'Beteiligte',
offset=offset,
max_size=max_size
)
entities = result['list']
if not entities:
break
# Process entities...
offset += len(entities)
if len(entities) < max_size:
break # Last page
```
### Rate Limiting
Optional Redis-based rate limiting can be implemented:
```python
# Check rate limit before API call
rate_limit_key = f'espocrm:rate_limit:{entity_type}'
if espo.redis_client:
count = espo.redis_client.incr(rate_limit_key)
espo.redis_client.expire(rate_limit_key, 60) # 1 minute window
if count > 100: # Max 100 requests per minute
raise Exception("Rate limit exceeded")
```
## Testing
```python
import pytest
from services.espocrm import EspoCRMAPI
@pytest.mark.asyncio
async def test_get_entity():
espo = EspoCRMAPI()
# Mock or use test entity ID
result = await espo.get_entity('Contact', 'test-id-123')
assert 'id' in result
assert result['id'] == 'test-id-123'
```
## Logging
All operations are logged via context.logger:
```
[INFO] [EspoCRM] EspoCRM API initialized with base URL: https://crm.bitbylaw.com/api/v1
[DEBUG] [EspoCRM] API call: GET https://crm.bitbylaw.com/api/v1/Beteiligte/123
[DEBUG] [EspoCRM] Response status: 200
[INFO] [EspoCRM] Getting Beteiligte with ID: 123
```
## Related Files
- [services/espocrm.py](./espocrm.py) - Implementation
- [scripts/compare_beteiligte.py](../scripts/compare_beteiligte.py) - Comparison tool
- [steps/crm-bbl-vmh-reset-nextcall_step.py](../../steps/crm-bbl-vmh-reset-nextcall_step.py) - Example usage
- [config.py](../config.py) - Configuration
## EspoCRM API Documentation
Official docs: https://docs.espocrm.com/development/api/
**Key Concepts:**
- RESTful API with JSON
- Entity-based operations
- Filter operators: `equals`, `notEquals`, `greaterThan`, `lessThan`, `like`, `contains`, `in`, `isNull`, `isNotNull`
- Boolean operators: `and` (default), `or`
- Metadata API: `/Metadata` (for entity definitions)

4
bitbylaw/services/advoware.py
View File

@@ -122,7 +122,9 @@ class AdvowareAPI:
params: Optional[Dict] = None, json_data: Optional[Dict] = None,
files: Optional[Any] = None, data: Optional[Any] = None,
timeout_seconds: Optional[int] = None) -> Any:
url = self.API_BASE_URL + endpoint
# Bereinige doppelte Slashes
endpoint = endpoint.lstrip('/')
url = self.API_BASE_URL.rstrip('/') + '/' + endpoint
effective_timeout = aiohttp.ClientTimeout(total=timeout_seconds or Config.ADVOWARE_API_TIMEOUT_SECONDS)
token = self.get_access_token() # Sync call
effective_headers = headers.copy() if headers else {}

276
bitbylaw/services/espocrm.py Normal file
View File

@@ -0,0 +1,276 @@
import aiohttp
import asyncio
import logging
import redis
from typing import Optional, Dict, Any, List
from config import Config
logger = logging.getLogger(__name__)
class EspoCRMError(Exception):
"""Base exception for EspoCRM API errors"""
pass
class EspoCRMAuthError(EspoCRMError):
"""Authentication error"""
pass
class EspoCRMAPI:
"""
EspoCRM API Client for bitbylaw integration.
Supports:
- API Key authentication (X-Api-Key header)
- Standard REST operations (GET, POST, PUT, DELETE)
- Entity management (Beteiligte, CVmhErstgespraech, etc.)
"""
def __init__(self, context=None):
self.context = context
self._log("EspoCRMAPI __init__ started", level='debug')
# Configuration
self.api_base_url = Config.ESPOCRM_API_BASE_URL
self.api_key = Config.ESPOCRM_API_KEY
if not self.api_key:
raise EspoCRMAuthError("ESPOCRM_MARVIN_API_KEY not configured in environment")
self._log(f"EspoCRM API initialized with base URL: {self.api_base_url}")
# Optional Redis for caching/rate limiting
try:
self.redis_client = redis.Redis(
host=Config.REDIS_HOST,
port=int(Config.REDIS_PORT),
db=int(Config.REDIS_DB_ADVOWARE_CACHE),
socket_timeout=Config.REDIS_TIMEOUT_SECONDS,
socket_connect_timeout=Config.REDIS_TIMEOUT_SECONDS,
decode_responses=True
)
self.redis_client.ping()
self._log("Connected to Redis for EspoCRM operations")
except Exception as e:
self._log(f"Could not connect to Redis: {e}. Continuing without caching.", level='warning')
self.redis_client = None
def _log(self, message: str, level: str = 'info'):
"""Log message via context.logger if available, otherwise use module logger"""
log_func = getattr(logger, level, logger.info)
if self.context and hasattr(self.context, 'logger'):
ctx_log_func = getattr(self.context.logger, level, self.context.logger.info)
ctx_log_func(f"[EspoCRM] {message}")
else:
log_func(f"[EspoCRM] {message}")
def _get_headers(self) -> Dict[str, str]:
"""Generate request headers with API key"""
return {
'X-Api-Key': self.api_key,
'Content-Type': 'application/json',
'Accept': 'application/json'
}
async def api_call(
self,
endpoint: str,
method: str = 'GET',
params: Optional[Dict] = None,
json_data: Optional[Dict] = None,
timeout_seconds: Optional[int] = None
) -> Any:
"""
Make an API call to EspoCRM.
Args:
endpoint: API endpoint (e.g., '/Beteiligte/123' or '/CVmhErstgespraech')
method: HTTP method (GET, POST, PUT, DELETE)
params: Query parameters
json_data: JSON body for POST/PUT
timeout_seconds: Request timeout
Returns:
Parsed JSON response or None
Raises:
EspoCRMError: On API errors
"""
# Ensure endpoint starts with /
if not endpoint.startswith('/'):
endpoint = '/' + endpoint
url = self.api_base_url.rstrip('/') + endpoint
headers = self._get_headers()
effective_timeout = aiohttp.ClientTimeout(
total=timeout_seconds or Config.ESPOCRM_API_TIMEOUT_SECONDS
)
self._log(f"API call: {method} {url}", level='debug')
if params:
self._log(f"Params: {params}", level='debug')
async with aiohttp.ClientSession(timeout=effective_timeout) as session:
try:
async with session.request(
method,
url,
headers=headers,
params=params,
json=json_data
) as response:
# Log response status
self._log(f"Response status: {response.status}", level='debug')
# Handle errors
if response.status == 401:
raise EspoCRMAuthError("Authentication failed - check API key")
elif response.status == 403:
raise EspoCRMError("Access forbidden")
elif response.status == 404:
raise EspoCRMError(f"Resource not found: {endpoint}")
elif response.status >= 400:
error_text = await response.text()
raise EspoCRMError(f"API error {response.status}: {error_text}")
# Parse response
if response.content_type == 'application/json':
result = await response.json()
self._log(f"Response received", level='debug')
return result
else:
# For DELETE or other non-JSON responses
return None
except aiohttp.ClientError as e:
self._log(f"API call failed: {e}", level='error')
raise EspoCRMError(f"Request failed: {e}") from e
async def get_entity(self, entity_type: str, entity_id: str) -> Dict[str, Any]:
"""
Get a single entity by ID.
Args:
entity_type: Entity type (e.g., 'Beteiligte', 'CVmhErstgespraech')
entity_id: Entity ID
Returns:
Entity data as dict
"""
self._log(f"Getting {entity_type} with ID: {entity_id}")
return await self.api_call(f"/{entity_type}/{entity_id}", method='GET')
async def list_entities(
self,
entity_type: str,
where: Optional[List[Dict]] = None,
select: Optional[str] = None,
order_by: Optional[str] = None,
offset: int = 0,
max_size: int = 50
) -> Dict[str, Any]:
"""
List entities with filtering and pagination.
Args:
entity_type: Entity type
where: Filter conditions (EspoCRM format)
select: Comma-separated field list
order_by: Sort field
offset: Pagination offset
max_size: Max results per page
Returns:
Dict with 'list' and 'total' keys
"""
params = {
'offset': offset,
'maxSize': max_size
}
if where:
params['where'] = where
if select:
params['select'] = select
if order_by:
params['orderBy'] = order_by
self._log(f"Listing {entity_type} entities")
return await self.api_call(f"/{entity_type}", method='GET', params=params)
async def create_entity(
self,
entity_type: str,
data: Dict[str, Any]
) -> Dict[str, Any]:
"""
Create a new entity.
Args:
entity_type: Entity type
data: Entity data
Returns:
Created entity with ID
"""
self._log(f"Creating {entity_type} entity")
return await self.api_call(f"/{entity_type}", method='POST', json_data=data)
async def update_entity(
self,
entity_type: str,
entity_id: str,
data: Dict[str, Any]
) -> Dict[str, Any]:
"""
Update an existing entity.
Args:
entity_type: Entity type
entity_id: Entity ID
data: Updated fields
Returns:
Updated entity
"""
self._log(f"Updating {entity_type} with ID: {entity_id}")
return await self.api_call(f"/{entity_type}/{entity_id}", method='PUT', json_data=data)
async def delete_entity(self, entity_type: str, entity_id: str) -> bool:
"""
Delete an entity.
Args:
entity_type: Entity type
entity_id: Entity ID
Returns:
True if successful
"""
self._log(f"Deleting {entity_type} with ID: {entity_id}")
await self.api_call(f"/{entity_type}/{entity_id}", method='DELETE')
return True
async def search_entities(
self,
entity_type: str,
query: str,
fields: Optional[List[str]] = None
) -> List[Dict[str, Any]]:
"""
Search entities by text query.
Args:
entity_type: Entity type
query: Search query
fields: Fields to search in
Returns:
List of matching entities
"""
where = [{
'type': 'textFilter',
'value': query
}]
result = await self.list_entities(entity_type, where=where)
return result.get('list', [])