bsiggel/motia
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

feat: Add EspoCRM and Advoware integration for Beteiligte comparison

Browse Source 
- Implemented `compare_beteiligte.py` script for comparing Beteiligte structures between EspoCRM and Advoware.
- Created `beteiligte_comparison_result.json` to store comparison results.
- Developed `EspoCRMAPI` service for handling API interactions with EspoCRM.
- Added comprehensive documentation for the EspoCRM API service.
- Included error handling and logging for API operations.
- Enhanced entity management with CRUD operations and search capabilities.
This commit is contained in:
bitbylaw
2026-02-07 14:42:58 +00:00
parent 36552903e7
commit e6ab22d5f4
12 changed files with 2143 additions and 1426 deletions
Download Patch File Download Diff File Expand all files Collapse all files

624
bitbylaw/docs/DEPLOYMENT.md
View File

@@ -1,624 +0,0 @@
# Deployment Guide
## Production Deployment
### Prerequisites
- Root/sudo access zum Server
- Ubuntu/Debian Linux (tested on Ubuntu 22.04+)
- Internet-Zugang für Package-Installation
### Installation Steps
#### 1. System Dependencies
```bash
# Update system
sudo apt-get update
sudo apt-get upgrade -y
# Install Node.js 18.x
curl -fsSL https://deb.nodesource.com/setup_18.x | sudo -E bash -
sudo apt-get install -y nodejs
# Install Python 3.13
sudo apt-get install -y python3.13 python3.13-venv python3.13-dev
# Install Redis
sudo apt-get install -y redis-server
# Install Git
sudo apt-get install -y git
# Start Redis
sudo systemctl enable redis-server
sudo systemctl start redis-server
```
#### 2. Application Setup
```bash
# Create application directory
sudo mkdir -p /opt/motia-app
cd /opt/motia-app
# Clone repository (oder rsync von Development)
git clone <repository-url> bitbylaw
cd bitbylaw
# Create www-data user if not exists
sudo useradd -r -s /bin/bash www-data || true
# Set ownership
sudo chown -R www-data:www-data /opt/motia-app
```
#### 3. Node.js Dependencies
```bash
# Als www-data user
sudo -u www-data bash
cd /opt/motia-app/bitbylaw
# Install Node.js packages
npm install
# Build TypeScript (falls nötig)
npm run build
```
#### 4. Python Dependencies
```bash
# Als www-data user
cd /opt/motia-app/bitbylaw
# Create virtual environment
python3.13 -m venv python_modules
# Activate
source python_modules/bin/activate
# Install dependencies
pip install -r requirements.txt
# Deactivate
deactivate
```
#### 5. Service Account Setup
```bash
# Copy service account JSON
sudo cp service-account.json /opt/motia-app/service-account.json
# Set secure permissions
sudo chmod 600 /opt/motia-app/service-account.json
sudo chown www-data:www-data /opt/motia-app/service-account.json
```
Siehe auch: [GOOGLE_SETUP_README.md](../GOOGLE_SETUP_README.md)
#### 6. systemd Service
Erstellen Sie `/etc/systemd/system/motia.service`:
```ini
[Unit]
Description=Motia Backend Framework
After=network.target redis-server.service
[Service]
Type=simple
User=www-data
WorkingDirectory=/opt/motia-app/bitbylaw
# Environment Variables
Environment=NODE_ENV=production
Environment=NODE_OPTIONS=--max-old-space-size=8192 --inspect --heapsnapshot-signal=SIGUSR2
Environment=HOST=0.0.0.0
Environment=MOTIA_LOG_LEVEL=info
Environment=NPM_CONFIG_CACHE=/opt/motia-app/.npm-cache
# Advoware Configuration (ADJUST VALUES!)
Environment=ADVOWARE_API_BASE_URL=https://www2.advo-net.net:90/
Environment=ADVOWARE_PRODUCT_ID=64
Environment=ADVOWARE_APP_ID=your_app_id
Environment=ADVOWARE_API_KEY=your_api_key_base64
Environment=ADVOWARE_KANZLEI=your_kanzlei
Environment=ADVOWARE_DATABASE=your_database
Environment=ADVOWARE_USER=your_user
Environment=ADVOWARE_ROLE=2
Environment=ADVOWARE_PASSWORD=your_password
Environment=ADVOWARE_WRITE_PROTECTION=false
# Redis Configuration
Environment=REDIS_HOST=localhost
Environment=REDIS_PORT=6379
Environment=REDIS_DB_ADVOWARE_CACHE=1
Environment=REDIS_DB_CALENDAR_SYNC=2
# Google Calendar
Environment=GOOGLE_CALENDAR_SERVICE_ACCOUNT_PATH=/opt/motia-app/service-account.json
# EspoCRM (if used)
Environment=ESPOCRM_MARVIN_API_KEY=your_webhook_key
# Start Command
ExecStart=/bin/bash -c 'source /opt/motia-app/python_modules/bin/activate && /usr/bin/npm start'
# Restart Policy
Restart=always
RestartSec=10
# Security
NoNewPrivileges=true
PrivateTmp=true
[Install]
WantedBy=multi-user.target
```
**WICHTIG**: Passen Sie alle `your_*` Werte an!
#### 7. Enable and Start Service
```bash
# Reload systemd
sudo systemctl daemon-reload
# Enable service (autostart)
sudo systemctl enable motia.service
# Start service
sudo systemctl start motia.service
# Check status
sudo systemctl status motia.service
```
#### 8. Verify Installation
```bash
# Check logs
sudo journalctl -u motia.service -f
# Test API
curl http://localhost:3000/health # (wenn implementiert)
# Test Advoware Proxy
curl "http://localhost:3000/advoware/proxy?endpoint=employees"
```
## Reverse Proxy Setup (nginx)
### Install nginx
```bash
sudo apt-get install -y nginx
```
### Configure
`/etc/nginx/sites-available/motia`:
```nginx
upstream motia_backend {
server 127.0.0.1:3000;
}
server {
listen 80;
server_name your-domain.com;
# Redirect to HTTPS
return 301 https://$server_name$request_uri;
}
server {
listen 443 ssl http2;
server_name your-domain.com;
# SSL Configuration (Let's Encrypt)
ssl_certificate /etc/letsencrypt/live/your-domain.com/fullchain.pem;
ssl_certificate_key /etc/letsencrypt/live/your-domain.com/privkey.pem;
# Security Headers
add_header X-Frame-Options "SAMEORIGIN" always;
add_header X-Content-Type-Options "nosniff" always;
add_header X-XSS-Protection "1; mode=block" always;
# Proxy Settings
location / {
proxy_pass http://motia_backend;
proxy_http_version 1.1;
proxy_set_header Upgrade $http_upgrade;
proxy_set_header Connection 'upgrade';
proxy_set_header Host $host;
proxy_cache_bypass $http_upgrade;
proxy_set_header X-Real-IP $remote_addr;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
proxy_set_header X-Forwarded-Proto $scheme;
# Timeouts
proxy_connect_timeout 60s;
proxy_send_timeout 60s;
proxy_read_timeout 60s;
}
# Access Log
access_log /var/log/nginx/motia-access.log;
error_log /var/log/nginx/motia-error.log;
}
```
### Enable and Restart
```bash
# Enable site
sudo ln -s /etc/nginx/sites-available/motia /etc/nginx/sites-enabled/
# Test configuration
sudo nginx -t
# Restart nginx
sudo systemctl restart nginx
```
### SSL Certificate (Let's Encrypt)
```bash
# Install certbot
sudo apt-get install -y certbot python3-certbot-nginx
# Obtain certificate
sudo certbot --nginx -d your-domain.com
# Auto-renewal is configured automatically
```
## Firewall Configuration
```bash
# Allow SSH
sudo ufw allow 22/tcp
# Allow HTTP/HTTPS (if using nginx)
sudo ufw allow 80/tcp
sudo ufw allow 443/tcp
# Enable firewall
sudo ufw enable
```
**Wichtig**: Port 3000 NICHT öffentlich öffnen (nur via nginx reverse proxy)
## Monitoring
### systemd Service Status
```bash
# Status anzeigen
sudo systemctl status motia.service
# Ist enabled?
sudo systemctl is-enabled motia.service
# Ist aktiv?
sudo systemctl is-active motia.service
```
### Logs
```bash
# Live logs
sudo journalctl -u motia.service -f
# Last 100 lines
sudo journalctl -u motia.service -n 100
# Since today
sudo journalctl -u motia.service --since today
# Filter by priority (error only)
sudo journalctl -u motia.service -p err
```
### Resource Usage
```bash
# CPU and Memory
sudo systemctl status motia.service
# Detailed process info
ps aux | grep motia
# Memory usage
sudo pmap $(pgrep -f "motia start") | tail -n 1
```
### Redis Monitoring
```bash
# Connect to Redis
redis-cli
# Show info
INFO
# Show database sizes
INFO keyspace
# Monitor commands (real-time)
MONITOR
# Show memory usage
MEMORY USAGE <key>
```
## Backup Strategy
### Application Code
```bash
# Git-based backup
cd /opt/motia-app/bitbylaw
git pull origin main
# Or: rsync backup
rsync -av /opt/motia-app/bitbylaw/ /backup/motia-app/
```
### Redis Data
```bash
# RDB snapshot (automatic by Redis)
# Location: /var/lib/redis/dump.rdb
# Manual backup
sudo cp /var/lib/redis/dump.rdb /backup/redis-dump-$(date +%Y%m%d).rdb
# Restore
sudo systemctl stop redis-server
sudo cp /backup/redis-dump-20260207.rdb /var/lib/redis/dump.rdb
sudo chown redis:redis /var/lib/redis/dump.rdb
sudo systemctl start redis-server
```
### Configuration
```bash
# Backup systemd service
sudo cp /etc/systemd/system/motia.service /backup/motia.service
# Backup nginx config
sudo cp /etc/nginx/sites-available/motia /backup/nginx-motia.conf
# Backup service account
sudo cp /opt/motia-app/service-account.json /backup/service-account.json.backup
```
## Updates & Maintenance
### Application Update
```bash
# 1. Pull latest code
cd /opt/motia-app/bitbylaw
sudo -u www-data git pull origin main
# 2. Update dependencies
sudo -u www-data npm install
sudo -u www-data bash -c 'source python_modules/bin/activate && pip install -r requirements.txt'
# 3. Restart service
sudo systemctl restart motia.service
# 4. Verify
sudo journalctl -u motia.service -f
```
### Zero-Downtime Deployment
Für zukünftige Implementierung mit Blue-Green Deployment:
```bash
# 1. Deploy to staging directory
# 2. Run health checks
# 3. Switch symlink
# 4. Reload service
# 5. Rollback if issues
```
### Database Migrations
**Aktuell**: Keine Datenbank-Migrationen (nur Redis)
**Zukünftig** (PostgreSQL):
```bash
# Run migrations
python manage.py migrate
```
## Security Hardening
### File Permissions
```bash
# Application files
sudo chown -R www-data:www-data /opt/motia-app
sudo chmod 755 /opt/motia-app
sudo chmod 755 /opt/motia-app/bitbylaw
# Service account
sudo chmod 600 /opt/motia-app/service-account.json
sudo chown www-data:www-data /opt/motia-app/service-account.json
# No world-readable secrets
sudo find /opt/motia-app -type f -name "*.json" -exec chmod 600 {} \;
```
### Redis Security
```bash
# Edit Redis config
sudo nano /etc/redis/redis.conf
# Bind to localhost only
bind 127.0.0.1 ::1
# Disable dangerous commands (optional)
rename-command FLUSHDB ""
rename-command FLUSHALL ""
rename-command CONFIG ""
# Restart Redis
sudo systemctl restart redis-server
```
### systemd Hardening
Bereits in Service-Datei enthalten:
- `NoNewPrivileges=true` - Verhindert Privilege-Escalation
- `PrivateTmp=true` - Isoliertes /tmp
- User: `www-data` (non-root)
Weitere Optionen:
```ini
[Service]
ProtectSystem=strict
ProtectHome=true
ReadWritePaths=/opt/motia-app
```
## Disaster Recovery
### Service Crashed
```bash
# Check status
sudo systemctl status motia.service
# View logs
sudo journalctl -u motia.service -n 100
# Restart
sudo systemctl restart motia.service
# If still failing, check:
# - Redis is running
# - Service account file exists
# - Environment variables are set
```
### Redis Data Loss
```bash
# Restore from backup
sudo systemctl stop redis-server
sudo cp /backup/redis-dump-latest.rdb /var/lib/redis/dump.rdb
sudo chown redis:redis /var/lib/redis/dump.rdb
sudo systemctl start redis-server
# Clear specific data if corrupted
redis-cli -n 1 FLUSHDB # Advoware cache
redis-cli -n 2 FLUSHDB # Calendar sync
```
### Complete System Failure
```bash
# 1. Fresh server setup (siehe Installation Steps)
# 2. Restore application code from Git/Backup
# 3. Restore configuration (systemd, nginx)
# 4. Restore service-account.json
# 5. Restore Redis data (optional, will rebuild)
# 6. Start services
```
## Performance Tuning
### Node.js Memory
In systemd service:
```ini
Environment=NODE_OPTIONS=--max-old-space-size=8192 # 8GB
```
### Redis Memory
In `/etc/redis/redis.conf`:
```
maxmemory 2gb
maxmemory-policy allkeys-lru
```
### Linux Kernel
```bash
# Increase file descriptors
echo "fs.file-max = 65536" | sudo tee -a /etc/sysctl.conf
sudo sysctl -p
# For www-data user
sudo nano /etc/security/limits.conf
# Add:
www-data soft nofile 65536
www-data hard nofile 65536
```
## Health Checks
### Automated Monitoring
Cron job für Health Checks:
```bash
# /usr/local/bin/motia-health-check.sh
#!/bin/bash
if ! systemctl is-active --quiet motia.service; then
echo "Motia service is down!" | mail -s "ALERT: Motia Down" admin@example.com
systemctl start motia.service
fi
```
```bash
# Add to crontab
sudo crontab -e
# Add line:
*/5 * * * * /usr/local/bin/motia-health-check.sh
```
### External Monitoring
Services wie Uptime Robot, Pingdom, etc. können verwendet werden:
- HTTP Endpoint: `https://your-domain.com/health`
- Check-Interval: 5 Minuten
- Alert via Email/SMS
## Rollback Procedure
```bash
# 1. Stop current service
sudo systemctl stop motia.service
# 2. Revert to previous version
cd /opt/motia-app/bitbylaw
sudo -u www-data git log # Find previous commit
sudo -u www-data git reset --hard <commit-hash>
# 3. Restore dependencies (if needed)
sudo -u www-data npm install
# 4. Start service
sudo systemctl start motia.service
# 5. Verify
sudo journalctl -u motia.service -f
```
## Related Documentation
- [Architecture](ARCHITECTURE.md)
- [Configuration](CONFIGURATION.md)
- [Troubleshooting](TROUBLESHOOTING.md)

800
bitbylaw/docs/TROUBLESHOOTING.md
View File

@@ -1,800 +0,0 @@
# Troubleshooting Guide
## Service Issues
### Service Won't Start
**Symptoms**: `systemctl start motia.service` schlägt fehl
**Diagnose**:
```bash
# Check service status
sudo systemctl status motia.service
# View detailed logs
sudo journalctl -u motia.service -n 100 --no-pager
# Check for port conflicts
sudo netstat -tlnp | grep 3000
```
**Häufige Ursachen**:
1. **Port 3000 bereits belegt**:
```bash
# Find process
sudo lsof -i :3000
# Kill process
sudo kill -9 <PID>
```
2. **Fehlende Dependencies**:
```bash
cd /opt/motia-app/bitbylaw
sudo -u www-data npm install
sudo -u www-data bash -c 'source python_modules/bin/activate && pip install -r requirements.txt'
```
3. **Falsche Permissions**:
```bash
sudo chown -R www-data:www-data /opt/motia-app
sudo chmod 600 /opt/motia-app/service-account.json
```
4. **Environment Variables fehlen**:
```bash
# Check systemd environment
sudo systemctl show motia.service -p Environment
# Verify required vars
sudo systemctl cat motia.service | grep Environment
```
### Service Keeps Crashing
**Symptoms**: Service startet, crashed aber nach kurzer Zeit
**Diagnose**:
```bash
# Watch logs in real-time
sudo journalctl -u motia.service -f
# Check for OOM (Out of Memory)
dmesg | grep -i "out of memory"
sudo grep -i "killed process" /var/log/syslog
```
**Solutions**:
1. **Memory Limit erhöhen**:
```ini
# In /etc/systemd/system/motia.service
Environment=NODE_OPTIONS=--max-old-space-size=8192
```
2. **Python Memory Leak**:
```bash
# Check memory usage
ps aux | grep python
# Restart service periodically (workaround)
# Add to crontab:
0 3 * * * systemctl restart motia.service
```
3. **Unhandled Exception**:
```bash
# Check error logs
sudo journalctl -u motia.service -p err
# Add try-catch in problematic step
```
## Redis Issues
### Redis Connection Failed
**Symptoms**: "Redis connection failed" in logs
**Diagnose**:
```bash
# Check Redis status
sudo systemctl status redis-server
# Test connection
redis-cli ping
# Check config
redis-cli CONFIG GET bind
redis-cli CONFIG GET port
```
**Solutions**:
1. **Redis not running**:
```bash
sudo systemctl start redis-server
sudo systemctl enable redis-server
```
2. **Wrong host/port**:
```bash
# Check environment
echo $REDIS_HOST
echo $REDIS_PORT
# Test connection
redis-cli -h $REDIS_HOST -p $REDIS_PORT ping
```
3. **Permission denied**:
```bash
# Check Redis log
sudo tail -f /var/log/redis/redis-server.log
# Fix permissions
sudo chown redis:redis /var/lib/redis
sudo chmod 750 /var/lib/redis
```
### Redis Out of Memory
**Symptoms**: "OOM command not allowed" errors
**Diagnose**:
```bash
# Check memory usage
redis-cli INFO memory
# Check maxmemory setting
redis-cli CONFIG GET maxmemory
```
**Solutions**:
1. **Increase maxmemory**:
```bash
# In /etc/redis/redis.conf
maxmemory 2gb
maxmemory-policy allkeys-lru
sudo systemctl restart redis-server
```
2. **Clear old data**:
```bash
# Clear cache (safe for Advoware tokens)
redis-cli -n 1 FLUSHDB
# Clear calendar sync state
redis-cli -n 2 FLUSHDB
```
3. **Check for memory leaks**:
```bash
# Find large keys
redis-cli --bigkeys
# Check specific key size
redis-cli MEMORY USAGE <key>
```
## Advoware API Issues
### Authentication Failed
**Symptoms**: "401 Unauthorized" oder "HMAC signature invalid"
**Diagnose**:
```bash
# Check logs for auth errors
sudo journalctl -u motia.service | grep -i "auth\|token\|401"
# Test token fetch manually
python3 << 'EOF'
from services.advoware import AdvowareAPI
api = AdvowareAPI()
token = api.get_access_token(force_refresh=True)
print(f"Token: {token[:20]}...")
EOF
```
**Solutions**:
1. **Invalid API Key**:
```bash
# Verify API Key is Base64
echo $ADVOWARE_API_KEY | base64 -d
# Re-encode if needed
echo -n "raw_key" | base64
```
2. **Wrong credentials**:
```bash
# Verify environment variables
sudo systemctl show motia.service -p Environment | grep ADVOWARE
# Update in systemd service
sudo nano /etc/systemd/system/motia.service
sudo systemctl daemon-reload
sudo systemctl restart motia.service
```
3. **Token expired**:
```bash
# Clear cached token
redis-cli -n 1 DEL advoware_access_token advoware_token_timestamp
# Retry request (will fetch new token)
```
### API Timeout
**Symptoms**: "Request timeout" oder "API call failed"
**Diagnose**:
```bash
# Check API response time
time curl "http://localhost:3000/advoware/proxy?endpoint=employees"
# Check network connectivity
ping www2.advo-net.net
curl -I https://www2.advo-net.net:90/
```
**Solutions**:
1. **Increase timeout**:
```bash
# In environment
export ADVOWARE_API_TIMEOUT_SECONDS=60
# Or in systemd service
Environment=ADVOWARE_API_TIMEOUT_SECONDS=60
```
2. **Network issues**:
```bash
# Check firewall
sudo ufw status
# Test direct connection
curl -v https://www2.advo-net.net:90/
```
3. **Advoware API down**:
```bash
# Wait and retry
# Implement exponential backoff in code
```
## Google Calendar Issues
### Service Account Not Found
**Symptoms**: "service-account.json not found"
**Diagnose**:
```bash
# Check file exists
ls -la /opt/motia-app/service-account.json
# Check permissions
ls -la /opt/motia-app/service-account.json
# Check environment variable
echo $GOOGLE_CALENDAR_SERVICE_ACCOUNT_PATH
```
**Solutions**:
1. **File missing**:
```bash
# Copy from backup
sudo cp /backup/service-account.json /opt/motia-app/
# Set permissions
sudo chmod 600 /opt/motia-app/service-account.json
sudo chown www-data:www-data /opt/motia-app/service-account.json
```
2. **Wrong path**:
```bash
# Update environment
# In /etc/systemd/system/motia.service:
Environment=GOOGLE_CALENDAR_SERVICE_ACCOUNT_PATH=/opt/motia-app/service-account.json
sudo systemctl daemon-reload
sudo systemctl restart motia.service
```
### Calendar API Rate Limit
**Symptoms**: "403 Rate limit exceeded" oder "429 Too Many Requests"
**Diagnose**:
```bash
# Check rate limiting in logs
sudo journalctl -u motia.service | grep -i "rate\|403\|429"
# Check Redis rate limit tokens
redis-cli -n 2 GET google_calendar_api_tokens
```
**Solutions**:
1. **Wait for rate limit reset**:
```bash
# Rate limit resets every minute
# Wait 60 seconds and retry
```
2. **Adjust rate limit settings**:
```python
# In calendar_sync_event_step.py
MAX_TOKENS = 7 # Decrease if hitting limits
REFILL_RATE_PER_MS = 7 / 1000
```
3. **Request quota increase**:
- Go to Google Cloud Console
- Navigate to "APIs & Services" → "Quotas"
- Request increase for Calendar API
### Calendar Access Denied
**Symptoms**: "Access denied" oder "Insufficient permissions"
**Diagnose**:
```bash
# Check service account email
python3 << 'EOF'
import json
with open('/opt/motia-app/service-account.json') as f:
data = json.load(f)
print(f"Service Account: {data['client_email']}")
EOF
# Test API access
python3 << 'EOF'
from google.oauth2 import service_account
from googleapiclient.discovery import build
creds = service_account.Credentials.from_service_account_file(
'/opt/motia-app/service-account.json',
scopes=['https://www.googleapis.com/auth/calendar']
)
service = build('calendar', 'v3', credentials=creds)
result = service.calendarList().list().execute()
print(f"Calendars: {len(result.get('items', []))}")
EOF
```
**Solutions**:
1. **Calendar not shared**:
```bash
# Share calendar with service account email
# In Google Calendar UI: Settings → Share → Add service account email
```
2. **Wrong scopes**:
```bash
# Verify scopes in code
# Should be: https://www.googleapis.com/auth/calendar
```
3. **Domain-wide delegation**:
```bash
# For G Suite, enable domain-wide delegation
# See GOOGLE_SETUP_README.md
```
## Calendar Sync Issues
### Sync Not Running
**Symptoms**: Keine Calendar-Updates, keine Sync-Logs
**Diagnose**:
```bash
# Check if cron is triggering
sudo journalctl -u motia.service | grep -i "calendar_sync_cron"
# Manually trigger sync
curl -X POST "http://localhost:3000/advoware/calendar/sync" \
-H "Content-Type: application/json" \
-d '{"full_content": true}'
# Check for locks
redis-cli -n 1 KEYS "calendar_sync:lock:*"
```
**Solutions**:
1. **Cron not configured**:
```python
# Verify calendar_sync_cron_step.py has correct schedule
config = {
'schedule': '0 2 * * *', # Daily at 2 AM
}
```
2. **Lock stuck**:
```bash
# Clear all locks
python /opt/motia-app/bitbylaw/delete_employee_locks.py
# Or manually
redis-cli -n 1 DEL calendar_sync:lock:SB
```
3. **Errors in sync**:
```bash
# Check error logs
sudo journalctl -u motia.service -p err | grep calendar
```
### Duplicate Events
**Symptoms**: Events erscheinen mehrfach in Google Calendar
**Diagnose**:
```bash
# Check for concurrent syncs
redis-cli -n 1 KEYS "calendar_sync:lock:*"
# Check logs for duplicate processing
sudo journalctl -u motia.service | grep -i "duplicate\|already exists"
```
**Solutions**:
1. **Locking not working**:
```bash
# Verify Redis lock TTL
redis-cli -n 1 TTL calendar_sync:lock:SB
# Should return positive number if locked
```
2. **Manual cleanup**:
```bash
# Delete duplicates in Google Calendar UI
# Or use cleanup script (if available)
```
3. **Improve deduplication logic**:
```python
# In calendar_sync_event_step.py
# Add better event matching logic
```
### Events Not Syncing
**Symptoms**: Advoware events nicht in Google Calendar
**Diagnose**:
```bash
# Check specific employee
curl -X POST "http://localhost:3000/advoware/calendar/sync" \
-H "Content-Type: application/json" \
-d '{"kuerzel": "SB", "full_content": true}'
# Check logs for that employee
sudo journalctl -u motia.service | grep "SB"
# Check if calendar exists
python3 << 'EOF'
from google.oauth2 import service_account
from googleapiclient.discovery import build
creds = service_account.Credentials.from_service_account_file(
'/opt/motia-app/service-account.json',
scopes=['https://www.googleapis.com/auth/calendar']
)
service = build('calendar', 'v3', credentials=creds)
result = service.calendarList().list().execute()
for cal in result.get('items', []):
if 'AW-SB' in cal['summary']:
print(f"Found: {cal['summary']} - {cal['id']}")
EOF
```
**Solutions**:
1. **Calendar doesn't exist**:
```bash
# Will be auto-created on first sync
# Force sync to trigger creation
```
2. **Date range mismatch**:
```python
# Check FETCH_FROM and FETCH_TO in calendar_sync_event_step.py
# Default: Previous year to 9 years ahead
```
3. **Write protection enabled**:
```bash
# Check environment
echo $ADVOWARE_WRITE_PROTECTION
# Should be "false" for two-way sync
```
## Webhook Issues
### Webhooks Not Received
**Symptoms**: EspoCRM sendet Webhooks, aber keine Verarbeitung
**Diagnose**:
```bash
# Check if endpoint reachable
curl -X POST "http://localhost:3000/vmh/webhook/beteiligte/create" \
-H "Content-Type: application/json" \
-d '[{"id": "test-123"}]'
# Check firewall
sudo ufw status
# Check nginx logs (if using reverse proxy)
sudo tail -f /var/log/nginx/motia-access.log
sudo tail -f /var/log/nginx/motia-error.log
```
**Solutions**:
1. **Firewall blocking**:
```bash
# Allow port (if direct access)
sudo ufw allow 3000/tcp
# Or use reverse proxy (recommended)
```
2. **Wrong URL in EspoCRM**:
```bash
# Verify URL in EspoCRM webhook configuration
# Should be: https://your-domain.com/vmh/webhook/beteiligte/create
```
3. **SSL certificate issues**:
```bash
# Check certificate
openssl s_client -connect your-domain.com:443
# Renew certificate
sudo certbot renew
```
### Webhook Deduplication Not Working
**Symptoms**: Mehrfache Verarbeitung derselben Webhooks
**Diagnose**:
```bash
# Check Redis dedup sets
redis-cli -n 1 SMEMBERS vmh:beteiligte:create_pending
redis-cli -n 1 SMEMBERS vmh:beteiligte:update_pending
redis-cli -n 1 SMEMBERS vmh:beteiligte:delete_pending
# Check for concurrent webhook processing
sudo journalctl -u motia.service | grep "Webhook.*received"
```
**Solutions**:
1. **Redis SET not working**:
```bash
# Test Redis SET operations
redis-cli -n 1 SADD test_set "value1"
redis-cli -n 1 SMEMBERS test_set
redis-cli -n 1 DEL test_set
```
2. **Clear dedup sets**:
```bash
# If corrupted
redis-cli -n 1 DEL vmh:beteiligte:create_pending
redis-cli -n 1 DEL vmh:beteiligte:update_pending
redis-cli -n 1 DEL vmh:beteiligte:delete_pending
```
## Performance Issues
### High CPU Usage
**Diagnose**:
```bash
# Check CPU usage
top -p $(pgrep -f "motia start")
# Profile with Node.js
# Already enabled with --inspect flag
# Connect to chrome://inspect
```
**Solutions**:
1. **Too many parallel syncs**:
```bash
# Reduce concurrent syncs
# Adjust DEBUG_KUERZEL to process fewer employees
```
2. **Infinite loop**:
```bash
# Check logs for repeated patterns
sudo journalctl -u motia.service | tail -n 1000 | sort | uniq -c | sort -rn
```
### High Memory Usage
**Diagnose**:
```bash
# Check memory
ps aux | grep motia | awk '{print $6}'
# Heap snapshot (if enabled)
kill -SIGUSR2 $(pgrep -f "motia start")
# Snapshot saved to current directory
```
**Solutions**:
1. **Increase memory limit**:
```ini
# In systemd service
Environment=NODE_OPTIONS=--max-old-space-size=16384
```
2. **Memory leak**:
```bash
# Restart service periodically
# Add to crontab:
0 3 * * * systemctl restart motia.service
```
### Slow API Responses
**Diagnose**:
```bash
# Measure response time
time curl "http://localhost:3000/advoware/proxy?endpoint=employees"
# Check for database/Redis latency
redis-cli --latency
```
**Solutions**:
1. **Redis slow**:
```bash
# Check slow log
redis-cli SLOWLOG GET 10
# Optimize Redis
redis-cli CONFIG SET tcp-backlog 511
```
2. **Advoware API slow**:
```bash
# Increase timeout
export ADVOWARE_API_TIMEOUT_SECONDS=60
# Add caching layer
```
## Debugging Tools
### Enable Debug Logging
```bash
# Set in systemd service
Environment=MOTIA_LOG_LEVEL=debug
sudo systemctl daemon-reload
sudo systemctl restart motia.service
```
### Redis Debugging
```bash
# Connect to Redis
redis-cli
# Monitor all commands
MONITOR
# Slow log
SLOWLOG GET 10
# Info
INFO all
```
### Python Debugging
```python
# Add to step code
import pdb; pdb.set_trace()
# Or use logging
context.logger.debug(f"Variable value: {variable}")
```
### Node.js Debugging
```bash
# Connect to inspector
# Chrome DevTools: chrome://inspect
# VSCode: Attach to Process
```
## Getting Help
### Check Logs First
```bash
# Last 100 lines
sudo journalctl -u motia.service -n 100
# Errors only
sudo journalctl -u motia.service -p err
# Specific time range
sudo journalctl -u motia.service --since "1 hour ago"
```
### Common Log Patterns
**Success**:
```
[INFO] Calendar sync completed for SB
[INFO] VMH Webhook received
```
**Warning**:
```
[WARNING] Rate limit approaching
[WARNING] Lock already exists for SB
```
**Error**:
```
[ERROR] Redis connection failed
[ERROR] API call failed: 401 Unauthorized
[ERROR] Unexpected error: ...
```
### Collect Debug Information
```bash
# System info
uname -a
node --version
python3 --version
# Service status
sudo systemctl status motia.service
# Recent logs
sudo journalctl -u motia.service -n 200 > motia-logs.txt
# Redis info
redis-cli INFO > redis-info.txt
# Configuration (redact secrets!)
sudo systemctl show motia.service -p Environment > env.txt
```
## Related Documentation
- [Architecture](ARCHITECTURE.md)
- [Configuration](CONFIGURATION.md)
- [Deployment](DEPLOYMENT.md)
- [Development Guide](DEVELOPMENT.md)