657 lines
14 KiB
Markdown
657 lines
14 KiB
Markdown
# Development Guide
|
|
|
|
## Setup
|
|
|
|
### Prerequisites
|
|
|
|
- **Node.js**: 18.x oder höher
|
|
- **Python**: 3.13 oder höher
|
|
- **Redis**: 6.x oder höher
|
|
- **Git**: Für Version Control
|
|
- **Motia CLI**: Wird automatisch via npm installiert
|
|
|
|
### Initial Setup
|
|
|
|
```bash
|
|
# 1. Repository navigieren
|
|
cd /opt/motia-app/bitbylaw
|
|
|
|
# 2. Node.js Dependencies installieren
|
|
npm install
|
|
|
|
# 3. Python Virtual Environment erstellen (falls nicht vorhanden)
|
|
python3.13 -m venv python_modules
|
|
|
|
# 4. Python Virtual Environment aktivieren
|
|
source python_modules/bin/activate
|
|
|
|
# 5. Python Dependencies installieren
|
|
pip install -r requirements.txt
|
|
|
|
# 6. Redis starten (falls nicht läuft)
|
|
sudo systemctl start redis-server
|
|
|
|
# 7. Environment Variables konfigurieren (siehe CONFIGURATION.md)
|
|
# Erstellen Sie eine .env Datei oder setzen Sie in systemd
|
|
|
|
# 8. Development Mode starten
|
|
npm run dev
|
|
```
|
|
|
|
### Entwicklungsumgebung
|
|
|
|
**Empfohlene IDE**: VS Code mit Extensions:
|
|
- Python (Microsoft)
|
|
- TypeScript (Built-in)
|
|
- ESLint
|
|
- Prettier
|
|
|
|
**VS Code Settings** (`.vscode/settings.json`):
|
|
```json
|
|
{
|
|
"python.defaultInterpreterPath": "${workspaceFolder}/python_modules/bin/python",
|
|
"python.linting.enabled": true,
|
|
"python.linting.pylintEnabled": false,
|
|
"python.linting.flake8Enabled": true,
|
|
"editor.formatOnSave": true,
|
|
"files.exclude": {
|
|
"**/__pycache__": true,
|
|
"**/node_modules": true
|
|
}
|
|
}
|
|
```
|
|
|
|
## Projektstruktur
|
|
|
|
```
|
|
bitbylaw/
|
|
├── docs/ # Dokumentation
|
|
│ ├── ARCHITECTURE.md # System-Architektur
|
|
│ ├── DEVELOPMENT.md # Dieser Guide
|
|
│ ├── API.md # API-Referenz
|
|
│ ├── CONFIGURATION.md # Environment & Config
|
|
│ ├── DEPLOYMENT.md # Deployment-Guide
|
|
│ └── TROUBLESHOOTING.md # Fehlerbehebung
|
|
├── steps/ # Motia Steps (Business Logic)
|
|
│ ├── advoware_proxy/ # API Proxy Steps
|
|
│ │ ├── README.md # Modul-Dokumentation
|
|
│ │ ├── *.py # Step-Implementierungen
|
|
│ │ └── *.md # Step-Detail-Doku
|
|
│ ├── advoware_cal_sync/ # Calendar Sync Steps
|
|
│ │ ├── README.md
|
|
│ │ ├── *.py
|
|
│ │ └── *.md
|
|
│ └── vmh/ # VMH Webhook Steps
|
|
│ ├── README.md
|
|
│ ├── webhook/ # Webhook Receiver
|
|
│ └── *.py
|
|
├── services/ # Shared Services
|
|
│ └── advoware.py # Advoware API Client
|
|
├── config.py # Configuration Loader
|
|
├── package.json # Node.js Dependencies
|
|
├── requirements.txt # Python Dependencies
|
|
├── tsconfig.json # TypeScript Config
|
|
├── motia-workbench.json # Motia Flow Definitions
|
|
└── README.md # Projekt-Übersicht
|
|
```
|
|
|
|
### Konventionen
|
|
|
|
**Verzeichnisse**:
|
|
- `steps/` - Motia Steps (Handler-Funktionen)
|
|
- `services/` - Wiederverwendbare Service-Layer
|
|
- `docs/` - Dokumentation
|
|
- `python_modules/` - Python Virtual Environment (nicht committen)
|
|
- `node_modules/` - Node.js Dependencies (nicht committen)
|
|
|
|
**Dateinamen**:
|
|
- Steps: `{module}_{action}_step.py` (z.B. `calendar_sync_cron_step.py`)
|
|
- Services: `{service_name}.py` (z.B. `advoware.py`)
|
|
- Dokumentation: `{STEP_NAME}.md` oder `{TOPIC}.md`
|
|
|
|
## Coding Standards
|
|
|
|
### Python
|
|
|
|
**Style Guide**: PEP 8 mit folgenden Anpassungen:
|
|
- Line length: 120 Zeichen (statt 79)
|
|
- String quotes: Single quotes bevorzugt
|
|
|
|
**Linting**:
|
|
```bash
|
|
# Flake8 check
|
|
flake8 steps/ services/
|
|
|
|
# Autopep8 formatting
|
|
autopep8 --in-place --aggressive --aggressive steps/**/*.py
|
|
```
|
|
|
|
**Type Hints**:
|
|
```python
|
|
from typing import Dict, List, Optional, Any
|
|
|
|
async def handler(req: Dict[str, Any], context: Any) -> Dict[str, Any]:
|
|
pass
|
|
```
|
|
|
|
**Docstrings**:
|
|
```python
|
|
def function_name(param1: str, param2: int) -> bool:
|
|
"""
|
|
Brief description of function.
|
|
|
|
Args:
|
|
param1: Description of param1
|
|
param2: Description of param2
|
|
|
|
Returns:
|
|
Description of return value
|
|
|
|
Raises:
|
|
ValueError: When something goes wrong
|
|
"""
|
|
pass
|
|
```
|
|
|
|
### TypeScript/JavaScript
|
|
|
|
**Style Guide**: Standard mit Motia-Konventionen
|
|
|
|
**Formatting**: Prettier (automatisch via Motia)
|
|
|
|
### Naming Conventions
|
|
|
|
**Variables**: `snake_case` (Python), `camelCase` (TypeScript)
|
|
**Constants**: `UPPER_CASE`
|
|
**Classes**: `PascalCase`
|
|
**Functions**: `snake_case` (Python), `camelCase` (TypeScript)
|
|
**Files**: `snake_case.py`, `kebab-case.ts`
|
|
|
|
### Error Handling
|
|
|
|
**Pattern**:
|
|
```python
|
|
async def handler(req, context):
|
|
try:
|
|
# Main logic
|
|
result = await some_operation()
|
|
return {'status': 200, 'body': {'result': result}}
|
|
|
|
except SpecificError as e:
|
|
# Handle known errors
|
|
context.logger.error(f"Specific error: {e}")
|
|
return {'status': 400, 'body': {'error': 'Bad request'}}
|
|
|
|
except Exception as e:
|
|
# Catch-all
|
|
context.logger.error(f"Unexpected error: {e}", exc_info=True)
|
|
return {'status': 500, 'body': {'error': 'Internal error'}}
|
|
```
|
|
|
|
**Logging**:
|
|
```python
|
|
# Use context.logger for Motia Workbench integration
|
|
context.logger.debug("Detailed information")
|
|
context.logger.info("Normal operation")
|
|
context.logger.warning("Warning message")
|
|
context.logger.error("Error message", exc_info=True) # Include stack trace
|
|
```
|
|
|
|
## Motia Step Development
|
|
|
|
### Step Structure
|
|
|
|
Every Step must have:
|
|
1. **Config Dictionary**: Defines step metadata
|
|
2. **Handler Function**: Implements business logic
|
|
|
|
**Minimal Example**:
|
|
```python
|
|
config = {
|
|
'type': 'api', # api|event|cron
|
|
'name': 'My API Step',
|
|
'description': 'Brief description',
|
|
'path': '/api/my-endpoint', # For API steps
|
|
'method': 'GET', # For API steps
|
|
'schedule': '0 2 * * *', # For cron steps
|
|
'emits': ['topic.name'], # Events this step emits
|
|
'subscribes': ['other.topic'], # Events this step subscribes to (event steps)
|
|
'flows': ['my-flow'] # Flow membership
|
|
}
|
|
|
|
async def handler(req, context):
|
|
"""Handler function - must be async."""
|
|
# req: Request object (API) or Event data (event step)
|
|
# context: Motia context (logger, emit, etc.)
|
|
|
|
# Business logic here
|
|
|
|
# For API steps: return HTTP response
|
|
return {'status': 200, 'body': {'result': 'success'}}
|
|
|
|
# For event steps: no return value (or None)
|
|
```
|
|
|
|
### Step Types
|
|
|
|
**1. API Steps** (`type: 'api'`):
|
|
```python
|
|
config = {
|
|
'type': 'api',
|
|
'name': 'My Endpoint',
|
|
'path': '/api/resource',
|
|
'method': 'POST',
|
|
'emits': [],
|
|
'flows': ['main']
|
|
}
|
|
|
|
async def handler(req, context):
|
|
# Access request data
|
|
body = req.get('body')
|
|
query_params = req.get('queryParams')
|
|
headers = req.get('headers')
|
|
|
|
# Return HTTP response
|
|
return {
|
|
'status': 200,
|
|
'body': {'data': 'response'},
|
|
'headers': {'X-Custom': 'value'}
|
|
}
|
|
```
|
|
|
|
**2. Event Steps** (`type: 'event'`):
|
|
```python
|
|
config = {
|
|
'type': 'event',
|
|
'name': 'Process Event',
|
|
'subscribes': ['my.topic'],
|
|
'emits': ['other.topic'],
|
|
'flows': ['main']
|
|
}
|
|
|
|
async def handler(event_data, context):
|
|
# Process event
|
|
entity_id = event_data.get('entity_id')
|
|
|
|
# Emit new event
|
|
await context.emit({
|
|
'topic': 'other.topic',
|
|
'data': {'processed': True}
|
|
})
|
|
|
|
# No return value needed
|
|
```
|
|
|
|
**3. Cron Steps** (`type: 'cron'`):
|
|
```python
|
|
config = {
|
|
'type': 'cron',
|
|
'name': 'Daily Job',
|
|
'schedule': '0 2 * * *', # Cron expression
|
|
'emits': ['job.complete'],
|
|
'flows': ['main']
|
|
}
|
|
|
|
async def handler(req, context):
|
|
# Scheduled logic
|
|
context.logger.info("Cron job triggered")
|
|
|
|
# Emit event to start pipeline
|
|
await context.emit({
|
|
'topic': 'job.complete',
|
|
'data': {}
|
|
})
|
|
```
|
|
|
|
### Context API
|
|
|
|
**Available Methods**:
|
|
```python
|
|
# Logging
|
|
context.logger.debug(msg)
|
|
context.logger.info(msg)
|
|
context.logger.warning(msg)
|
|
context.logger.error(msg, exc_info=True)
|
|
|
|
# Event Emission
|
|
await context.emit({
|
|
'topic': 'my.topic',
|
|
'data': {'key': 'value'}
|
|
})
|
|
|
|
# Flow information
|
|
context.flow_id # Current flow ID
|
|
context.step_name # Current step name
|
|
```
|
|
|
|
## Testing
|
|
|
|
### Unit Tests
|
|
|
|
**Location**: Tests neben dem Code (z.B. `*_test.py`)
|
|
|
|
**Framework**: pytest
|
|
|
|
```python
|
|
# test_my_step.py
|
|
import pytest
|
|
from unittest.mock import AsyncMock, MagicMock
|
|
from my_step import handler, config
|
|
|
|
@pytest.mark.asyncio
|
|
async def test_handler_success():
|
|
# Arrange
|
|
req = {'body': {'key': 'value'}}
|
|
context = MagicMock()
|
|
context.logger = MagicMock()
|
|
|
|
# Act
|
|
result = await handler(req, context)
|
|
|
|
# Assert
|
|
assert result['status'] == 200
|
|
assert 'result' in result['body']
|
|
```
|
|
|
|
**Run Tests**:
|
|
```bash
|
|
pytest steps/
|
|
```
|
|
|
|
### Integration Tests
|
|
|
|
**Manual Testing mit curl**:
|
|
|
|
```bash
|
|
# API Step testen
|
|
curl -X POST "http://localhost:3000/api/my-endpoint" \
|
|
-H "Content-Type: application/json" \
|
|
-d '{"key": "value"}'
|
|
|
|
# Mit Query Parameters
|
|
curl -X GET "http://localhost:3000/advoware/proxy?endpoint=employees"
|
|
```
|
|
|
|
**Motia Workbench**: Nutzen Sie die Workbench UI zum Testen und Debugging
|
|
|
|
### Test-Daten
|
|
|
|
**Redis Mock Data**:
|
|
```bash
|
|
# Set test token
|
|
redis-cli -n 1 SET advoware_access_token "test_token" EX 3600
|
|
|
|
# Set test lock
|
|
redis-cli -n 1 SET "calendar_sync:lock:TEST" "1" EX 300
|
|
|
|
# Check dedup set
|
|
redis-cli -n 1 SMEMBERS "vmh:beteiligte:create_pending"
|
|
```
|
|
|
|
## Debugging
|
|
|
|
### Local Development
|
|
|
|
**Start in Dev Mode**:
|
|
```bash
|
|
npm run dev
|
|
```
|
|
|
|
**Enable Debug Logging**:
|
|
```bash
|
|
export MOTIA_LOG_LEVEL=debug
|
|
npm start
|
|
```
|
|
|
|
**Node.js Inspector**:
|
|
```bash
|
|
# Already enabled in systemd (--inspect)
|
|
# Connect with Chrome DevTools: chrome://inspect
|
|
```
|
|
|
|
### Motia Workbench
|
|
|
|
**Access**: `http://localhost:3000/workbench` (wenn verfügbar)
|
|
|
|
**Features**:
|
|
- Live logs
|
|
- Flow visualization
|
|
- Event traces
|
|
- Step execution history
|
|
|
|
### Redis Debugging
|
|
|
|
```bash
|
|
# Connect to Redis
|
|
redis-cli
|
|
|
|
# Switch database
|
|
SELECT 1
|
|
|
|
# List all keys
|
|
KEYS *
|
|
|
|
# Get value
|
|
GET advoware_access_token
|
|
|
|
# Check SET members
|
|
SMEMBERS vmh:beteiligte:create_pending
|
|
|
|
# Monitor live commands
|
|
MONITOR
|
|
```
|
|
|
|
---
|
|
|
|
## Utility Scripts
|
|
|
|
### Calendar Sync Utilities
|
|
|
|
Helper-Scripts für Wartung und Debugging der Calendar-Sync-Funktionalität.
|
|
|
|
**Standort**: `scripts/calendar_sync/`
|
|
|
|
**Verfügbare Scripts**:
|
|
|
|
```bash
|
|
# Alle Employee-Locks in Redis löschen (bei hängenden Syncs)
|
|
python3 scripts/calendar_sync/delete_employee_locks.py
|
|
|
|
# Alle Google Kalender löschen (außer Primary) - VORSICHT!
|
|
python3 scripts/calendar_sync/delete_all_calendars.py
|
|
```
|
|
|
|
**Use Cases**:
|
|
- **Lock Cleanup**: Wenn ein Sync-Prozess abgestürzt ist und Locks nicht aufgeräumt wurden
|
|
- **Calendar Reset**: Bei fehlerhafter Synchronisation oder Tests
|
|
- **Debugging**: Untersuchung von Sync-Problemen
|
|
|
|
**Dokumentation**: [scripts/calendar_sync/README.md](../scripts/calendar_sync/README.md)
|
|
|
|
**⚠️ Wichtig**:
|
|
- Immer Motia Service stoppen vor Cleanup: `sudo systemctl stop motia`
|
|
- Nach Cleanup Service neu starten: `sudo systemctl start motia`
|
|
- `delete_all_calendars.py` löscht unwiderruflich alle Kalender!
|
|
|
|
---
|
|
|
|
### Common Issues
|
|
|
|
**1. Import Errors**:
|
|
```bash
|
|
# Ensure PYTHONPATH is set
|
|
export PYTHONPATH=/opt/motia-app/bitbylaw
|
|
source python_modules/bin/activate
|
|
```
|
|
|
|
**2. Redis Connection Errors**:
|
|
```bash
|
|
# Check Redis is running
|
|
sudo systemctl status redis-server
|
|
|
|
# Test connection
|
|
redis-cli ping
|
|
```
|
|
|
|
**3. Token Errors**:
|
|
```bash
|
|
# Clear cached token
|
|
redis-cli -n 1 DEL advoware_access_token advoware_token_timestamp
|
|
```
|
|
|
|
## Git Workflow
|
|
|
|
### Branch Strategy
|
|
|
|
- `main` - Production code
|
|
- `develop` - Integration branch
|
|
- `feature/*` - Feature branches
|
|
- `fix/*` - Bugfix branches
|
|
|
|
### Commit Messages
|
|
|
|
**Format**: `<type>: <subject>`
|
|
|
|
**Types**:
|
|
- `feat`: New feature
|
|
- `fix`: Bug fix
|
|
- `docs`: Documentation only
|
|
- `refactor`: Code refactoring
|
|
- `test`: Adding tests
|
|
- `chore`: Maintenance tasks
|
|
|
|
**Examples**:
|
|
```
|
|
feat: add calendar sync retry logic
|
|
fix: prevent duplicate webhook processing
|
|
docs: update API documentation
|
|
refactor: extract common validation logic
|
|
```
|
|
|
|
### Pull Request Process
|
|
|
|
1. Create feature branch from `develop`
|
|
2. Implement changes
|
|
3. Write/update tests
|
|
4. Update documentation
|
|
5. Create PR with description
|
|
6. Code review
|
|
7. Merge to `develop`
|
|
8. Deploy to staging
|
|
9. Merge to `main` (production)
|
|
|
|
## Performance Optimization
|
|
|
|
### Profiling
|
|
|
|
**Python Memory Profiling**:
|
|
```bash
|
|
# Install memory_profiler
|
|
pip install memory_profiler
|
|
|
|
# Profile a function
|
|
python -m memory_profiler steps/my_step.py
|
|
```
|
|
|
|
**Node.js Profiling**:
|
|
```bash
|
|
# Already enabled with --inspect flag
|
|
# Use Chrome DevTools Performance tab
|
|
```
|
|
|
|
### Best Practices
|
|
|
|
**Async/Await**:
|
|
```python
|
|
# Good: Concurrent requests
|
|
results = await asyncio.gather(
|
|
fetch_data_1(),
|
|
fetch_data_2()
|
|
)
|
|
|
|
# Bad: Sequential (slow)
|
|
result1 = await fetch_data_1()
|
|
result2 = await fetch_data_2()
|
|
```
|
|
|
|
**Redis Pipelining**:
|
|
```python
|
|
# Good: Batch operations
|
|
pipe = redis.pipeline()
|
|
pipe.get('key1')
|
|
pipe.get('key2')
|
|
results = pipe.execute()
|
|
|
|
# Bad: Multiple round-trips
|
|
val1 = redis.get('key1')
|
|
val2 = redis.get('key2')
|
|
```
|
|
|
|
**Avoid N+1 Queries**:
|
|
```python
|
|
# Good: Batch fetch
|
|
employee_ids = [1, 2, 3]
|
|
employees = await advoware.api_call(
|
|
'/employees',
|
|
params={'ids': ','.join(map(str, employee_ids))}
|
|
)
|
|
|
|
# Bad: Loop with API calls
|
|
employees = []
|
|
for emp_id in employee_ids:
|
|
emp = await advoware.api_call(f'/employees/{emp_id}')
|
|
employees.append(emp)
|
|
```
|
|
|
|
## Code Review Checklist
|
|
|
|
- [ ] Code follows style guide
|
|
- [ ] Type hints present (Python)
|
|
- [ ] Error handling implemented
|
|
- [ ] Logging added at key points
|
|
- [ ] Tests written/updated
|
|
- [ ] Documentation updated
|
|
- [ ] No secrets in code
|
|
- [ ] Performance considered
|
|
- [ ] Redis keys documented
|
|
- [ ] Events documented
|
|
|
|
## Deployment
|
|
|
|
See [DEPLOYMENT.md](DEPLOYMENT.md) for detailed deployment instructions.
|
|
|
|
**Quick Deploy to Production**:
|
|
```bash
|
|
# 1. Pull latest code
|
|
git pull origin main
|
|
|
|
# 2. Install dependencies
|
|
npm install
|
|
pip install -r requirements.txt
|
|
|
|
# 3. Restart service
|
|
sudo systemctl restart motia.service
|
|
|
|
# 4. Check status
|
|
sudo systemctl status motia.service
|
|
|
|
# 5. Monitor logs
|
|
sudo journalctl -u motia.service -f
|
|
```
|
|
|
|
## Resources
|
|
|
|
### Documentation
|
|
- [Motia Framework](https://motia.dev)
|
|
- [Advoware API](docs/advoware/) (internal)
|
|
- [Google Calendar API](https://developers.google.com/calendar)
|
|
|
|
### Tools
|
|
- [Redis Commander](http://localhost:8081) (if installed)
|
|
- [Motia Workbench](http://localhost:3000/workbench)
|
|
|
|
### Team Contacts
|
|
- Architecture Questions: [Lead Developer]
|
|
- Deployment Issues: [DevOps Team]
|
|
- API Access: [Integration Team]
|