cleanup
This commit is contained in:
656
bitbylaw/docs/DEVELOPMENT.md
Normal file
656
bitbylaw/docs/DEVELOPMENT.md
Normal file
@@ -0,0 +1,656 @@
|
||||
# Development Guide
|
||||
|
||||
## Setup
|
||||
|
||||
### Prerequisites
|
||||
|
||||
- **Node.js**: 18.x oder höher
|
||||
- **Python**: 3.13 oder höher
|
||||
- **Redis**: 6.x oder höher
|
||||
- **Git**: Für Version Control
|
||||
- **Motia CLI**: Wird automatisch via npm installiert
|
||||
|
||||
### Initial Setup
|
||||
|
||||
```bash
|
||||
# 1. Repository navigieren
|
||||
cd /opt/motia-app/bitbylaw
|
||||
|
||||
# 2. Node.js Dependencies installieren
|
||||
npm install
|
||||
|
||||
# 3. Python Virtual Environment erstellen (falls nicht vorhanden)
|
||||
python3.13 -m venv python_modules
|
||||
|
||||
# 4. Python Virtual Environment aktivieren
|
||||
source python_modules/bin/activate
|
||||
|
||||
# 5. Python Dependencies installieren
|
||||
pip install -r requirements.txt
|
||||
|
||||
# 6. Redis starten (falls nicht läuft)
|
||||
sudo systemctl start redis-server
|
||||
|
||||
# 7. Environment Variables konfigurieren (siehe CONFIGURATION.md)
|
||||
# Erstellen Sie eine .env Datei oder setzen Sie in systemd
|
||||
|
||||
# 8. Development Mode starten
|
||||
npm run dev
|
||||
```
|
||||
|
||||
### Entwicklungsumgebung
|
||||
|
||||
**Empfohlene IDE**: VS Code mit Extensions:
|
||||
- Python (Microsoft)
|
||||
- TypeScript (Built-in)
|
||||
- ESLint
|
||||
- Prettier
|
||||
|
||||
**VS Code Settings** (`.vscode/settings.json`):
|
||||
```json
|
||||
{
|
||||
"python.defaultInterpreterPath": "${workspaceFolder}/python_modules/bin/python",
|
||||
"python.linting.enabled": true,
|
||||
"python.linting.pylintEnabled": false,
|
||||
"python.linting.flake8Enabled": true,
|
||||
"editor.formatOnSave": true,
|
||||
"files.exclude": {
|
||||
"**/__pycache__": true,
|
||||
"**/node_modules": true
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
## Projektstruktur
|
||||
|
||||
```
|
||||
bitbylaw/
|
||||
├── docs/ # Dokumentation
|
||||
│ ├── ARCHITECTURE.md # System-Architektur
|
||||
│ ├── DEVELOPMENT.md # Dieser Guide
|
||||
│ ├── API.md # API-Referenz
|
||||
│ ├── CONFIGURATION.md # Environment & Config
|
||||
│ ├── DEPLOYMENT.md # Deployment-Guide
|
||||
│ └── TROUBLESHOOTING.md # Fehlerbehebung
|
||||
├── steps/ # Motia Steps (Business Logic)
|
||||
│ ├── advoware_proxy/ # API Proxy Steps
|
||||
│ │ ├── README.md # Modul-Dokumentation
|
||||
│ │ ├── *.py # Step-Implementierungen
|
||||
│ │ └── *.md # Step-Detail-Doku
|
||||
│ ├── advoware_cal_sync/ # Calendar Sync Steps
|
||||
│ │ ├── README.md
|
||||
│ │ ├── *.py
|
||||
│ │ └── *.md
|
||||
│ └── vmh/ # VMH Webhook Steps
|
||||
│ ├── README.md
|
||||
│ ├── webhook/ # Webhook Receiver
|
||||
│ └── *.py
|
||||
├── services/ # Shared Services
|
||||
│ └── advoware.py # Advoware API Client
|
||||
├── config.py # Configuration Loader
|
||||
├── package.json # Node.js Dependencies
|
||||
├── requirements.txt # Python Dependencies
|
||||
├── tsconfig.json # TypeScript Config
|
||||
├── motia-workbench.json # Motia Flow Definitions
|
||||
└── README.md # Projekt-Übersicht
|
||||
```
|
||||
|
||||
### Konventionen
|
||||
|
||||
**Verzeichnisse**:
|
||||
- `steps/` - Motia Steps (Handler-Funktionen)
|
||||
- `services/` - Wiederverwendbare Service-Layer
|
||||
- `docs/` - Dokumentation
|
||||
- `python_modules/` - Python Virtual Environment (nicht committen)
|
||||
- `node_modules/` - Node.js Dependencies (nicht committen)
|
||||
|
||||
**Dateinamen**:
|
||||
- Steps: `{module}_{action}_step.py` (z.B. `calendar_sync_cron_step.py`)
|
||||
- Services: `{service_name}.py` (z.B. `advoware.py`)
|
||||
- Dokumentation: `{STEP_NAME}.md` oder `{TOPIC}.md`
|
||||
|
||||
## Coding Standards
|
||||
|
||||
### Python
|
||||
|
||||
**Style Guide**: PEP 8 mit folgenden Anpassungen:
|
||||
- Line length: 120 Zeichen (statt 79)
|
||||
- String quotes: Single quotes bevorzugt
|
||||
|
||||
**Linting**:
|
||||
```bash
|
||||
# Flake8 check
|
||||
flake8 steps/ services/
|
||||
|
||||
# Autopep8 formatting
|
||||
autopep8 --in-place --aggressive --aggressive steps/**/*.py
|
||||
```
|
||||
|
||||
**Type Hints**:
|
||||
```python
|
||||
from typing import Dict, List, Optional, Any
|
||||
|
||||
async def handler(req: Dict[str, Any], context: Any) -> Dict[str, Any]:
|
||||
pass
|
||||
```
|
||||
|
||||
**Docstrings**:
|
||||
```python
|
||||
def function_name(param1: str, param2: int) -> bool:
|
||||
"""
|
||||
Brief description of function.
|
||||
|
||||
Args:
|
||||
param1: Description of param1
|
||||
param2: Description of param2
|
||||
|
||||
Returns:
|
||||
Description of return value
|
||||
|
||||
Raises:
|
||||
ValueError: When something goes wrong
|
||||
"""
|
||||
pass
|
||||
```
|
||||
|
||||
### TypeScript/JavaScript
|
||||
|
||||
**Style Guide**: Standard mit Motia-Konventionen
|
||||
|
||||
**Formatting**: Prettier (automatisch via Motia)
|
||||
|
||||
### Naming Conventions
|
||||
|
||||
**Variables**: `snake_case` (Python), `camelCase` (TypeScript)
|
||||
**Constants**: `UPPER_CASE`
|
||||
**Classes**: `PascalCase`
|
||||
**Functions**: `snake_case` (Python), `camelCase` (TypeScript)
|
||||
**Files**: `snake_case.py`, `kebab-case.ts`
|
||||
|
||||
### Error Handling
|
||||
|
||||
**Pattern**:
|
||||
```python
|
||||
async def handler(req, context):
|
||||
try:
|
||||
# Main logic
|
||||
result = await some_operation()
|
||||
return {'status': 200, 'body': {'result': result}}
|
||||
|
||||
except SpecificError as e:
|
||||
# Handle known errors
|
||||
context.logger.error(f"Specific error: {e}")
|
||||
return {'status': 400, 'body': {'error': 'Bad request'}}
|
||||
|
||||
except Exception as e:
|
||||
# Catch-all
|
||||
context.logger.error(f"Unexpected error: {e}", exc_info=True)
|
||||
return {'status': 500, 'body': {'error': 'Internal error'}}
|
||||
```
|
||||
|
||||
**Logging**:
|
||||
```python
|
||||
# Use context.logger for Motia Workbench integration
|
||||
context.logger.debug("Detailed information")
|
||||
context.logger.info("Normal operation")
|
||||
context.logger.warning("Warning message")
|
||||
context.logger.error("Error message", exc_info=True) # Include stack trace
|
||||
```
|
||||
|
||||
## Motia Step Development
|
||||
|
||||
### Step Structure
|
||||
|
||||
Every Step must have:
|
||||
1. **Config Dictionary**: Defines step metadata
|
||||
2. **Handler Function**: Implements business logic
|
||||
|
||||
**Minimal Example**:
|
||||
```python
|
||||
config = {
|
||||
'type': 'api', # api|event|cron
|
||||
'name': 'My API Step',
|
||||
'description': 'Brief description',
|
||||
'path': '/api/my-endpoint', # For API steps
|
||||
'method': 'GET', # For API steps
|
||||
'schedule': '0 2 * * *', # For cron steps
|
||||
'emits': ['topic.name'], # Events this step emits
|
||||
'subscribes': ['other.topic'], # Events this step subscribes to (event steps)
|
||||
'flows': ['my-flow'] # Flow membership
|
||||
}
|
||||
|
||||
async def handler(req, context):
|
||||
"""Handler function - must be async."""
|
||||
# req: Request object (API) or Event data (event step)
|
||||
# context: Motia context (logger, emit, etc.)
|
||||
|
||||
# Business logic here
|
||||
|
||||
# For API steps: return HTTP response
|
||||
return {'status': 200, 'body': {'result': 'success'}}
|
||||
|
||||
# For event steps: no return value (or None)
|
||||
```
|
||||
|
||||
### Step Types
|
||||
|
||||
**1. API Steps** (`type: 'api'`):
|
||||
```python
|
||||
config = {
|
||||
'type': 'api',
|
||||
'name': 'My Endpoint',
|
||||
'path': '/api/resource',
|
||||
'method': 'POST',
|
||||
'emits': [],
|
||||
'flows': ['main']
|
||||
}
|
||||
|
||||
async def handler(req, context):
|
||||
# Access request data
|
||||
body = req.get('body')
|
||||
query_params = req.get('queryParams')
|
||||
headers = req.get('headers')
|
||||
|
||||
# Return HTTP response
|
||||
return {
|
||||
'status': 200,
|
||||
'body': {'data': 'response'},
|
||||
'headers': {'X-Custom': 'value'}
|
||||
}
|
||||
```
|
||||
|
||||
**2. Event Steps** (`type: 'event'`):
|
||||
```python
|
||||
config = {
|
||||
'type': 'event',
|
||||
'name': 'Process Event',
|
||||
'subscribes': ['my.topic'],
|
||||
'emits': ['other.topic'],
|
||||
'flows': ['main']
|
||||
}
|
||||
|
||||
async def handler(event_data, context):
|
||||
# Process event
|
||||
entity_id = event_data.get('entity_id')
|
||||
|
||||
# Emit new event
|
||||
await context.emit({
|
||||
'topic': 'other.topic',
|
||||
'data': {'processed': True}
|
||||
})
|
||||
|
||||
# No return value needed
|
||||
```
|
||||
|
||||
**3. Cron Steps** (`type: 'cron'`):
|
||||
```python
|
||||
config = {
|
||||
'type': 'cron',
|
||||
'name': 'Daily Job',
|
||||
'schedule': '0 2 * * *', # Cron expression
|
||||
'emits': ['job.complete'],
|
||||
'flows': ['main']
|
||||
}
|
||||
|
||||
async def handler(req, context):
|
||||
# Scheduled logic
|
||||
context.logger.info("Cron job triggered")
|
||||
|
||||
# Emit event to start pipeline
|
||||
await context.emit({
|
||||
'topic': 'job.complete',
|
||||
'data': {}
|
||||
})
|
||||
```
|
||||
|
||||
### Context API
|
||||
|
||||
**Available Methods**:
|
||||
```python
|
||||
# Logging
|
||||
context.logger.debug(msg)
|
||||
context.logger.info(msg)
|
||||
context.logger.warning(msg)
|
||||
context.logger.error(msg, exc_info=True)
|
||||
|
||||
# Event Emission
|
||||
await context.emit({
|
||||
'topic': 'my.topic',
|
||||
'data': {'key': 'value'}
|
||||
})
|
||||
|
||||
# Flow information
|
||||
context.flow_id # Current flow ID
|
||||
context.step_name # Current step name
|
||||
```
|
||||
|
||||
## Testing
|
||||
|
||||
### Unit Tests
|
||||
|
||||
**Location**: Tests neben dem Code (z.B. `*_test.py`)
|
||||
|
||||
**Framework**: pytest
|
||||
|
||||
```python
|
||||
# test_my_step.py
|
||||
import pytest
|
||||
from unittest.mock import AsyncMock, MagicMock
|
||||
from my_step import handler, config
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_handler_success():
|
||||
# Arrange
|
||||
req = {'body': {'key': 'value'}}
|
||||
context = MagicMock()
|
||||
context.logger = MagicMock()
|
||||
|
||||
# Act
|
||||
result = await handler(req, context)
|
||||
|
||||
# Assert
|
||||
assert result['status'] == 200
|
||||
assert 'result' in result['body']
|
||||
```
|
||||
|
||||
**Run Tests**:
|
||||
```bash
|
||||
pytest steps/
|
||||
```
|
||||
|
||||
### Integration Tests
|
||||
|
||||
**Manual Testing mit curl**:
|
||||
|
||||
```bash
|
||||
# API Step testen
|
||||
curl -X POST "http://localhost:3000/api/my-endpoint" \
|
||||
-H "Content-Type: application/json" \
|
||||
-d '{"key": "value"}'
|
||||
|
||||
# Mit Query Parameters
|
||||
curl -X GET "http://localhost:3000/advoware/proxy?endpoint=employees"
|
||||
```
|
||||
|
||||
**Motia Workbench**: Nutzen Sie die Workbench UI zum Testen und Debugging
|
||||
|
||||
### Test-Daten
|
||||
|
||||
**Redis Mock Data**:
|
||||
```bash
|
||||
# Set test token
|
||||
redis-cli -n 1 SET advoware_access_token "test_token" EX 3600
|
||||
|
||||
# Set test lock
|
||||
redis-cli -n 1 SET "calendar_sync:lock:TEST" "1" EX 300
|
||||
|
||||
# Check dedup set
|
||||
redis-cli -n 1 SMEMBERS "vmh:beteiligte:create_pending"
|
||||
```
|
||||
|
||||
## Debugging
|
||||
|
||||
### Local Development
|
||||
|
||||
**Start in Dev Mode**:
|
||||
```bash
|
||||
npm run dev
|
||||
```
|
||||
|
||||
**Enable Debug Logging**:
|
||||
```bash
|
||||
export MOTIA_LOG_LEVEL=debug
|
||||
npm start
|
||||
```
|
||||
|
||||
**Node.js Inspector**:
|
||||
```bash
|
||||
# Already enabled in systemd (--inspect)
|
||||
# Connect with Chrome DevTools: chrome://inspect
|
||||
```
|
||||
|
||||
### Motia Workbench
|
||||
|
||||
**Access**: `http://localhost:3000/workbench` (wenn verfügbar)
|
||||
|
||||
**Features**:
|
||||
- Live logs
|
||||
- Flow visualization
|
||||
- Event traces
|
||||
- Step execution history
|
||||
|
||||
### Redis Debugging
|
||||
|
||||
```bash
|
||||
# Connect to Redis
|
||||
redis-cli
|
||||
|
||||
# Switch database
|
||||
SELECT 1
|
||||
|
||||
# List all keys
|
||||
KEYS *
|
||||
|
||||
# Get value
|
||||
GET advoware_access_token
|
||||
|
||||
# Check SET members
|
||||
SMEMBERS vmh:beteiligte:create_pending
|
||||
|
||||
# Monitor live commands
|
||||
MONITOR
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Utility Scripts
|
||||
|
||||
### Calendar Sync Utilities
|
||||
|
||||
Helper-Scripts für Wartung und Debugging der Calendar-Sync-Funktionalität.
|
||||
|
||||
**Standort**: `scripts/calendar_sync/`
|
||||
|
||||
**Verfügbare Scripts**:
|
||||
|
||||
```bash
|
||||
# Alle Employee-Locks in Redis löschen (bei hängenden Syncs)
|
||||
python3 scripts/calendar_sync/delete_employee_locks.py
|
||||
|
||||
# Alle Google Kalender löschen (außer Primary) - VORSICHT!
|
||||
python3 scripts/calendar_sync/delete_all_calendars.py
|
||||
```
|
||||
|
||||
**Use Cases**:
|
||||
- **Lock Cleanup**: Wenn ein Sync-Prozess abgestürzt ist und Locks nicht aufgeräumt wurden
|
||||
- **Calendar Reset**: Bei fehlerhafter Synchronisation oder Tests
|
||||
- **Debugging**: Untersuchung von Sync-Problemen
|
||||
|
||||
**Dokumentation**: [scripts/calendar_sync/README.md](../scripts/calendar_sync/README.md)
|
||||
|
||||
**⚠️ Wichtig**:
|
||||
- Immer Motia Service stoppen vor Cleanup: `sudo systemctl stop motia`
|
||||
- Nach Cleanup Service neu starten: `sudo systemctl start motia`
|
||||
- `delete_all_calendars.py` löscht unwiderruflich alle Kalender!
|
||||
|
||||
---
|
||||
|
||||
### Common Issues
|
||||
|
||||
**1. Import Errors**:
|
||||
```bash
|
||||
# Ensure PYTHONPATH is set
|
||||
export PYTHONPATH=/opt/motia-app/bitbylaw
|
||||
source python_modules/bin/activate
|
||||
```
|
||||
|
||||
**2. Redis Connection Errors**:
|
||||
```bash
|
||||
# Check Redis is running
|
||||
sudo systemctl status redis-server
|
||||
|
||||
# Test connection
|
||||
redis-cli ping
|
||||
```
|
||||
|
||||
**3. Token Errors**:
|
||||
```bash
|
||||
# Clear cached token
|
||||
redis-cli -n 1 DEL advoware_access_token advoware_token_timestamp
|
||||
```
|
||||
|
||||
## Git Workflow
|
||||
|
||||
### Branch Strategy
|
||||
|
||||
- `main` - Production code
|
||||
- `develop` - Integration branch
|
||||
- `feature/*` - Feature branches
|
||||
- `fix/*` - Bugfix branches
|
||||
|
||||
### Commit Messages
|
||||
|
||||
**Format**: `<type>: <subject>`
|
||||
|
||||
**Types**:
|
||||
- `feat`: New feature
|
||||
- `fix`: Bug fix
|
||||
- `docs`: Documentation only
|
||||
- `refactor`: Code refactoring
|
||||
- `test`: Adding tests
|
||||
- `chore`: Maintenance tasks
|
||||
|
||||
**Examples**:
|
||||
```
|
||||
feat: add calendar sync retry logic
|
||||
fix: prevent duplicate webhook processing
|
||||
docs: update API documentation
|
||||
refactor: extract common validation logic
|
||||
```
|
||||
|
||||
### Pull Request Process
|
||||
|
||||
1. Create feature branch from `develop`
|
||||
2. Implement changes
|
||||
3. Write/update tests
|
||||
4. Update documentation
|
||||
5. Create PR with description
|
||||
6. Code review
|
||||
7. Merge to `develop`
|
||||
8. Deploy to staging
|
||||
9. Merge to `main` (production)
|
||||
|
||||
## Performance Optimization
|
||||
|
||||
### Profiling
|
||||
|
||||
**Python Memory Profiling**:
|
||||
```bash
|
||||
# Install memory_profiler
|
||||
pip install memory_profiler
|
||||
|
||||
# Profile a function
|
||||
python -m memory_profiler steps/my_step.py
|
||||
```
|
||||
|
||||
**Node.js Profiling**:
|
||||
```bash
|
||||
# Already enabled with --inspect flag
|
||||
# Use Chrome DevTools Performance tab
|
||||
```
|
||||
|
||||
### Best Practices
|
||||
|
||||
**Async/Await**:
|
||||
```python
|
||||
# Good: Concurrent requests
|
||||
results = await asyncio.gather(
|
||||
fetch_data_1(),
|
||||
fetch_data_2()
|
||||
)
|
||||
|
||||
# Bad: Sequential (slow)
|
||||
result1 = await fetch_data_1()
|
||||
result2 = await fetch_data_2()
|
||||
```
|
||||
|
||||
**Redis Pipelining**:
|
||||
```python
|
||||
# Good: Batch operations
|
||||
pipe = redis.pipeline()
|
||||
pipe.get('key1')
|
||||
pipe.get('key2')
|
||||
results = pipe.execute()
|
||||
|
||||
# Bad: Multiple round-trips
|
||||
val1 = redis.get('key1')
|
||||
val2 = redis.get('key2')
|
||||
```
|
||||
|
||||
**Avoid N+1 Queries**:
|
||||
```python
|
||||
# Good: Batch fetch
|
||||
employee_ids = [1, 2, 3]
|
||||
employees = await advoware.api_call(
|
||||
'/employees',
|
||||
params={'ids': ','.join(map(str, employee_ids))}
|
||||
)
|
||||
|
||||
# Bad: Loop with API calls
|
||||
employees = []
|
||||
for emp_id in employee_ids:
|
||||
emp = await advoware.api_call(f'/employees/{emp_id}')
|
||||
employees.append(emp)
|
||||
```
|
||||
|
||||
## Code Review Checklist
|
||||
|
||||
- [ ] Code follows style guide
|
||||
- [ ] Type hints present (Python)
|
||||
- [ ] Error handling implemented
|
||||
- [ ] Logging added at key points
|
||||
- [ ] Tests written/updated
|
||||
- [ ] Documentation updated
|
||||
- [ ] No secrets in code
|
||||
- [ ] Performance considered
|
||||
- [ ] Redis keys documented
|
||||
- [ ] Events documented
|
||||
|
||||
## Deployment
|
||||
|
||||
See [DEPLOYMENT.md](DEPLOYMENT.md) for detailed deployment instructions.
|
||||
|
||||
**Quick Deploy to Production**:
|
||||
```bash
|
||||
# 1. Pull latest code
|
||||
git pull origin main
|
||||
|
||||
# 2. Install dependencies
|
||||
npm install
|
||||
pip install -r requirements.txt
|
||||
|
||||
# 3. Restart service
|
||||
sudo systemctl restart motia.service
|
||||
|
||||
# 4. Check status
|
||||
sudo systemctl status motia.service
|
||||
|
||||
# 5. Monitor logs
|
||||
sudo journalctl -u motia.service -f
|
||||
```
|
||||
|
||||
## Resources
|
||||
|
||||
### Documentation
|
||||
- [Motia Framework](https://motia.dev)
|
||||
- [Advoware API](docs/advoware/) (internal)
|
||||
- [Google Calendar API](https://developers.google.com/calendar)
|
||||
|
||||
### Tools
|
||||
- [Redis Commander](http://localhost:8081) (if installed)
|
||||
- [Motia Workbench](http://localhost:3000/workbench)
|
||||
|
||||
### Team Contacts
|
||||
- Architecture Questions: [Lead Developer]
|
||||
- Deployment Issues: [DevOps Team]
|
||||
- API Access: [Integration Team]
|
||||
Reference in New Issue
Block a user