171 lines
6.7 KiB
Markdown
171 lines
6.7 KiB
Markdown
# Motia Performance Degradation Bug Report
|
|
|
|
## Title
|
|
Exponential Performance Degradation in Event Processing Over Time
|
|
|
|
## Description
|
|
Motia exhibits exponential performance degradation when processing events continuously. Even in a clean, fresh installation with minimal custom code, the time required to emit and process batches of events grows exponentially, leading to severe slowdowns. This issue persists regardless of custom modifications, indicating a fundamental problem in the Motia framework's event handling or queue management.
|
|
|
|
## Steps to Reproduce
|
|
1. Create a completely new Motia project:
|
|
```bash
|
|
mkdir -p /opt/motia-app/motia-clean-test && cd /opt/motia-app/motia-clean-test
|
|
cd /opt/motia-app/motia-clean-test && npx motia@latest create
|
|
```
|
|
|
|
2. Add the following performance test steps (create these files in the `steps/` directory):
|
|
|
|
**steps/perf_cron_step.py**:
|
|
```python
|
|
import asyncio
|
|
from datetime import datetime, timezone
|
|
|
|
config = {
|
|
"type": "cron",
|
|
"cron": "*/1 * * * *", # Every minute
|
|
"name": "PerfTestCron",
|
|
"description": "Emits 100 perf-test events every minute",
|
|
"emits": ["perf-test-event"],
|
|
"flows": ["perf-test"],
|
|
}
|
|
|
|
async def handler(context):
|
|
start_time = datetime.now(timezone.utc)
|
|
context.logger.info(f"Starting perf test emission at {start_time}")
|
|
|
|
# Emit 100 events
|
|
for i in range(100):
|
|
await context.emit({
|
|
"topic": "perf-test-event",
|
|
"data": {
|
|
"event_id": i,
|
|
"batch_start": start_time.isoformat(),
|
|
"timestamp": datetime.now(timezone.utc).isoformat(),
|
|
},
|
|
})
|
|
|
|
end_time = datetime.now(timezone.utc)
|
|
duration = (end_time - start_time).total_seconds()
|
|
context.logger.info(f"Completed emitting 100 perf-test events in {duration:.3f}s")
|
|
```
|
|
|
|
**steps/perf_event_step.py**:
|
|
```python
|
|
import asyncio
|
|
from datetime import datetime, timezone
|
|
|
|
config = {
|
|
"type": "event",
|
|
"name": "PerfTestEventHandler",
|
|
"description": "Handles perf-test events with 1ms delay and logging",
|
|
"subscribes": ["perf-test-event"],
|
|
"emits": [],
|
|
"flows": ["perf-test"],
|
|
}
|
|
|
|
async def handler(event_data, context):
|
|
start_time = datetime.now(timezone.utc)
|
|
event_id = event_data.get("event_id")
|
|
batch_start = event_data.get("batch_start")
|
|
|
|
# Wait 1ms
|
|
await asyncio.sleep(0.001)
|
|
|
|
# Log completion with duration
|
|
end_time = datetime.now(timezone.utc)
|
|
duration = (end_time - start_time).total_seconds() * 1000 # in milliseconds
|
|
context.logger.info(f"Processed perf-test event {event_id} from batch {batch_start} in {duration:.2f}ms")
|
|
```
|
|
|
|
3. Start Motia and run for at least 15 minutes:
|
|
```bash
|
|
npm run dev
|
|
```
|
|
|
|
4. Monitor logs or use an analysis script to observe the degradation.
|
|
|
|
## Expected Behavior
|
|
- Each batch of 100 events should be emitted and processed consistently within ~1-5 seconds.
|
|
- Event processing times should remain stable at ~1-2ms per event.
|
|
- No exponential growth in processing times over time.
|
|
|
|
## Actual Behavior
|
|
- Emission times grow exponentially: starting at ~1s, reaching up to 193s by the 10th batch.
|
|
- Event processing remains constant (~1.7ms average), but the overall throughput degrades severely.
|
|
- The framework becomes unresponsive after several minutes of continuous operation.
|
|
|
|
## Environment
|
|
- **Motia Version**: Latest (0.17.11-beta.193) and 0.8.2-beta.139 and 0.7.2-beta.134 (all tested)
|
|
- **Node.js Version**: [Check with `node --version`]
|
|
- **Python Version**: 3.13
|
|
- **OS**: Linux
|
|
- **Dependencies**: Minimal (only Motia defaults)
|
|
- **Hardware**: [Standard server specs]
|
|
|
|
## Additional Context
|
|
### Version Comparison Results
|
|
The performance degradation issue exists in all tested versions of Motia, indicating this is a fundamental framework issue that has persisted across versions.
|
|
|
|
**Version 0.17.11-beta.193 (Latest):**
|
|
- Average emission time: 7.786s
|
|
- Min: 1.135s (Batch 1)
|
|
- Max: 14.074s (Batch 9)
|
|
|
|
**Version 0.8.2-beta.139:**
|
|
- Average emission time: 51.386s
|
|
- Min: 1.004s (Batch 1)
|
|
- Max: 193.355s (Batch 10)
|
|
|
|
**Version 0.7.2-beta.134:**
|
|
- Average emission time: 57.859s
|
|
- Min: 1.163s (Batch 1)
|
|
- Max: 260.849s (Batch 9)
|
|
|
|
All versions show identical exponential degradation patterns, confirming this is not a regression introduced in recent versions.
|
|
|
|
### Test Results from Clean Installation
|
|
The issue was reproduced in a completely fresh Motia installation with no custom code beyond the minimal performance test above.
|
|
|
|
**Latest Version (0.17.11-beta.193) Emission Statistics (from 9 batches):**
|
|
- Average emission time: 7.786s
|
|
- Min: 1.135s (Batch 1)
|
|
- Max: 14.074s (Batch 9)
|
|
|
|
**Event Processing Statistics:**
|
|
- Average processing time: 1.688ms
|
|
- Min: 1.030ms
|
|
- Max: 4.810ms
|
|
|
|
### Detailed Measurements (Latest Version)
|
|
| Batch | Emission Time (s) | Events Processed | Total Processing Time (s) |
|
|
|-------|-------------------|-----------------|---------------------------|
|
|
| 1 | 1.135 | 100 | ~4 |
|
|
| 2 | ~8 | 100 | ~8 |
|
|
| 3 | ~9 | 100 | ~9 |
|
|
| 4 | ~11 | 100 | ~11 |
|
|
| 5 | ~12 | 100 | ~12 |
|
|
| 6 | ~13 | 100 | ~13 |
|
|
| 7 | ~14 | 100 | ~14 |
|
|
| 8 | ~14 | 100 | ~14 |
|
|
| 9 | 14.074 | 100 | ~14 |
|
|
|
|
### Analysis
|
|
- Individual event processing remains constant, indicating the bottleneck is in the framework's event queue or scheduling system.
|
|
- The exponential growth suggests a memory leak, queue accumulation, or inefficient resource management in Motia's core event handling.
|
|
- Logs show no errors; the degradation is purely performance-related.
|
|
|
|
### Files Attached
|
|
- `perf_test_data.csv`: Raw timing data
|
|
- `perf_test_analysis.png`: Performance plots
|
|
- `motia.log`: Full log output from the test run
|
|
|
|
## Possible Root Causes
|
|
- Event queue not being properly cleared or garbage collected.
|
|
- Redis state accumulation (if used for event persistence).
|
|
- Node.js event loop blocking or memory pressure.
|
|
- Python asyncio event handling inefficiencies in the Motia bridge.
|
|
|
|
## Impact
|
|
- Makes Motia unsuitable for continuous, high-throughput event processing.
|
|
- Severe degradation occurs within minutes, preventing production use for event-driven applications.
|