added detailed readme
This commit is contained in:
Admin
459
README.md
459
README.md
@@ -1,5 +1,456 @@
|
|||||||
# prefect-flows
|
# BBL API-Gateway Setup mit Prefect, Kong, EspoCRM und Gitea
|
||||||
|
|
||||||
# Test
|
Dieses Repository enthält die Flows für Prefect und dient als zentrale Dokumentation für das gesamte Setup. Das System integriert einen API-Gateway (Kong) für Prefect und EspoCRM, einen Git-Server (Gitea) für Code-Management und Docker für die Container-Orchestrierung. Traefik handhabt das externe Routing (einschließlich SSL), und Portainer/Watchtower managen Updates.
|
||||||
# Test
|
|
||||||
dasadass
|
## Einführung
|
||||||
|
|
||||||
|
Dieses Setup ermöglicht:
|
||||||
|
- **Kong als API-Gateway**: Routet Anfragen an Prefect (`api.bitbylaw.com:5000/prefect`) und EspoCRM (`api.bitbylaw.com:5000/espocrm`).
|
||||||
|
- **Prefect für Workflows**: Flows werden in diesem Repository erstellt, geändert, gepusht, deployed und über die API getriggert.
|
||||||
|
- **Gitea als Git-Server**: Zentrale Code-Speicherung für Prefect-Flows, integriert mit VS Code.
|
||||||
|
- **EspoCRM**: CRM-System, integriert mit Kong für API-Zugriff.
|
||||||
|
- **Docker**: Alle Komponenten laufen in Containern mit einem gemeinsamen Netzwerk (`bbl-shared-net`) für interne Kommunikation.
|
||||||
|
- **Traefik**: Manuelle Konfiguration für URLs wie `api.bitbylaw.com`, `ui.api.bitbylaw.com`, `git.bitbylaw.com`, `prefect.bitbylaw.com`, `espocrm.bitbylaw.com`.
|
||||||
|
- **Sicherheit**: Authentifizierung über Kong (z.B. `basic-auth` oder `key-auth`), SSL via Traefik, SSH-Keys für Git.
|
||||||
|
|
||||||
|
## Voraussetzungen
|
||||||
|
|
||||||
|
- **VM/Host**: Ubuntu/Debian mit Docker, Docker Compose, Portainer, Watchtower und Traefik installiert.
|
||||||
|
- **Netzwerk**: Externes Docker-Netzwerk `bbl-shared-net` (`docker network create bbl-shared-net`).
|
||||||
|
- **Domains**: `api.bitbylaw.com` (Kong Proxy), `ui.api.bitbylaw.com` (Kong Manager), `git.bitbylaw.com` (Gitea), `prefect.bitbylaw.com` (Prefect UI), `espocrm.bitbylaw.com` (EspoCRM).
|
||||||
|
- **VS Code**: Installiert auf der VM, mit Python-Extension und GitLens für Git-Integration.
|
||||||
|
- **Python**: 3.8+ in einer virtuellen Umgebung (`venv` unter `~/bbl-code`).
|
||||||
|
- **Gitea-Credentials**: Admin-User (`admin:strongpass`).
|
||||||
|
- **Prefect-API**: Ohne Auth (für Tests); für Produktion `basic-auth` aktivieren.
|
||||||
|
|
||||||
|
## Docker-Konfiguration
|
||||||
|
|
||||||
|
Alle Container laufen in `bbl-shared-net` für interne Kommunikation. Watchtower-Labels für automatische Updates. Ports werden über Traefik geroutet.
|
||||||
|
|
||||||
|
### Kong (API-Gateway)
|
||||||
|
`docker-compose-kong.yml`:
|
||||||
|
```yaml
|
||||||
|
version: '3.8'
|
||||||
|
|
||||||
|
networks:
|
||||||
|
bbl-shared-net:
|
||||||
|
external: true
|
||||||
|
|
||||||
|
volumes:
|
||||||
|
kong_data: {}
|
||||||
|
|
||||||
|
services:
|
||||||
|
kong-database:
|
||||||
|
image: postgres:16
|
||||||
|
restart: always
|
||||||
|
networks:
|
||||||
|
- bbl-shared-net
|
||||||
|
environment:
|
||||||
|
POSTGRES_USER: kong
|
||||||
|
POSTGRES_PASSWORD: kong
|
||||||
|
POSTGRES_DB: kong
|
||||||
|
volumes:
|
||||||
|
- kong_data:/var/lib/postgresql/data
|
||||||
|
ports:
|
||||||
|
- "5432:5432"
|
||||||
|
healthcheck:
|
||||||
|
test: ["CMD", "pg_isready", "-U", "kong"]
|
||||||
|
interval: 5s
|
||||||
|
timeout: 5s
|
||||||
|
retries: 5
|
||||||
|
labels:
|
||||||
|
com.centurylinklabs.watchtower.enable: "true"
|
||||||
|
|
||||||
|
kong-migration:
|
||||||
|
image: kong:latest
|
||||||
|
command: "kong migrations bootstrap"
|
||||||
|
networks:
|
||||||
|
- bbl-shared-net
|
||||||
|
restart: on-failure
|
||||||
|
environment:
|
||||||
|
KONG_PG_HOST: kong-database
|
||||||
|
KONG_PG_USER: kong
|
||||||
|
KONG_PG_PASSWORD: kong
|
||||||
|
KONG_PG_DATABASE: kong
|
||||||
|
KONG_DATABASE: postgres
|
||||||
|
KONG_LOG_LEVEL: debug
|
||||||
|
depends_on:
|
||||||
|
kong-database:
|
||||||
|
condition: service_healthy
|
||||||
|
|
||||||
|
kong:
|
||||||
|
image: kong:latest
|
||||||
|
restart: always
|
||||||
|
networks:
|
||||||
|
- bbl-shared-net
|
||||||
|
environment:
|
||||||
|
KONG_PG_HOST: kong-database
|
||||||
|
KONG_PG_USER: kong
|
||||||
|
KONG_PG_PASSWORD: kong
|
||||||
|
KONG_PG_DATABASE: kong
|
||||||
|
KONG_DATABASE: postgres
|
||||||
|
KONG_PG_TIMEOUT: 10000
|
||||||
|
KONG_PROXY_LISTEN: 0.0.0.0:5000
|
||||||
|
KONG_ADMIN_LISTEN: 0.0.0.0:8001
|
||||||
|
KONG_ADMIN_GUI_LISTEN: 0.0.0.0:5001
|
||||||
|
KONG_PORTAL: on
|
||||||
|
KONG_ADMIN_GUI_AUTH: basic-auth
|
||||||
|
KONG_ADMIN_GUI_USERNAME: admin
|
||||||
|
KONG_ADMIN_GUI_PASSWORD: adminpass
|
||||||
|
KONG_LOG_LEVEL: debug
|
||||||
|
depends_on:
|
||||||
|
- kong-migration
|
||||||
|
- kong-database
|
||||||
|
healthcheck:
|
||||||
|
test: ["CMD", "kong", "health"]
|
||||||
|
interval: 5s
|
||||||
|
timeout: 5s
|
||||||
|
retries: 20
|
||||||
|
ports:
|
||||||
|
- "5000:5000"
|
||||||
|
- "8001:8001"
|
||||||
|
- "5001:5001"
|
||||||
|
labels:
|
||||||
|
com.centurylinklabs.watchtower.enable: "true"
|
||||||
|
```
|
||||||
|
|
||||||
|
- **Konfiguration in Kong Manager**: Services und Routes für Prefect (`prefect-service`: `http://prefect-server:4200/api`, Route: `/prefect`) und EspoCRM (`espocrm-service`: `http://espocrm:80`, Route: `/espocrm`).
|
||||||
|
- **Authentifizierung**: `basic-auth` oder `key-auth` auf Service-Ebene (z.B. für `prefect-service`).
|
||||||
|
|
||||||
|
### Prefect (Workflow-Management)
|
||||||
|
`docker-compose-prefect.yml`:
|
||||||
|
```yaml
|
||||||
|
version: '3.8'
|
||||||
|
|
||||||
|
networks:
|
||||||
|
bbl-shared-net:
|
||||||
|
external: true
|
||||||
|
|
||||||
|
volumes:
|
||||||
|
postgres_data:
|
||||||
|
redis_data:
|
||||||
|
|
||||||
|
services:
|
||||||
|
postgres:
|
||||||
|
image: postgres:14
|
||||||
|
container_name: prefect-postgres
|
||||||
|
restart: unless-stopped
|
||||||
|
environment:
|
||||||
|
POSTGRES_USER: prefect
|
||||||
|
POSTGRES_PASSWORD: OujaoPhaiS5Eig7ohy3TeiKeecahhe
|
||||||
|
POSTGRES_DB: prefect
|
||||||
|
volumes:
|
||||||
|
- postgres_data:/var/lib/postgresql/data
|
||||||
|
healthcheck:
|
||||||
|
test: ["CMD-SHELL", "pg_isready -U prefect"]
|
||||||
|
interval: 5s
|
||||||
|
timeout: 5s
|
||||||
|
retries: 5
|
||||||
|
networks:
|
||||||
|
- bbl-shared-net
|
||||||
|
labels:
|
||||||
|
com.centurylinklabs.watchtower.enable: "true"
|
||||||
|
|
||||||
|
redis:
|
||||||
|
image: redis:7
|
||||||
|
container_name: prefect-redis
|
||||||
|
restart: unless-stopped
|
||||||
|
volumes:
|
||||||
|
- redis_data:/data
|
||||||
|
healthcheck:
|
||||||
|
test: ["CMD-SHELL", "redis-cli ping"]
|
||||||
|
networks:
|
||||||
|
- bbl-shared-net
|
||||||
|
labels:
|
||||||
|
com.centurylinklabs.watchtower.enable: "true"
|
||||||
|
|
||||||
|
prefect-server:
|
||||||
|
image: prefecthq/prefect:3-latest
|
||||||
|
container_name: prefect-server
|
||||||
|
restart: unless-stopped
|
||||||
|
depends_on:
|
||||||
|
postgres:
|
||||||
|
condition: service_healthy
|
||||||
|
redis:
|
||||||
|
condition: service_healthy
|
||||||
|
environment:
|
||||||
|
PREFECT_API_DATABASE_CONNECTION_URL: postgresql+asyncpg://prefect:OujaoPhaiS5Eig7ohy3TeiKeecahhe@postgres:5432/prefect
|
||||||
|
PREFECT_SERVER_API_HOST: 0.0.0.0
|
||||||
|
PREFECT_SERVER_API_PORT: 4200
|
||||||
|
PREFECT_UI_API_URL: http://prefect-server:4200/api
|
||||||
|
PREFECT_SERVER_CSRF_PROTECTION_ENABLED: "true"
|
||||||
|
PREFECT_SERVER_AUTHENTICATION_ENABLED: "true"
|
||||||
|
PREFECT_MESSAGING_BROKER: prefect_redis.messaging
|
||||||
|
PREFECT_MESSAGING_CACHE: prefect_redis.messaging
|
||||||
|
PREFECT_REDIS_MESSAGING_HOST: redis
|
||||||
|
PREFECT_REDIS_MESSAGING_PORT: 6379
|
||||||
|
PREFECT_REDIS_MESSAGING_DB: 0
|
||||||
|
command: prefect server start --no-services
|
||||||
|
ports:
|
||||||
|
- "4200:4200"
|
||||||
|
networks:
|
||||||
|
- bbl-shared-net
|
||||||
|
labels:
|
||||||
|
com.centurylinklabs.watchtower.enable: "true"
|
||||||
|
|
||||||
|
prefect-services:
|
||||||
|
image: prefecthq/prefect:3-latest
|
||||||
|
container_name: prefect-services
|
||||||
|
restart: unless-stopped
|
||||||
|
depends_on:
|
||||||
|
postgres:
|
||||||
|
condition: service_healthy
|
||||||
|
redis:
|
||||||
|
condition: service_healthy
|
||||||
|
environment:
|
||||||
|
PREFECT_API_DATABASE_CONNECTION_URL: postgresql+asyncpg://prefect:OujaoPhaiS5Eig7ohy3TeiKeecahhe@postgres:5432/prefect
|
||||||
|
PREFECT_MESSAGING_BROKER: prefect_redis.messaging
|
||||||
|
PREFECT_MESSAGING_CACHE: prefect_redis.messaging
|
||||||
|
PREFECT_REDIS_MESSAGING_HOST: redis
|
||||||
|
PREFECT_REDIS_MESSAGING_PORT: 6379
|
||||||
|
PREFECT_REDIS_MESSAGING_DB: 0
|
||||||
|
command: prefect server services start
|
||||||
|
networks:
|
||||||
|
- bbl-shared-net
|
||||||
|
labels:
|
||||||
|
com.centurylinklabs.watchtower.enable: "true"
|
||||||
|
|
||||||
|
prefect-worker:
|
||||||
|
image: prefecthq/prefect:3-latest
|
||||||
|
container_name: prefect-worker
|
||||||
|
restart: unless-stopped
|
||||||
|
depends_on:
|
||||||
|
prefect-server:
|
||||||
|
condition: service_started
|
||||||
|
environment:
|
||||||
|
PREFECT_API_URL: http://prefect-server:4200/api
|
||||||
|
command: prefect worker start --pool local-pool
|
||||||
|
networks:
|
||||||
|
- bbl-shared-net
|
||||||
|
labels:
|
||||||
|
com.centurylinklabs.watchtower.enable: "true"
|
||||||
|
```
|
||||||
|
|
||||||
|
- **Prefect UI**: `https://prefect.bitbylaw.com`.
|
||||||
|
- **API**: Über Kong `https://api.bitbylaw.com/prefect`.
|
||||||
|
|
||||||
|
### Git-Server (Gitea)
|
||||||
|
`docker-compose-gitea.yml`:
|
||||||
|
```yaml
|
||||||
|
version: '3.8'
|
||||||
|
|
||||||
|
networks:
|
||||||
|
bbl-shared-net:
|
||||||
|
external: true
|
||||||
|
|
||||||
|
volumes:
|
||||||
|
gitea-data: {}
|
||||||
|
|
||||||
|
services:
|
||||||
|
gitea:
|
||||||
|
image: gitea/gitea:latest
|
||||||
|
container_name: gitea
|
||||||
|
restart: always
|
||||||
|
environment:
|
||||||
|
- USER_UID=1000
|
||||||
|
- USER_GID=1000
|
||||||
|
- GITEA__server__ROOT_URL=http://git.bitbylaw.com:3000
|
||||||
|
- GITEA__database__DB_TYPE=sqlite3
|
||||||
|
- GITEA__server__SSH_DOMAIN=git.bitbylaw.com
|
||||||
|
volumes:
|
||||||
|
- gitea-data:/data
|
||||||
|
- /etc/timezone:/etc/timezone:ro
|
||||||
|
- /etc/localtime:/etc/localtime:ro
|
||||||
|
ports:
|
||||||
|
- "3000:3000"
|
||||||
|
- "2222:22"
|
||||||
|
networks:
|
||||||
|
- bbl-shared-net
|
||||||
|
labels:
|
||||||
|
com.centurylinklabs.watchtower.enable: "true"
|
||||||
|
```
|
||||||
|
|
||||||
|
- **Web-UI**: `https://git.bitbylaw.com`.
|
||||||
|
- **SSH**: `git@git.bitbylaw.com` (Port 2222).
|
||||||
|
- **Integration**: Flows werden in Repos wie `admin/prefect-flows` gepusht.
|
||||||
|
|
||||||
|
### EspoCRM
|
||||||
|
`docker-compose-espocrm.yml`:
|
||||||
|
```yaml
|
||||||
|
version: '3.8'
|
||||||
|
|
||||||
|
networks:
|
||||||
|
bbl-shared-net:
|
||||||
|
external: true
|
||||||
|
|
||||||
|
volumes:
|
||||||
|
espocrm-db:
|
||||||
|
espocrm:
|
||||||
|
|
||||||
|
services:
|
||||||
|
espocrm-db:
|
||||||
|
image: mariadb:latest
|
||||||
|
container_name: espocrm-db
|
||||||
|
environment:
|
||||||
|
MARIADB_ROOT_PASSWORD: root_password
|
||||||
|
MARIADB_DATABASE: espocrm
|
||||||
|
MARIADB_USER: espocrm
|
||||||
|
MARIADB_PASSWORD: database_password
|
||||||
|
volumes:
|
||||||
|
- espocrm-db:/var/lib/mysql
|
||||||
|
restart: always
|
||||||
|
healthcheck:
|
||||||
|
test: ["CMD", "healthcheck.sh", "--connect", "--innodb_initialized"]
|
||||||
|
interval: 20s
|
||||||
|
start_period: 10s
|
||||||
|
timeout: 10s
|
||||||
|
retries: 3
|
||||||
|
networks:
|
||||||
|
- bbl-shared-net
|
||||||
|
labels:
|
||||||
|
com.centurylinklabs.watchtower.enable: "true"
|
||||||
|
|
||||||
|
espocrm:
|
||||||
|
image: espocrm/espocrm:latest
|
||||||
|
container_name: espocrm
|
||||||
|
environment:
|
||||||
|
ESPOCRM_DATABASE_PLATFORM: Mysql
|
||||||
|
ESPOCRM_DATABASE_HOST: espocrm-db
|
||||||
|
ESPOCRM_DATABASE_USER: espocrm
|
||||||
|
ESPOCRM_DATABASE_PASSWORD: database_password
|
||||||
|
ESPOCRM_ADMIN_USERNAME: admin
|
||||||
|
ESPOCRM_ADMIN_PASSWORD: password
|
||||||
|
ESPOCRM_SITE_URL: "http://api.bitbylaw.com:5000/espocrm"
|
||||||
|
volumes:
|
||||||
|
- espocrm:/var/www/html
|
||||||
|
restart: always
|
||||||
|
depends_on:
|
||||||
|
espocrm-db:
|
||||||
|
condition: service_healthy
|
||||||
|
ports:
|
||||||
|
- "8080:80"
|
||||||
|
networks:
|
||||||
|
- bbl-shared-net
|
||||||
|
labels:
|
||||||
|
com.centurylinklabs.watchtower.enable: "true"
|
||||||
|
|
||||||
|
espocrm-daemon:
|
||||||
|
image: espocrm/espocrm:latest
|
||||||
|
container_name: espocrm-daemon
|
||||||
|
volumes:
|
||||||
|
- espocrm:/var/www/html
|
||||||
|
restart: always
|
||||||
|
entrypoint: docker-daemon.sh
|
||||||
|
networks:
|
||||||
|
- bbl-shared-net
|
||||||
|
labels:
|
||||||
|
com.centurylinklabs.watchtower.enable: "true"
|
||||||
|
|
||||||
|
espocrm-websocket:
|
||||||
|
image: espocrm/espocrm:latest
|
||||||
|
container_name: espocrm-websocket
|
||||||
|
environment:
|
||||||
|
ESPOCRM_CONFIG_USE_WEB_SOCKET: "true"
|
||||||
|
ESPOCRM_CONFIG_WEB_SOCKET_URL: "ws://api.bitbylaw.com:5000/espocrm/ws"
|
||||||
|
ESPOCRM_CONFIG_WEB_SOCKET_ZERO_M_Q_SUBSCRIBER_DSN: "tcp://*:7777"
|
||||||
|
ESPOCRM_CONFIG_WEB_SOCKET_ZERO_M_Q_SUBMISSION_DSN: "tcp://espocrm-websocket:7777"
|
||||||
|
volumes:
|
||||||
|
- espocrm:/var/www/html
|
||||||
|
restart: always
|
||||||
|
entrypoint: docker-websocket.sh
|
||||||
|
ports:
|
||||||
|
- "8081:8080"
|
||||||
|
networks:
|
||||||
|
- bbl-shared-net
|
||||||
|
labels:
|
||||||
|
com.centurylinklabs.watchtower.enable: "true"
|
||||||
|
```
|
||||||
|
|
||||||
|
- **UI**: `https://espocrm.bitbylaw.com`.
|
||||||
|
- **Integration**: Über Kong `https://api.bitbylaw.com/espocrm`.
|
||||||
|
|
||||||
|
### Flows erstellen, ändern und deployen
|
||||||
|
|
||||||
|
1. **Flow erstellen**:
|
||||||
|
- In VS Code: Öffne `~/bbl-code/prefect-flows`.
|
||||||
|
- Erstelle z.B. `flows/new_flow.py`:
|
||||||
|
```python
|
||||||
|
from prefect import flow, task
|
||||||
|
|
||||||
|
@task
|
||||||
|
def say_hello(name: str) -> str:
|
||||||
|
message = f"New Flow: Hello, {name}!"
|
||||||
|
print(message)
|
||||||
|
return message
|
||||||
|
|
||||||
|
@flow(name="new-flow")
|
||||||
|
def new_flow(name: str = "New World"):
|
||||||
|
result = say_hello(name)
|
||||||
|
return result
|
||||||
|
|
||||||
|
if __name__ == "__main__":
|
||||||
|
new_flow()
|
||||||
|
```
|
||||||
|
- Teste lokal: `python3 flows/new_flow.py`.
|
||||||
|
|
||||||
|
2. **Flow ändern**:
|
||||||
|
- Bearbeite `flows/hello_world.py` oder `new_flow.py` in VS Code.
|
||||||
|
- Speichere und teste lokal.
|
||||||
|
|
||||||
|
3. **Commit und Push**:
|
||||||
|
- In VS Code **Source Control** (Ctrl+Shift+G):
|
||||||
|
- Stage: `+` neben Datei.
|
||||||
|
- Commit: Nachricht eingeben (z.B. `Add new_flow.py`) → Häkchen.
|
||||||
|
- Push: `...` → **Push**.
|
||||||
|
- Alternativ Terminal:
|
||||||
|
```bash
|
||||||
|
git add flows/new_flow.py
|
||||||
|
git commit -m "Add new_flow.py"
|
||||||
|
git push origin main
|
||||||
|
```
|
||||||
|
|
||||||
|
4. **Deploy den Flow**:
|
||||||
|
- In VS Code Terminal:
|
||||||
|
```bash
|
||||||
|
cd ~/bbl-code/prefect-flows
|
||||||
|
prefect deploy
|
||||||
|
```
|
||||||
|
- Wähle das Deployment aus `prefect.yaml` (z.B. `new-flow-deployment`).
|
||||||
|
- Prefect pullt den Code aus Gitea in den Worker.
|
||||||
|
|
||||||
|
5. **Trigger den Flow**:
|
||||||
|
- Finde Deployment-ID:
|
||||||
|
```bash
|
||||||
|
curl -i https://api.bitbylaw.com/prefect/deployments
|
||||||
|
```
|
||||||
|
- Trigger:
|
||||||
|
```bash
|
||||||
|
curl -i -X POST \
|
||||||
|
-H "Content-Type: application/json" \
|
||||||
|
-d '{
|
||||||
|
"name": "new-flow-run",
|
||||||
|
"parameters": {"name": "New World"},
|
||||||
|
"state": {"type": "SCHEDULED"}
|
||||||
|
}' \
|
||||||
|
https://api.bitbylaw.com/prefect/deployments/<deployment-id>/flow_runs
|
||||||
|
```
|
||||||
|
- Prüfe:
|
||||||
|
```bash
|
||||||
|
curl -i https://api.bitbylaw.com/prefect/flow_runs/<flow-run-id>
|
||||||
|
```
|
||||||
|
|
||||||
|
### Sicherheit und Authentifizierung
|
||||||
|
|
||||||
|
- **Kong**: `basic-auth` oder `key-auth` auf Service-Ebene (z.B. `prefect-service`).
|
||||||
|
- **Gitea**: SSH-Keys für passwortlosen Zugriff (siehe vorherige Anleitung).
|
||||||
|
- **Traefik**: SSL für alle URLs.
|
||||||
|
- **Prefect**: Aktiviere Authentifizierung in `PREFECT_SERVER_AUTHENTICATION_ENABLED`.
|
||||||
|
|
||||||
|
### Troubleshooting
|
||||||
|
|
||||||
|
- **Git**: Prüfe Logs in Gitea (`docker logs gitea`).
|
||||||
|
- **Prefect Deployment**: Worker-Logs (`docker logs prefect-worker`).
|
||||||
|
- **Flow-Run**: UI (`https://prefect.bitbylaw.com`) oder API-Logs.
|
||||||
|
- **Traefik**: Logs (`docker logs traefik`).
|
||||||
|
|
||||||
|
Für Updates pushe Änderungen zu Gitea und deploye neu.
|
||||||
Reference in New Issue
Block a user