added detailed readme
This commit is contained in:
Admin
459
README.md
459
README.md
@@ -1,5 +1,456 @@
|
||||
# prefect-flows
|
||||
# BBL API-Gateway Setup mit Prefect, Kong, EspoCRM und Gitea
|
||||
|
||||
# Test
|
||||
# Test
|
||||
dasadass
|
||||
Dieses Repository enthält die Flows für Prefect und dient als zentrale Dokumentation für das gesamte Setup. Das System integriert einen API-Gateway (Kong) für Prefect und EspoCRM, einen Git-Server (Gitea) für Code-Management und Docker für die Container-Orchestrierung. Traefik handhabt das externe Routing (einschließlich SSL), und Portainer/Watchtower managen Updates.
|
||||
|
||||
## Einführung
|
||||
|
||||
Dieses Setup ermöglicht:
|
||||
- **Kong als API-Gateway**: Routet Anfragen an Prefect (`api.bitbylaw.com:5000/prefect`) und EspoCRM (`api.bitbylaw.com:5000/espocrm`).
|
||||
- **Prefect für Workflows**: Flows werden in diesem Repository erstellt, geändert, gepusht, deployed und über die API getriggert.
|
||||
- **Gitea als Git-Server**: Zentrale Code-Speicherung für Prefect-Flows, integriert mit VS Code.
|
||||
- **EspoCRM**: CRM-System, integriert mit Kong für API-Zugriff.
|
||||
- **Docker**: Alle Komponenten laufen in Containern mit einem gemeinsamen Netzwerk (`bbl-shared-net`) für interne Kommunikation.
|
||||
- **Traefik**: Manuelle Konfiguration für URLs wie `api.bitbylaw.com`, `ui.api.bitbylaw.com`, `git.bitbylaw.com`, `prefect.bitbylaw.com`, `espocrm.bitbylaw.com`.
|
||||
- **Sicherheit**: Authentifizierung über Kong (z.B. `basic-auth` oder `key-auth`), SSL via Traefik, SSH-Keys für Git.
|
||||
|
||||
## Voraussetzungen
|
||||
|
||||
- **VM/Host**: Ubuntu/Debian mit Docker, Docker Compose, Portainer, Watchtower und Traefik installiert.
|
||||
- **Netzwerk**: Externes Docker-Netzwerk `bbl-shared-net` (`docker network create bbl-shared-net`).
|
||||
- **Domains**: `api.bitbylaw.com` (Kong Proxy), `ui.api.bitbylaw.com` (Kong Manager), `git.bitbylaw.com` (Gitea), `prefect.bitbylaw.com` (Prefect UI), `espocrm.bitbylaw.com` (EspoCRM).
|
||||
- **VS Code**: Installiert auf der VM, mit Python-Extension und GitLens für Git-Integration.
|
||||
- **Python**: 3.8+ in einer virtuellen Umgebung (`venv` unter `~/bbl-code`).
|
||||
- **Gitea-Credentials**: Admin-User (`admin:strongpass`).
|
||||
- **Prefect-API**: Ohne Auth (für Tests); für Produktion `basic-auth` aktivieren.
|
||||
|
||||
## Docker-Konfiguration
|
||||
|
||||
Alle Container laufen in `bbl-shared-net` für interne Kommunikation. Watchtower-Labels für automatische Updates. Ports werden über Traefik geroutet.
|
||||
|
||||
### Kong (API-Gateway)
|
||||
`docker-compose-kong.yml`:
|
||||
```yaml
|
||||
version: '3.8'
|
||||
|
||||
networks:
|
||||
bbl-shared-net:
|
||||
external: true
|
||||
|
||||
volumes:
|
||||
kong_data: {}
|
||||
|
||||
services:
|
||||
kong-database:
|
||||
image: postgres:16
|
||||
restart: always
|
||||
networks:
|
||||
- bbl-shared-net
|
||||
environment:
|
||||
POSTGRES_USER: kong
|
||||
POSTGRES_PASSWORD: kong
|
||||
POSTGRES_DB: kong
|
||||
volumes:
|
||||
- kong_data:/var/lib/postgresql/data
|
||||
ports:
|
||||
- "5432:5432"
|
||||
healthcheck:
|
||||
test: ["CMD", "pg_isready", "-U", "kong"]
|
||||
interval: 5s
|
||||
timeout: 5s
|
||||
retries: 5
|
||||
labels:
|
||||
com.centurylinklabs.watchtower.enable: "true"
|
||||
|
||||
kong-migration:
|
||||
image: kong:latest
|
||||
command: "kong migrations bootstrap"
|
||||
networks:
|
||||
- bbl-shared-net
|
||||
restart: on-failure
|
||||
environment:
|
||||
KONG_PG_HOST: kong-database
|
||||
KONG_PG_USER: kong
|
||||
KONG_PG_PASSWORD: kong
|
||||
KONG_PG_DATABASE: kong
|
||||
KONG_DATABASE: postgres
|
||||
KONG_LOG_LEVEL: debug
|
||||
depends_on:
|
||||
kong-database:
|
||||
condition: service_healthy
|
||||
|
||||
kong:
|
||||
image: kong:latest
|
||||
restart: always
|
||||
networks:
|
||||
- bbl-shared-net
|
||||
environment:
|
||||
KONG_PG_HOST: kong-database
|
||||
KONG_PG_USER: kong
|
||||
KONG_PG_PASSWORD: kong
|
||||
KONG_PG_DATABASE: kong
|
||||
KONG_DATABASE: postgres
|
||||
KONG_PG_TIMEOUT: 10000
|
||||
KONG_PROXY_LISTEN: 0.0.0.0:5000
|
||||
KONG_ADMIN_LISTEN: 0.0.0.0:8001
|
||||
KONG_ADMIN_GUI_LISTEN: 0.0.0.0:5001
|
||||
KONG_PORTAL: on
|
||||
KONG_ADMIN_GUI_AUTH: basic-auth
|
||||
KONG_ADMIN_GUI_USERNAME: admin
|
||||
KONG_ADMIN_GUI_PASSWORD: adminpass
|
||||
KONG_LOG_LEVEL: debug
|
||||
depends_on:
|
||||
- kong-migration
|
||||
- kong-database
|
||||
healthcheck:
|
||||
test: ["CMD", "kong", "health"]
|
||||
interval: 5s
|
||||
timeout: 5s
|
||||
retries: 20
|
||||
ports:
|
||||
- "5000:5000"
|
||||
- "8001:8001"
|
||||
- "5001:5001"
|
||||
labels:
|
||||
com.centurylinklabs.watchtower.enable: "true"
|
||||
```
|
||||
|
||||
- **Konfiguration in Kong Manager**: Services und Routes für Prefect (`prefect-service`: `http://prefect-server:4200/api`, Route: `/prefect`) und EspoCRM (`espocrm-service`: `http://espocrm:80`, Route: `/espocrm`).
|
||||
- **Authentifizierung**: `basic-auth` oder `key-auth` auf Service-Ebene (z.B. für `prefect-service`).
|
||||
|
||||
### Prefect (Workflow-Management)
|
||||
`docker-compose-prefect.yml`:
|
||||
```yaml
|
||||
version: '3.8'
|
||||
|
||||
networks:
|
||||
bbl-shared-net:
|
||||
external: true
|
||||
|
||||
volumes:
|
||||
postgres_data:
|
||||
redis_data:
|
||||
|
||||
services:
|
||||
postgres:
|
||||
image: postgres:14
|
||||
container_name: prefect-postgres
|
||||
restart: unless-stopped
|
||||
environment:
|
||||
POSTGRES_USER: prefect
|
||||
POSTGRES_PASSWORD: OujaoPhaiS5Eig7ohy3TeiKeecahhe
|
||||
POSTGRES_DB: prefect
|
||||
volumes:
|
||||
- postgres_data:/var/lib/postgresql/data
|
||||
healthcheck:
|
||||
test: ["CMD-SHELL", "pg_isready -U prefect"]
|
||||
interval: 5s
|
||||
timeout: 5s
|
||||
retries: 5
|
||||
networks:
|
||||
- bbl-shared-net
|
||||
labels:
|
||||
com.centurylinklabs.watchtower.enable: "true"
|
||||
|
||||
redis:
|
||||
image: redis:7
|
||||
container_name: prefect-redis
|
||||
restart: unless-stopped
|
||||
volumes:
|
||||
- redis_data:/data
|
||||
healthcheck:
|
||||
test: ["CMD-SHELL", "redis-cli ping"]
|
||||
networks:
|
||||
- bbl-shared-net
|
||||
labels:
|
||||
com.centurylinklabs.watchtower.enable: "true"
|
||||
|
||||
prefect-server:
|
||||
image: prefecthq/prefect:3-latest
|
||||
container_name: prefect-server
|
||||
restart: unless-stopped
|
||||
depends_on:
|
||||
postgres:
|
||||
condition: service_healthy
|
||||
redis:
|
||||
condition: service_healthy
|
||||
environment:
|
||||
PREFECT_API_DATABASE_CONNECTION_URL: postgresql+asyncpg://prefect:OujaoPhaiS5Eig7ohy3TeiKeecahhe@postgres:5432/prefect
|
||||
PREFECT_SERVER_API_HOST: 0.0.0.0
|
||||
PREFECT_SERVER_API_PORT: 4200
|
||||
PREFECT_UI_API_URL: http://prefect-server:4200/api
|
||||
PREFECT_SERVER_CSRF_PROTECTION_ENABLED: "true"
|
||||
PREFECT_SERVER_AUTHENTICATION_ENABLED: "true"
|
||||
PREFECT_MESSAGING_BROKER: prefect_redis.messaging
|
||||
PREFECT_MESSAGING_CACHE: prefect_redis.messaging
|
||||
PREFECT_REDIS_MESSAGING_HOST: redis
|
||||
PREFECT_REDIS_MESSAGING_PORT: 6379
|
||||
PREFECT_REDIS_MESSAGING_DB: 0
|
||||
command: prefect server start --no-services
|
||||
ports:
|
||||
- "4200:4200"
|
||||
networks:
|
||||
- bbl-shared-net
|
||||
labels:
|
||||
com.centurylinklabs.watchtower.enable: "true"
|
||||
|
||||
prefect-services:
|
||||
image: prefecthq/prefect:3-latest
|
||||
container_name: prefect-services
|
||||
restart: unless-stopped
|
||||
depends_on:
|
||||
postgres:
|
||||
condition: service_healthy
|
||||
redis:
|
||||
condition: service_healthy
|
||||
environment:
|
||||
PREFECT_API_DATABASE_CONNECTION_URL: postgresql+asyncpg://prefect:OujaoPhaiS5Eig7ohy3TeiKeecahhe@postgres:5432/prefect
|
||||
PREFECT_MESSAGING_BROKER: prefect_redis.messaging
|
||||
PREFECT_MESSAGING_CACHE: prefect_redis.messaging
|
||||
PREFECT_REDIS_MESSAGING_HOST: redis
|
||||
PREFECT_REDIS_MESSAGING_PORT: 6379
|
||||
PREFECT_REDIS_MESSAGING_DB: 0
|
||||
command: prefect server services start
|
||||
networks:
|
||||
- bbl-shared-net
|
||||
labels:
|
||||
com.centurylinklabs.watchtower.enable: "true"
|
||||
|
||||
prefect-worker:
|
||||
image: prefecthq/prefect:3-latest
|
||||
container_name: prefect-worker
|
||||
restart: unless-stopped
|
||||
depends_on:
|
||||
prefect-server:
|
||||
condition: service_started
|
||||
environment:
|
||||
PREFECT_API_URL: http://prefect-server:4200/api
|
||||
command: prefect worker start --pool local-pool
|
||||
networks:
|
||||
- bbl-shared-net
|
||||
labels:
|
||||
com.centurylinklabs.watchtower.enable: "true"
|
||||
```
|
||||
|
||||
- **Prefect UI**: `https://prefect.bitbylaw.com`.
|
||||
- **API**: Über Kong `https://api.bitbylaw.com/prefect`.
|
||||
|
||||
### Git-Server (Gitea)
|
||||
`docker-compose-gitea.yml`:
|
||||
```yaml
|
||||
version: '3.8'
|
||||
|
||||
networks:
|
||||
bbl-shared-net:
|
||||
external: true
|
||||
|
||||
volumes:
|
||||
gitea-data: {}
|
||||
|
||||
services:
|
||||
gitea:
|
||||
image: gitea/gitea:latest
|
||||
container_name: gitea
|
||||
restart: always
|
||||
environment:
|
||||
- USER_UID=1000
|
||||
- USER_GID=1000
|
||||
- GITEA__server__ROOT_URL=http://git.bitbylaw.com:3000
|
||||
- GITEA__database__DB_TYPE=sqlite3
|
||||
- GITEA__server__SSH_DOMAIN=git.bitbylaw.com
|
||||
volumes:
|
||||
- gitea-data:/data
|
||||
- /etc/timezone:/etc/timezone:ro
|
||||
- /etc/localtime:/etc/localtime:ro
|
||||
ports:
|
||||
- "3000:3000"
|
||||
- "2222:22"
|
||||
networks:
|
||||
- bbl-shared-net
|
||||
labels:
|
||||
com.centurylinklabs.watchtower.enable: "true"
|
||||
```
|
||||
|
||||
- **Web-UI**: `https://git.bitbylaw.com`.
|
||||
- **SSH**: `git@git.bitbylaw.com` (Port 2222).
|
||||
- **Integration**: Flows werden in Repos wie `admin/prefect-flows` gepusht.
|
||||
|
||||
### EspoCRM
|
||||
`docker-compose-espocrm.yml`:
|
||||
```yaml
|
||||
version: '3.8'
|
||||
|
||||
networks:
|
||||
bbl-shared-net:
|
||||
external: true
|
||||
|
||||
volumes:
|
||||
espocrm-db:
|
||||
espocrm:
|
||||
|
||||
services:
|
||||
espocrm-db:
|
||||
image: mariadb:latest
|
||||
container_name: espocrm-db
|
||||
environment:
|
||||
MARIADB_ROOT_PASSWORD: root_password
|
||||
MARIADB_DATABASE: espocrm
|
||||
MARIADB_USER: espocrm
|
||||
MARIADB_PASSWORD: database_password
|
||||
volumes:
|
||||
- espocrm-db:/var/lib/mysql
|
||||
restart: always
|
||||
healthcheck:
|
||||
test: ["CMD", "healthcheck.sh", "--connect", "--innodb_initialized"]
|
||||
interval: 20s
|
||||
start_period: 10s
|
||||
timeout: 10s
|
||||
retries: 3
|
||||
networks:
|
||||
- bbl-shared-net
|
||||
labels:
|
||||
com.centurylinklabs.watchtower.enable: "true"
|
||||
|
||||
espocrm:
|
||||
image: espocrm/espocrm:latest
|
||||
container_name: espocrm
|
||||
environment:
|
||||
ESPOCRM_DATABASE_PLATFORM: Mysql
|
||||
ESPOCRM_DATABASE_HOST: espocrm-db
|
||||
ESPOCRM_DATABASE_USER: espocrm
|
||||
ESPOCRM_DATABASE_PASSWORD: database_password
|
||||
ESPOCRM_ADMIN_USERNAME: admin
|
||||
ESPOCRM_ADMIN_PASSWORD: password
|
||||
ESPOCRM_SITE_URL: "http://api.bitbylaw.com:5000/espocrm"
|
||||
volumes:
|
||||
- espocrm:/var/www/html
|
||||
restart: always
|
||||
depends_on:
|
||||
espocrm-db:
|
||||
condition: service_healthy
|
||||
ports:
|
||||
- "8080:80"
|
||||
networks:
|
||||
- bbl-shared-net
|
||||
labels:
|
||||
com.centurylinklabs.watchtower.enable: "true"
|
||||
|
||||
espocrm-daemon:
|
||||
image: espocrm/espocrm:latest
|
||||
container_name: espocrm-daemon
|
||||
volumes:
|
||||
- espocrm:/var/www/html
|
||||
restart: always
|
||||
entrypoint: docker-daemon.sh
|
||||
networks:
|
||||
- bbl-shared-net
|
||||
labels:
|
||||
com.centurylinklabs.watchtower.enable: "true"
|
||||
|
||||
espocrm-websocket:
|
||||
image: espocrm/espocrm:latest
|
||||
container_name: espocrm-websocket
|
||||
environment:
|
||||
ESPOCRM_CONFIG_USE_WEB_SOCKET: "true"
|
||||
ESPOCRM_CONFIG_WEB_SOCKET_URL: "ws://api.bitbylaw.com:5000/espocrm/ws"
|
||||
ESPOCRM_CONFIG_WEB_SOCKET_ZERO_M_Q_SUBSCRIBER_DSN: "tcp://*:7777"
|
||||
ESPOCRM_CONFIG_WEB_SOCKET_ZERO_M_Q_SUBMISSION_DSN: "tcp://espocrm-websocket:7777"
|
||||
volumes:
|
||||
- espocrm:/var/www/html
|
||||
restart: always
|
||||
entrypoint: docker-websocket.sh
|
||||
ports:
|
||||
- "8081:8080"
|
||||
networks:
|
||||
- bbl-shared-net
|
||||
labels:
|
||||
com.centurylinklabs.watchtower.enable: "true"
|
||||
```
|
||||
|
||||
- **UI**: `https://espocrm.bitbylaw.com`.
|
||||
- **Integration**: Über Kong `https://api.bitbylaw.com/espocrm`.
|
||||
|
||||
### Flows erstellen, ändern und deployen
|
||||
|
||||
1. **Flow erstellen**:
|
||||
- In VS Code: Öffne `~/bbl-code/prefect-flows`.
|
||||
- Erstelle z.B. `flows/new_flow.py`:
|
||||
```python
|
||||
from prefect import flow, task
|
||||
|
||||
@task
|
||||
def say_hello(name: str) -> str:
|
||||
message = f"New Flow: Hello, {name}!"
|
||||
print(message)
|
||||
return message
|
||||
|
||||
@flow(name="new-flow")
|
||||
def new_flow(name: str = "New World"):
|
||||
result = say_hello(name)
|
||||
return result
|
||||
|
||||
if __name__ == "__main__":
|
||||
new_flow()
|
||||
```
|
||||
- Teste lokal: `python3 flows/new_flow.py`.
|
||||
|
||||
2. **Flow ändern**:
|
||||
- Bearbeite `flows/hello_world.py` oder `new_flow.py` in VS Code.
|
||||
- Speichere und teste lokal.
|
||||
|
||||
3. **Commit und Push**:
|
||||
- In VS Code **Source Control** (Ctrl+Shift+G):
|
||||
- Stage: `+` neben Datei.
|
||||
- Commit: Nachricht eingeben (z.B. `Add new_flow.py`) → Häkchen.
|
||||
- Push: `...` → **Push**.
|
||||
- Alternativ Terminal:
|
||||
```bash
|
||||
git add flows/new_flow.py
|
||||
git commit -m "Add new_flow.py"
|
||||
git push origin main
|
||||
```
|
||||
|
||||
4. **Deploy den Flow**:
|
||||
- In VS Code Terminal:
|
||||
```bash
|
||||
cd ~/bbl-code/prefect-flows
|
||||
prefect deploy
|
||||
```
|
||||
- Wähle das Deployment aus `prefect.yaml` (z.B. `new-flow-deployment`).
|
||||
- Prefect pullt den Code aus Gitea in den Worker.
|
||||
|
||||
5. **Trigger den Flow**:
|
||||
- Finde Deployment-ID:
|
||||
```bash
|
||||
curl -i https://api.bitbylaw.com/prefect/deployments
|
||||
```
|
||||
- Trigger:
|
||||
```bash
|
||||
curl -i -X POST \
|
||||
-H "Content-Type: application/json" \
|
||||
-d '{
|
||||
"name": "new-flow-run",
|
||||
"parameters": {"name": "New World"},
|
||||
"state": {"type": "SCHEDULED"}
|
||||
}' \
|
||||
https://api.bitbylaw.com/prefect/deployments/<deployment-id>/flow_runs
|
||||
```
|
||||
- Prüfe:
|
||||
```bash
|
||||
curl -i https://api.bitbylaw.com/prefect/flow_runs/<flow-run-id>
|
||||
```
|
||||
|
||||
### Sicherheit und Authentifizierung
|
||||
|
||||
- **Kong**: `basic-auth` oder `key-auth` auf Service-Ebene (z.B. `prefect-service`).
|
||||
- **Gitea**: SSH-Keys für passwortlosen Zugriff (siehe vorherige Anleitung).
|
||||
- **Traefik**: SSL für alle URLs.
|
||||
- **Prefect**: Aktiviere Authentifizierung in `PREFECT_SERVER_AUTHENTICATION_ENABLED`.
|
||||
|
||||
### Troubleshooting
|
||||
|
||||
- **Git**: Prüfe Logs in Gitea (`docker logs gitea`).
|
||||
- **Prefect Deployment**: Worker-Logs (`docker logs prefect-worker`).
|
||||
- **Flow-Run**: UI (`https://prefect.bitbylaw.com`) oder API-Logs.
|
||||
- **Traefik**: Logs (`docker logs traefik`).
|
||||
|
||||
Für Updates pushe Änderungen zu Gitea und deploye neu.
|
||||
Reference in New Issue
Block a user