diff --git a/README.md b/README.md index ea15ca7..401e809 100644 --- a/README.md +++ b/README.md @@ -1,5 +1,456 @@ -# prefect-flows +# BBL API-Gateway Setup mit Prefect, Kong, EspoCRM und Gitea -# Test -# Test -dasadass \ No newline at end of file +Dieses Repository enthält die Flows für Prefect und dient als zentrale Dokumentation für das gesamte Setup. Das System integriert einen API-Gateway (Kong) für Prefect und EspoCRM, einen Git-Server (Gitea) für Code-Management und Docker für die Container-Orchestrierung. Traefik handhabt das externe Routing (einschließlich SSL), und Portainer/Watchtower managen Updates. + +## Einführung + +Dieses Setup ermöglicht: +- **Kong als API-Gateway**: Routet Anfragen an Prefect (`api.bitbylaw.com:5000/prefect`) und EspoCRM (`api.bitbylaw.com:5000/espocrm`). +- **Prefect für Workflows**: Flows werden in diesem Repository erstellt, geändert, gepusht, deployed und über die API getriggert. +- **Gitea als Git-Server**: Zentrale Code-Speicherung für Prefect-Flows, integriert mit VS Code. +- **EspoCRM**: CRM-System, integriert mit Kong für API-Zugriff. +- **Docker**: Alle Komponenten laufen in Containern mit einem gemeinsamen Netzwerk (`bbl-shared-net`) für interne Kommunikation. +- **Traefik**: Manuelle Konfiguration für URLs wie `api.bitbylaw.com`, `ui.api.bitbylaw.com`, `git.bitbylaw.com`, `prefect.bitbylaw.com`, `espocrm.bitbylaw.com`. +- **Sicherheit**: Authentifizierung über Kong (z.B. `basic-auth` oder `key-auth`), SSL via Traefik, SSH-Keys für Git. + +## Voraussetzungen + +- **VM/Host**: Ubuntu/Debian mit Docker, Docker Compose, Portainer, Watchtower und Traefik installiert. +- **Netzwerk**: Externes Docker-Netzwerk `bbl-shared-net` (`docker network create bbl-shared-net`). +- **Domains**: `api.bitbylaw.com` (Kong Proxy), `ui.api.bitbylaw.com` (Kong Manager), `git.bitbylaw.com` (Gitea), `prefect.bitbylaw.com` (Prefect UI), `espocrm.bitbylaw.com` (EspoCRM). +- **VS Code**: Installiert auf der VM, mit Python-Extension und GitLens für Git-Integration. +- **Python**: 3.8+ in einer virtuellen Umgebung (`venv` unter `~/bbl-code`). +- **Gitea-Credentials**: Admin-User (`admin:strongpass`). +- **Prefect-API**: Ohne Auth (für Tests); für Produktion `basic-auth` aktivieren. + +## Docker-Konfiguration + +Alle Container laufen in `bbl-shared-net` für interne Kommunikation. Watchtower-Labels für automatische Updates. Ports werden über Traefik geroutet. + +### Kong (API-Gateway) +`docker-compose-kong.yml`: +```yaml +version: '3.8' + +networks: + bbl-shared-net: + external: true + +volumes: + kong_data: {} + +services: + kong-database: + image: postgres:16 + restart: always + networks: + - bbl-shared-net + environment: + POSTGRES_USER: kong + POSTGRES_PASSWORD: kong + POSTGRES_DB: kong + volumes: + - kong_data:/var/lib/postgresql/data + ports: + - "5432:5432" + healthcheck: + test: ["CMD", "pg_isready", "-U", "kong"] + interval: 5s + timeout: 5s + retries: 5 + labels: + com.centurylinklabs.watchtower.enable: "true" + + kong-migration: + image: kong:latest + command: "kong migrations bootstrap" + networks: + - bbl-shared-net + restart: on-failure + environment: + KONG_PG_HOST: kong-database + KONG_PG_USER: kong + KONG_PG_PASSWORD: kong + KONG_PG_DATABASE: kong + KONG_DATABASE: postgres + KONG_LOG_LEVEL: debug + depends_on: + kong-database: + condition: service_healthy + + kong: + image: kong:latest + restart: always + networks: + - bbl-shared-net + environment: + KONG_PG_HOST: kong-database + KONG_PG_USER: kong + KONG_PG_PASSWORD: kong + KONG_PG_DATABASE: kong + KONG_DATABASE: postgres + KONG_PG_TIMEOUT: 10000 + KONG_PROXY_LISTEN: 0.0.0.0:5000 + KONG_ADMIN_LISTEN: 0.0.0.0:8001 + KONG_ADMIN_GUI_LISTEN: 0.0.0.0:5001 + KONG_PORTAL: on + KONG_ADMIN_GUI_AUTH: basic-auth + KONG_ADMIN_GUI_USERNAME: admin + KONG_ADMIN_GUI_PASSWORD: adminpass + KONG_LOG_LEVEL: debug + depends_on: + - kong-migration + - kong-database + healthcheck: + test: ["CMD", "kong", "health"] + interval: 5s + timeout: 5s + retries: 20 + ports: + - "5000:5000" + - "8001:8001" + - "5001:5001" + labels: + com.centurylinklabs.watchtower.enable: "true" +``` + +- **Konfiguration in Kong Manager**: Services und Routes für Prefect (`prefect-service`: `http://prefect-server:4200/api`, Route: `/prefect`) und EspoCRM (`espocrm-service`: `http://espocrm:80`, Route: `/espocrm`). +- **Authentifizierung**: `basic-auth` oder `key-auth` auf Service-Ebene (z.B. für `prefect-service`). + +### Prefect (Workflow-Management) +`docker-compose-prefect.yml`: +```yaml +version: '3.8' + +networks: + bbl-shared-net: + external: true + +volumes: + postgres_data: + redis_data: + +services: + postgres: + image: postgres:14 + container_name: prefect-postgres + restart: unless-stopped + environment: + POSTGRES_USER: prefect + POSTGRES_PASSWORD: OujaoPhaiS5Eig7ohy3TeiKeecahhe + POSTGRES_DB: prefect + volumes: + - postgres_data:/var/lib/postgresql/data + healthcheck: + test: ["CMD-SHELL", "pg_isready -U prefect"] + interval: 5s + timeout: 5s + retries: 5 + networks: + - bbl-shared-net + labels: + com.centurylinklabs.watchtower.enable: "true" + + redis: + image: redis:7 + container_name: prefect-redis + restart: unless-stopped + volumes: + - redis_data:/data + healthcheck: + test: ["CMD-SHELL", "redis-cli ping"] + networks: + - bbl-shared-net + labels: + com.centurylinklabs.watchtower.enable: "true" + + prefect-server: + image: prefecthq/prefect:3-latest + container_name: prefect-server + restart: unless-stopped + depends_on: + postgres: + condition: service_healthy + redis: + condition: service_healthy + environment: + PREFECT_API_DATABASE_CONNECTION_URL: postgresql+asyncpg://prefect:OujaoPhaiS5Eig7ohy3TeiKeecahhe@postgres:5432/prefect + PREFECT_SERVER_API_HOST: 0.0.0.0 + PREFECT_SERVER_API_PORT: 4200 + PREFECT_UI_API_URL: http://prefect-server:4200/api + PREFECT_SERVER_CSRF_PROTECTION_ENABLED: "true" + PREFECT_SERVER_AUTHENTICATION_ENABLED: "true" + PREFECT_MESSAGING_BROKER: prefect_redis.messaging + PREFECT_MESSAGING_CACHE: prefect_redis.messaging + PREFECT_REDIS_MESSAGING_HOST: redis + PREFECT_REDIS_MESSAGING_PORT: 6379 + PREFECT_REDIS_MESSAGING_DB: 0 + command: prefect server start --no-services + ports: + - "4200:4200" + networks: + - bbl-shared-net + labels: + com.centurylinklabs.watchtower.enable: "true" + + prefect-services: + image: prefecthq/prefect:3-latest + container_name: prefect-services + restart: unless-stopped + depends_on: + postgres: + condition: service_healthy + redis: + condition: service_healthy + environment: + PREFECT_API_DATABASE_CONNECTION_URL: postgresql+asyncpg://prefect:OujaoPhaiS5Eig7ohy3TeiKeecahhe@postgres:5432/prefect + PREFECT_MESSAGING_BROKER: prefect_redis.messaging + PREFECT_MESSAGING_CACHE: prefect_redis.messaging + PREFECT_REDIS_MESSAGING_HOST: redis + PREFECT_REDIS_MESSAGING_PORT: 6379 + PREFECT_REDIS_MESSAGING_DB: 0 + command: prefect server services start + networks: + - bbl-shared-net + labels: + com.centurylinklabs.watchtower.enable: "true" + + prefect-worker: + image: prefecthq/prefect:3-latest + container_name: prefect-worker + restart: unless-stopped + depends_on: + prefect-server: + condition: service_started + environment: + PREFECT_API_URL: http://prefect-server:4200/api + command: prefect worker start --pool local-pool + networks: + - bbl-shared-net + labels: + com.centurylinklabs.watchtower.enable: "true" +``` + +- **Prefect UI**: `https://prefect.bitbylaw.com`. +- **API**: Über Kong `https://api.bitbylaw.com/prefect`. + +### Git-Server (Gitea) +`docker-compose-gitea.yml`: +```yaml +version: '3.8' + +networks: + bbl-shared-net: + external: true + +volumes: + gitea-data: {} + +services: + gitea: + image: gitea/gitea:latest + container_name: gitea + restart: always + environment: + - USER_UID=1000 + - USER_GID=1000 + - GITEA__server__ROOT_URL=http://git.bitbylaw.com:3000 + - GITEA__database__DB_TYPE=sqlite3 + - GITEA__server__SSH_DOMAIN=git.bitbylaw.com + volumes: + - gitea-data:/data + - /etc/timezone:/etc/timezone:ro + - /etc/localtime:/etc/localtime:ro + ports: + - "3000:3000" + - "2222:22" + networks: + - bbl-shared-net + labels: + com.centurylinklabs.watchtower.enable: "true" +``` + +- **Web-UI**: `https://git.bitbylaw.com`. +- **SSH**: `git@git.bitbylaw.com` (Port 2222). +- **Integration**: Flows werden in Repos wie `admin/prefect-flows` gepusht. + +### EspoCRM +`docker-compose-espocrm.yml`: +```yaml +version: '3.8' + +networks: + bbl-shared-net: + external: true + +volumes: + espocrm-db: + espocrm: + +services: + espocrm-db: + image: mariadb:latest + container_name: espocrm-db + environment: + MARIADB_ROOT_PASSWORD: root_password + MARIADB_DATABASE: espocrm + MARIADB_USER: espocrm + MARIADB_PASSWORD: database_password + volumes: + - espocrm-db:/var/lib/mysql + restart: always + healthcheck: + test: ["CMD", "healthcheck.sh", "--connect", "--innodb_initialized"] + interval: 20s + start_period: 10s + timeout: 10s + retries: 3 + networks: + - bbl-shared-net + labels: + com.centurylinklabs.watchtower.enable: "true" + + espocrm: + image: espocrm/espocrm:latest + container_name: espocrm + environment: + ESPOCRM_DATABASE_PLATFORM: Mysql + ESPOCRM_DATABASE_HOST: espocrm-db + ESPOCRM_DATABASE_USER: espocrm + ESPOCRM_DATABASE_PASSWORD: database_password + ESPOCRM_ADMIN_USERNAME: admin + ESPOCRM_ADMIN_PASSWORD: password + ESPOCRM_SITE_URL: "http://api.bitbylaw.com:5000/espocrm" + volumes: + - espocrm:/var/www/html + restart: always + depends_on: + espocrm-db: + condition: service_healthy + ports: + - "8080:80" + networks: + - bbl-shared-net + labels: + com.centurylinklabs.watchtower.enable: "true" + + espocrm-daemon: + image: espocrm/espocrm:latest + container_name: espocrm-daemon + volumes: + - espocrm:/var/www/html + restart: always + entrypoint: docker-daemon.sh + networks: + - bbl-shared-net + labels: + com.centurylinklabs.watchtower.enable: "true" + + espocrm-websocket: + image: espocrm/espocrm:latest + container_name: espocrm-websocket + environment: + ESPOCRM_CONFIG_USE_WEB_SOCKET: "true" + ESPOCRM_CONFIG_WEB_SOCKET_URL: "ws://api.bitbylaw.com:5000/espocrm/ws" + ESPOCRM_CONFIG_WEB_SOCKET_ZERO_M_Q_SUBSCRIBER_DSN: "tcp://*:7777" + ESPOCRM_CONFIG_WEB_SOCKET_ZERO_M_Q_SUBMISSION_DSN: "tcp://espocrm-websocket:7777" + volumes: + - espocrm:/var/www/html + restart: always + entrypoint: docker-websocket.sh + ports: + - "8081:8080" + networks: + - bbl-shared-net + labels: + com.centurylinklabs.watchtower.enable: "true" +``` + +- **UI**: `https://espocrm.bitbylaw.com`. +- **Integration**: Über Kong `https://api.bitbylaw.com/espocrm`. + +### Flows erstellen, ändern und deployen + +1. **Flow erstellen**: + - In VS Code: Öffne `~/bbl-code/prefect-flows`. + - Erstelle z.B. `flows/new_flow.py`: + ```python + from prefect import flow, task + + @task + def say_hello(name: str) -> str: + message = f"New Flow: Hello, {name}!" + print(message) + return message + + @flow(name="new-flow") + def new_flow(name: str = "New World"): + result = say_hello(name) + return result + + if __name__ == "__main__": + new_flow() + ``` + - Teste lokal: `python3 flows/new_flow.py`. + +2. **Flow ändern**: + - Bearbeite `flows/hello_world.py` oder `new_flow.py` in VS Code. + - Speichere und teste lokal. + +3. **Commit und Push**: + - In VS Code **Source Control** (Ctrl+Shift+G): + - Stage: `+` neben Datei. + - Commit: Nachricht eingeben (z.B. `Add new_flow.py`) → Häkchen. + - Push: `...` → **Push**. + - Alternativ Terminal: + ```bash + git add flows/new_flow.py + git commit -m "Add new_flow.py" + git push origin main + ``` + +4. **Deploy den Flow**: + - In VS Code Terminal: + ```bash + cd ~/bbl-code/prefect-flows + prefect deploy + ``` + - Wähle das Deployment aus `prefect.yaml` (z.B. `new-flow-deployment`). + - Prefect pullt den Code aus Gitea in den Worker. + +5. **Trigger den Flow**: + - Finde Deployment-ID: + ```bash + curl -i https://api.bitbylaw.com/prefect/deployments + ``` + - Trigger: + ```bash + curl -i -X POST \ + -H "Content-Type: application/json" \ + -d '{ + "name": "new-flow-run", + "parameters": {"name": "New World"}, + "state": {"type": "SCHEDULED"} + }' \ + https://api.bitbylaw.com/prefect/deployments//flow_runs + ``` + - Prüfe: + ```bash + curl -i https://api.bitbylaw.com/prefect/flow_runs/ + ``` + +### Sicherheit und Authentifizierung + +- **Kong**: `basic-auth` oder `key-auth` auf Service-Ebene (z.B. `prefect-service`). +- **Gitea**: SSH-Keys für passwortlosen Zugriff (siehe vorherige Anleitung). +- **Traefik**: SSL für alle URLs. +- **Prefect**: Aktiviere Authentifizierung in `PREFECT_SERVER_AUTHENTICATION_ENABLED`. + +### Troubleshooting + +- **Git**: Prüfe Logs in Gitea (`docker logs gitea`). +- **Prefect Deployment**: Worker-Logs (`docker logs prefect-worker`). +- **Flow-Run**: UI (`https://prefect.bitbylaw.com`) oder API-Logs. +- **Traefik**: Logs (`docker logs traefik`). + +Für Updates pushe Änderungen zu Gitea und deploye neu. \ No newline at end of file