bsiggel/prefect-flows
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity
Files
main
prefect-flows/README.md
Admin a56ded352f added detailed readme
2025-09-30 21:14:57 +00:00

13 KiB
Raw Permalink Blame History

BBL API-Gateway Setup mit Prefect, Kong, EspoCRM und Gitea

Dieses Repository enthält die Flows für Prefect und dient als zentrale Dokumentation für das gesamte Setup. Das System integriert einen API-Gateway (Kong) für Prefect und EspoCRM, einen Git-Server (Gitea) für Code-Management und Docker für die Container-Orchestrierung. Traefik handhabt das externe Routing (einschließlich SSL), und Portainer/Watchtower managen Updates.

Einführung

Dieses Setup ermöglicht:

  • Kong als API-Gateway: Routet Anfragen an Prefect (api.bitbylaw.com:5000/prefect) und EspoCRM (api.bitbylaw.com:5000/espocrm).
  • Prefect für Workflows: Flows werden in diesem Repository erstellt, geändert, gepusht, deployed und über die API getriggert.
  • Gitea als Git-Server: Zentrale Code-Speicherung für Prefect-Flows, integriert mit VS Code.
  • EspoCRM: CRM-System, integriert mit Kong für API-Zugriff.
  • Docker: Alle Komponenten laufen in Containern mit einem gemeinsamen Netzwerk (bbl-shared-net) für interne Kommunikation.
  • Traefik: Manuelle Konfiguration für URLs wie api.bitbylaw.com, ui.api.bitbylaw.com, git.bitbylaw.com, prefect.bitbylaw.com, espocrm.bitbylaw.com.
  • Sicherheit: Authentifizierung über Kong (z.B. basic-auth oder key-auth), SSL via Traefik, SSH-Keys für Git.

Voraussetzungen

  • VM/Host: Ubuntu/Debian mit Docker, Docker Compose, Portainer, Watchtower und Traefik installiert.
  • Netzwerk: Externes Docker-Netzwerk bbl-shared-net (docker network create bbl-shared-net).
  • Domains: api.bitbylaw.com (Kong Proxy), ui.api.bitbylaw.com (Kong Manager), git.bitbylaw.com (Gitea), prefect.bitbylaw.com (Prefect UI), espocrm.bitbylaw.com (EspoCRM).
  • VS Code: Installiert auf der VM, mit Python-Extension und GitLens für Git-Integration.
  • Python: 3.8+ in einer virtuellen Umgebung (venv unter ~/bbl-code).
  • Gitea-Credentials: Admin-User (admin:strongpass).
  • Prefect-API: Ohne Auth (für Tests); für Produktion basic-auth aktivieren.

Docker-Konfiguration

Alle Container laufen in bbl-shared-net für interne Kommunikation. Watchtower-Labels für automatische Updates. Ports werden über Traefik geroutet.

Kong (API-Gateway)

docker-compose-kong.yml:

version: '3.8'

networks:
  bbl-shared-net:
    external: true

volumes:
  kong_data: {}

services:
  kong-database:
    image: postgres:16
    restart: always
    networks:
      - bbl-shared-net
    environment:
      POSTGRES_USER: kong
      POSTGRES_PASSWORD: kong
      POSTGRES_DB: kong
    volumes:
      - kong_data:/var/lib/postgresql/data
    ports:
      - "5432:5432"
    healthcheck:
      test: ["CMD", "pg_isready", "-U", "kong"]
      interval: 5s
      timeout: 5s
      retries: 5
    labels:
      com.centurylinklabs.watchtower.enable: "true"

  kong-migration:
    image: kong:latest
    command: "kong migrations bootstrap"
    networks:
      - bbl-shared-net
    restart: on-failure
    environment:
      KONG_PG_HOST: kong-database
      KONG_PG_USER: kong
      KONG_PG_PASSWORD: kong
      KONG_PG_DATABASE: kong
      KONG_DATABASE: postgres
      KONG_LOG_LEVEL: debug
    depends_on:
      kong-database:
        condition: service_healthy

  kong:
    image: kong:latest
    restart: always
    networks:
      - bbl-shared-net
    environment:
      KONG_PG_HOST: kong-database
      KONG_PG_USER: kong
      KONG_PG_PASSWORD: kong
      KONG_PG_DATABASE: kong
      KONG_DATABASE: postgres
      KONG_PG_TIMEOUT: 10000
      KONG_PROXY_LISTEN: 0.0.0.0:5000
      KONG_ADMIN_LISTEN: 0.0.0.0:8001
      KONG_ADMIN_GUI_LISTEN: 0.0.0.0:5001
      KONG_PORTAL: on
      KONG_ADMIN_GUI_AUTH: basic-auth
      KONG_ADMIN_GUI_USERNAME: admin
      KONG_ADMIN_GUI_PASSWORD: adminpass
      KONG_LOG_LEVEL: debug
    depends_on:
      - kong-migration
      - kong-database
    healthcheck:
      test: ["CMD", "kong", "health"]
      interval: 5s
      timeout: 5s
      retries: 20
    ports:
      - "5000:5000"
      - "8001:8001"
      - "5001:5001"
    labels:
      com.centurylinklabs.watchtower.enable: "true"
  • Konfiguration in Kong Manager: Services und Routes für Prefect (prefect-service: http://prefect-server:4200/api, Route: /prefect) und EspoCRM (espocrm-service: http://espocrm:80, Route: /espocrm).
  • Authentifizierung: basic-auth oder key-auth auf Service-Ebene (z.B. für prefect-service).

Prefect (Workflow-Management)

docker-compose-prefect.yml:

version: '3.8'

networks:
  bbl-shared-net:
    external: true

volumes:
  postgres_data:
  redis_data:

services:
  postgres:
    image: postgres:14
    container_name: prefect-postgres
    restart: unless-stopped
    environment:
      POSTGRES_USER: prefect
      POSTGRES_PASSWORD: OujaoPhaiS5Eig7ohy3TeiKeecahhe
      POSTGRES_DB: prefect
    volumes:
      - postgres_data:/var/lib/postgresql/data
    healthcheck:
      test: ["CMD-SHELL", "pg_isready -U prefect"]
      interval: 5s
      timeout: 5s
      retries: 5
    networks:
      - bbl-shared-net
    labels:
      com.centurylinklabs.watchtower.enable: "true"

  redis:
    image: redis:7
    container_name: prefect-redis
    restart: unless-stopped
    volumes:
      - redis_data:/data
    healthcheck:
      test: ["CMD-SHELL", "redis-cli ping"]
    networks:
      - bbl-shared-net
    labels:
      com.centurylinklabs.watchtower.enable: "true"

  prefect-server:
    image: prefecthq/prefect:3-latest
    container_name: prefect-server
    restart: unless-stopped
    depends_on:
      postgres:
        condition: service_healthy
      redis:
        condition: service_healthy
    environment:
      PREFECT_API_DATABASE_CONNECTION_URL: postgresql+asyncpg://prefect:OujaoPhaiS5Eig7ohy3TeiKeecahhe@postgres:5432/prefect
      PREFECT_SERVER_API_HOST: 0.0.0.0
      PREFECT_SERVER_API_PORT: 4200
      PREFECT_UI_API_URL: http://prefect-server:4200/api
      PREFECT_SERVER_CSRF_PROTECTION_ENABLED: "true"
      PREFECT_SERVER_AUTHENTICATION_ENABLED: "true"
      PREFECT_MESSAGING_BROKER: prefect_redis.messaging
      PREFECT_MESSAGING_CACHE: prefect_redis.messaging
      PREFECT_REDIS_MESSAGING_HOST: redis
      PREFECT_REDIS_MESSAGING_PORT: 6379
      PREFECT_REDIS_MESSAGING_DB: 0
    command: prefect server start --no-services
    ports:
      - "4200:4200"
    networks:
      - bbl-shared-net
    labels:
      com.centurylinklabs.watchtower.enable: "true"

  prefect-services:
    image: prefecthq/prefect:3-latest
    container_name: prefect-services
    restart: unless-stopped
    depends_on:
      postgres:
        condition: service_healthy
      redis:
        condition: service_healthy
    environment:
      PREFECT_API_DATABASE_CONNECTION_URL: postgresql+asyncpg://prefect:OujaoPhaiS5Eig7ohy3TeiKeecahhe@postgres:5432/prefect
      PREFECT_MESSAGING_BROKER: prefect_redis.messaging
      PREFECT_MESSAGING_CACHE: prefect_redis.messaging
      PREFECT_REDIS_MESSAGING_HOST: redis
      PREFECT_REDIS_MESSAGING_PORT: 6379
      PREFECT_REDIS_MESSAGING_DB: 0
    command: prefect server services start
    networks:
      - bbl-shared-net
    labels:
      com.centurylinklabs.watchtower.enable: "true"

  prefect-worker:
    image: prefecthq/prefect:3-latest
    container_name: prefect-worker
    restart: unless-stopped
    depends_on:
      prefect-server:
        condition: service_started
    environment:
      PREFECT_API_URL: http://prefect-server:4200/api
    command: prefect worker start --pool local-pool
    networks:
      - bbl-shared-net
    labels:
      com.centurylinklabs.watchtower.enable: "true"
  • Prefect UI: https://prefect.bitbylaw.com.
  • API: Über Kong https://api.bitbylaw.com/prefect.

Git-Server (Gitea)

docker-compose-gitea.yml:

version: '3.8'

networks:
  bbl-shared-net:
    external: true

volumes:
  gitea-data: {}

services:
  gitea:
    image: gitea/gitea:latest
    container_name: gitea
    restart: always
    environment:
      - USER_UID=1000
      - USER_GID=1000
      - GITEA__server__ROOT_URL=http://git.bitbylaw.com:3000
      - GITEA__database__DB_TYPE=sqlite3
      - GITEA__server__SSH_DOMAIN=git.bitbylaw.com
    volumes:
      - gitea-data:/data
      - /etc/timezone:/etc/timezone:ro
      - /etc/localtime:/etc/localtime:ro
    ports:
      - "3000:3000"
      - "2222:22"
    networks:
      - bbl-shared-net
    labels:
      com.centurylinklabs.watchtower.enable: "true"
  • Web-UI: https://git.bitbylaw.com.
  • SSH: git@git.bitbylaw.com (Port 2222).
  • Integration: Flows werden in Repos wie admin/prefect-flows gepusht.

EspoCRM

docker-compose-espocrm.yml:

version: '3.8'

networks:
  bbl-shared-net:
    external: true

volumes:
  espocrm-db:
  espocrm:

services:
  espocrm-db:
    image: mariadb:latest
    container_name: espocrm-db
    environment:
      MARIADB_ROOT_PASSWORD: root_password
      MARIADB_DATABASE: espocrm
      MARIADB_USER: espocrm
      MARIADB_PASSWORD: database_password
    volumes:
      - espocrm-db:/var/lib/mysql
    restart: always
    healthcheck:
      test: ["CMD", "healthcheck.sh", "--connect", "--innodb_initialized"]
      interval: 20s
      start_period: 10s
      timeout: 10s
      retries: 3
    networks:
      - bbl-shared-net
    labels:
      com.centurylinklabs.watchtower.enable: "true"

  espocrm:
    image: espocrm/espocrm:latest
    container_name: espocrm
    environment:
      ESPOCRM_DATABASE_PLATFORM: Mysql
      ESPOCRM_DATABASE_HOST: espocrm-db
      ESPOCRM_DATABASE_USER: espocrm
      ESPOCRM_DATABASE_PASSWORD: database_password
      ESPOCRM_ADMIN_USERNAME: admin
      ESPOCRM_ADMIN_PASSWORD: password
      ESPOCRM_SITE_URL: "http://api.bitbylaw.com:5000/espocrm"
    volumes:
      - espocrm:/var/www/html
    restart: always
    depends_on:
      espocrm-db:
        condition: service_healthy
    ports:
      - "8080:80"
    networks:
      - bbl-shared-net
    labels:
      com.centurylinklabs.watchtower.enable: "true"

  espocrm-daemon:
    image: espocrm/espocrm:latest
    container_name: espocrm-daemon
    volumes:
      - espocrm:/var/www/html
    restart: always
    entrypoint: docker-daemon.sh
    networks:
      - bbl-shared-net
    labels:
      com.centurylinklabs.watchtower.enable: "true"

  espocrm-websocket:
    image: espocrm/espocrm:latest
    container_name: espocrm-websocket
    environment:
      ESPOCRM_CONFIG_USE_WEB_SOCKET: "true"
      ESPOCRM_CONFIG_WEB_SOCKET_URL: "ws://api.bitbylaw.com:5000/espocrm/ws"
      ESPOCRM_CONFIG_WEB_SOCKET_ZERO_M_Q_SUBSCRIBER_DSN: "tcp://*:7777"
      ESPOCRM_CONFIG_WEB_SOCKET_ZERO_M_Q_SUBMISSION_DSN: "tcp://espocrm-websocket:7777"
    volumes:
      - espocrm:/var/www/html
    restart: always
    entrypoint: docker-websocket.sh
    ports:
      - "8081:8080"
    networks:
      - bbl-shared-net
    labels:
      com.centurylinklabs.watchtower.enable: "true"
  • UI: https://espocrm.bitbylaw.com.
  • Integration: Über Kong https://api.bitbylaw.com/espocrm.

Flows erstellen, ändern und deployen

  1. Flow erstellen:

    • In VS Code: Öffne ~/bbl-code/prefect-flows.
    • Erstelle z.B. flows/new_flow.py: 
      from prefect import flow, task

@task
def say_hello(name: str) -> str:
    message = f"New Flow: Hello, {name}!"
    print(message)
    return message

@flow(name="new-flow")
def new_flow(name: str = "New World"):
    result = say_hello(name)
    return result

if __name__ == "__main__":
    new_flow()
    • Teste lokal: python3 flows/new_flow.py.

  2. Flow ändern:

    • Bearbeite flows/hello_world.py oder new_flow.py in VS Code.
    • Speichere und teste lokal.

  3. Commit und Push:

    • In VS Code Source Control (Ctrl+Shift+G):
      • Stage: + neben Datei.
      • Commit: Nachricht eingeben (z.B. Add new_flow.py) → Häkchen.
      • Push: ... → Push.
    • Alternativ Terminal: 
      git add flows/new_flow.py
git commit -m "Add new_flow.py"
git push origin main

  4. Deploy den Flow:

    • In VS Code Terminal: 
      cd ~/bbl-code/prefect-flows
prefect deploy
    • Wähle das Deployment aus prefect.yaml (z.B. new-flow-deployment).
    • Prefect pullt den Code aus Gitea in den Worker.

  5. Trigger den Flow:

    • Finde Deployment-ID: 
      curl -i https://api.bitbylaw.com/prefect/deployments
    • Trigger: 
      curl -i -X POST \
  -H "Content-Type: application/json" \
  -d '{
    "name": "new-flow-run",
    "parameters": {"name": "New World"},
    "state": {"type": "SCHEDULED"}
  }' \
  https://api.bitbylaw.com/prefect/deployments/<deployment-id>/flow_runs
    • Prüfe: 
      curl -i https://api.bitbylaw.com/prefect/flow_runs/<flow-run-id>

Sicherheit und Authentifizierung

  • Kong: basic-auth oder key-auth auf Service-Ebene (z.B. prefect-service).
  • Gitea: SSH-Keys für passwortlosen Zugriff (siehe vorherige Anleitung).
  • Traefik: SSL für alle URLs.
  • Prefect: Aktiviere Authentifizierung in PREFECT_SERVER_AUTHENTICATION_ENABLED.

Troubleshooting

  • Git: Prüfe Logs in Gitea (docker logs gitea).
  • Prefect Deployment: Worker-Logs (docker logs prefect-worker).
  • Flow-Run: UI (https://prefect.bitbylaw.com) oder API-Logs.
  • Traefik: Logs (docker logs traefik).

Für Updates pushe Änderungen zu Gitea und deploye neu.

Reference in New Issue View Git Blame Copy Permalink