BBL API-Gateway Setup mit Prefect, Kong, EspoCRM und Gitea

Dieses Repository enthält die Flows für Prefect und dient als zentrale Dokumentation für das gesamte Setup. Das System integriert einen API-Gateway (Kong) für Prefect und EspoCRM, einen Git-Server (Gitea) für Code-Management und Docker für die Container-Orchestrierung. Traefik handhabt das externe Routing (einschließlich SSL), und Portainer/Watchtower managen Updates.

Einführung

Dieses Setup ermöglicht:

• Kong als API-Gateway: Routet Anfragen an Prefect (api.bitbylaw.com:5000/prefect) und EspoCRM (api.bitbylaw.com:5000/espocrm).
• Prefect für Workflows: Flows werden in diesem Repository erstellt, geändert, gepusht, deployed und über die API getriggert.
• Gitea als Git-Server: Zentrale Code-Speicherung für Prefect-Flows, integriert mit VS Code.
• EspoCRM: CRM-System, integriert mit Kong für API-Zugriff.
• Docker: Alle Komponenten laufen in Containern mit einem gemeinsamen Netzwerk (bbl-shared-net) für interne Kommunikation.
• Traefik: Manuelle Konfiguration für URLs wie api.bitbylaw.com, ui.api.bitbylaw.com, git.bitbylaw.com, prefect.bitbylaw.com, espocrm.bitbylaw.com.
• Sicherheit: Authentifizierung über Kong (z.B. basic-auth oder key-auth), SSL via Traefik, SSH-Keys für Git.

Voraussetzungen

• VM/Host: Ubuntu/Debian mit Docker, Docker Compose, Portainer, Watchtower und Traefik installiert.
• Netzwerk: Externes Docker-Netzwerk bbl-shared-net (docker network create bbl-shared-net).
• Domains: api.bitbylaw.com (Kong Proxy), ui.api.bitbylaw.com (Kong Manager), git.bitbylaw.com (Gitea), prefect.bitbylaw.com (Prefect UI), espocrm.bitbylaw.com (EspoCRM).
• VS Code: Installiert auf der VM, mit Python-Extension und GitLens für Git-Integration.
• Python: 3.8+ in einer virtuellen Umgebung (venv unter ~/bbl-code).
• Gitea-Credentials: Admin-User (admin:strongpass).
• Prefect-API: Ohne Auth (für Tests); für Produktion basic-auth aktivieren.

Docker-Konfiguration

Alle Container laufen in bbl-shared-net für interne Kommunikation. Watchtower-Labels für automatische Updates. Ports werden über Traefik geroutet.

Kong (API-Gateway)

docker-compose-kong.yml:

version: '3.8'

networks:
  bbl-shared-net:
    external: true

volumes:
  kong_data: {}

services:
  kong-database:
    image: postgres:16
    restart: always
    networks:
      - bbl-shared-net
    environment:
      POSTGRES_USER: kong
      POSTGRES_PASSWORD: kong
      POSTGRES_DB: kong
    volumes:
      - kong_data:/var/lib/postgresql/data
    ports:
      - "5432:5432"
    healthcheck:
      test: ["CMD", "pg_isready", "-U", "kong"]
      interval: 5s
      timeout: 5s
      retries: 5
    labels:
      com.centurylinklabs.watchtower.enable: "true"

  kong-migration:
    image: kong:latest
    command: "kong migrations bootstrap"
    networks:
      - bbl-shared-net
    restart: on-failure
    environment:
      KONG_PG_HOST: kong-database
      KONG_PG_USER: kong
      KONG_PG_PASSWORD: kong
      KONG_PG_DATABASE: kong
      KONG_DATABASE: postgres
      KONG_LOG_LEVEL: debug
    depends_on:
      kong-database:
        condition: service_healthy

  kong:
    image: kong:latest
    restart: always
    networks:
      - bbl-shared-net
    environment:
      KONG_PG_HOST: kong-database
      KONG_PG_USER: kong
      KONG_PG_PASSWORD: kong
      KONG_PG_DATABASE: kong
      KONG_DATABASE: postgres
      KONG_PG_TIMEOUT: 10000
      KONG_PROXY_LISTEN: 0.0.0.0:5000
      KONG_ADMIN_LISTEN: 0.0.0.0:8001
      KONG_ADMIN_GUI_LISTEN: 0.0.0.0:5001
      KONG_PORTAL: on
      KONG_ADMIN_GUI_AUTH: basic-auth
      KONG_ADMIN_GUI_USERNAME: admin
      KONG_ADMIN_GUI_PASSWORD: adminpass
      KONG_LOG_LEVEL: debug
    depends_on:
      - kong-migration
      - kong-database
    healthcheck:
      test: ["CMD", "kong", "health"]
      interval: 5s
      timeout: 5s
      retries: 20
    ports:
      - "5000:5000"
      - "8001:8001"
      - "5001:5001"
    labels:
      com.centurylinklabs.watchtower.enable: "true"

• Konfiguration in Kong Manager: Services und Routes für Prefect (prefect-service: http://prefect-server:4200/api, Route: /prefect) und EspoCRM (espocrm-service: http://espocrm:80, Route: /espocrm).
• Authentifizierung: basic-auth oder key-auth auf Service-Ebene (z.B. für prefect-service).

Prefect (Workflow-Management)

docker-compose-prefect.yml:

version: '3.8'

networks:
  bbl-shared-net:
    external: true

volumes:
  postgres_data:
  redis_data:

services:
  postgres:
    image: postgres:14
    container_name: prefect-postgres
    restart: unless-stopped
    environment:
      POSTGRES_USER: prefect
      POSTGRES_PASSWORD: OujaoPhaiS5Eig7ohy3TeiKeecahhe
      POSTGRES_DB: prefect
    volumes:
      - postgres_data:/var/lib/postgresql/data
    healthcheck:
      test: ["CMD-SHELL", "pg_isready -U prefect"]
      interval: 5s
      timeout: 5s
      retries: 5
    networks:
      - bbl-shared-net
    labels:
      com.centurylinklabs.watchtower.enable: "true"

  redis:
    image: redis:7
    container_name: prefect-redis
    restart: unless-stopped
    volumes:
      - redis_data:/data
    healthcheck:
      test: ["CMD-SHELL", "redis-cli ping"]
    networks:
      - bbl-shared-net
    labels:
      com.centurylinklabs.watchtower.enable: "true"

  prefect-server:
    image: prefecthq/prefect:3-latest
    container_name: prefect-server
    restart: unless-stopped
    depends_on:
      postgres:
        condition: service_healthy
      redis:
        condition: service_healthy
    environment:
      PREFECT_API_DATABASE_CONNECTION_URL: postgresql+asyncpg://prefect:OujaoPhaiS5Eig7ohy3TeiKeecahhe@postgres:5432/prefect
      PREFECT_SERVER_API_HOST: 0.0.0.0
      PREFECT_SERVER_API_PORT: 4200
      PREFECT_UI_API_URL: http://prefect-server:4200/api
      PREFECT_SERVER_CSRF_PROTECTION_ENABLED: "true"
      PREFECT_SERVER_AUTHENTICATION_ENABLED: "true"
      PREFECT_MESSAGING_BROKER: prefect_redis.messaging
      PREFECT_MESSAGING_CACHE: prefect_redis.messaging
      PREFECT_REDIS_MESSAGING_HOST: redis
      PREFECT_REDIS_MESSAGING_PORT: 6379
      PREFECT_REDIS_MESSAGING_DB: 0
    command: prefect server start --no-services
    ports:
      - "4200:4200"
    networks:
      - bbl-shared-net
    labels:
      com.centurylinklabs.watchtower.enable: "true"

  prefect-services:
    image: prefecthq/prefect:3-latest
    container_name: prefect-services
    restart: unless-stopped
    depends_on:
      postgres:
        condition: service_healthy
      redis:
        condition: service_healthy
    environment:
      PREFECT_API_DATABASE_CONNECTION_URL: postgresql+asyncpg://prefect:OujaoPhaiS5Eig7ohy3TeiKeecahhe@postgres:5432/prefect
      PREFECT_MESSAGING_BROKER: prefect_redis.messaging
      PREFECT_MESSAGING_CACHE: prefect_redis.messaging
      PREFECT_REDIS_MESSAGING_HOST: redis
      PREFECT_REDIS_MESSAGING_PORT: 6379
      PREFECT_REDIS_MESSAGING_DB: 0
    command: prefect server services start
    networks:
      - bbl-shared-net
    labels:
      com.centurylinklabs.watchtower.enable: "true"

  prefect-worker:
    image: prefecthq/prefect:3-latest
    container_name: prefect-worker
    restart: unless-stopped
    depends_on:
      prefect-server:
        condition: service_started
    environment:
      PREFECT_API_URL: http://prefect-server:4200/api
    command: prefect worker start --pool local-pool
    networks:
      - bbl-shared-net
    labels:
      com.centurylinklabs.watchtower.enable: "true"

• Prefect UI: https://prefect.bitbylaw.com.
• API: Über Kong https://api.bitbylaw.com/prefect.

Git-Server (Gitea)

docker-compose-gitea.yml:

version: '3.8'

networks:
  bbl-shared-net:
    external: true

volumes:
  gitea-data: {}

services:
  gitea:
    image: gitea/gitea:latest
    container_name: gitea
    restart: always
    environment:
      - USER_UID=1000
      - USER_GID=1000
      - GITEA__server__ROOT_URL=http://git.bitbylaw.com:3000
      - GITEA__database__DB_TYPE=sqlite3
      - GITEA__server__SSH_DOMAIN=git.bitbylaw.com
    volumes:
      - gitea-data:/data
      - /etc/timezone:/etc/timezone:ro
      - /etc/localtime:/etc/localtime:ro
    ports:
      - "3000:3000"
      - "2222:22"
    networks:
      - bbl-shared-net
    labels:
      com.centurylinklabs.watchtower.enable: "true"

• Web-UI: https://git.bitbylaw.com.
• SSH: git@git.bitbylaw.com (Port 2222).
• Integration: Flows werden in Repos wie admin/prefect-flows gepusht.

EspoCRM

docker-compose-espocrm.yml:

version: '3.8'

networks:
  bbl-shared-net:
    external: true

volumes:
  espocrm-db:
  espocrm:

services:
  espocrm-db:
    image: mariadb:latest
    container_name: espocrm-db
    environment:
      MARIADB_ROOT_PASSWORD: root_password
      MARIADB_DATABASE: espocrm
      MARIADB_USER: espocrm
      MARIADB_PASSWORD: database_password
    volumes:
      - espocrm-db:/var/lib/mysql
    restart: always
    healthcheck:
      test: ["CMD", "healthcheck.sh", "--connect", "--innodb_initialized"]
      interval: 20s
      start_period: 10s
      timeout: 10s
      retries: 3
    networks:
      - bbl-shared-net
    labels:
      com.centurylinklabs.watchtower.enable: "true"

  espocrm:
    image: espocrm/espocrm:latest
    container_name: espocrm
    environment:
      ESPOCRM_DATABASE_PLATFORM: Mysql
      ESPOCRM_DATABASE_HOST: espocrm-db
      ESPOCRM_DATABASE_USER: espocrm
      ESPOCRM_DATABASE_PASSWORD: database_password
      ESPOCRM_ADMIN_USERNAME: admin
      ESPOCRM_ADMIN_PASSWORD: password
      ESPOCRM_SITE_URL: "http://api.bitbylaw.com:5000/espocrm"
    volumes:
      - espocrm:/var/www/html
    restart: always
    depends_on:
      espocrm-db:
        condition: service_healthy
    ports:
      - "8080:80"
    networks:
      - bbl-shared-net
    labels:
      com.centurylinklabs.watchtower.enable: "true"

  espocrm-daemon:
    image: espocrm/espocrm:latest
    container_name: espocrm-daemon
    volumes:
      - espocrm:/var/www/html
    restart: always
    entrypoint: docker-daemon.sh
    networks:
      - bbl-shared-net
    labels:
      com.centurylinklabs.watchtower.enable: "true"

  espocrm-websocket:
    image: espocrm/espocrm:latest
    container_name: espocrm-websocket
    environment:
      ESPOCRM_CONFIG_USE_WEB_SOCKET: "true"
      ESPOCRM_CONFIG_WEB_SOCKET_URL: "ws://api.bitbylaw.com:5000/espocrm/ws"
      ESPOCRM_CONFIG_WEB_SOCKET_ZERO_M_Q_SUBSCRIBER_DSN: "tcp://*:7777"
      ESPOCRM_CONFIG_WEB_SOCKET_ZERO_M_Q_SUBMISSION_DSN: "tcp://espocrm-websocket:7777"
    volumes:
      - espocrm:/var/www/html
    restart: always
    entrypoint: docker-websocket.sh
    ports:
      - "8081:8080"
    networks:
      - bbl-shared-net
    labels:
      com.centurylinklabs.watchtower.enable: "true"

• UI: https://espocrm.bitbylaw.com.
• Integration: Über Kong https://api.bitbylaw.com/espocrm.

Flows erstellen, ändern und deployen

1. Flow erstellen:

   • In VS Code: Öffne ~/bbl-code/prefect-flows.
   • Erstelle z.B. flows/new_flow.py:

     from prefect import flow, task
     
     @task
     def say_hello(name: str) -> str:
         message = f"New Flow: Hello, {name}!"
         print(message)
         return message
     
     @flow(name="new-flow")
     def new_flow(name: str = "New World"):
         result = say_hello(name)
         return result
     
     if __name__ == "__main__":
         new_flow()
     

   • Teste lokal: python3 flows/new_flow.py.

2. Flow ändern:

   • Bearbeite flows/hello_world.py oder new_flow.py in VS Code.
   • Speichere und teste lokal.

3. Commit und Push:

   • In VS Code Source Control (Ctrl+Shift+G):
     • Stage: + neben Datei.
     • Commit: Nachricht eingeben (z.B. Add new_flow.py) → Häkchen.
     • Push: ... → Push.
   • Alternativ Terminal:

     git add flows/new_flow.py
     git commit -m "Add new_flow.py"
     git push origin main
     

4. Deploy den Flow:

   • In VS Code Terminal:

     cd ~/bbl-code/prefect-flows
     prefect deploy
     

   • Wähle das Deployment aus prefect.yaml (z.B. new-flow-deployment).
   • Prefect pullt den Code aus Gitea in den Worker.

5. Trigger den Flow:

   • Finde Deployment-ID:

     curl -i https://api.bitbylaw.com/prefect/deployments
     

   • Trigger:

     curl -i -X POST \
       -H "Content-Type: application/json" \
       -d '{
         "name": "new-flow-run",
         "parameters": {"name": "New World"},
         "state": {"type": "SCHEDULED"}
       }' \
       https://api.bitbylaw.com/prefect/deployments/<deployment-id>/flow_runs
     

   • Prüfe:

     curl -i https://api.bitbylaw.com/prefect/flow_runs/<flow-run-id>
     

Sicherheit und Authentifizierung

• Kong: basic-auth oder key-auth auf Service-Ebene (z.B. prefect-service).
• Gitea: SSH-Keys für passwortlosen Zugriff (siehe vorherige Anleitung).
• Traefik: SSL für alle URLs.
• Prefect: Aktiviere Authentifizierung in PREFECT_SERVER_AUTHENTICATION_ENABLED.

Troubleshooting

• Git: Prüfe Logs in Gitea (docker logs gitea).
• Prefect Deployment: Worker-Logs (docker logs prefect-worker).
• Flow-Run: UI (https://prefect.bitbylaw.com) oder API-Logs.
• Traefik: Logs (docker logs traefik).

Für Updates pushe Änderungen zu Gitea und deploye neu.