# BBL API-Gateway Setup mit Prefect, Kong, EspoCRM und Gitea

Dieses Repository enthält die Flows für Prefect und dient als zentrale Dokumentation für das gesamte Setup. Das System integriert einen API-Gateway (Kong) für Prefect und EspoCRM, einen Git-Server (Gitea) für Code-Management und Docker für die Container-Orchestrierung. Traefik handhabt das externe Routing (einschließlich SSL), und Portainer/Watchtower managen Updates.

## Einführung

Dieses Setup ermöglicht:
- **Kong als API-Gateway**: Routet Anfragen an Prefect (`api.bitbylaw.com:5000/prefect`) und EspoCRM (`api.bitbylaw.com:5000/espocrm`).
- **Prefect für Workflows**: Flows werden in diesem Repository erstellt, geändert, gepusht, deployed und über die API getriggert.
- **Gitea als Git-Server**: Zentrale Code-Speicherung für Prefect-Flows, integriert mit VS Code.
- **EspoCRM**: CRM-System, integriert mit Kong für API-Zugriff.
- **Docker**: Alle Komponenten laufen in Containern mit einem gemeinsamen Netzwerk (`bbl-shared-net`) für interne Kommunikation.
- **Traefik**: Manuelle Konfiguration für URLs wie `api.bitbylaw.com`, `ui.api.bitbylaw.com`, `git.bitbylaw.com`, `prefect.bitbylaw.com`, `espocrm.bitbylaw.com`.
- **Sicherheit**: Authentifizierung über Kong (z.B. `basic-auth` oder `key-auth`), SSL via Traefik, SSH-Keys für Git.

## Voraussetzungen

- **VM/Host**: Ubuntu/Debian mit Docker, Docker Compose, Portainer, Watchtower und Traefik installiert.
- **Netzwerk**: Externes Docker-Netzwerk `bbl-shared-net` (`docker network create bbl-shared-net`).
- **Domains**: `api.bitbylaw.com` (Kong Proxy), `ui.api.bitbylaw.com` (Kong Manager), `git.bitbylaw.com` (Gitea), `prefect.bitbylaw.com` (Prefect UI), `espocrm.bitbylaw.com` (EspoCRM).
- **VS Code**: Installiert auf der VM, mit Python-Extension und GitLens für Git-Integration.
- **Python**: 3.8+ in einer virtuellen Umgebung (`venv` unter `~/bbl-code`).
- **Gitea-Credentials**: Admin-User (`admin:strongpass`).
- **Prefect-API**: Ohne Auth (für Tests); für Produktion `basic-auth` aktivieren.

## Docker-Konfiguration

Alle Container laufen in `bbl-shared-net` für interne Kommunikation. Watchtower-Labels für automatische Updates. Ports werden über Traefik geroutet.

### Kong (API-Gateway)
`docker-compose-kong.yml`:
```yaml
version: '3.8'

networks:
  bbl-shared-net:
    external: true

volumes:
  kong_data: {}

services:
  kong-database:
    image: postgres:16
    restart: always
    networks:
      - bbl-shared-net
    environment:
      POSTGRES_USER: kong
      POSTGRES_PASSWORD: kong
      POSTGRES_DB: kong
    volumes:
      - kong_data:/var/lib/postgresql/data
    ports:
      - "5432:5432"
    healthcheck:
      test: ["CMD", "pg_isready", "-U", "kong"]
      interval: 5s
      timeout: 5s
      retries: 5
    labels:
      com.centurylinklabs.watchtower.enable: "true"

  kong-migration:
    image: kong:latest
    command: "kong migrations bootstrap"
    networks:
      - bbl-shared-net
    restart: on-failure
    environment:
      KONG_PG_HOST: kong-database
      KONG_PG_USER: kong
      KONG_PG_PASSWORD: kong
      KONG_PG_DATABASE: kong
      KONG_DATABASE: postgres
      KONG_LOG_LEVEL: debug
    depends_on:
      kong-database:
        condition: service_healthy

  kong:
    image: kong:latest
    restart: always
    networks:
      - bbl-shared-net
    environment:
      KONG_PG_HOST: kong-database
      KONG_PG_USER: kong
      KONG_PG_PASSWORD: kong
      KONG_PG_DATABASE: kong
      KONG_DATABASE: postgres
      KONG_PG_TIMEOUT: 10000
      KONG_PROXY_LISTEN: 0.0.0.0:5000
      KONG_ADMIN_LISTEN: 0.0.0.0:8001
      KONG_ADMIN_GUI_LISTEN: 0.0.0.0:5001
      KONG_PORTAL: on
      KONG_ADMIN_GUI_AUTH: basic-auth
      KONG_ADMIN_GUI_USERNAME: admin
      KONG_ADMIN_GUI_PASSWORD: adminpass
      KONG_LOG_LEVEL: debug
    depends_on:
      - kong-migration
      - kong-database
    healthcheck:
      test: ["CMD", "kong", "health"]
      interval: 5s
      timeout: 5s
      retries: 20
    ports:
      - "5000:5000"
      - "8001:8001"
      - "5001:5001"
    labels:
      com.centurylinklabs.watchtower.enable: "true"
```

- **Konfiguration in Kong Manager**: Services und Routes für Prefect (`prefect-service`: `http://prefect-server:4200/api`, Route: `/prefect`) und EspoCRM (`espocrm-service`: `http://espocrm:80`, Route: `/espocrm`).
- **Authentifizierung**: `basic-auth` oder `key-auth` auf Service-Ebene (z.B. für `prefect-service`).

### Prefect (Workflow-Management)
`docker-compose-prefect.yml`:
```yaml
version: '3.8'

networks:
  bbl-shared-net:
    external: true

volumes:
  postgres_data:
  redis_data:

services:
  postgres:
    image: postgres:14
    container_name: prefect-postgres
    restart: unless-stopped
    environment:
      POSTGRES_USER: prefect
      POSTGRES_PASSWORD: OujaoPhaiS5Eig7ohy3TeiKeecahhe
      POSTGRES_DB: prefect
    volumes:
      - postgres_data:/var/lib/postgresql/data
    healthcheck:
      test: ["CMD-SHELL", "pg_isready -U prefect"]
      interval: 5s
      timeout: 5s
      retries: 5
    networks:
      - bbl-shared-net
    labels:
      com.centurylinklabs.watchtower.enable: "true"

  redis:
    image: redis:7
    container_name: prefect-redis
    restart: unless-stopped
    volumes:
      - redis_data:/data
    healthcheck:
      test: ["CMD-SHELL", "redis-cli ping"]
    networks:
      - bbl-shared-net
    labels:
      com.centurylinklabs.watchtower.enable: "true"

  prefect-server:
    image: prefecthq/prefect:3-latest
    container_name: prefect-server
    restart: unless-stopped
    depends_on:
      postgres:
        condition: service_healthy
      redis:
        condition: service_healthy
    environment:
      PREFECT_API_DATABASE_CONNECTION_URL: postgresql+asyncpg://prefect:OujaoPhaiS5Eig7ohy3TeiKeecahhe@postgres:5432/prefect
      PREFECT_SERVER_API_HOST: 0.0.0.0
      PREFECT_SERVER_API_PORT: 4200
      PREFECT_UI_API_URL: http://prefect-server:4200/api
      PREFECT_SERVER_CSRF_PROTECTION_ENABLED: "true"
      PREFECT_SERVER_AUTHENTICATION_ENABLED: "true"
      PREFECT_MESSAGING_BROKER: prefect_redis.messaging
      PREFECT_MESSAGING_CACHE: prefect_redis.messaging
      PREFECT_REDIS_MESSAGING_HOST: redis
      PREFECT_REDIS_MESSAGING_PORT: 6379
      PREFECT_REDIS_MESSAGING_DB: 0
    command: prefect server start --no-services
    ports:
      - "4200:4200"
    networks:
      - bbl-shared-net
    labels:
      com.centurylinklabs.watchtower.enable: "true"

  prefect-services:
    image: prefecthq/prefect:3-latest
    container_name: prefect-services
    restart: unless-stopped
    depends_on:
      postgres:
        condition: service_healthy
      redis:
        condition: service_healthy
    environment:
      PREFECT_API_DATABASE_CONNECTION_URL: postgresql+asyncpg://prefect:OujaoPhaiS5Eig7ohy3TeiKeecahhe@postgres:5432/prefect
      PREFECT_MESSAGING_BROKER: prefect_redis.messaging
      PREFECT_MESSAGING_CACHE: prefect_redis.messaging
      PREFECT_REDIS_MESSAGING_HOST: redis
      PREFECT_REDIS_MESSAGING_PORT: 6379
      PREFECT_REDIS_MESSAGING_DB: 0
    command: prefect server services start
    networks:
      - bbl-shared-net
    labels:
      com.centurylinklabs.watchtower.enable: "true"

  prefect-worker:
    image: prefecthq/prefect:3-latest
    container_name: prefect-worker
    restart: unless-stopped
    depends_on:
      prefect-server:
        condition: service_started
    environment:
      PREFECT_API_URL: http://prefect-server:4200/api
    command: prefect worker start --pool local-pool
    networks:
      - bbl-shared-net
    labels:
      com.centurylinklabs.watchtower.enable: "true"
```

- **Prefect UI**: `https://prefect.bitbylaw.com`.
- **API**: Über Kong `https://api.bitbylaw.com/prefect`.

### Git-Server (Gitea)
`docker-compose-gitea.yml`:
```yaml
version: '3.8'

networks:
  bbl-shared-net:
    external: true

volumes:
  gitea-data: {}

services:
  gitea:
    image: gitea/gitea:latest
    container_name: gitea
    restart: always
    environment:
      - USER_UID=1000
      - USER_GID=1000
      - GITEA__server__ROOT_URL=http://git.bitbylaw.com:3000
      - GITEA__database__DB_TYPE=sqlite3
      - GITEA__server__SSH_DOMAIN=git.bitbylaw.com
    volumes:
      - gitea-data:/data
      - /etc/timezone:/etc/timezone:ro
      - /etc/localtime:/etc/localtime:ro
    ports:
      - "3000:3000"
      - "2222:22"
    networks:
      - bbl-shared-net
    labels:
      com.centurylinklabs.watchtower.enable: "true"
```

- **Web-UI**: `https://git.bitbylaw.com`.
- **SSH**: `git@git.bitbylaw.com` (Port 2222).
- **Integration**: Flows werden in Repos wie `admin/prefect-flows` gepusht.

### EspoCRM
`docker-compose-espocrm.yml`:
```yaml
version: '3.8'

networks:
  bbl-shared-net:
    external: true

volumes:
  espocrm-db:
  espocrm:

services:
  espocrm-db:
    image: mariadb:latest
    container_name: espocrm-db
    environment:
      MARIADB_ROOT_PASSWORD: root_password
      MARIADB_DATABASE: espocrm
      MARIADB_USER: espocrm
      MARIADB_PASSWORD: database_password
    volumes:
      - espocrm-db:/var/lib/mysql
    restart: always
    healthcheck:
      test: ["CMD", "healthcheck.sh", "--connect", "--innodb_initialized"]
      interval: 20s
      start_period: 10s
      timeout: 10s
      retries: 3
    networks:
      - bbl-shared-net
    labels:
      com.centurylinklabs.watchtower.enable: "true"

  espocrm:
    image: espocrm/espocrm:latest
    container_name: espocrm
    environment:
      ESPOCRM_DATABASE_PLATFORM: Mysql
      ESPOCRM_DATABASE_HOST: espocrm-db
      ESPOCRM_DATABASE_USER: espocrm
      ESPOCRM_DATABASE_PASSWORD: database_password
      ESPOCRM_ADMIN_USERNAME: admin
      ESPOCRM_ADMIN_PASSWORD: password
      ESPOCRM_SITE_URL: "http://api.bitbylaw.com:5000/espocrm"
    volumes:
      - espocrm:/var/www/html
    restart: always
    depends_on:
      espocrm-db:
        condition: service_healthy
    ports:
      - "8080:80"
    networks:
      - bbl-shared-net
    labels:
      com.centurylinklabs.watchtower.enable: "true"

  espocrm-daemon:
    image: espocrm/espocrm:latest
    container_name: espocrm-daemon
    volumes:
      - espocrm:/var/www/html
    restart: always
    entrypoint: docker-daemon.sh
    networks:
      - bbl-shared-net
    labels:
      com.centurylinklabs.watchtower.enable: "true"

  espocrm-websocket:
    image: espocrm/espocrm:latest
    container_name: espocrm-websocket
    environment:
      ESPOCRM_CONFIG_USE_WEB_SOCKET: "true"
      ESPOCRM_CONFIG_WEB_SOCKET_URL: "ws://api.bitbylaw.com:5000/espocrm/ws"
      ESPOCRM_CONFIG_WEB_SOCKET_ZERO_M_Q_SUBSCRIBER_DSN: "tcp://*:7777"
      ESPOCRM_CONFIG_WEB_SOCKET_ZERO_M_Q_SUBMISSION_DSN: "tcp://espocrm-websocket:7777"
    volumes:
      - espocrm:/var/www/html
    restart: always
    entrypoint: docker-websocket.sh
    ports:
      - "8081:8080"
    networks:
      - bbl-shared-net
    labels:
      com.centurylinklabs.watchtower.enable: "true"
```

- **UI**: `https://espocrm.bitbylaw.com`.
- **Integration**: Über Kong `https://api.bitbylaw.com/espocrm`.

### Flows erstellen, ändern und deployen

1. **Flow erstellen**:
   - In VS Code: Öffne `~/bbl-code/prefect-flows`.
   - Erstelle z.B. `flows/new_flow.py`:
     ```python
     from prefect import flow, task

     @task
     def say_hello(name: str) -> str:
         message = f"New Flow: Hello, {name}!"
         print(message)
         return message

     @flow(name="new-flow")
     def new_flow(name: str = "New World"):
         result = say_hello(name)
         return result

     if __name__ == "__main__":
         new_flow()
     ```
   - Teste lokal: `python3 flows/new_flow.py`.

2. **Flow ändern**:
   - Bearbeite `flows/hello_world.py` oder `new_flow.py` in VS Code.
   - Speichere und teste lokal.

3. **Commit und Push**:
   - In VS Code **Source Control** (Ctrl+Shift+G):
     - Stage: `+` neben Datei.
     - Commit: Nachricht eingeben (z.B. `Add new_flow.py`) → Häkchen.
     - Push: `...` → **Push**.
   - Alternativ Terminal:
     ```bash
     git add flows/new_flow.py
     git commit -m "Add new_flow.py"
     git push origin main
     ```

4. **Deploy den Flow**:
   - In VS Code Terminal:
     ```bash
     cd ~/bbl-code/prefect-flows
     prefect deploy
     ```
   - Wähle das Deployment aus `prefect.yaml` (z.B. `new-flow-deployment`).
   - Prefect pullt den Code aus Gitea in den Worker.

5. **Trigger den Flow**:
   - Finde Deployment-ID:
     ```bash
     curl -i https://api.bitbylaw.com/prefect/deployments
     ```
   - Trigger:
     ```bash
     curl -i -X POST \
       -H "Content-Type: application/json" \
       -d '{
         "name": "new-flow-run",
         "parameters": {"name": "New World"},
         "state": {"type": "SCHEDULED"}
       }' \
       https://api.bitbylaw.com/prefect/deployments/<deployment-id>/flow_runs
     ```
   - Prüfe:
     ```bash
     curl -i https://api.bitbylaw.com/prefect/flow_runs/<flow-run-id>
     ```

### Sicherheit und Authentifizierung

- **Kong**: `basic-auth` oder `key-auth` auf Service-Ebene (z.B. `prefect-service`).
- **Gitea**: SSH-Keys für passwortlosen Zugriff (siehe vorherige Anleitung).
- **Traefik**: SSL für alle URLs.
- **Prefect**: Aktiviere Authentifizierung in `PREFECT_SERVER_AUTHENTICATION_ENABLED`.

### Troubleshooting

- **Git**: Prüfe Logs in Gitea (`docker logs gitea`).
- **Prefect Deployment**: Worker-Logs (`docker logs prefect-worker`).
- **Flow-Run**: UI (`https://prefect.bitbylaw.com`) oder API-Logs.
- **Traefik**: Logs (`docker logs traefik`).

Für Updates pushe Änderungen zu Gitea und deploye neu.