added detailed readme

README.md

@@ -1,4 +1,456 @@
-# prefect-flows
+# BBL API-Gateway Setup mit Prefect, Kong, EspoCRM und Gitea
 
-# Test
-# Test
+Dieses Repository enthält die Flows für Prefect und dient als zentrale Dokumentation für das gesamte Setup. Das System integriert einen API-Gateway (Kong) für Prefect und EspoCRM, einen Git-Server (Gitea) für Code-Management und Docker für die Container-Orchestrierung. Traefik handhabt das externe Routing (einschließlich SSL), und Portainer/Watchtower managen Updates.
+
+## Einführung
+
+Dieses Setup ermöglicht:
+- **Kong als API-Gateway**: Routet Anfragen an Prefect (`api.bitbylaw.com:5000/prefect`) und EspoCRM (`api.bitbylaw.com:5000/espocrm`).
+- **Prefect für Workflows**: Flows werden in diesem Repository erstellt, geändert, gepusht, deployed und über die API getriggert.
+- **Gitea als Git-Server**: Zentrale Code-Speicherung für Prefect-Flows, integriert mit VS Code.
+- **EspoCRM**: CRM-System, integriert mit Kong für API-Zugriff.
+- **Docker**: Alle Komponenten laufen in Containern mit einem gemeinsamen Netzwerk (`bbl-shared-net`) für interne Kommunikation.
+- **Traefik**: Manuelle Konfiguration für URLs wie `api.bitbylaw.com`, `ui.api.bitbylaw.com`, `git.bitbylaw.com`, `prefect.bitbylaw.com`, `espocrm.bitbylaw.com`.
+- **Sicherheit**: Authentifizierung über Kong (z.B. `basic-auth` oder `key-auth`), SSL via Traefik, SSH-Keys für Git.
+
+## Voraussetzungen
+
+- **VM/Host**: Ubuntu/Debian mit Docker, Docker Compose, Portainer, Watchtower und Traefik installiert.
+- **Netzwerk**: Externes Docker-Netzwerk `bbl-shared-net` (`docker network create bbl-shared-net`).
+- **Domains**: `api.bitbylaw.com` (Kong Proxy), `ui.api.bitbylaw.com` (Kong Manager), `git.bitbylaw.com` (Gitea), `prefect.bitbylaw.com` (Prefect UI), `espocrm.bitbylaw.com` (EspoCRM).
+- **VS Code**: Installiert auf der VM, mit Python-Extension und GitLens für Git-Integration.
+- **Python**: 3.8+ in einer virtuellen Umgebung (`venv` unter `~/bbl-code`).
+- **Gitea-Credentials**: Admin-User (`admin:strongpass`).
+- **Prefect-API**: Ohne Auth (für Tests); für Produktion `basic-auth` aktivieren.
+
+## Docker-Konfiguration
+
+Alle Container laufen in `bbl-shared-net` für interne Kommunikation. Watchtower-Labels für automatische Updates. Ports werden über Traefik geroutet.
+
+### Kong (API-Gateway)
+`docker-compose-kong.yml`:
+```yaml
+version: '3.8'
+
+networks:
+  bbl-shared-net:
+    external: true
+
+volumes:
+  kong_data: {}
+
+services:
+  kong-database:
+    image: postgres:16
+    restart: always
+    networks:
+      - bbl-shared-net
+    environment:
+      POSTGRES_USER: kong
+      POSTGRES_PASSWORD: kong
+      POSTGRES_DB: kong
+    volumes:
+      - kong_data:/var/lib/postgresql/data
+    ports:
+      - "5432:5432"
+    healthcheck:
+      test: ["CMD", "pg_isready", "-U", "kong"]
+      interval: 5s
+      timeout: 5s
+      retries: 5
+    labels:
+      com.centurylinklabs.watchtower.enable: "true"
+
+  kong-migration:
+    image: kong:latest
+    command: "kong migrations bootstrap"
+    networks:
+      - bbl-shared-net
+    restart: on-failure
+    environment:
+      KONG_PG_HOST: kong-database
+      KONG_PG_USER: kong
+      KONG_PG_PASSWORD: kong
+      KONG_PG_DATABASE: kong
+      KONG_DATABASE: postgres
+      KONG_LOG_LEVEL: debug
+    depends_on:
+      kong-database:
+        condition: service_healthy
+
+  kong:
+    image: kong:latest
+    restart: always
+    networks:
+      - bbl-shared-net
+    environment:
+      KONG_PG_HOST: kong-database
+      KONG_PG_USER: kong
+      KONG_PG_PASSWORD: kong
+      KONG_PG_DATABASE: kong
+      KONG_DATABASE: postgres
+      KONG_PG_TIMEOUT: 10000
+      KONG_PROXY_LISTEN: 0.0.0.0:5000
+      KONG_ADMIN_LISTEN: 0.0.0.0:8001
+      KONG_ADMIN_GUI_LISTEN: 0.0.0.0:5001
+      KONG_PORTAL: on
+      KONG_ADMIN_GUI_AUTH: basic-auth
+      KONG_ADMIN_GUI_USERNAME: admin
+      KONG_ADMIN_GUI_PASSWORD: adminpass
+      KONG_LOG_LEVEL: debug
+    depends_on:
+      - kong-migration
+      - kong-database
+    healthcheck:
+      test: ["CMD", "kong", "health"]
+      interval: 5s
+      timeout: 5s
+      retries: 20
+    ports:
+      - "5000:5000"
+      - "8001:8001"
+      - "5001:5001"
+    labels:
+      com.centurylinklabs.watchtower.enable: "true"
+```
+
+- **Konfiguration in Kong Manager**: Services und Routes für Prefect (`prefect-service`: `http://prefect-server:4200/api`, Route: `/prefect`) und EspoCRM (`espocrm-service`: `http://espocrm:80`, Route: `/espocrm`).
+- **Authentifizierung**: `basic-auth` oder `key-auth` auf Service-Ebene (z.B. für `prefect-service`).
+
+### Prefect (Workflow-Management)
+`docker-compose-prefect.yml`:
+```yaml
+version: '3.8'
+
+networks:
+  bbl-shared-net:
+    external: true
+
+volumes:
+  postgres_data:
+  redis_data:
+
+services:
+  postgres:
+    image: postgres:14
+    container_name: prefect-postgres
+    restart: unless-stopped
+    environment:
+      POSTGRES_USER: prefect
+      POSTGRES_PASSWORD: OujaoPhaiS5Eig7ohy3TeiKeecahhe
+      POSTGRES_DB: prefect
+    volumes:
+      - postgres_data:/var/lib/postgresql/data
+    healthcheck:
+      test: ["CMD-SHELL", "pg_isready -U prefect"]
+      interval: 5s
+      timeout: 5s
+      retries: 5
+    networks:
+      - bbl-shared-net
+    labels:
+      com.centurylinklabs.watchtower.enable: "true"
+
+  redis:
+    image: redis:7
+    container_name: prefect-redis
+    restart: unless-stopped
+    volumes:
+      - redis_data:/data
+    healthcheck:
+      test: ["CMD-SHELL", "redis-cli ping"]
+    networks:
+      - bbl-shared-net
+    labels:
+      com.centurylinklabs.watchtower.enable: "true"
+
+  prefect-server:
+    image: prefecthq/prefect:3-latest
+    container_name: prefect-server
+    restart: unless-stopped
+    depends_on:
+      postgres:
+        condition: service_healthy
+      redis:
+        condition: service_healthy
+    environment:
+      PREFECT_API_DATABASE_CONNECTION_URL: postgresql+asyncpg://prefect:OujaoPhaiS5Eig7ohy3TeiKeecahhe@postgres:5432/prefect
+      PREFECT_SERVER_API_HOST: 0.0.0.0
+      PREFECT_SERVER_API_PORT: 4200
+      PREFECT_UI_API_URL: http://prefect-server:4200/api
+      PREFECT_SERVER_CSRF_PROTECTION_ENABLED: "true"
+      PREFECT_SERVER_AUTHENTICATION_ENABLED: "true"
+      PREFECT_MESSAGING_BROKER: prefect_redis.messaging
+      PREFECT_MESSAGING_CACHE: prefect_redis.messaging
+      PREFECT_REDIS_MESSAGING_HOST: redis
+      PREFECT_REDIS_MESSAGING_PORT: 6379
+      PREFECT_REDIS_MESSAGING_DB: 0
+    command: prefect server start --no-services
+    ports:
+      - "4200:4200"
+    networks:
+      - bbl-shared-net
+    labels:
+      com.centurylinklabs.watchtower.enable: "true"
+
+  prefect-services:
+    image: prefecthq/prefect:3-latest
+    container_name: prefect-services
+    restart: unless-stopped
+    depends_on:
+      postgres:
+        condition: service_healthy
+      redis:
+        condition: service_healthy
+    environment:
+      PREFECT_API_DATABASE_CONNECTION_URL: postgresql+asyncpg://prefect:OujaoPhaiS5Eig7ohy3TeiKeecahhe@postgres:5432/prefect
+      PREFECT_MESSAGING_BROKER: prefect_redis.messaging
+      PREFECT_MESSAGING_CACHE: prefect_redis.messaging
+      PREFECT_REDIS_MESSAGING_HOST: redis
+      PREFECT_REDIS_MESSAGING_PORT: 6379
+      PREFECT_REDIS_MESSAGING_DB: 0
+    command: prefect server services start
+    networks:
+      - bbl-shared-net
+    labels:
+      com.centurylinklabs.watchtower.enable: "true"
+
+  prefect-worker:
+    image: prefecthq/prefect:3-latest
+    container_name: prefect-worker
+    restart: unless-stopped
+    depends_on:
+      prefect-server:
+        condition: service_started
+    environment:
+      PREFECT_API_URL: http://prefect-server:4200/api
+    command: prefect worker start --pool local-pool
+    networks:
+      - bbl-shared-net
+    labels:
+      com.centurylinklabs.watchtower.enable: "true"
+```
+
+- **Prefect UI**: `https://prefect.bitbylaw.com`.
+- **API**: Über Kong `https://api.bitbylaw.com/prefect`.
+
+### Git-Server (Gitea)
+`docker-compose-gitea.yml`:
+```yaml
+version: '3.8'
+
+networks:
+  bbl-shared-net:
+    external: true
+
+volumes:
+  gitea-data: {}
+
+services:
+  gitea:
+    image: gitea/gitea:latest
+    container_name: gitea
+    restart: always
+    environment:
+      - USER_UID=1000
+      - USER_GID=1000
+      - GITEA__server__ROOT_URL=http://git.bitbylaw.com:3000
+      - GITEA__database__DB_TYPE=sqlite3
+      - GITEA__server__SSH_DOMAIN=git.bitbylaw.com
+    volumes:
+      - gitea-data:/data
+      - /etc/timezone:/etc/timezone:ro
+      - /etc/localtime:/etc/localtime:ro
+    ports:
+      - "3000:3000"
+      - "2222:22"
+    networks:
+      - bbl-shared-net
+    labels:
+      com.centurylinklabs.watchtower.enable: "true"
+```
+
+- **Web-UI**: `https://git.bitbylaw.com`.
+- **SSH**: `git@git.bitbylaw.com` (Port 2222).
+- **Integration**: Flows werden in Repos wie `admin/prefect-flows` gepusht.
+
+### EspoCRM
+`docker-compose-espocrm.yml`:
+```yaml
+version: '3.8'
+
+networks:
+  bbl-shared-net:
+    external: true
+
+volumes:
+  espocrm-db:
+  espocrm:
+
+services:
+  espocrm-db:
+    image: mariadb:latest
+    container_name: espocrm-db
+    environment:
+      MARIADB_ROOT_PASSWORD: root_password
+      MARIADB_DATABASE: espocrm
+      MARIADB_USER: espocrm
+      MARIADB_PASSWORD: database_password
+    volumes:
+      - espocrm-db:/var/lib/mysql
+    restart: always
+    healthcheck:
+      test: ["CMD", "healthcheck.sh", "--connect", "--innodb_initialized"]
+      interval: 20s
+      start_period: 10s
+      timeout: 10s
+      retries: 3
+    networks:
+      - bbl-shared-net
+    labels:
+      com.centurylinklabs.watchtower.enable: "true"
+
+  espocrm:
+    image: espocrm/espocrm:latest
+    container_name: espocrm
+    environment:
+      ESPOCRM_DATABASE_PLATFORM: Mysql
+      ESPOCRM_DATABASE_HOST: espocrm-db
+      ESPOCRM_DATABASE_USER: espocrm
+      ESPOCRM_DATABASE_PASSWORD: database_password
+      ESPOCRM_ADMIN_USERNAME: admin
+      ESPOCRM_ADMIN_PASSWORD: password
+      ESPOCRM_SITE_URL: "http://api.bitbylaw.com:5000/espocrm"
+    volumes:
+      - espocrm:/var/www/html
+    restart: always
+    depends_on:
+      espocrm-db:
+        condition: service_healthy
+    ports:
+      - "8080:80"
+    networks:
+      - bbl-shared-net
+    labels:
+      com.centurylinklabs.watchtower.enable: "true"
+
+  espocrm-daemon:
+    image: espocrm/espocrm:latest
+    container_name: espocrm-daemon
+    volumes:
+      - espocrm:/var/www/html
+    restart: always
+    entrypoint: docker-daemon.sh
+    networks:
+      - bbl-shared-net
+    labels:
+      com.centurylinklabs.watchtower.enable: "true"
+
+  espocrm-websocket:
+    image: espocrm/espocrm:latest
+    container_name: espocrm-websocket
+    environment:
+      ESPOCRM_CONFIG_USE_WEB_SOCKET: "true"
+      ESPOCRM_CONFIG_WEB_SOCKET_URL: "ws://api.bitbylaw.com:5000/espocrm/ws"
+      ESPOCRM_CONFIG_WEB_SOCKET_ZERO_M_Q_SUBSCRIBER_DSN: "tcp://*:7777"
+      ESPOCRM_CONFIG_WEB_SOCKET_ZERO_M_Q_SUBMISSION_DSN: "tcp://espocrm-websocket:7777"
+    volumes:
+      - espocrm:/var/www/html
+    restart: always
+    entrypoint: docker-websocket.sh
+    ports:
+      - "8081:8080"
+    networks:
+      - bbl-shared-net
+    labels:
+      com.centurylinklabs.watchtower.enable: "true"
+```
+
+- **UI**: `https://espocrm.bitbylaw.com`.
+- **Integration**: Über Kong `https://api.bitbylaw.com/espocrm`.
+
+### Flows erstellen, ändern und deployen
+
+1. **Flow erstellen**:
+   - In VS Code: Öffne `~/bbl-code/prefect-flows`.
+   - Erstelle z.B. `flows/new_flow.py`:
+     ```python
+     from prefect import flow, task
+
+     @task
+     def say_hello(name: str) -> str:
+         message = f"New Flow: Hello, {name}!"
+         print(message)
+         return message
+
+     @flow(name="new-flow")
+     def new_flow(name: str = "New World"):
+         result = say_hello(name)
+         return result
+
+     if __name__ == "__main__":
+         new_flow()
+     ```
+   - Teste lokal: `python3 flows/new_flow.py`.
+
+2. **Flow ändern**:
+   - Bearbeite `flows/hello_world.py` oder `new_flow.py` in VS Code.
+   - Speichere und teste lokal.
+
+3. **Commit und Push**:
+   - In VS Code **Source Control** (Ctrl+Shift+G):
+     - Stage: `+` neben Datei.
+     - Commit: Nachricht eingeben (z.B. `Add new_flow.py`) → Häkchen.
+     - Push: `...` → **Push**.
+   - Alternativ Terminal:
+     ```bash
+     git add flows/new_flow.py
+     git commit -m "Add new_flow.py"
+     git push origin main
+     ```
+
+4. **Deploy den Flow**:
+   - In VS Code Terminal:
+     ```bash
+     cd ~/bbl-code/prefect-flows
+     prefect deploy
+     ```
+   - Wähle das Deployment aus `prefect.yaml` (z.B. `new-flow-deployment`).
+   - Prefect pullt den Code aus Gitea in den Worker.
+
+5. **Trigger den Flow**:
+   - Finde Deployment-ID:
+     ```bash
+     curl -i https://api.bitbylaw.com/prefect/deployments
+     ```
+   - Trigger:
+     ```bash
+     curl -i -X POST \
+       -H "Content-Type: application/json" \
+       -d '{
+         "name": "new-flow-run",
+         "parameters": {"name": "New World"},
+         "state": {"type": "SCHEDULED"}
+       }' \
+       https://api.bitbylaw.com/prefect/deployments/<deployment-id>/flow_runs
+     ```
+   - Prüfe:
+     ```bash
+     curl -i https://api.bitbylaw.com/prefect/flow_runs/<flow-run-id>
+     ```
+
+### Sicherheit und Authentifizierung
+
+- **Kong**: `basic-auth` oder `key-auth` auf Service-Ebene (z.B. `prefect-service`).
+- **Gitea**: SSH-Keys für passwortlosen Zugriff (siehe vorherige Anleitung).
+- **Traefik**: SSL für alle URLs.
+- **Prefect**: Aktiviere Authentifizierung in `PREFECT_SERVER_AUTHENTICATION_ENABLED`.
+
+### Troubleshooting
+
+- **Git**: Prüfe Logs in Gitea (`docker logs gitea`).
+- **Prefect Deployment**: Worker-Logs (`docker logs prefect-worker`).
+- **Flow-Run**: UI (`https://prefect.bitbylaw.com`) oder API-Logs.
+- **Traefik**: Logs (`docker logs traefik`).
+
+Für Updates pushe Änderungen zu Gitea und deploye neu.