Observability

self-hosting monitoring prometheus metrics observability

Observability

This guide covers monitoring and observability for self-hosted Huitzo deployments, including Prometheus metrics, logging, and alerting.

Built-in Metrics Endpoint

Huitzo exposes metrics in Prometheus format at /metrics:

curl http://localhost:8080/metrics

Example output:

# HELP huitzo_command_duration_seconds Command execution duration
# TYPE huitzo_command_duration_seconds histogram
huitzo_command_duration_seconds_bucket{namespace="analytics",command="analyze",le="0.1"} 12
huitzo_command_duration_seconds_bucket{namespace="analytics",command="analyze",le="0.5"} 45
huitzo_command_duration_seconds_bucket{namespace="analytics",command="analyze",le="1.0"} 78
...

# HELP huitzo_command_total Total commands executed
# TYPE huitzo_command_total counter
huitzo_command_total{namespace="analytics",command="analyze",status="success"} 156
huitzo_command_total{namespace="analytics",command="analyze",status="error"} 3

Key Metrics

Command Execution

Metric Type Labels Description
huitzo_command_duration_seconds histogram namespace, command Command execution duration
huitzo_command_total counter namespace, command, status Total commands executed
huitzo_active_tasks gauge queue Currently running tasks per queue
huitzo_queued_tasks gauge queue Tasks waiting in queue

Storage Operations

Metric Type Labels Description
huitzo_storage_operations_total counter operation, status Storage operations (save, get, delete)
huitzo_storage_operation_duration_seconds histogram operation Storage operation latency
huitzo_storage_bytes_total counter operation Bytes read/written

System Health

Metric Type Labels Description
huitzo_http_requests_total counter method, endpoint, status HTTP request count
huitzo_http_request_duration_seconds histogram method, endpoint HTTP request latency
huitzo_worker_status gauge worker_id Worker health (1=healthy, 0=unhealthy)
huitzo_database_connections gauge pool Active database connections

LLM Usage

Metric Type Labels Description
huitzo_llm_requests_total counter provider, model, status LLM API calls
huitzo_llm_tokens_total counter provider, model, type Tokens consumed (input/output)
huitzo_llm_request_duration_seconds histogram provider, model LLM request latency

Prometheus Configuration

Basic Setup

Add Prometheus to your docker-compose.yml:

# docker-compose.yml
services:
  # ... existing services ...

  prometheus:
    image: prom/prometheus:latest
    ports:
      - "9090:9090"
    volumes:
      - ./prometheus.yml:/etc/prometheus/prometheus.yml
      - prometheus_data:/prometheus
    command:
      - '--config.file=/etc/prometheus/prometheus.yml'
      - '--storage.tsdb.retention.time=15d'

volumes:
  prometheus_data:

Prometheus Configuration

# prometheus.yml
global:
  scrape_interval: 15s
  evaluation_interval: 15s

scrape_configs:
  - job_name: 'huitzo'
    static_configs:
      - targets: ['app:8080']
    metrics_path: /metrics

  - job_name: 'huitzo-workers'
    static_configs:
      - targets: ['worker:9100']  # If worker exposes metrics

  - job_name: 'redis'
    static_configs:
      - targets: ['redis:9121']  # Redis exporter

  - job_name: 'postgres'
    static_configs:
      - targets: ['postgres:9187']  # Postgres exporter

Adding Exporters

For database and cache metrics, add exporters:

# docker-compose.yml additions
services:
  redis-exporter:
    image: oliver006/redis_exporter:latest
    environment:
      - REDIS_ADDR=redis://redis:6379
    ports:
      - "9121:9121"

  postgres-exporter:
    image: prometheuscommunity/postgres-exporter:latest
    environment:
      - DATA_SOURCE_NAME=postgresql://huitzo:${POSTGRES_PASSWORD}@postgres:5432/huitzo?sslmode=disable
    ports:
      - "9187:9187"

Grafana Dashboard

Setup

Add Grafana to your stack:

# docker-compose.yml
services:
  grafana:
    image: grafana/grafana:latest
    ports:
      - "3000:3000"
    environment:
      - GF_SECURITY_ADMIN_PASSWORD=admin
      - GF_USERS_ALLOW_SIGN_UP=false
    volumes:
      - grafana_data:/var/lib/grafana
      - ./grafana/dashboards:/etc/grafana/provisioning/dashboards
      - ./grafana/datasources:/etc/grafana/provisioning/datasources

volumes:
  grafana_data:

Datasource Configuration

# grafana/datasources/prometheus.yml
apiVersion: 1

datasources:
  - name: Prometheus
    type: prometheus
    access: proxy
    url: http://prometheus:9090
    isDefault: true

Dashboard JSON

Create a Huitzo dashboard at grafana/dashboards/huitzo.json:

{
  "dashboard": {
    "title": "Huitzo Platform",
    "panels": [
      {
        "title": "Command Execution Rate",
        "type": "graph",
        "targets": [
          {
            "expr": "rate(huitzo_command_total[5m])",
            "legendFormat": "{{namespace}}.{{command}} - {{status}}"
          }
        ]
      },
      {
        "title": "Command Duration (p95)",
        "type": "graph",
        "targets": [
          {
            "expr": "histogram_quantile(0.95, rate(huitzo_command_duration_seconds_bucket[5m]))",
            "legendFormat": "{{namespace}}.{{command}}"
          }
        ]
      },
      {
        "title": "Active Tasks",
        "type": "gauge",
        "targets": [
          {
            "expr": "sum(huitzo_active_tasks)",
            "legendFormat": "Active"
          }
        ]
      },
      {
        "title": "Error Rate",
        "type": "graph",
        "targets": [
          {
            "expr": "sum(rate(huitzo_command_total{status=\"error\"}[5m])) / sum(rate(huitzo_command_total[5m])) * 100",
            "legendFormat": "Error %"
          }
        ]
      },
      {
        "title": "LLM Token Usage",
        "type": "graph",
        "targets": [
          {
            "expr": "sum(rate(huitzo_llm_tokens_total[1h])) by (provider, type)",
            "legendFormat": "{{provider}} - {{type}}"
          }
        ]
      },
      {
        "title": "Storage Operations",
        "type": "graph",
        "targets": [
          {
            "expr": "rate(huitzo_storage_operations_total[5m])",
            "legendFormat": "{{operation}}"
          }
        ]
      }
    ]
  }
}

AlertManager Rules

Setup

Add AlertManager:

# docker-compose.yml
services:
  alertmanager:
    image: prom/alertmanager:latest
    ports:
      - "9093:9093"
    volumes:
      - ./alertmanager.yml:/etc/alertmanager/alertmanager.yml

Alert Rules

Create alert rules in prometheus-rules.yml:

# prometheus-rules.yml
groups:
  - name: huitzo-alerts
    rules:
      # High error rate
      - alert: HuitzoHighErrorRate
        expr: |
          sum(rate(huitzo_command_total{status="error"}[5m]))
          / sum(rate(huitzo_command_total[5m])) > 0.05
        for: 5m
        labels:
          severity: warning
        annotations:
          summary: "High command error rate"
          description: "Error rate is {{ $value | humanizePercentage }} (threshold: 5%)"

      # Command latency spike
      - alert: HuitzoHighLatency
        expr: |
          histogram_quantile(0.95, rate(huitzo_command_duration_seconds_bucket[5m])) > 30
        for: 10m
        labels:
          severity: warning
        annotations:
          summary: "High command latency"
          description: "p95 latency is {{ $value | humanizeDuration }}"

      # Worker down
      - alert: HuitzoWorkerDown
        expr: huitzo_worker_status == 0
        for: 2m
        labels:
          severity: critical
        annotations:
          summary: "Huitzo worker is down"
          description: "Worker {{ $labels.worker_id }} is unhealthy"

      # Queue backup
      - alert: HuitzoQueueBacklog
        expr: huitzo_queued_tasks > 100
        for: 10m
        labels:
          severity: warning
        annotations:
          summary: "Task queue backlog"
          description: "{{ $value }} tasks waiting in {{ $labels.queue }} queue"

      # Database connection exhaustion
      - alert: HuitzoDatabaseConnectionsHigh
        expr: huitzo_database_connections / huitzo_database_connections_max > 0.8
        for: 5m
        labels:
          severity: warning
        annotations:
          summary: "Database connection pool nearly exhausted"
          description: "{{ $value | humanizePercentage }} of connections in use"

      # LLM provider errors
      - alert: HuitzoLLMErrors
        expr: |
          sum(rate(huitzo_llm_requests_total{status="error"}[5m]))
          / sum(rate(huitzo_llm_requests_total[5m])) > 0.1
        for: 5m
        labels:
          severity: warning
        annotations:
          summary: "High LLM error rate"
          description: "LLM error rate is {{ $value | humanizePercentage }}"

AlertManager Configuration

# alertmanager.yml
global:
  smtp_smarthost: 'smtp.example.com:587'
  smtp_from: '[email protected]'
  smtp_auth_username: '[email protected]'
  smtp_auth_password: 'password'

route:
  group_by: ['alertname']
  group_wait: 30s
  group_interval: 5m
  repeat_interval: 1h
  receiver: 'email-notifications'

receivers:
  - name: 'email-notifications'
    email_configs:
      - to: '[email protected]'

  - name: 'slack-notifications'
    slack_configs:
      - api_url: 'https://hooks.slack.com/services/xxx/xxx/xxx'
        channel: '#alerts'

Structured Logging

Log Configuration

# Environment variables
LOG_LEVEL=INFO
LOG_FORMAT=json        # json | text
LOG_OUTPUT=stdout      # stdout | file
LOG_FILE_PATH=/var/log/huitzo/app.log

Log Format

JSON logs include correlation IDs for request tracing:

{
  "timestamp": "2026-01-22T10:30:00.000Z",
  "level": "INFO",
  "logger": "huitzo.executor",
  "message": "Command completed",
  "correlation_id": "abc-123-def",
  "tenant_id": "org_456",
  "user_id": "user_789",
  "command": "analytics.analyze",
  "duration_ms": 1234,
  "status": "success"
}

Log Aggregation

For log aggregation, use Loki or your preferred solution:

# docker-compose.yml
services:
  loki:
    image: grafana/loki:latest
    ports:
      - "3100:3100"
    volumes:
      - loki_data:/loki

  promtail:
    image: grafana/promtail:latest
    volumes:
      - /var/log:/var/log
      - ./promtail.yml:/etc/promtail/promtail.yml

Health Checks

Endpoints

Endpoint Description
GET /health Overall system health
GET /health/live Liveness probe (is process running?)
GET /health/ready Readiness probe (can accept requests?)

Response Format

{
  "status": "healthy",
  "version": "2.0.0",
  "timestamp": "2026-01-22T10:30:00Z",
  "services": {
    "database": {"status": "healthy", "latency_ms": 2},
    "redis": {"status": "healthy", "latency_ms": 1},
    "worker": {"status": "healthy", "active_tasks": 5}
  }
}

Kubernetes Probes

# kubernetes deployment
spec:
  containers:
    - name: huitzo
      livenessProbe:
        httpGet:
          path: /health/live
          port: 8080
        initialDelaySeconds: 10
        periodSeconds: 10
      readinessProbe:
        httpGet:
          path: /health/ready
          port: 8080
        initialDelaySeconds: 5
        periodSeconds: 5

Complete Observability Stack

Full docker-compose with all observability components:

# docker-compose.observability.yml
services:
  prometheus:
    image: prom/prometheus:latest
    ports:
      - "9090:9090"
    volumes:
      - ./prometheus.yml:/etc/prometheus/prometheus.yml
      - ./prometheus-rules.yml:/etc/prometheus/rules.yml
      - prometheus_data:/prometheus

  grafana:
    image: grafana/grafana:latest
    ports:
      - "3000:3000"
    environment:
      - GF_SECURITY_ADMIN_PASSWORD=${GRAFANA_PASSWORD:-admin}
    volumes:
      - grafana_data:/var/lib/grafana
      - ./grafana/provisioning:/etc/grafana/provisioning

  alertmanager:
    image: prom/alertmanager:latest
    ports:
      - "9093:9093"
    volumes:
      - ./alertmanager.yml:/etc/alertmanager/alertmanager.yml

  loki:
    image: grafana/loki:latest
    ports:
      - "3100:3100"
    volumes:
      - loki_data:/loki

volumes:
  prometheus_data:
  grafana_data:
  loki_data:

Run with your main stack:

docker compose -f docker-compose.yml -f docker-compose.observability.yml up -d