Self-Hosted Deployment

self-hosting deployment docker

Self-Hosted Deployment

Deploy Huitzo on your own infrastructure with full control over your data. Target deployment time: <30 minutes.

Prerequisites

See Requirements for detailed hardware specifications, platform compatibility, and scaling guidelines.

Quick reference: - 4+ cores, 8+ GB RAM, 50+ GB SSD recommended - Docker 28.0+ and Docker Compose 2.28+ required

Platform Compatibility

Huitzo supports multiple architectures via multi-arch Docker images:

Platform Architecture Supported Notes
Linux x86_64 (amd64) ✅ Primary development platform
Linux arm64 (aarch64) ✅ AWS Graviton, Oracle Ampere
macOS Intel (x86_64) ✅ Docker Desktop
macOS Apple Silicon (arm64) ✅ Docker Desktop, native performance

Software Requirements

  • Docker 28.0+ (29.x recommended)
  • Docker Compose v2.28+
  • Linux (Ubuntu 24.04 LTS recommended) or macOS

ARM Architecture Notes

Huitzo Docker images are built for both amd64 and arm64 architectures:

# Images automatically select correct architecture
docker pull huitzo/huitzo:latest

# Verify architecture
docker inspect huitzo/huitzo:latest | grep Architecture

ARM-specific considerations:

  1. Performance: Native ARM images run efficiently on ARM hosts (no emulation)
  2. Dependencies: All Python dependencies have ARM wheels available
  3. PostgreSQL/Redis: Official images support ARM natively
  4. Build from source: If building custom images, use multi-arch builds:
# Build multi-arch image
docker buildx build \
  --platform linux/amd64,linux/arm64 \
  -t huitzo/huitzo:latest \
  --push .

Self-hosted on ARM:

# docker-compose.yml works unchanged on ARM
# Images auto-select correct architecture
services:
  app:
    image: huitzo/huitzo:latest  # Works on both amd64 and arm64
    # ... rest of configuration

License Key

You'll need a valid Huitzo license key. Contact [email protected] or visit huitzo.com/pricing.

Quick Start

1. Download Configuration

# Create deployment directory
mkdir huitzo && cd huitzo

# Download docker-compose.yml and configuration
curl -sSL https://get.huitzo.com/docker-compose.yml -o docker-compose.yml
curl -sSL https://get.huitzo.com/.env.example -o .env

2. Configure Environment

Edit .env with your settings:

# .env

# License
HUITZO_LICENSE_KEY=your-license-key-here

# Security (generate a random key)
HUITZO_SECRET_KEY=your-random-secret-key-change-me

# Database
POSTGRES_PASSWORD=your-secure-database-password

# Optional: Email (for notifications)
SENDGRID_API_KEY=SG.xxxxx
HUITZO_EMAIL_FROM=[email protected]

# Optional: LLM providers
OPENAI_API_KEY=sk-xxxxx
ANTHROPIC_API_KEY=sk-ant-xxxxx

Generate a secure secret key:

openssl rand -hex 32

3. Start Services

docker compose up -d

4. Verify Installation

# Check all services are running
docker compose ps

# Check health endpoint
curl http://localhost:8080/health

Expected output:

{
  "status": "healthy",
  "version": "2.0.0",
  "services": {
    "database": "healthy",
    "redis": "healthy",
    "worker": "healthy"
  }
}

5. Access Huitzo

  • WebCLI: http://localhost:8080
  • API Docs: http://localhost:8080/docs
  • Health: http://localhost:8080/health

Docker Compose Configuration

Full docker-compose.yml

services:
  app:
    image: huitzo/huitzo:latest
    restart: unless-stopped
    ports:
      - "8080:8080"
    environment:
      - DATABASE_URL=postgresql://huitzo:${POSTGRES_PASSWORD}@postgres:5432/huitzo
      - REDIS_URL=redis://redis:6379/0
      - SECRET_KEY=${HUITZO_SECRET_KEY}
      - LICENSE_KEY=${HUITZO_LICENSE_KEY}
      - LOG_LEVEL=${LOG_LEVEL:-INFO}
      # Optional integrations
      - OPENAI_API_KEY=${OPENAI_API_KEY:-}
      - ANTHROPIC_API_KEY=${ANTHROPIC_API_KEY:-}
      - SENDGRID_API_KEY=${SENDGRID_API_KEY:-}
      - EMAIL_FROM=${HUITZO_EMAIL_FROM:-}
    depends_on:
      postgres:
        condition: service_healthy
      redis:
        condition: service_healthy
    healthcheck:
      test: ["CMD", "curl", "-f", "http://localhost:8080/health"]
      interval: 30s
      timeout: 10s
      retries: 3
      start_period: 40s
    volumes:
      - ./packs:/app/packs:ro
      - ./data/uploads:/app/uploads

  worker:
    image: huitzo/huitzo:latest
    command: ["celery", "-A", "huitzo.worker", "worker", "-l", "info"]
    restart: unless-stopped
    environment:
      - DATABASE_URL=postgresql://huitzo:${POSTGRES_PASSWORD}@postgres:5432/huitzo
      - REDIS_URL=redis://redis:6379/0
      - SECRET_KEY=${HUITZO_SECRET_KEY}
      - LICENSE_KEY=${HUITZO_LICENSE_KEY}
    depends_on:
      - app
      - redis
    deploy:
      resources:
        limits:
          memory: 512M

  postgres:
    image: postgres:17-alpine
    restart: unless-stopped
    environment:
      - POSTGRES_USER=huitzo
      - POSTGRES_PASSWORD=${POSTGRES_PASSWORD}
      - POSTGRES_DB=huitzo
    volumes:
      - ./data/postgres:/var/lib/postgresql/data
    healthcheck:
      test: ["CMD-SHELL", "pg_isready -U huitzo"]
      interval: 10s
      timeout: 5s
      retries: 5

  redis:
    image: redis:8-alpine
    restart: unless-stopped
    command: redis-server --appendonly yes
    volumes:
      - ./data/redis:/data
    healthcheck:
      test: ["CMD", "redis-cli", "ping"]
      interval: 10s
      timeout: 5s
      retries: 5

volumes:
  postgres_data:
  redis_data:

Resource Allocation (Containerized Services)

Service Memory CPU Purpose
app 1.5 GB 1.5 cores API server, WebCLI
worker 512 MB 0.5 cores Background tasks
postgres 1 GB 0.75 cores Data storage
redis 256 MB 0.25 cores Cache, task queue

Using External Services

PostgreSQL and Redis do not need to be containers. You can use:

  • Cloud-managed services: AWS RDS, Azure Database for PostgreSQL, Google Cloud SQL, Amazon ElastiCache, Azure Cache for Redis, etc.
  • On-premises servers: Dedicated PostgreSQL/Redis servers in your datacenter
  • Existing infrastructure: Connect to databases you already manage

External Services Configuration

When using external services, provide connection strings via environment variables:

# .env for external services
DATABASE_URL=postgresql://user:[email protected]:5432/huitzo
REDIS_URL=redis://:[email protected]:6379/0

# With SSL (recommended for cloud services)
DATABASE_URL=postgresql://user:[email protected]:5432/huitzo?sslmode=require
REDIS_URL=rediss://:[email protected]:6379/0

docker-compose for External Services

When using external PostgreSQL and Redis, use a simplified docker-compose:

# docker-compose.yml (external services)
services:
  app:
    image: huitzo/huitzo:latest
    restart: unless-stopped
    ports:
      - "8080:8080"
    environment:
      - HUITZO_DEPLOYMENT_MODE=self_hosted
      # External database (no container dependency)
      - DATABASE_URL=${DATABASE_URL}
      # External Redis (no container dependency)
      - REDIS_URL=${REDIS_URL}
      - SECRET_KEY=${HUITZO_SECRET_KEY}
      - LICENSE_KEY=${HUITZO_LICENSE_KEY}
      - LOG_LEVEL=${LOG_LEVEL:-INFO}
      - OPENAI_API_KEY=${OPENAI_API_KEY:-}
      - ANTHROPIC_API_KEY=${ANTHROPIC_API_KEY:-}
    healthcheck:
      test: ["CMD", "curl", "-f", "http://localhost:8080/health"]
      interval: 30s
      timeout: 10s
      retries: 3
      start_period: 40s
    volumes:
      - ./packs:/app/packs:ro
      - ./data/uploads:/app/uploads

  worker:
    image: huitzo/huitzo:latest
    command: ["celery", "-A", "huitzo.worker", "worker", "-l", "info", "-Q", "default,fast,medium,long"]
    restart: unless-stopped
    environment:
      - HUITZO_DEPLOYMENT_MODE=self_hosted
      - DATABASE_URL=${DATABASE_URL}
      - REDIS_URL=${REDIS_URL}
      - SECRET_KEY=${HUITZO_SECRET_KEY}
      - LICENSE_KEY=${HUITZO_LICENSE_KEY}
      - OPENAI_API_KEY=${OPENAI_API_KEY:-}
      - ANTHROPIC_API_KEY=${ANTHROPIC_API_KEY:-}
    volumes:
      - ./packs:/app/packs:ro
      - ./data/uploads:/app/uploads

  beat:
    image: huitzo/huitzo:latest
    command: ["celery", "-A", "huitzo.worker", "beat", "-l", "info"]
    restart: unless-stopped
    environment:
      - HUITZO_DEPLOYMENT_MODE=self_hosted
      - DATABASE_URL=${DATABASE_URL}
      - REDIS_URL=${REDIS_URL}
      - SECRET_KEY=${HUITZO_SECRET_KEY}
      - LICENSE_KEY=${HUITZO_LICENSE_KEY}

Note: No depends_on for postgres/redis since they're external services.

Cloud Provider Examples

AWS RDS + ElastiCache:

# .env
DATABASE_URL=postgresql://huitzo:[email protected]:5432/huitzo?sslmode=require
REDIS_URL=rediss://huitzo-cache.abc123.cache.amazonaws.com:6379/0

Azure Database + Cache:

# .env
DATABASE_URL=postgresql://huitzo@huitzo-server:[email protected]:5432/huitzo?sslmode=require
REDIS_URL=rediss://:[email protected]:6380/0

Google Cloud SQL + Memorystore:

# .env
DATABASE_URL=postgresql://huitzo:MySecurePass@/huitzo?host=/cloudsql/project:region:instance
REDIS_URL=redis://10.0.0.5:6379/0

Database Requirements

When using external PostgreSQL:

Requirement Value
Version PostgreSQL 14+ (17 recommended)
Extensions uuid-ossp, pgcrypto (usually pre-installed)
User permissions CREATE, SELECT, INSERT, UPDATE, DELETE on database
SSL Recommended for cloud connections

When using external Redis:

Requirement Value
Version Redis 6+ (8 recommended)
Mode Standalone or Sentinel (Cluster not yet supported)
Persistence AOF recommended for task queue durability
Memory 256 MB minimum, 1 GB recommended

Verifying External Connections

Before starting Huitzo, verify connectivity:

# Test PostgreSQL connection
psql "${DATABASE_URL}" -c "SELECT version();"

# Test Redis connection
redis-cli -u "${REDIS_URL}" ping

File Storage Configuration

User uploads and pack output files require storage. Options:

Backend Best For Configuration
Local filesystem Single-instance, dev/test Default, volume mount
S3-compatible Production, multi-instance AWS S3, MinIO, Spaces
Azure Blob Azure deployments Native integration
Google Cloud Storage GCP deployments Native integration

Local Filesystem (Default)

The default configuration stores files on the local filesystem via Docker volume:

# docker-compose.yml (default)
services:
  app:
    volumes:
      - ./data/uploads:/app/uploads

Limitation: Does not work for multi-instance deployments (files not shared between containers).

S3-Compatible Storage (Production)

For production and multi-instance deployments, use S3-compatible storage.

AWS S3:

# .env
HUITZO_FILE_STORAGE_BACKEND=s3
HUITZO_S3_BUCKET=mycompany-huitzo-uploads
HUITZO_S3_REGION=us-east-1
HUITZO_S3_ACCESS_KEY_ID=AKIAIOSFODNN7EXAMPLE
HUITZO_S3_SECRET_ACCESS_KEY=wJalrXUtnFEMI/K7MDENG/bPxRfiCYEXAMPLEKEY

MinIO (Self-Hosted S3-Compatible):

Add MinIO to your docker-compose.yml:

# docker-compose.yml
services:
  minio:
    image: minio/minio:latest
    command: server /data --console-address ":9001"
    restart: unless-stopped
    ports:
      - "9000:9000"   # S3 API
      - "9001:9001"   # Console
    environment:
      - MINIO_ROOT_USER=${MINIO_ACCESS_KEY:-minioadmin}
      - MINIO_ROOT_PASSWORD=${MINIO_SECRET_KEY:-minioadmin}
    volumes:
      - ./data/minio:/data
    healthcheck:
      test: ["CMD", "curl", "-f", "http://localhost:9000/minio/health/live"]
      interval: 30s
      timeout: 10s
      retries: 3

Configure Huitzo to use MinIO:

# .env
HUITZO_FILE_STORAGE_BACKEND=s3
HUITZO_S3_BUCKET=uploads
HUITZO_S3_ENDPOINT=http://minio:9000
HUITZO_S3_ACCESS_KEY_ID=minioadmin
HUITZO_S3_SECRET_ACCESS_KEY=minioadmin
HUITZO_S3_USE_SSL=false
HUITZO_S3_PATH_STYLE=true

Create the bucket after starting MinIO:

# Using MinIO client
docker compose exec minio mc alias set local http://localhost:9000 minioadmin minioadmin
docker compose exec minio mc mb local/uploads

Azure Blob Storage

# .env
HUITZO_FILE_STORAGE_BACKEND=azure
HUITZO_AZURE_STORAGE_ACCOUNT=mycompanystorage
HUITZO_AZURE_STORAGE_KEY=base64encodedkey==
HUITZO_AZURE_CONTAINER=huitzo-uploads

Google Cloud Storage

# .env
HUITZO_FILE_STORAGE_BACKEND=gcs
HUITZO_GCS_BUCKET=mycompany-huitzo-uploads
HUITZO_GCS_PROJECT=my-gcp-project
HUITZO_GCS_CREDENTIALS_FILE=/secrets/gcs-credentials.json

Mount the credentials file:

# docker-compose.yml
services:
  app:
    volumes:
      - ./secrets/gcs-credentials.json:/secrets/gcs-credentials.json:ro

When to Use Each Backend

Deployment Scenario Recommended Backend
Single instance, dev/test Local filesystem
Single instance, production Local or S3 (your choice)
Multi-instance / HA S3-compatible (required)
AWS infrastructure AWS S3
Azure infrastructure Azure Blob
GCP infrastructure Google Cloud Storage
Air-gapped / on-premises MinIO

See File Storage Backends for detailed configuration and migration guides.

HTTP Security for External Integrations

Packs that integrate with external APIs (e.g., Maxerience, Stripe) declare allowed domains in their manifests.

Configuring Global Trusted Domains

For internal services not declared by packs:

# docker-compose.yml
environment:
  HUITZO_HTTP_GLOBAL_ALLOWED_DOMAINS: "api.internal.corp,*.partner.com"

Security Model

  1. Pack declares allowed_domains in huitzo.yaml
  2. Platform enforces domain restrictions at runtime
  3. Requests to unlisted domains are blocked with HTTPSecurityError
  4. Global allowlist supplements (not overrides) pack declarations

See Configuration Reference for all HTTP options.

Installing Packs

From Files

Place pack files in the ./packs directory:

huitzo/
├── docker-compose.yml
├── .env
└── packs/
    ├── my-pack-1.0.0.tar.gz
    └── another-pack-2.0.0.tar.gz

Restart to load packs:

docker compose restart app worker

From Registry

# Using the CLI inside the container
docker compose exec app huitzo pack install my-pack

# Or via API
curl -X POST http://localhost:8080/api/v1/packs/install \
  -H "Authorization: Bearer YOUR_TOKEN" \
  -d '{"name": "my-pack", "version": "1.0.0"}'

Configuration Reference

Deployment Mode

Self-hosted deployments use the same Docker image as Huitzo Cloud, with behavior controlled by the HUITZO_DEPLOYMENT_MODE parameter:

environment:
  - HUITZO_DEPLOYMENT_MODE=self_hosted
Mode Description
cloud Multi-tenant SaaS (Huitzo-managed)
self_hosted Single-tenant, customer-managed infrastructure
edge Offline-capable edge devices (Year 3+)

The codebase is modular—self-hosted activates license validation and single-tenant logic while excluding cloud-specific code paths (billing, multi-tenant isolation). See Build Architecture for details.

Environment Variables

Variable Required Default Description
HUITZO_DEPLOYMENT_MODE ✅ cloud Must be self_hosted for self-hosted deployments
HUITZO_LICENSE_KEY ✅ - License key from Huitzo
HUITZO_SECRET_KEY ✅ - Secret key for JWT signing
HUITZO_DEFAULT_TENANT_ID - Auto-generated Fixed tenant ID for self-hosted
POSTGRES_PASSWORD ✅ - PostgreSQL password
DATABASE_URL - Auto Full database connection URL
REDIS_URL - Auto Full Redis connection URL
LOG_LEVEL - INFO Logging level
OPENAI_API_KEY - - OpenAI API key
ANTHROPIC_API_KEY - - Anthropic API key
SENDGRID_API_KEY - - SendGrid API key

Volume Mounts

Path Purpose
./data/postgres PostgreSQL data (persistent)
./data/redis Redis data (persistent)
./data/uploads User uploaded files
./packs Intelligence packs

Operations

Viewing Logs

# All services
docker compose logs -f

# Specific service
docker compose logs -f app
docker compose logs -f worker

Stopping Services

# Stop all
docker compose down

# Stop and remove volumes (⚠️ DATA LOSS)
docker compose down -v

Updating

# Pull latest images
docker compose pull

# Restart with new images
docker compose up -d

Backup

# Backup PostgreSQL
docker compose exec postgres pg_dump -U huitzo huitzo > backup.sql

# Backup all data
tar -czvf huitzo-backup-$(date +%Y%m%d).tar.gz ./data

Restore

# Restore PostgreSQL
cat backup.sql | docker compose exec -T postgres psql -U huitzo huitzo

# Restore all data
tar -xzvf huitzo-backup-20240101.tar.gz
docker compose restart

Production Recommendations

1. Use HTTPS

Put Huitzo behind a reverse proxy with TLS:

# /etc/nginx/sites-available/huitzo
server {
    listen 443 ssl http2;
    server_name huitzo.yourdomain.com;

    ssl_certificate /etc/letsencrypt/live/huitzo.yourdomain.com/fullchain.pem;
    ssl_certificate_key /etc/letsencrypt/live/huitzo.yourdomain.com/privkey.pem;

    location / {
        proxy_pass http://localhost:8080;
        proxy_http_version 1.1;
        proxy_set_header Upgrade $http_upgrade;
        proxy_set_header Connection "upgrade";
        proxy_set_header Host $host;
        proxy_set_header X-Real-IP $remote_addr;
    }
}

2. Configure Backups

Add automated daily backups:

# /etc/cron.daily/huitzo-backup
#!/bin/bash
cd /opt/huitzo
./backup.sh

3. Monitor Health

Set up monitoring with your preferred tool:

# Health check endpoint
curl -f http://localhost:8080/health || alert "Huitzo is down"

4. Log Rotation

Configure Docker log rotation in /etc/docker/daemon.json:

{
  "log-driver": "json-file",
  "log-opts": {
    "max-size": "10m",
    "max-file": "3"
  }
}

Troubleshooting

Container Won't Start

# Check logs
docker compose logs app

# Common issues:
# - Invalid license key
# - Database connection failed
# - Missing environment variables

Database Connection Failed

# Check PostgreSQL is running
docker compose ps postgres

# Check connectivity
docker compose exec app pg_isready -h postgres -U huitzo

License Validation Failed

  1. Verify your license key is correct in .env
  2. Check internet connectivity (for online validation)
  3. Contact [email protected] if issues persist

Worker Not Processing Tasks

# Check worker logs
docker compose logs worker

# Restart worker
docker compose restart worker

# Check Redis connectivity
docker compose exec worker redis-cli -h redis ping

Licensing

Self-hosted Huitzo requires a valid license key. Licenses are validated through keygen.sh integration.

License Validation Flow

┌─────────────────────────────────────────────────────────────────┐
│                     Startup                                     │
│                        │                                        │
│                        ▼                                        │
│              ┌─────────────────┐                                │
│              │ Online Check    │                                │
│              │ (keygen.sh)     │                                │
│              └────────┬────────┘                                │
│                       │                                         │
│         ┌─────────────┼─────────────┐                           │
│         │ Success     │ Failure     │                           │
│         ▼             ▼             │                           │
│   ┌──────────┐  ┌──────────────┐    │                           │
│   │ Cache    │  │ Use Cached   │    │                           │
│   │ Response │  │ Validation   │    │                           │
│   │ (7 days) │  │ (if exists)  │    │                           │
│   └──────────┘  └──────────────┘    │                           │
│         │             │             │                           │
│         ▼             ▼             │                           │
│   ┌──────────────────────────────┐  │                           │
│   │      Normal Operation        │  │                           │
│   └──────────────────────────────┘  │                           │
│                                     │                           │
│                                     ▼                           │
│                          ┌──────────────────┐                   │
│                          │ Cache Expired &  │                   │
│                          │ No Connection?   │                   │
│                          └────────┬─────────┘                   │
│                                   │                             │
│                                   ▼                             │
│                          ┌──────────────────┐                   │
│                          │  Read-Only Mode  │                   │
│                          │ (No new commands)│                   │
│                          └──────────────────┘                   │
└─────────────────────────────────────────────────────────────────┘

License Authentication

Huitzo uses license key authentication for self-hosted deployments:

# .env - Only one credential needed
HUITZO_LICENSE_KEY=ABC-123-DEF-456-789

How it works: 1. Customer receives license key from Huitzo 2. Backend validates license with Keygen.sh using license key authentication 3. Backend activates machines (hardware binding) using the same license key 4. All operations are scoped to that license only

Security Model:

Deployment Authentication Method Scope
Self-Hosted License Key (License <key>) Single license only
Cloud Product Token (Bearer <token>) Huitzo-managed (internal)

Important: Self-hosted deployments do NOT use product tokens. Product tokens grant full account management access and must remain Huitzo-side only. License key authentication follows Keygen best practices and provides the correct security boundaries for customer deployments.

License States

State Description Behavior
Valid License verified online Full functionality
Cached Offline, using cached validation Full functionality (7-day TTL)
Grace Cached validation expired Read-only mode, warning displayed
Invalid License revoked or expired Platform refuses to start

Offline Grace Period

Self-hosted deployments can operate offline for up to 7 days using cached license validation:

# Environment variables for offline mode
HUITZO_LICENSE_KEY=your-license-key
HUITZO_OFFLINE_GRACE_DAYS=7  # Default: 7 days

During the grace period: - All commands continue to work - New packs can be installed - Warning logs indicate offline status

After grace period expires: - New command executions are blocked - Existing data remains accessible - Platform logs error and suggests connectivity check

License Environment Variables

Variable Required Default Description
HUITZO_LICENSE_KEY ✅ - Your license key
HUITZO_LICENSE_VALIDATION_URL ❌ https://api.keygen.sh Validation endpoint
HUITZO_OFFLINE_GRACE_DAYS ❌ 7 Offline grace period
HUITZO_LICENSE_CHECK_INTERVAL ❌ 86400 Re-validation interval (seconds)

Troubleshooting License Issues

License validation failed:

# Check license key format
echo $HUITZO_LICENSE_KEY | head -c 10

# Test connectivity to keygen.sh
curl -I https://api.keygen.sh/v1/ping

# Check container logs
docker compose logs app | grep -i license

Stuck in read-only mode:

# Force license re-validation
docker compose exec app huitzo license validate

# Clear cached validation (will require online check)
docker compose exec app huitzo license clear-cache

SaaS vs Self-Hosted Licensing

Aspect SaaS Self-Hosted
Billing Stripe subscription Annual/monthly license fee
Validation Automatic (Stripe webhooks) keygen.sh API
Offline Not applicable 7-day grace period
Usage Limits Subscription tier None (customer manages infra)

Testing

Running License Validation Integration Tests

Integration tests for the licensing subsystem live at:

apps/backend/tests/integration/test_licensing_e2e.py

The tests use respx to intercept outbound HTTP requests, so they run without any live credentials by default:

# Run all mocked integration tests
uv run pytest apps/backend/tests/integration/test_licensing_e2e.py -v

# Run only a specific scenario group
uv run pytest apps/backend/tests/integration/test_licensing_e2e.py \
  -k "TestValidationFlow" -v

Test scenario groups

Class Covers
TestValidationFlow All keygen response codes (VALID, SUSPENDED, EXPIRED, OVERDUE, BANNED, NO_MACHINE)
TestStartupValidation Missing key, valid key, invalid key at boot time
TestMiddlewareResponses 402 on INVALID, X-License-Warning on GRACE, public-path bypass
TestCachePersistence Cache write/read, restart survival, corruption recovery, grace timing
TestNetworkFailureScenarios Offline with/without cache, HTTP timeout, 5xx fallback
TestKeygenSandboxE2E Live keygen.sh sandbox (skipped without credentials)

Live Sandbox Tests (optional)

To run the live TestKeygenSandboxE2E tests you need a keygen.sh sandbox account and the following environment variables:

export KEYGEN_SANDBOX_ACCOUNT_ID="your-sandbox-account-id"
export KEYGEN_SANDBOX_PRODUCT_ID="your-sandbox-product-id"
export KEYGEN_SANDBOX_PRODUCT_TOKEN="your-sandbox-product-token"
export KEYGEN_SANDBOX_VALID_LICENSE_KEY="a-valid-key-in-your-sandbox"
export KEYGEN_SANDBOX_SUSPENDED_LICENSE_KEY="a-suspended-key"  # optional

uv run pytest apps/backend/tests/integration/test_licensing_e2e.py \
  -k "TestKeygenSandboxE2E" -v

CI – Automated License Integration Tests

A dedicated CI workflow runs these tests automatically:

  • Scheduled: every Monday at 04:00 UTC
  • On push: when licensing code changes land on main
  • Manual: via the GitHub Actions "License Integration Tests" workflow

Configure the following repository secrets to enable live sandbox tests in CI:

Secret Description
KEYGEN_SANDBOX_ACCOUNT_ID Sandbox account ID
KEYGEN_SANDBOX_PRODUCT_ID Sandbox product ID
KEYGEN_SANDBOX_PRODUCT_TOKEN Sandbox product token
KEYGEN_SANDBOX_VALID_LICENSE_KEY A valid license key
KEYGEN_SANDBOX_SUSPENDED_LICENSE_KEY A suspended license key (optional)

Next Steps