File Storage Backends

Last updated: 2026-05-07 06:53:50

storage files s3 azure gcs minio

File Storage Backends¶

Huitzo uses a pluggable file storage backend architecture to support different deployment scenarios. The ctx.files API in your packs works identically regardless of which backend is configured.

Overview¶

Storage Abstraction¶

Pack developers use the ctx.files API without knowing which storage backend is configured:

# This code works with any backend (local, S3, Azure, GCS)
@command("process-data", namespace="analytics")
async def process_data(args: Args, ctx: Context) -> dict:
    # Read uploaded file
    df = await ctx.files.read_excel(args.file_path)

    # Process data...
    result = analyze(df)

    # Write output
    await ctx.files.write("output/report.csv", result.to_csv())

    return {"status": "complete"}

Backend Selection¶

The storage backend is selected via the HUITZO_FILE_STORAGE_BACKEND environment variable:

Value	Backend	Best For
`local`	Local filesystem	Single-instance, dev/test
`s3`	S3-compatible	Production, multi-instance, AWS
`azure`	Azure Blob Storage	Azure deployments
`gcs`	Google Cloud Storage	GCP deployments

Default: local (simplest setup, no external dependencies)

Backend Options¶

Local Filesystem (Default)¶

The simplest option—files are stored on the local filesystem.

When to use: - Single-instance deployments - Development and testing - When HA/multi-instance is not required

Configuration:

# .env
HUITZO_FILE_STORAGE_BACKEND=local
HUITZO_FILE_STORAGE_PATH=/app/uploads  # Optional, this is the default

Docker volume mount:

# docker-compose.yml
services:
  app:
    volumes:
      - ./data/uploads:/app/uploads

Limitations: - Does not work for multi-instance deployments (files not shared) - Requires backup strategy for the volume - Limited by local disk space

S3-Compatible Storage¶

Works with AWS S3, MinIO, DigitalOcean Spaces, Backblaze B2, and any S3-compatible service.

When to use: - Multi-instance / HA deployments - AWS deployments (native S3) - Self-hosted with MinIO - Any production deployment requiring shared file access

Configuration (AWS S3):

# .env
HUITZO_FILE_STORAGE_BACKEND=s3
HUITZO_S3_BUCKET=mycompany-huitzo-uploads
HUITZO_S3_REGION=us-east-1
HUITZO_S3_ACCESS_KEY_ID=AKIAIOSFODNN7EXAMPLE
HUITZO_S3_SECRET_ACCESS_KEY=wJalrXUtnFEMI/K7MDENG/bPxRfiCYEXAMPLEKEY

Configuration (MinIO - Self-Hosted):

# .env
HUITZO_FILE_STORAGE_BACKEND=s3
HUITZO_S3_BUCKET=uploads
HUITZO_S3_ENDPOINT=http://minio:9000
HUITZO_S3_ACCESS_KEY_ID=minioadmin
HUITZO_S3_SECRET_ACCESS_KEY=minioadmin
HUITZO_S3_USE_SSL=false
HUITZO_S3_PATH_STYLE=true  # Required for MinIO

Configuration (DigitalOcean Spaces):

# .env
HUITZO_FILE_STORAGE_BACKEND=s3
HUITZO_S3_BUCKET=my-space-name
HUITZO_S3_REGION=nyc3
HUITZO_S3_ENDPOINT=https://nyc3.digitaloceanspaces.com
HUITZO_S3_ACCESS_KEY_ID=your-spaces-key
HUITZO_S3_SECRET_ACCESS_KEY=your-spaces-secret

All S3 Environment Variables:

Variable	Required	Default	Description
`HUITZO_S3_BUCKET`	Yes	-	Bucket name
`HUITZO_S3_REGION`	Yes*	`us-east-1`	AWS region (*required for AWS)
`HUITZO_S3_ENDPOINT`	No	AWS default	Custom endpoint for MinIO/Spaces
`HUITZO_S3_ACCESS_KEY_ID`	Yes	-	Access key
`HUITZO_S3_SECRET_ACCESS_KEY`	Yes	-	Secret key
`HUITZO_S3_USE_SSL`	No	`true`	Use HTTPS
`HUITZO_S3_PATH_STYLE`	No	`false`	Use path-style URLs (for MinIO)

Azure Blob Storage¶

Native Azure Blob Storage integration for Azure deployments.

When to use: - Azure-hosted deployments - Existing Azure infrastructure - Azure-native features required (lifecycle management, etc.)

Configuration:

# .env
HUITZO_FILE_STORAGE_BACKEND=azure
HUITZO_AZURE_STORAGE_ACCOUNT=mycompanystorage
HUITZO_AZURE_STORAGE_KEY=base64encodedkey==
HUITZO_AZURE_CONTAINER=huitzo-uploads

Using Managed Identity (recommended for Azure VMs/AKS):

# .env
HUITZO_FILE_STORAGE_BACKEND=azure
HUITZO_AZURE_STORAGE_ACCOUNT=mycompanystorage
HUITZO_AZURE_CONTAINER=huitzo-uploads
HUITZO_AZURE_USE_MANAGED_IDENTITY=true

All Azure Environment Variables:

Variable	Required	Default	Description
`HUITZO_AZURE_STORAGE_ACCOUNT`	Yes	-	Storage account name
`HUITZO_AZURE_STORAGE_KEY`	Yes*	-	Storage account key
`HUITZO_AZURE_CONTAINER`	Yes	-	Container name
`HUITZO_AZURE_USE_MANAGED_IDENTITY`	No	`false`	Use managed identity instead of key
`HUITZO_AZURE_CONNECTION_STRING`	No	-	Full connection string (alternative)

*Required unless using managed identity or connection string.

Google Cloud Storage¶

Native GCS integration for Google Cloud deployments.

When to use: - GCP-hosted deployments - Existing GCP infrastructure - GCS-native features required

Configuration (Service Account):

# .env
HUITZO_FILE_STORAGE_BACKEND=gcs
HUITZO_GCS_BUCKET=mycompany-huitzo-uploads
HUITZO_GCS_PROJECT=my-gcp-project
HUITZO_GCS_CREDENTIALS_FILE=/secrets/gcs-credentials.json

Using Workload Identity (recommended for GKE):

# .env
HUITZO_FILE_STORAGE_BACKEND=gcs
HUITZO_GCS_BUCKET=mycompany-huitzo-uploads
HUITZO_GCS_PROJECT=my-gcp-project
# No credentials needed - uses workload identity

All GCS Environment Variables:

Variable	Required	Default	Description
`HUITZO_GCS_BUCKET`	Yes	-	Bucket name
`HUITZO_GCS_PROJECT`	Yes	-	GCP project ID
`HUITZO_GCS_CREDENTIALS_FILE`	No*	-	Path to service account JSON
`HUITZO_GCS_CREDENTIALS_JSON`	No*	-	Inline JSON credentials

*Not required if using workload identity or default credentials.

Multi-Instance Deployments¶

Why Local Filesystem Doesn't Work¶

In multi-instance deployments, each container has its own filesystem:

Instance 1                    Instance 2
┌─────────────────┐          ┌─────────────────┐
│ /app/uploads/   │          │ /app/uploads/   │
│ └── file-a.csv  │          │ (empty)         │
└─────────────────┘          └─────────────────┘

User uploads file-a.csv to Instance 1
User request routed to Instance 2 → File not found!

Shared Storage Solution¶

Object storage provides a shared backend accessible by all instances:

Instance 1                    Instance 2
┌─────────────────┐          ┌─────────────────┐
│ Huitzo App      │          │ Huitzo App      │
└────────┬────────┘          └────────┬────────┘
         │                            │
         └──────────┬─────────────────┘
                    │
                    ▼
         ┌─────────────────┐
         │   S3 / MinIO    │
         │ ┌─────────────┐ │
         │ │ file-a.csv  │ │
         │ │ file-b.pdf  │ │
         │ └─────────────┘ │
         └─────────────────┘

All instances see the same files!

Tenant Isolation¶

Files are automatically isolated by tenant using path prefixes:

bucket/
├── tenant/
│   ├── {tenant-id-1}/
│   │   ├── uploads/
│   │   │   └── user-file.csv
│   │   └── output/
│   │       └── report.pdf
│   └── {tenant-id-2}/
│       ├── uploads/
│       └── output/
└── system/
    └── packs/

Pack code uses relative paths—the platform automatically prepends the tenant prefix:

# Pack code writes to "output/report.pdf"
await ctx.files.write("output/report.pdf", data)

# Actually stored at: tenant/{tenant_id}/output/report.pdf

File Limits and Quotas¶

Default Limits¶

Limit	Default	Configurable
Max file size	100 MB	Yes
Max files per user	1000	Yes
Storage quota per tenant	10 GB	Yes
Allowed file types	All	Yes

Configuration¶

# .env
HUITZO_FILE_MAX_SIZE_MB=100
HUITZO_FILE_MAX_PER_USER=1000
HUITZO_FILE_QUOTA_GB=10
HUITZO_FILE_ALLOWED_TYPES=.csv,.xlsx,.json,.pdf,.txt,.png,.jpg

Pack-Level Restrictions¶

Packs can declare their own file requirements in the manifest:

# huitzo.yaml
services:
  files:
    max_size_mb: 50
    allowed_types:
      - .csv
      - .xlsx
      - .json

Migration¶

Local to S3 Migration¶

Create S3 bucket: bash aws s3 mb s3://mycompany-huitzo-uploads
Sync existing files: bash aws s3 sync ./data/uploads s3://mycompany-huitzo-uploads/
Update configuration: bash # .env HUITZO_FILE_STORAGE_BACKEND=s3 HUITZO_S3_BUCKET=mycompany-huitzo-uploads HUITZO_S3_REGION=us-east-1 HUITZO_S3_ACCESS_KEY_ID=xxx HUITZO_S3_SECRET_ACCESS_KEY=xxx
Restart services: bash docker compose up -d
Verify: bash # Test file access through the API curl http://localhost:8080/health

Security¶

Encryption at Rest¶

Backend	Encryption
Local	Use encrypted filesystem (LUKS, BitLocker)
S3	Enable SSE-S3 or SSE-KMS
Azure	Enabled by default (Azure Storage encryption)
GCS	Enabled by default (Google-managed keys)

S3 with SSE-KMS:

HUITZO_S3_SSE_TYPE=aws:kms
HUITZO_S3_SSE_KMS_KEY_ID=arn:aws:kms:us-east-1:123456789:key/xxx

Encryption in Transit¶

All backends use TLS/HTTPS by default. For S3-compatible backends:

HUITZO_S3_USE_SSL=true  # Default

Access Control¶

Tenant isolation: Automatic path-based isolation
User isolation: Optional, configurable per pack
Presigned URLs: Time-limited access for direct downloads

Troubleshooting¶

S3 Connection Errors¶

# Test S3 connectivity
aws s3 ls s3://${HUITZO_S3_BUCKET}/ --region ${HUITZO_S3_REGION}

# For MinIO
mc alias set myminio ${HUITZO_S3_ENDPOINT} ${HUITZO_S3_ACCESS_KEY_ID} ${HUITZO_S3_SECRET_ACCESS_KEY}
mc ls myminio/${HUITZO_S3_BUCKET}

Azure Connection Errors¶

# Test Azure connectivity
az storage blob list --account-name ${HUITZO_AZURE_STORAGE_ACCOUNT} --container-name ${HUITZO_AZURE_CONTAINER}

GCS Connection Errors¶

# Test GCS connectivity
gsutil ls gs://${HUITZO_GCS_BUCKET}/

Permission Denied Errors¶

Verify credentials are correct
Check bucket/container exists
Verify IAM permissions include read/write
Check tenant prefix path exists (or can be created)

Configuration Reference – All environment variables
Self-Hosting Deployment – Deployment guide
Files Integration – SDK ctx.files API
Architecture Overview – System architecture