Security Architecture¶

Huitzo prioritizes the "Security Nightmare" of third-party code execution through a multi-stage isolation strategy and comprehensive security model.

Authentication¶

JWT Access Tokens¶

All authenticated API requests use short-lived JWT access tokens:

• Algorithm: HS256 (symmetric, configurable via HUITZO_JWT_ALGORITHM)
• Lifetime: 15 minutes (configurable via HUITZO_ACCESS_TOKEN_EXPIRE_MINUTES)
• Signed with: HUITZO_JWT_SECRET_KEY (must be cryptographically random in production)

JWT Payload:

{
  "sub": "<user_id>",
  "tenant_id": "<tenant_id>",
  "role": "member|admin|owner",
  "tier": "free|developer_preview|pro|enterprise",
  "exp": 1708000000,
  "type": "access"
}

The tier claim enables rate limiting decisions at the middleware layer without database lookups. Tier changes (e.g., after Stripe subscription upgrade) take effect on next token refresh.

Refresh Tokens¶

Refresh tokens are opaque random strings (two concatenated UUID4 hex values). They are:

• Stored server-side: Hashed with HMAC-SHA256 in the sessions table
• Lifetime: 7 days
• Single-use: Each refresh generates a new refresh token (rotation)
• Revocable: Logout invalidates the session and its refresh token

Password Hashing¶

User passwords are hashed with bcrypt using automatically generated salts. The bcrypt work factor uses the library default (currently 12 rounds).

API Keys (Machine-to-Machine)¶

API keys enable CI/CD pipelines and scripts to authenticate without interactive login. Format: sk-huitzo-<64 hex chars> (256-bit entropy via secrets.token_hex(32)).

• Hashing: argon2id (OWASP-recommended parameters via argon2-cffi)
• Storage: Only the argon2id hash is persisted; plaintext returned once at creation
• Scopes: ["commands:execute"] (v1 allowlist, validated at creation)
• Lookup: By key_prefix (first 16 chars), then constant-time argon2id verification
• Revocation: Soft-delete via revoked_at timestamp
• Audit: last_used_at updated fire-and-forget on each use
• Self-management: Keys cannot create, list, or revoke other keys (JWT required)

Auth Middleware Flow¶

Request
  │
  ├─ Path in PUBLIC_PATHS? → Skip auth → call_next()
  │
  ├─ Missing/malformed Authorization header? → 401
  │
  ├─ Token starts with "sk-huitzo-"?
  │    ├─ Format invalid (not 64 hex after prefix)? → 401
  │    ├─ Hash lookup → no match or revoked? → 401 (no JWT fallthrough)
  │    └─ Match → UserContext(role="member", tier="free", _from_api_key=True) → call_next()
  │
  └─ Decode JWT
       ├─ Invalid/expired → 401
       └─ Valid → UserContext(user_id, tenant_id, role, tier) → call_next()

The sk-huitzo- prefix is checked before JWT decode — cheaper and avoids timing-oracle confusion between key and token failures.

Public Paths (no authentication required):

Tenant Isolation¶

Database Level — Row-Level Security (RLS)¶

PostgreSQL Row-Level Security provides tenant isolation at the database level. Every tenant-scoped table has RLS policies enforced via SET LOCAL:

-- Enable RLS on tenant-scoped tables
ALTER TABLE pack_data ENABLE ROW LEVEL SECURITY;
ALTER TABLE pack_data FORCE ROW LEVEL SECURITY;

-- Tenant isolation policy
CREATE POLICY tenant_isolation ON pack_data
    FOR ALL TO app_user
    USING (tenant_id = current_setting('app.tenant_id')::uuid)
    WITH CHECK (tenant_id = current_setting('app.tenant_id')::uuid);

How RLS context is set:

Every database session begins with:

SET LOCAL app.tenant_id = '<tenant_id>';

This is transaction-scoped (SET LOCAL) so it cannot leak between requests. The get_session(tenant_id=...) context manager handles this automatically.

Bootstrap policies: Some tables (e.g., billing_events) have additional policies that allow INSERT without tenant context. This enables event tracking during registration (before a user has a session with tenant context).

Admin RLS Sentinel¶

Platform-staff admin endpoints (/api/v1/admin/*, /api/v1/metrics/*) need to read across tenants — for example, listing every paid signup or computing the cross-tenant funnel from §2 of the paid-registration roadmap. Granting these endpoints BYPASSRLS would erase the audit story for cross-tenant reads.

Instead, an explicit sentinel value is used. Admin-context database sessions set:

-- pseudocode — admin sentinel
SET LOCAL app.tenant_id = '__admin__';

Each admin-readable table carries an additional policy:

-- pseudocode — admin read policy
CREATE POLICY admin_read ON <table>
    FOR SELECT TO huitzo_app
    USING (current_setting('app.tenant_id', true) = '__admin__');

The sentinel '__admin__' is intentionally not a UUID, so it cannot collide with any real tenant id. The with_admin_context dependency (PR-PR04 in apps/backend/src/backend/auth/dependencies.py) is the only application-side path that sets it; every use writes a row to admin_audit_log per the roadmap audit hook. Cross-tenant writes are not granted by this policy — admin writes go through normal application logic that sets per-target-tenant context, so an admin "grant a comp code to user X" is recorded against X's tenant scope, not the admin sentinel.

This pattern keeps RLS as the single source of truth for cross-tenant access while making the staff-vs-tenant distinction visible in every query plan.

Multi-Tenancy Model¶

Tables Without RLS¶

Some tables are intentionally NOT RLS-protected:

• access_codes — Admin-only, accessed during registration without tenant context
• users — Queried by Stripe customer_id in webhook handlers (cross-tenant)
• tenants — Updated by webhook handlers

These tables use explicit authorization checks (e.g., require_role("owner", "admin")) instead of RLS.

Access Code Security¶

Access codes control platform access during early launch. Security considerations:

• Cryptographic generation: Codes use secrets.token_hex(2).upper() per segment (format: HUITZO-XXXX-XXXX), providing 32 bits of entropy per code
• Use limits: Each code has max_uses (default 1) and use_count tracking
• Expiration: Optional expires_at timestamp
• Deactivation: Admins can deactivate codes via DELETE /api/v1/access-codes/{code_id}
• Required for registration: No public signup without a valid access code
• Validation endpoint is public: POST /api/v1/access-codes/validate allows pre-registration validation without authentication

Founding Customer Tracking¶

Access codes carry a cohort tag (e.g., founding-feb15) that is permanently stored on the user record. This provides:

• Audit trail of how each user gained access
• Permanent founding customer identification for future benefits
• Cohort-based analytics in the metrics dashboard

Stripe Webhook Security¶

Stripe webhooks bypass JWT authentication and use Stripe signature verification instead:

stripe.Webhook.construct_event(
    payload=raw_body,
    sig_header=request.headers["stripe-signature"],
    secret=settings.stripe_webhook_secret,
)

• Signature algorithm: Stripe uses HMAC-SHA256 with a per-endpoint webhook secret
• Timestamp tolerance: Stripe SDK rejects events older than 5 minutes (replay protection)
• No JWT required: The /api/v1/billing/webhook path is in PUBLIC_PATHS
• Idempotency: Webhook handlers are designed to be idempotent (safe to replay)

Rate Limiting¶

Rate limiting uses Redis INCR/EXPIRE counters keyed by user ID and time window:

rate:{user_id}:{endpoint_type}:{minute_key}

Tier Limits¶

Design Decisions¶

• Simple counters (not sliding window): INCR + EXPIRE for simplicity. Acceptable for launch; upgrade to sliding window post-launch if needed.
• Fail-open: If Redis is unavailable, requests are allowed through (availability over strictness).
• Kill switch: HUITZO_RATE_LIMIT_ENABLED=false disables all rate limiting.
• Exempt paths: Health checks, metrics, docs, and OpenAPI endpoints are never rate-limited.
• Only authenticated requests: Unauthenticated requests (public endpoints) are not rate-limited.

Rate Limit Headers¶

Responses include:

Secrets Management¶

Three-Tier Model¶

Platform secrets are managed via environment variables and never exposed to pack code.

Pack secrets are defined in the pack manifest and configured per-deployment.

User secrets (v0.2+) will be encrypted at rest with AES-256-GCM, scoped to user + pack, and accessible only via ctx.secrets.

Pack Data Plaintext Storage Policy¶

The pack_data table stores pack state as unencrypted PostgreSQL JSONB. RLS provides tenant isolation but not confidentiality — any row is readable by anyone with direct database access.

Policy: Packs MUST NOT store secrets, API keys, tokens, or credentials in ctx.storage. Sensitive values must use ctx.secrets instead.

Enforcement (partial): The storage backend rejects save() calls where the data dict contains keys ending with _key, _token, _secret, _password, _credential, or _credentials (checked recursively). This catches the most common accidental cases.

# pseudocode
if any field in data matches *_key, *_token, *_secret, *_password, *_credential, *_credentials:
    raise StorageError("use ctx.secrets for sensitive values")

Known enforcement gap: The suffix check is heuristic. A pack can still store secrets under non-conventional field names (e.g., auth, bearer, creds). Column-level encryption is required to close this gap fully.

Tracking: Column-level encryption for pack_data.value is a planned post-v0.2 requirement. Until then, pack authors are responsible for not placing secrets in storage.

Column-level encryption¶

Selected sensitive columns are encrypted at rest with AES-256-GCM via the EncryptedString SQLAlchemy TypeDecorator (apps/backend/src/backend/core/models/encrypted_type.py).

• Algorithm: AES-256-GCM with a fresh 96-bit random nonce per write. Ciphertext is stored as <nonce_b64>:<ciphertext_b64> in a TEXT column. Each write produces a distinct ciphertext — the encoding does not leak equality.
• Key source: HUITZO_SECRETS_ENCRYPTION_KEY (64-char hex = 32 bytes). The same key is used for both AES-GCM encryption and HMAC-SHA256 lookup hashing; a 32-byte key is sufficient for both primitives and avoids the operational cost of introducing a second secret.
• Tamper detection: GCM's authenticator tag guarantees integrity. On decrypt, cryptography.exceptions.InvalidTag, binascii.Error, and UnicodeDecodeError are caught, a warning is logged (no PII), and the field returns None. A single corrupted row cannot crash a SELECT.
• Type safety: process_bind_param rejects non-str inputs with TypeError before encrypting — no silent coercion.

Equality lookups via HMAC-SHA256 sidecar: Randomized AES-GCM means WHERE encrypted_col = ? cannot work. For columns that need equality lookups (e.g. Stripe webhook customer_id → user) we add a sibling *_lookup_hash VARCHAR(64) column populated with hmac.new(key, value.utf-8, sha256).hexdigest() and a btree index on it. The application keeps the two columns in sync via a SQLAlchemy @validates hook on the model, and lookups are rewritten to compare the hash.

Why HMAC sidecar, not AES-SIV: AES-256-SIV requires a 64-byte key; the existing platform key is 32 bytes. The sidecar pattern stays within the 32-byte envelope, keeps ciphertext indistinguishable under chosen-plaintext attack (AES-GCM is IND-CPA, AES-SIV is not), and remains auditable.

Current users:

See apps/backend/src/backend/core/models/user.py for the User model and the @validates("stripe_customer_id") sync hook.

Stripe Customer ID Plaintext Storage¶

Resolved in alembic revision 020 — users.stripe_customer_id is now encrypted at rest via AES-256-GCM, with an HMAC-SHA256 sidecar (stripe_customer_id_lookup_hash) preserving indexed equality lookups for webhook handlers. See Column-level encryption.

Defense-in-depth mitigations that remain in force: - The field is never returned in any API response (enforced via dedicated response schemas in billing.py) - Database access is restricted to service-account credentials; no direct public access - Stripe API calls require the platform's secret key in addition to the customer ID

Isolation Strategy¶

Phase 1: Docker Containerization (Current)¶

All packs execute within resource-limited Docker containers. Environment variables are strictly scoped, and the filesystem is isolated via tenant-specific volume mounts.

Security Evolution: As Huitzo moves toward a public marketplace, Docker isolation may not be sufficient for high-trust environments. Goal: Transition from Docker to microVMs (e.g., Firecracker) for pack execution. Reason: MicroVMs provide hardware-level isolation, ensuring a malicious Pack cannot read host environment variables or escape to the host kernel.

Authorization¶

Role-Based Access Control¶

Role checks use the require_role() dependency:

@router.get("/admin-only")
async def admin_endpoint(user: UserContext = Depends(require_role("owner", "admin"))):
    ...

Role-Based Data Visibility¶

Beyond capability gating, role determines the scope of data visibility for tenant-scoped resources:

Design rationale: Admin and owner roles carry operational responsibility for their tenant. Tenant-wide observability (e.g., seeing stuck executions, orphaned sessions) is a necessary operational capability. This is explicitly authorized and scoped to the tenant — it does not extend across tenant boundaries.

Cross-tenant isolation is absolute: PostgreSQL RLS enforces tenant_id on all tenant-scoped tables. No role — including admin and owner — can read data from a different tenant. This invariant is enforced at the database level and cannot be bypassed by application code.

Security invariants:

See System Monitor and Dev Sessions for per-resource access control tables.

Subscription-Based Access Control¶

The subscription_tier in the JWT payload determines:

• Rate limits (see Rate Limiting above)
• Feature access (future: premium-only commands)
• Command execution quotas

Code Protection (Self-Hosted)¶

• Nuitka compilation: Platform compiled to optimized binary
• Pack signing: Hash verification prevents tampering
• License validation: keygen.sh for key management

Context Injection¶

Every command receives a Context object with deployment-aware services:

@dataclass
class Context:
    # Identity
    user_id: UUID
    tenant_id: UUID
    session_id: UUID
    correlation_id: str

    # Deployment
    deployment_mode: DeploymentMode = DeploymentMode.CLOUD

    # Services (backend selected based on deployment_mode)
    storage: StorageBackend
    llm: LLMService
    email: EmailService
    files: FileService
    http: HTTPService
    log: LogService

Commands can check ctx.deployment_mode to adjust behavior, though most commands should work identically across all modes. The correlation_id enables distributed tracing across command execution.

HTTP Integration Threat Model¶

When a tenant configures an HTTP integration, the integration's base_url host is unconditionally permitted for outbound requests issued by that tenant's packs (in addition to any allowed_domains declared in the pack manifest or the integration). The implication is that a tenant who can write integration config can direct any pack they install to issue HTTPS requests to whatever host they name.

These guardrails still hold and run before the base_url-host permit:

• Cloud metadata blocklist — requests to 169.254.169.254 (AWS / Azure IMDS), metadata.google.internal (GCP), and 100.100.100.200 (Alibaba) are rejected even if a tenant sets one of those as base_url. SSRF to cloud instance credentials is not reachable.
• HTTPS-only — a base_url starting with http:// passes the config-layer validator (^https?:// accepts both schemes) and can be stored, but it is blocked at request time by the SDK's _check_domain, before the host is contacted. Plain-HTTP downgrade is not reachable from pack code regardless of the stored value.
• Tenant scoping — integration rows are filtered by tenant_id in the resolver, so one tenant's base_url cannot be read or invoked by another tenant's command.

What is not blocked and is accepted under this threat model:

• RFC 1918 / link-local hosts — a self-hosted tenant who runs the Huitzo runtime inside a private VPC can name internal HTTPS endpoints (https://10.0.0.5/..., https://kubernetes.default.svc/...) as their base_url. This is intentional for the "tenant runs Huitzo on the same VPC as their internal API" use case and is the same actor as the would-be attack target.
• Request-time logging of basic-auth userinfo — if a tenant sets base_url=https://user:pass@host, the credentials may surface in the HTTP backend's retry log prefix. Tenants are advised to use bearer_token / basic_auth integration secrets instead, which are applied at request time rather than embedded in the URL.

Pack developers consuming this model: see SDK Integrations — Relative Paths and base_url for the public-facing contract and the SDK's enforcement order.

Production Security Checklist¶

Related Documentation¶

• Architecture Overview - High-level platform architecture
• Deployment Modes - Mode comparison
• Storage Backends - Storage isolation
• Self-Hosting Guide - Deployment security
• Billing Guide - Subscription tiers and Stripe integration