Storage Backends Reference

Storage Backends Reference

Huitzo uses a pluggable storage backend architecture to support different deployment modes. This document describes the storage backend protocol and available implementations.

Overview

Huitzo uses PostgreSQL as the single database for all deployments. This "simple by design, scalable by architecture" approach minimizes self-hosted dependencies while providing enterprise-grade capabilities through PostgreSQL's JSONB and RLS features.

Backend Deployment Mode Status Use Case
PostgreSQLBackend Cloud, Self-Hosted Production All data: user, tenant, pack registry, pack data (JSONB)
SQLiteBackend Edge Stubbed Offline local storage (Year 3+)

Storage Backend Protocol

All storage backends implement a common protocol:

from typing import Protocol, Optional, List, Any

class StorageBackend(Protocol):
    """Abstract storage interface enabling multiple backends."""

    async def save(self, key: str, data: dict, **options) -> str:
        """
        Save data to storage.

        Args:
            key: Storage key (max 256 characters)
            data: JSON-serializable dictionary
            **options: Backend-specific options
                - ttl: Time-to-live in seconds (optional)
                - scope: "user" | "tenant" | "pack" (default: "user")
                - encrypt: Enable field-level encryption (future)

        Returns:
            The storage identifier (usually the key)

        Raises:
            StorageError: If save operation fails
        """
        ...

    async def get(self, key: str) -> Optional[dict]:
        """
        Retrieve data from storage.

        Args:
            key: Storage key to retrieve

        Returns:
            The stored data, or None if not found

        Raises:
            StorageError: If retrieval fails
        """
        ...

    async def list(self, prefix: str, **filters) -> List[dict]:
        """
        List data matching prefix and filters.

        Args:
            prefix: Key prefix to match
            **filters: Backend-specific filters
                - limit: Maximum results (default: 100)
                - offset: Pagination offset (default: 0)
                - metadata: Metadata filters (optional)

        Returns:
            List of matching records

        Raises:
            StorageError: If listing fails
        """
        ...

    async def delete(self, key: str) -> bool:
        """
        Delete data from storage.

        Args:
            key: Storage key to delete

        Returns:
            True if deleted, False if not found

        Raises:
            StorageError: If deletion fails
        """
        ...

PostgreSQLBackend

The primary backend for cloud and self-hosted deployments. Uses PostgreSQL 17 with JSONB columns and Row-Level Security (RLS).

Configuration

# Environment variables
HUITZO_STORAGE_BACKEND: "postgresql"
DATABASE_URL: "postgresql://user:pass@host:5432/huitzo"

Features

  • Row-Level Security: Automatic tenant isolation at database level
  • JSONB Storage: Efficient storage and querying of JSON data
  • Full-Text Search: PostgreSQL GIN indexes for searching within JSONB
  • Transactions: ACID-compliant transactions via ctx.storage.transaction()
  • TTL Support: Automatic cleanup of expired keys

Schema

CREATE TABLE pack_data (
    id UUID PRIMARY KEY DEFAULT gen_random_uuid(),
    tenant_id UUID NOT NULL,
    user_id UUID NOT NULL,
    pack_id VARCHAR(256) NOT NULL,
    key VARCHAR(256) NOT NULL,
    value JSONB NOT NULL,
    metadata JSONB DEFAULT '{}',
    created_at TIMESTAMP WITH TIME ZONE DEFAULT NOW(),
    updated_at TIMESTAMP WITH TIME ZONE DEFAULT NOW(),
    expires_at TIMESTAMP WITH TIME ZONE,

    UNIQUE(tenant_id, user_id, pack_id, key)
);

-- Row-Level Security
ALTER TABLE pack_data ENABLE ROW LEVEL SECURITY;
ALTER TABLE pack_data FORCE ROW LEVEL SECURITY;

CREATE POLICY tenant_isolation ON pack_data
    FOR ALL TO app_user
    USING (tenant_id = current_setting('app.tenant_id')::uuid)
    WITH CHECK (tenant_id = current_setting('app.tenant_id')::uuid);

Usage

# The SDK automatically selects PostgreSQLBackend for cloud/self-hosted

@command("save-data", namespace="example")
async def save_data(args: Args, ctx: Context) -> dict:
    # Storage is automatically backed by PostgreSQL
    await ctx.storage.save("my-key", {"field": "value"})
    data = await ctx.storage.get("my-key")
    return data

SQLiteBackend (Future - Year 3+)

Local storage backend for edge deployments. Enables offline operation with sync-when-connected capability.

Status

Current: Stubbed - raises NotImplementedError

class SQLiteBackend(StorageBackend):
    """Edge-capable local storage. Available in Year 3+."""

    async def save(self, key: str, data: dict, **options) -> str:
        raise NotImplementedError("Edge mode not yet available")

    async def get(self, key: str) -> Optional[dict]:
        raise NotImplementedError("Edge mode not yet available")

    async def list(self, prefix: str, **filters) -> List[dict]:
        raise NotImplementedError("Edge mode not yet available")

    async def delete(self, key: str) -> bool:
        raise NotImplementedError("Edge mode not yet available")

Planned Features (Year 3+)

  • Offline Operation: Full functionality without network
  • Local Encryption: Data encrypted at rest on edge device
  • Sync Protocol: Bidirectional sync when connected
  • Conflict Resolution: Last-write-wins or CRDT-based (TBD)

Trigger for Implementation

SQLiteBackend will be implemented when: - First customer requires air-gapped deployment - Data sovereignty requirements prevent cloud storage - Edge device deployment validated with design partners


Backend Selection

The platform selects storage backend based on deployment mode:

def get_storage_backend(ctx: CommandContext) -> StorageBackend:
    """Select appropriate storage backend."""

    if ctx.deployment_mode == DeploymentMode.EDGE:
        # Future: Return SQLite backend for edge deployments
        raise NotImplementedError("Edge mode not yet available")

    # PostgreSQL is the single database for all cloud/self-hosted deployments
    # Simple by design, scalable by architecture
    return PostgreSQLBackend(
        dsn=settings.DATABASE_URL,
        tenant_id=ctx.tenant_id,
        user_id=ctx.user_id
    )

Configuration Reference

Variable Default Description
DATABASE_URL Required PostgreSQL connection string

Writing Backend-Agnostic Code

Intelligence packs should work across all backends. Follow these guidelines:

Do: Use Standard Operations

# Works on all backends
await ctx.storage.save("key", {"data": "value"})
data = await ctx.storage.get("key")
await ctx.storage.delete("key")

Do: Use Supported Options

# TTL is supported on all backends
await ctx.storage.save("cache:key", data, ttl=3600)

# Scope is supported on all backends
await ctx.storage.save("shared:key", data, scope="tenant")

Avoid: Backend-Specific Features

# Avoid: PostgreSQL-specific queries
# These won't work on SQLite when edge mode is available
await ctx.storage._raw_query("SELECT * FROM pack_data WHERE value->>'field' = 'x'")

Check Deployment Mode (When Necessary)

@command("sync-data", namespace="example")
async def sync_data(args: Args, ctx: Context) -> dict:
    if ctx.deployment_mode == DeploymentMode.EDGE:
        # Edge-specific behavior (future)
        return {"status": "offline", "queued_for_sync": True}
    else:
        # Cloud/self-hosted behavior
        await ctx.storage.save("data", args.payload)
        return {"status": "saved"}

Error Handling

All backends raise consistent errors:

from huitzo_sdk.errors import StorageError

try:
    await ctx.storage.save("key", data)
except StorageError as e:
    ctx.log.error(f"Storage operation failed: {e.message}")
    # e.operation: "save" | "get" | "list" | "delete"
    # e.key: The key that failed
    # e.backend: "postgresql" | "sqlite"

Future Requirements

FR-EDGE-001: Offline Execution Capability

  • Priority: P3 (Design Year 1, Build Year 3+)
  • Trigger: First air-gapped customer requirement
  • Implementation: SQLiteBackend + local state management

FR-REG-001: Field-Level Encryption

  • Priority: P2 (Design Year 1, Build Year 2)
  • Trigger: First HIPAA or financial services customer
  • Implementation: encrypt=True parameter in save() method
# Future: Field-level encryption
await ctx.storage.save("sensitive-data", data, encrypt=True, key_id="tenant-kek")