Storage Backends Reference
Storage Backends Reference¶
Huitzo uses a pluggable storage backend architecture to support different deployment modes. This document describes the storage backend protocol and available implementations.
Overview¶
Huitzo uses PostgreSQL as the single database for all deployments. This "simple by design, scalable by architecture" approach minimizes self-hosted dependencies while providing enterprise-grade capabilities through PostgreSQL's JSONB and RLS features.
| Backend | Deployment Mode | Status | Use Case |
|---|---|---|---|
PostgreSQLBackend |
Cloud, Self-Hosted | Production | All data: user, tenant, pack registry, pack data (JSONB) |
SQLiteBackend |
Edge | Stubbed | Offline local storage (Year 3+) |
Storage Backend Protocol¶
All storage backends implement a common protocol:
from typing import Protocol, Optional, List, Any
class StorageBackend(Protocol):
"""Abstract storage interface enabling multiple backends."""
async def save(self, key: str, data: dict, **options) -> str:
"""
Save data to storage.
Args:
key: Storage key (max 256 characters)
data: JSON-serializable dictionary
**options: Backend-specific options
- ttl: Time-to-live in seconds (optional)
- scope: "user" | "tenant" | "pack" (default: "user")
- encrypt: Enable field-level encryption (future)
Returns:
The storage identifier (usually the key)
Raises:
StorageError: If save operation fails
"""
...
async def get(self, key: str) -> Optional[dict]:
"""
Retrieve data from storage.
Args:
key: Storage key to retrieve
Returns:
The stored data, or None if not found
Raises:
StorageError: If retrieval fails
"""
...
async def list(self, prefix: str, **filters) -> List[dict]:
"""
List data matching prefix and filters.
Args:
prefix: Key prefix to match
**filters: Backend-specific filters
- limit: Maximum results (default: 100)
- offset: Pagination offset (default: 0)
- metadata: Metadata filters (optional)
Returns:
List of matching records
Raises:
StorageError: If listing fails
"""
...
async def delete(self, key: str) -> bool:
"""
Delete data from storage.
Args:
key: Storage key to delete
Returns:
True if deleted, False if not found
Raises:
StorageError: If deletion fails
"""
...
PostgreSQLBackend¶
The primary backend for cloud and self-hosted deployments. Uses PostgreSQL 17 with JSONB columns and Row-Level Security (RLS).
Configuration¶
# Environment variables
HUITZO_STORAGE_BACKEND: "postgresql"
DATABASE_URL: "postgresql://user:pass@host:5432/huitzo"
Features¶
- Row-Level Security: Automatic tenant isolation at database level
- JSONB Storage: Efficient storage and querying of JSON data
- Full-Text Search: PostgreSQL GIN indexes for searching within JSONB
- Transactions: ACID-compliant transactions via
ctx.storage.transaction() - TTL Support: Automatic cleanup of expired keys
Schema¶
CREATE TABLE pack_data (
id UUID PRIMARY KEY DEFAULT gen_random_uuid(),
tenant_id UUID NOT NULL,
user_id UUID NOT NULL,
pack_id VARCHAR(256) NOT NULL,
key VARCHAR(256) NOT NULL,
value JSONB NOT NULL,
metadata JSONB DEFAULT '{}',
created_at TIMESTAMP WITH TIME ZONE DEFAULT NOW(),
updated_at TIMESTAMP WITH TIME ZONE DEFAULT NOW(),
expires_at TIMESTAMP WITH TIME ZONE,
UNIQUE(tenant_id, user_id, pack_id, key)
);
-- Row-Level Security
ALTER TABLE pack_data ENABLE ROW LEVEL SECURITY;
ALTER TABLE pack_data FORCE ROW LEVEL SECURITY;
CREATE POLICY tenant_isolation ON pack_data
FOR ALL TO app_user
USING (tenant_id = current_setting('app.tenant_id')::uuid)
WITH CHECK (tenant_id = current_setting('app.tenant_id')::uuid);
Usage¶
# The SDK automatically selects PostgreSQLBackend for cloud/self-hosted
@command("save-data", namespace="example")
async def save_data(args: Args, ctx: Context) -> dict:
# Storage is automatically backed by PostgreSQL
await ctx.storage.save("my-key", {"field": "value"})
data = await ctx.storage.get("my-key")
return data
SQLiteBackend (Future - Year 3+)¶
Local storage backend for edge deployments. Enables offline operation with sync-when-connected capability.
Status¶
Current: Stubbed - raises NotImplementedError
class SQLiteBackend(StorageBackend):
"""Edge-capable local storage. Available in Year 3+."""
async def save(self, key: str, data: dict, **options) -> str:
raise NotImplementedError("Edge mode not yet available")
async def get(self, key: str) -> Optional[dict]:
raise NotImplementedError("Edge mode not yet available")
async def list(self, prefix: str, **filters) -> List[dict]:
raise NotImplementedError("Edge mode not yet available")
async def delete(self, key: str) -> bool:
raise NotImplementedError("Edge mode not yet available")
Planned Features (Year 3+)¶
- Offline Operation: Full functionality without network
- Local Encryption: Data encrypted at rest on edge device
- Sync Protocol: Bidirectional sync when connected
- Conflict Resolution: Last-write-wins or CRDT-based (TBD)
Trigger for Implementation¶
SQLiteBackend will be implemented when: - First customer requires air-gapped deployment - Data sovereignty requirements prevent cloud storage - Edge device deployment validated with design partners
Backend Selection¶
The platform selects storage backend based on deployment mode:
def get_storage_backend(ctx: CommandContext) -> StorageBackend:
"""Select appropriate storage backend."""
if ctx.deployment_mode == DeploymentMode.EDGE:
# Future: Return SQLite backend for edge deployments
raise NotImplementedError("Edge mode not yet available")
# PostgreSQL is the single database for all cloud/self-hosted deployments
# Simple by design, scalable by architecture
return PostgreSQLBackend(
dsn=settings.DATABASE_URL,
tenant_id=ctx.tenant_id,
user_id=ctx.user_id
)
Configuration Reference¶
| Variable | Default | Description |
|---|---|---|
DATABASE_URL |
Required | PostgreSQL connection string |
Writing Backend-Agnostic Code¶
Intelligence packs should work across all backends. Follow these guidelines:
Do: Use Standard Operations¶
# Works on all backends
await ctx.storage.save("key", {"data": "value"})
data = await ctx.storage.get("key")
await ctx.storage.delete("key")
Do: Use Supported Options¶
# TTL is supported on all backends
await ctx.storage.save("cache:key", data, ttl=3600)
# Scope is supported on all backends
await ctx.storage.save("shared:key", data, scope="tenant")
Avoid: Backend-Specific Features¶
# Avoid: PostgreSQL-specific queries
# These won't work on SQLite when edge mode is available
await ctx.storage._raw_query("SELECT * FROM pack_data WHERE value->>'field' = 'x'")
Check Deployment Mode (When Necessary)¶
@command("sync-data", namespace="example")
async def sync_data(args: Args, ctx: Context) -> dict:
if ctx.deployment_mode == DeploymentMode.EDGE:
# Edge-specific behavior (future)
return {"status": "offline", "queued_for_sync": True}
else:
# Cloud/self-hosted behavior
await ctx.storage.save("data", args.payload)
return {"status": "saved"}
Error Handling¶
All backends raise consistent errors:
from huitzo_sdk.errors import StorageError
try:
await ctx.storage.save("key", data)
except StorageError as e:
ctx.log.error(f"Storage operation failed: {e.message}")
# e.operation: "save" | "get" | "list" | "delete"
# e.key: The key that failed
# e.backend: "postgresql" | "sqlite"
Future Requirements¶
FR-EDGE-001: Offline Execution Capability¶
- Priority: P3 (Design Year 1, Build Year 3+)
- Trigger: First air-gapped customer requirement
- Implementation: SQLiteBackend + local state management
FR-REG-001: Field-Level Encryption¶
- Priority: P2 (Design Year 1, Build Year 2)
- Trigger: First HIPAA or financial services customer
- Implementation:
encrypt=Trueparameter insave()method
# Future: Field-level encryption
await ctx.storage.save("sensitive-data", data, encrypt=True, key_id="tenant-kek")
Related Documentation¶
- Storage Reference – High-level storage API
- Context Reference – Context object and services
- Configuration – Environment variables
- Future Requirements – Edge and compliance roadmap