Self-Hosted Operations

Last updated: 2026-05-07 06:53:50

Self-Hosted Operations¶

This guide covers day-2 operations for running Huitzo in production: maintenance, upgrades, backup/restore, and troubleshooting.

Service Management¶

Starting and Stopping¶

# Start all services
docker compose up -d

# Stop all services (preserves data)
docker compose down

# Restart a specific service
docker compose restart app
docker compose restart worker

# Stop and remove volumes (DATA LOSS)
docker compose down -v

Viewing Status¶

# Service status
docker compose ps

# Resource usage
docker stats

# Service health
curl http://localhost:8080/health

Viewing Logs¶

# All services (follow)
docker compose logs -f

# Specific service
docker compose logs -f app
docker compose logs -f worker
docker compose logs -f postgres

# Last 100 lines
docker compose logs --tail=100 app

# Since timestamp
docker compose logs --since="2024-01-01T00:00:00" app

Updates and Upgrades¶

Checking for Updates¶

# Check current version
docker compose exec app huitzo --version

# Check for available updates
docker compose pull --dry-run

Performing Updates¶

Standard Update (Minor/Patch):

# Pull latest images
docker compose pull

# Restart with new images
docker compose up -d

# Verify health
curl http://localhost:8080/health

Major Update (Breaking Changes):

Backup first: bash ./backup.sh
Review release notes: Check changelog for migration requirements.
Run database migrations: bash docker compose exec app huitzo db migrate
Update images: bash docker compose pull docker compose up -d
Verify: bash docker compose logs app | tail -50 curl http://localhost:8080/health

Rollback Procedure¶

If an update causes issues:

# Stop services
docker compose down

# Restore from backup
./restore.sh backup-20240101.tar.gz

# Pin to previous version
# Edit docker-compose.yml:
# image: huitzo/huitzo:2.0.1  # Instead of :latest

# Start with previous version
docker compose up -d

Backup and Restore¶

Backup Strategy¶

Component	Frequency	Retention	Method
PostgreSQL	Daily	30 days	pg_dump
Redis	Weekly	7 days	RDB snapshot
Configuration	On change	Forever	Git/copy
Uploads	Daily	90 days	tar/rsync

Automated Backup Script¶

Create backup.sh:

#!/bin/bash
set -e

BACKUP_DIR="./backups"
DATE=$(date +%Y%m%d_%H%M%S)
BACKUP_NAME="huitzo-backup-${DATE}"

mkdir -p "${BACKUP_DIR}"

echo "Starting backup: ${BACKUP_NAME}"

# Backup PostgreSQL
echo "Backing up PostgreSQL..."
docker compose exec -T postgres pg_dump -U huitzo huitzo > "${BACKUP_DIR}/${BACKUP_NAME}.sql"

# Backup configuration
echo "Backing up configuration..."
cp .env "${BACKUP_DIR}/${BACKUP_NAME}.env"
cp docker-compose.yml "${BACKUP_DIR}/${BACKUP_NAME}.docker-compose.yml"

# Backup uploads
echo "Backing up uploads..."
tar -czf "${BACKUP_DIR}/${BACKUP_NAME}-uploads.tar.gz" ./data/uploads 2>/dev/null || true

# Create combined archive
echo "Creating archive..."
cd "${BACKUP_DIR}"
tar -czf "${BACKUP_NAME}.tar.gz" \
    "${BACKUP_NAME}.sql" \
    "${BACKUP_NAME}.env" \
    "${BACKUP_NAME}.docker-compose.yml" \
    "${BACKUP_NAME}-uploads.tar.gz" 2>/dev/null || true

# Cleanup individual files
rm -f "${BACKUP_NAME}.sql" "${BACKUP_NAME}.env" \
      "${BACKUP_NAME}.docker-compose.yml" "${BACKUP_NAME}-uploads.tar.gz"

echo "Backup complete: ${BACKUP_DIR}/${BACKUP_NAME}.tar.gz"

# Cleanup old backups (keep last 30)
ls -t *.tar.gz | tail -n +31 | xargs -r rm -f

Automated Backup Schedule¶

Add to crontab:

# Daily backup at 2 AM
0 2 * * * cd /opt/huitzo && ./backup.sh >> /var/log/huitzo-backup.log 2>&1

Manual Backup¶

# PostgreSQL only
docker compose exec -T postgres pg_dump -U huitzo huitzo > backup.sql

# Full data directory
tar -czf huitzo-data-$(date +%Y%m%d).tar.gz ./data

Restore Procedure¶

Create restore.sh:

#!/bin/bash
set -e

if [ -z "$1" ]; then
    echo "Usage: ./restore.sh backup-file.tar.gz"
    exit 1
fi

BACKUP_FILE="$1"
RESTORE_DIR="./restore_temp"

echo "Restoring from: ${BACKUP_FILE}"

# Stop services
docker compose down

# Extract backup
mkdir -p "${RESTORE_DIR}"
tar -xzf "${BACKUP_FILE}" -C "${RESTORE_DIR}"

# Find SQL file
SQL_FILE=$(find "${RESTORE_DIR}" -name "*.sql" | head -1)

# Restore PostgreSQL
echo "Restoring PostgreSQL..."
docker compose up -d postgres
sleep 5  # Wait for PostgreSQL to start

# Drop and recreate database
docker compose exec -T postgres psql -U huitzo -c "DROP DATABASE IF EXISTS huitzo_restore;"
docker compose exec -T postgres psql -U huitzo -c "CREATE DATABASE huitzo_restore;"
cat "${SQL_FILE}" | docker compose exec -T postgres psql -U huitzo huitzo_restore
docker compose exec -T postgres psql -U huitzo -c "DROP DATABASE huitzo;"
docker compose exec -T postgres psql -U huitzo -c "ALTER DATABASE huitzo_restore RENAME TO huitzo;"

# Restore uploads if present
UPLOADS_FILE=$(find "${RESTORE_DIR}" -name "*-uploads.tar.gz" | head -1)
if [ -n "${UPLOADS_FILE}" ]; then
    echo "Restoring uploads..."
    tar -xzf "${UPLOADS_FILE}" -C ./
fi

# Cleanup
rm -rf "${RESTORE_DIR}"

# Start all services
docker compose up -d

echo "Restore complete!"

Database Management¶

Database Health¶

# Check PostgreSQL status
docker compose exec postgres pg_isready -U huitzo

# Check database size
docker compose exec postgres psql -U huitzo -c "
  SELECT pg_size_pretty(pg_database_size('huitzo'));
"

# Check table sizes
docker compose exec postgres psql -U huitzo -c "
  SELECT relname, pg_size_pretty(pg_total_relation_size(relid))
  FROM pg_catalog.pg_statio_user_tables
  ORDER BY pg_total_relation_size(relid) DESC
  LIMIT 10;
"

Database Maintenance¶

# Vacuum (reclaim space)
docker compose exec postgres psql -U huitzo -c "VACUUM ANALYZE;"

# Reindex (improve query performance)
docker compose exec postgres psql -U huitzo -c "REINDEX DATABASE huitzo;"

Database Migrations¶

# Check migration status
docker compose exec app huitzo db status

# Run pending migrations
docker compose exec app huitzo db migrate

# Rollback last migration
docker compose exec app huitzo db rollback

Cache Management¶

Redis Operations¶

# Check Redis status
docker compose exec redis redis-cli ping

# Check memory usage
docker compose exec redis redis-cli info memory

# Clear all cache (use with caution)
docker compose exec redis redis-cli FLUSHDB

# Clear specific pattern
docker compose exec redis redis-cli KEYS "cache:*" | xargs -r docker compose exec redis redis-cli DEL

License Management¶

License Status¶

# Check license status
docker compose exec app huitzo license status

# Force license re-validation
docker compose exec app huitzo license validate

# Clear cached validation (requires online check)
docker compose exec app huitzo license clear-cache

Offline Operation¶

Self-hosted deployments can operate offline for up to 7 days:

# Check offline grace period remaining
docker compose exec app huitzo license offline-status

Pack Management¶

Installing Packs¶

# From registry
docker compose exec app huitzo pack install my-pack

# From file
cp my-pack-1.0.0.tar.gz ./packs/
docker compose restart app worker

Updating Packs¶

# Update specific pack
docker compose exec app huitzo pack update my-pack

# Update all packs
docker compose exec app huitzo pack update --all

Removing Packs¶

# Remove pack
docker compose exec app huitzo pack uninstall my-pack

Troubleshooting¶

Service Won't Start¶

# Check logs for errors
docker compose logs app | tail -100

# Check resource usage
docker stats --no-stream

# Check disk space
df -h

# Check memory
free -m

Database Connection Issues¶

# Test PostgreSQL connectivity
docker compose exec app pg_isready -h postgres -U huitzo

# Check PostgreSQL logs
docker compose logs postgres | tail -50

# Restart PostgreSQL
docker compose restart postgres

Worker Not Processing¶

# Check worker logs
docker compose logs worker | tail -100

# Check Redis connectivity
docker compose exec worker redis-cli -h redis ping

# Check Celery queue
docker compose exec app celery -A huitzo.worker inspect active

License Validation Failed¶

# Check connectivity to keygen.sh
curl -I https://api.keygen.sh/v1/ping

# Check license key format
echo $HUITZO_LICENSE_KEY | head -c 10

# View license-related logs
docker compose logs app | grep -i license

High Memory Usage¶

# Check memory by service
docker stats --no-stream

# Restart memory-heavy service
docker compose restart worker

# Check for memory leaks in logs
docker compose logs app | grep -i "memory\|oom"

Slow Performance¶

# Check database query times
docker compose exec postgres psql -U huitzo -c "
  SELECT query, calls, mean_time, total_time
  FROM pg_stat_statements
  ORDER BY mean_time DESC
  LIMIT 10;
"

# Check Redis latency
docker compose exec redis redis-cli --latency

# Check worker queue depth
docker compose exec app celery -A huitzo.worker inspect reserved

Health Checks¶

Manual Health Check¶

# API health
curl -s http://localhost:8080/health | jq

# Detailed health
curl -s http://localhost:8080/health/detailed | jq

Automated Health Monitoring¶

Add to monitoring system:

#!/bin/bash
# health-check.sh
HEALTH=$(curl -s http://localhost:8080/health)
STATUS=$(echo $HEALTH | jq -r '.status')

if [ "$STATUS" != "healthy" ]; then
    echo "ALERT: Huitzo unhealthy - $HEALTH"
    # Send alert (email, Slack, PagerDuty, etc.)
fi

Log Management¶

Log Rotation¶

Configure Docker log rotation in /etc/docker/daemon.json:

{
  "log-driver": "json-file",
  "log-opts": {
    "max-size": "10m",
    "max-file": "5"
  }
}

Centralized Logging¶

Forward logs to your logging infrastructure:

# docker-compose.override.yml
services:
  app:
    logging:
      driver: "syslog"
      options:
        syslog-address: "udp://your-log-server:514"
        tag: "huitzo-app"

Deployment Guide – Initial deployment
Requirements – Hardware and network requirements
Observability – Monitoring and metrics
Configuration – Environment variables