Connected Installation¶

The supported self-hosted install: one Linux host with internet access, Docker Compose, served over HTTPS. install.sh takes a clean host to a healthy stack with a first admin, and is safe to re-run.

Air-gapped installs are not supported at launch. For a short walk-through of the same install, see the Quick start.

Support matrix¶

install.sh stops on an unsupported architecture, Docker Engine or Compose version, or a missing cosign, and warns on an operating system other than Ubuntu 24.04. Sizing, ports and the outbound allow-list are in the partner requirements document (Requirements).

What you receive¶

Release assets¶

Each release is published at https://get.huitzo.com/self-hosted/<version>/ (/self-hosted/stable/ redirects to the current stable release). The version is the platform release number that /health reports. Images are built for x86_64 (amd64) only.

Download one release into an empty directory:

# pseudocode: operator commands
v=<version>; base=https://get.huitzo.com/self-hosted/$v
for f in install.sh docker-compose.yml .env.template Caddyfile backup.sh \
         values.yaml RELEASE-NOTES.md SHA256SUMS SHA256SUMS.cosign.bundle; do
  curl -fsSLO "$base/$f"
done

Pre-run check¶

install.sh cannot vouch for itself, so check the assets before running it: the signature over SHA256SUMS must come from Huitzo's release workflow on main, then every file must match its checksum.

# pseudocode: operator commands, in the download directory
cosign verify-blob SHA256SUMS --bundle SHA256SUMS.cosign.bundle \
  --certificate-identity https://github.com/Huitzo-Inc/huitzo/.github/workflows/customer-release.yml@refs/heads/main \
  --certificate-oidc-issuer https://token.actions.githubusercontent.com
sha256sum -c SHA256SUMS
chmod +x install.sh backup.sh

Stop if either command fails. Any other signing identity (another workflow, branch or repository) fails verify-blob.

Install¶

A license names the deployment ID it is for, and a deployment ID exists only after the first boot (ADR-008 D3). A first install therefore runs twice:

# pseudocode: operator commands; adjust paths and values
# 1. No license yet: setup mode on this host only; prints the deployment ID.
./install.sh \
  --pull-credentials pull-credentials.json \
  --domain huitzo.example.com \
  --acme-email [email protected] \
  --admin-email [email protected]

# 2. Request a license for the printed deployment ID, then:
/opt/huitzo/install.sh --license license.json

The script runs these steps in order and stops at the first failure:

1. Checks the support matrix.
2. Copies the release assets into the install directory (/opt/huitzo): docker-compose.yml, Caddyfile, .env.template, install.sh and backup.sh, replaced on every run, plus empty branding/ and certs/ directories. It never touches an existing .env, except for the two one-time upgrades in the next step.
3. Creates .env on the first run (Environment file). An existing .env with the single HUITZO_DB_PASSWORD of an older release is split into per-role passwords after step 7 and before step 8 (Upgrading). An existing .env without the database backup secrets gets them appended once (Backup secrets on an existing install).
4. Logs in to the registry. The credentials path is passed to Python as an argument and the token goes to docker login on stdin, into a temporary Docker configuration directory that the image verification and pull use and that is deleted when the script exits; your own Docker configuration and logins are never touched (ADR-045).
5. Verifies the cosign signature of every Huitzo image in the compose file (migrate, backend, worker, beat, dashboard, backup, backup-restore, and any other ghcr.io/huitzo-ai image) with the exact identity of the pre-run check and the annotation release=<version> of this release (an older release's digest is refused), then pulls the digest-pinned images. An image without a digest, a signature from any other identity, or a missing cosign stops the install. The verified digests are recorded with the signer identity in .verified-images; a later run skips the verification and pull only for a recorded digest whose image is still present locally.
6. Runs the license preflight on the file you passed. An INVALID license stops the install with the live license/license.json unchanged and no service started; a usable one is written over license/license.json in place. Without --license and with no license placed yet, the run continues in setup mode.
7. With the HTTPS profile and a license, starts Caddy on ports 80 and 443 and records the certificate key pin (HTTPS profile). Nothing of the application is reachable through it yet. Setup mode skips this step.
8. Starts the backend, the workers and beat (the setup status checks that it ticks): migrate runs the schema migrations and sets the database role password first. When a license was placed but the backend still reports setup mode (it booted before the file existed), the backend is restarted once; if it is still in setup mode, the run stops and prints the deployment ID and the reason.
9. Creates the first admin from inside the backend container.
10. Starts the dashboard on 127.0.0.1:3000 (the route Caddy forwards to), restarting it when the backend (re)started after it so its proxy follows the new backend, and prints the URL and what to back up. In setup mode it starts the dashboard only when its port is bound to 127.0.0.1, then prints the deployment ID instead (after step 11) and exits.
11. Prepares and starts the database backup service: it creates or updates the backup roles, generates the manifest signing key once, and starts backup. It runs after the dashboard, so a failure here stops the run with an error but never leaves the Hub without its proxy.

Setup mode¶

Without a license the backend boots in setup mode (ADR-008 setup mode): it serves the health probes, /api/v1/setup/*, the license status and reload routes, and answers every other route with 402 LICENSE_INVALID; the workers run nothing. install.sh without --license uses it to prepare the deployment before a license exists:

• it generates .env, pulls the images and starts the backend and workers;
• it creates the first admin with the one-time setup token;
• it starts the dashboard only when its port is bound to 127.0.0.1 (127.0.0.1:3000 by default) and keeps Caddy down, stopping one left running by an earlier install, so nothing listens beyond this host;
• it prints the deployment ID (hzd_ plus 26 characters, shown in groups of four for reading aloud; the license request takes it without spaces) and exits.

The ID is also on GET /api/v1/setup/status (no sign-in needed), on the dashboard's setup page (/setup) and on the admin license card. Request the license for it, then run install.sh --license license.json, which checks the file against this deployment's ID, places it and starts the HTTPS profile.

The backend also enters setup mode when the license names another deployment ID (including a license issued before deployment IDs existed, whose free-text deployment_id never matches), is not active yet (not_before in the future), or carries an issued_at more than 24 hours ahead of the host clock. Any other unusable license (expired past grace, tampered, signed by an untrusted key) still stops the backend from booting, and its log prints the local deployment ID. A matching license leaves setup mode on the next re-read (at most hourly), on POST /api/v1/license/reload (it needs an admin or owner token; signing in is refused in setup mode, but the bootstrap-admin response carries one), or on a backend restart, which is what install.sh --license does. A reload, each backend boot and a re-read that finds a usable license all tell the workers to re-read before they refuse work. The customer compose runs one backend; with several backend replicas (Kubernetes), a reload re-reads the file only on the replica that serves the call, and the others follow at their next hourly re-read or a restart.

Serving this host only is a property of install.sh, not of setup mode itself. Starting the stack by hand without a license (for example docker compose --profile https up -d) publishes the setup routes through Caddy: bootstrap-admin is then reachable remotely, still behind the one-time setup token and its failure limits. Prefer install.sh, or keep the https profile down until the license is installed.

Remote plain HTTP is unsupported¶

Session cookies are always Secure, so a browser on another machine that reaches the dashboard over plain HTTP is signed out on every page reload. install.sh therefore accepts exactly three access modes:

In every mode except --external-tls the script refuses to start when .env publishes port 3000 on anything other than 127.0.0.1. Docker-published ports bypass host firewalls such as ufw and firewalld, so if your proxy runs on another machine, block port 3000 in the network or security group.

Environment file¶

On the first run install.sh writes .env from .env.template with mode 0600 and fills the required secrets, the metrics token and the database backup secrets with random values (32 bytes each; hex, except the control-plane key, which is base64):

Hex keeps every password safe inside the database and Redis URLs. The access-mode flags are written to .env too (HUITZO_TLS_MODE, HUITZO_DOMAIN, HUITZO_ACME_EMAIL, COMPOSE_PROFILES, and HUITZO_EDGE_TRUSTED_PROXIES with the https profile; see HTTPS profile). It never writes HUITZO_BACKUP_AGE_RECIPIENTS: the encryption keys are yours to create, off this host (Encryption keys). On later runs the existing .env is the source of truth: it stays byte-identical, and the script stops if --domain, --acme-email or the TLS mode flags contradict it. To change them, edit .env and re-run.

Losing .env loses HUITZO_SECRETS_ENCRYPTION_KEY, and with it every stored secret. Back it up (Backups).

Metrics token on an existing install¶

An .env written before install.sh generated HUITZO_METRICS_TOKEN has no such line, so /metrics answers 503. Re-running install.sh does not add it, because the script never rewrites an existing .env. Generate a value, add the line to .env, and recreate the backend:

cd /opt/huitzo
python3 -c 'import secrets; print(secrets.token_hex(32))'
# add the printed value to .env as HUITZO_METRICS_TOKEN=<value>
docker compose up -d

Scraping is covered in Observability.

Backup secrets on an existing install¶

An .env written by a release without the backup service lacks some of HUITZO_DB_BACKUP_PASSWORD, HUITZO_DB_BACKUP_AUDIT_PASSWORD, HUITZO_BACKUP_CTL_HMAC_KEY, HUITZO_BACKUP_TICKET_KEY and the empty HUITZO_BACKUP_AGE_RECIPIENTS= line. The first run of a newer install.sh copies .env to .env.bak (mode 0600), then appends exactly the missing lines with fresh random values, in one atomic rewrite; every existing line stays as it was. A later run finds them all and leaves .env byte-identical. Back up the new .env and delete .env.bak once that backup exists.

Database role passwords¶

Each database role has its own password in .env:

install.sh creates both backup roles, or brings them back to their exact shape and password, on every run without --no-up, after migrate: in a one-off migrate container it runs huitzo-db db-backup-role with the two passwords in owner-only files that the container deletes on exit, so they are never printed or put on a command line (roles). To rotate one, put a new value in .env and re-run install.sh.

The huitzo_app password no longer opens the superuser, and backend, worker and beat never receive the superuser URL (Which service holds the admin URL); separate passwords are what make that placement a boundary. The bundle has no separate huitzo_admin role: the migration admin is the superuser.

After alembic upgrade head, the migrate service sets the huitzo_app password from HUITZO_DB_APP_PASSWORD on every run. It sends a SCRAM-SHA-256 verifier computed in the container, never the cleartext, quoted by PostgreSQL's format('%L'); nothing secret is logged. Every password must be printable ASCII.

To rotate the huitzo_app password, put a new value (python3 -c 'import secrets; print(secrets.token_hex(32))') in HUITZO_DB_APP_PASSWORD in .env and run docker compose up -d from the install directory: migrate moves the role to it before the backend and workers reconnect.

To rotate the superuser password, from the install directory:

1. docker compose exec postgres psql -U huitzo -d huitzo -c "\password huitzo" and enter the new value.
2. Put the same value in HUITZO_DB_SUPERUSER_PASSWORD in .env.
3. docker compose up -d.

Upgrading an install with one database password¶

Releases before the split kept one HUITZO_DB_PASSWORD for both roles. The first run of a newer install.sh over such an install splits it after the HTTPS proxy step, which needs no database, and before it starts the backend, workers and beat. A failure before list item 1 below leaves the previous release's backend, workers and beat running and the database unchanged; from item 1 until the backend, workers and beat start again, the application is down.

1. It stops the previous release's backend, worker and beat and copies .env to .env.bak (mode 0600), so none of them is left with a password the database no longer accepts.
2. It generates the three new passwords and keeps them in .env.db-rotation (mode 0600) until the switch, so an interrupted run reuses them.
3. In a one-off migrate container, connected as huitzo with the old password, it runs ALTER ROLE huitzo_app PASSWORD … and ALTER ROLE huitzo PASSWORD … in one transaction (SCRAM verifiers, as above). If the old password is refused because an earlier run already changed it, it connects with the new one instead.
4. Only then does it replace the HUITZO_DB_PASSWORD line in .env with the three new variables and delete .env.db-rotation.

A failure while the roles move (item 3) leaves .env unchanged and matching the database. A failure in the final rewrite (item 4) leaves .env on the old password while the roles already use the new ones, kept in .env.db-rotation, which the next run reuses. Either way the application stays stopped: re-run install.sh. A later run finds no HUITZO_DB_PASSWORD and changes nothing. With --no-up the split waits for the next run without it, because it needs the database running. No password is printed.

After the upgrade, back up the new .env (Backups) and delete .env.bak once that backup exists: .env.bak holds the old password, which no role accepts any more, and every other secret.

Development compose is never for customers¶

deploy/compose/docker-compose.yml in the repository is the development and internal stack. It publishes PostgreSQL on port 5432 with the fixed credentials huitzo/huitzo and Redis on 6379. Never run it for a customer or on a host reachable from a network you do not control: install with install.sh and the release assets, whose compose file publishes no database port and generates every password.

Which service holds the admin URL¶

Only the run-only migrate service receives HUITZO_DATABASE_ADMIN_URL. backend, worker and beat get HUITZO_DATABASE_URL (the huitzo_app role) and nothing else for the database. They run pack code: pipeline stages, nested command bodies and the fast queue run in-process, and without the sandbox helper an isolated pack child runs as the same user as its parent and can read the parent's environment from /proc/<ppid>/environ (ADR-039 §f, Sandbox UID), so a variable in their environment is readable by any installed pack.

No runtime code path reads the admin URL. It is read only by alembic (apps/backend/alembic/env.py and the extension chains), by the app_role step that migrate runs, by huitzo-db db-backup-role, which install.sh runs in a one-off migrate container (backup service), and by the break-glass rerun-aad-rotation command, which therefore runs in a one-off migrate container. With the variable unset, settings fall back to HUITZO_DATABASE_URL, so a command run in backend connects as huitzo_app. The run-only backup-restore service holds the superuser credential under its own name, HUITZO_DATABASE_RESTORE_URL (Restore); it runs no pack code and up -d never starts it.

This is a boundary because the two roles have separate passwords (HUITZO_DB_SUPERUSER_PASSWORD and HUITZO_DB_APP_PASSWORD, above): the app role's URL carries nothing that opens the superuser. An install that still has the single HUITZO_DB_PASSWORD of an older release gets the boundary when install.sh splits it (upgrade).

The same rule holds in the Helm chart: the migration Job and the backend's migration init container receive the admin URL; the backend, worker and beat Deployments do not. There it protects the admin credential as long as HUITZO_DATABASE_URL names huitzo_app with its own password, as the chart's example Secret does (deploy/k8s/README.md).

License preflight¶

Before it replaces anything, install.sh checks the license file you passed with the release backend image itself, with no network:

# pseudocode: what install.sh runs (settings validation gets throwaway secrets)
docker run --rm --network none
  -e HUITZO_DEPLOYMENT_MODE, HUITZO_DATABASE_URL (never contacted), HUITZO_LICENSE_PATH,
     HUITZO_LICENSE_DATA_DIR=/app/license-data,
     HUITZO_JWT_SECRET_KEY, HUITZO_AUDIT_PEPPER, HUITZO_SECRETS_ENCRYPTION_KEY (random)
  -v <your license file>:/etc/huitzo/license/license.json:ro
  -v huitzo-customer_huitzo-license-data:/app/license-data:ro   # only once the volume exists
  <backend image> python -c 'import sys; from backend.licensing.verify import main; sys.exit(main())'

Once the deployment has booted, the preflight also reads its deployment ID from the huitzo-license-data volume (read-only) and refuses a license issued for another ID, naming both. Before the first boot there is no ID yet: it prints deployment ID: not generated yet and checks everything else.

The entry is called with python -c because the release image ships compiled modules, which python -m cannot run.

It prints the state (VALID, GRACE or INVALID), tier, customer, licensed deployment, this deployment's ID, expiry date and days left. An INVALID license (expired past grace, tampered, signed by a key this release does not trust such as the dev key, issued for another deployment ID, or malformed) prints the reason and stops the install with a non-zero exit, before the live license/license.json is touched. The file is staged with mode 0644 first, so the image's user can read it whatever its mode on your host. A GRACE license (already expired, still inside its grace window) is refused too, printing the state, expiry date and days past expiry, unless you pass --allow-grace; the entry exits 0 for VALID, 1 for INVALID and 2 for GRACE.

HTTPS profile¶

--domain turns on the https compose profile: a Caddy service publishing ports 80 and 443 in front of the dashboard, which then listens on 127.0.0.1:3000 only. Caddy reads HUITZO_DOMAIN, HUITZO_ACME_EMAIL and HUITZO_TLS_MODE from .env:

Keep the key across renewals. Huitzo CLI runners pin the SPKI hash of the Hub's TLS key. Caddy is configured with reuse_private_keys, so a renewal issues a new certificate for the same key and the pin stays valid. The key lives in the caddy-data volume; back it up with the rest (Backups).

The pin. After Caddy serves its first certificate, install.sh computes the leaf key's SPKI SHA-256 and writes it to hub-pin.env in the install directory as HUITZO_RUNNER_HUB_CERT_SPKI_SHA256. The backend reads that file and ships the pin, root-signed, in the runner install manifest. Caddy starts, and the pin is recorded, before the backend, so the backend boots with it. On a re-run the script keeps hub-pin.env as long as it contains the served pin (a rotation file with two pins stays as it is), and warns and rewrites it when the served key is not in it.

With a byo certificate, replace the files by re-running install.sh with --tls-cert and --tls-key: it restarts Caddy when they change (a change left by an interrupted or --no-up run is applied on the next run) and records the new pin (with a warning when the key changed; plan that as a key rotation). With --external-tls, write the pin yourself:

# pseudocode: pin = sha256(DER SubjectPublicKeyInfo of the served leaf)
openssl s_client -connect <host>:443 -servername <host>  -> leaf certificate
openssl x509 -pubkey -noout | openssl pkey -pubin -outform DER | sha256sum  -> <hex>
hub-pin.env:  HUITZO_RUNNER_HUB_CERT_SPKI_SHA256='["sha256:<hex>"]'
(from the install directory) docker compose up -d backend

Per-client sign-in limits behind Caddy. Caddy has the fixed address HUITZO_CADDY_ADDRESS (172.16.231.10) on huitzo-network, whose subnet is HUITZO_NETWORK_SUBNET (172.16.231.0/24); containers without a fixed address get one from HUITZO_NETWORK_DYNAMIC_RANGE (172.16.231.128/25). On the first run with --domain, install.sh writes HUITZO_EDGE_TRUSTED_PROXIES with Caddy's address to .env, so the dashboard trusts Caddy alone (one hop) and each user keeps their own sign-in rate-limit bucket. A later run never edits .env, so with an .env written before the variable existed it warns: add HUITZO_EDGE_TRUSTED_PROXIES=172.16.231.10 and run docker compose down && docker compose up -d from the install directory, which also recreates the network with the fixed subnet.

• Load balancer in front of Caddy. The shipped Caddyfile trusts no upstream. Caddy replaces the load balancer's X-Forwarded-For, so all users behind that load balancer share its bucket.
• --external-tls. The variable stays empty. Set it to the address your proxy reaches the dashboard from. For a proxy on this host that is the huitzo-network gateway, the first address of HUITZO_NETWORK_SUBNET (172.16.231.1 by default), when Docker's userland proxy carries the loopback-published port (the default); check the peer in the dashboard log. Trusting the gateway trusts every process on this host that can reach port 3000.
• Subnet collision. If 172.16.231.0/24 overlaps a network this host or its users reach, up fails with a pool overlap, or that network loses its route to the host. Run install.sh with --no-up first, set the three network variables and HUITZO_EDGE_TRUSTED_PROXIES in .env to another private range, then run it again.
• Checked on every run. install.sh stops before starting anything when the subnet is not inside 10.0.0.0/8, 172.16.0.0/12 or 192.168.0.0/16 (the backend's default trusted ranges), the dynamic range is not inside it, or Caddy's address is outside the subnet, inside the dynamic range, or the subnet's network, gateway or broadcast address. With the https profile it warns when HUITZO_EDGE_TRUSTED_PROXIES is empty or does not list Caddy.

See client IP behind an HTTPS proxy.

Rotating the key¶

Rotating the key is a manual step: put the new and the old pin in hub-pin.env ('["sha256:<new>","sha256:<old>"]'), run docker compose up -d backend from the install directory, let runners refresh their manifest, then swap the key and drop the old pin. See Runner Cert-Pin Rotation.

First admin¶

The setup API creates the first admin while no user exists, and only with the one-time setup token: the backend writes it at first boot to setup-token on the huitzo-license-data volume (mode 0600, readable only from the host), never returns it from an API and deletes it once the first admin exists. install.sh reads it with docker compose exec and calls POST /api/v1/setup/bootstrap-admin from inside the backend container, on the backend's own loopback, before the dashboard (the only route to the backend) starts. It uses --admin-email and a generated password, printed once before the call so a lost response never hides it. Sign in and change it. If the deployment already has a user, the step is skipped.

A missing or wrong token gets 403. Wrong tokens are counted per client and across the deployment; past the limit they get 429, while the correct token still succeeds. The dashboard's setup page (/setup) asks for the token too.

Doing it by hand, while the dashboard is still stopped:

# pseudocode: run from the install directory
docker compose exec backend  cat /app/license-data/setup-token
docker compose exec backend  GET  http://127.0.0.1:8000/api/v1/setup/status
docker compose exec backend  POST http://127.0.0.1:8000/api/v1/setup/bootstrap-admin
                             {"email": "<admin email>", "password": "<8 to 72 characters>",
                              "setup_token": "<the token>"}

Backups¶

Releases with Hub-managed backups. This section describes the release that adds the backup service and the /admin/backups page; its release notes say so. On an older release, backup.sh --db-only still takes a plain pg_dump (huitzo_db_*.sql.gz): store it encrypted, and restore it with step 3 of the older procedure in that release's copy of this page.

Database backups are managed by the Hub. A backup service in the compose project takes them on a schedule, encrypts each one with age to the public keys you configure, signs its manifest, and keeps them in its own volume. Platform admins see them, start one, verify one or download one on /admin/backups. Restoring is a command on the host, never a button.

Encryption keys¶

Backups are encrypted before they touch the disk. Before the first backup:

1. On a machine other than this host, create an age X25519 identity with the age tools. Keep the private identity there, and a second copy offline or in a password manager, encrypted with a passphrase.
2. Put the public recipient (age1…) in .env as HUITZO_BACKUP_AGE_RECIPIENTS. List several, separated by commas, to let more than one key holder decrypt.
3. Re-run install.sh (or docker compose up -d backup).

The age tools' key generator writes the identity file and prints its public key (age1…); age -p makes the passphrase-encrypted copy. Run both on your workstation, never on the Huitzo host; install.sh prints the exact commands while no recipient is set.

install.sh never creates an age identity and never asks for one: it only reads the public recipients. It checks each one is an age1… recipient and stops on anything else, including a pasted private identity (AGE-SECRET-KEY-…). While the list is empty it says so on every run.

With no recipient, the backup service refuses to take backups and alerts: it never writes an unencrypted dump. Turning the schedule on in the Hub does not change that: every scheduled backup fails, visibly, until a recipient is set. Losing every private identity loses every backup.

Also record the backup service's signer public key off the host (/admin/backups shows its id; huitzo-db db-backup-list --json prints it). You need it to restore on a different host.

The backup service¶

Networks, all but huitzo-network internal (no route out):

The workers and beat get no backup setting, volume or network: pack code runs in the workers, so nothing there may read a backup or its credentials.

On each run without --no-up (step 11, after the dashboard; with --backups-only, right after migrate), install.sh:

1. Runs huitzo-db db-backup-role in a one-off migrate container (roles).
2. Generates the Ed25519 manifest signing key into the huitzo-backup-keys volume once, in a throwaway backup container with no network (mode 0400, UID 10950), and prints its public key: record that off the host. A key that exists is never replaced.
3. Starts backup and waits for it to be healthy.

Downloads from /admin/backups are on in this bundle (Schedule and retention). The dashboard's proxy serves them from an internal location that only a backend download response can reach.

What a complete backup holds¶

Without .env, encrypted data such as stored API keys cannot be recovered. The files and volumes stay with backup.sh until the Hub takes them over in a later release.

Schedule and retention¶

On /admin/backups a platform admin turns the schedule on (hourly by default) and sets how many scheduled, daily and pre-upgrade backups to keep. The daily backup is the first verified scheduled backup of each UTC day. .env sets the floors no admin can go below (HUITZO_BACKUP_RETENTION_MIN_SCHEDULED, HUITZO_BACKUP_RETENTION_MIN_DAILY, HUITZO_BACKUP_RETENTION_MIN_PREDEPLOY), and the newest verified backup is never deleted. A deleted backup stays recoverable in the volume's .trash/ for 72 hours.

Downloads from the Hub are on in this bundle (HUITZO_BACKUP_DOWNLOAD_ENABLED=true); set it to false in .env to turn them off. A download is the encrypted file, <id>.dump.age.

backup.sh¶

install.sh places backup.sh in the install directory:

# pseudocode: operator commands; adjust paths and values
/opt/huitzo/backup.sh                                   # database backup + files, to ~/huitzo-backups
/opt/huitzo/backup.sh --output /mnt/backups             # another destination
/opt/huitzo/backup.sh --db-only                         # a Hub database backup now, nothing else
/opt/huitzo/backup.sh --files-only                      # volumes and files, no database
/srv/huitzo/backup.sh --install-dir /srv/huitzo         # a non-default --prefix

--db-only asks the backup service for a backup now, so the result is encrypted, signed and listed in the Hub like any other; it writes no database file of its own. The full run does the same and then archives the files and volumes; when the database backup fails (for example while HUITZO_BACKUP_AGE_RECIPIENTS is empty) it still archives the files, then exits non-zero. --db-only needs no output directory and exits non-zero on a failed backup. backup.sh targets the huitzo-customer compose project in the install directory (default /opt/huitzo), creates every file owner-only (mode 0600), and writes a SHA-256 checksum per file. The files archive holds .env and keys in the clear: store it encrypted.

Copying backups off the host¶

Backups live on this host, so losing the host or its disk loses them. Copy huitzo-backups (the .dump.age, .manifest.json and .chunks files of each backup) to another machine or a NAS on a schedule. This is your own job: the Hub does not copy backups anywhere in this release.

For a NAS:

• Encryption first. Copy only the encrypted .dump.age files with their .manifest.json and .chunks files, never a backup.sh files archive in the clear, and never the reports/ directory.
• Make the copy hard to erase. Prefer a copy the NAS pulls from this host, or a write-once target (NAS snapshots, S3 Object Lock), so that someone who takes over this host cannot delete the copies as well.
• A NAS copy is trusted for storage only. Restoring from it still checks the signature and every chunk of the backup, so a file swapped on the NAS is refused.
• NFS: export only to this host's IP; all_squash with anonuid/anongid set to a dedicated account; prefer NFSv4 with Kerberos (sec=krb5p).
• SMB: SMB3 with encryption required.
• Mount with nofail, so a missing NAS never stops the host from booting.
• Copy, don't point the backup service's volume at the NAS: its locking and free-space checks assume a local filesystem.

Restore¶

Test a restore on a separate host before go-live. The database restore is huitzo-db db-restore, in two steps: --prepare checks the backup's signature and checksum, the deployment ID, the PostgreSQL and schema versions, and that the app is stopped, then takes a backup of the current database (skipped for a database that has never booted, such as one prepared with --backups-only); --apply reads the decrypted backup on its input, restores it into a scratch database, checks it, and swaps it in. Decrypt wherever the private key is and pipe the result in over ssh, so the key never has to be on the host. Before pg_restore reads any part of the decrypted stream, --apply checks it against the chunk checksums the backup service signed, so a backup file swapped somewhere along the way is refused.

# pseudocode: restore onto a fresh host from a backup.sh set + a Hub database backup
1. mkdir /opt/huitzo; copy the backed-up .env, hub-pin.env and certs/key.pem,
   certs/cert.pem (when present) into it, dropping backup.sh's
   huitzo_file_<name>_<timestamp> naming (huitzo_file_certs_key.pem_* becomes
   certs/key.pem); keep .env and key.pem at mode 0600. Add the old host's
   signer public key to HUITZO_BACKUP_TRUSTED_SIGNERS in .env. Download the
   release assets of the same or a newer version and run
   ./install.sh --license <backed-up license.json> --pull-credentials <file> --no-up:
   it places the compose file, Caddyfile and backup.sh, keeps the restored
   .env, places the license with mode 0644, verifies and pulls the images, and
   starts nothing (so no new deployment ID is minted).
2. For each huitzo_vol_<name>_*.tar.gz:
     docker volume create huitzo-customer_<name>
     docker run --rm -i -v huitzo-customer_<name>:/data <postgres image> tar xzf - -C /data
       < huitzo_vol_<name>_*.tar.gz
   Not huitzo-backups: the backup service creates that volume in step 3.
3. ./install.sh with the step 1 flags, --backups-only in place of --no-up:
   migrate brings the empty database to head (it creates huitzo_app and
   huitzo_auth, which the dump grants to), then the backup roles with their
   .env passwords, the signing key and the backup service start, in a
   huitzo-backups volume the backup service owns (UID 10950, 0700); the
   backend never does, so no deployment ID is minted. From the install
   directory, copy the backup's three files in as the backup service's user,
   so they are its own (a root-owned copy is refused):
     tar cf - <id>.dump.age <id>.manifest.json <id>.chunks
       | docker compose run --rm --no-deps -T backup
           sh -c 'cd /var/lib/huitzo/backups && tar xf - && chmod 0400 <id>.*'
   docker compose run --rm backup-restore huitzo-db db-restore <id> --prepare
     (backup-restore starts redis for its serving check)
   Then, from the machine that holds the private key:
     ssh <host> 'docker compose … exec -T backup huitzo-db db-backup-cat <id>'
       | age -d -i <identity>
       | ssh <host> 'docker compose … run --rm -T backup-restore huitzo-db db-restore <id> --apply <approval>'
4. Run install.sh with the same flags: it keeps the restored .env, migrate is a
   no-op at head and sets the huitzo_app password, and the first-admin step is
   skipped because users exist.

After the restore every user signs in again: --apply deletes all sessions, and re-applies revocations made after the backup was taken when the old database is still there to read them from. On the same host, only the two db-restore commands of step 3 are needed: stop worker, beat and backend, run them, then docker compose up -d.

Deployment ID on restore¶

The deployment ID is stored twice: in the database (authoritative) and in the huitzo-license-data volume (ADR-008 D3).

• Both copies restored (a new host included): the ID is unchanged and the license stays valid. Restoring onto new hardware needs no reissue.
• One copy restored: the backend restores the other from it at boot and logs a warning.
• Copies that disagree (for example the database of one install and the volume of another): the backend refuses to boot and logs both IDs and the recovery steps: restore both copies from the same backup, or remove the wrong deployment-id file from the volume so it is rewritten from the database.
• A fresh install gets a new ID, so it needs a reissued license (reason: reissue on the request).
• Two running copies of one install break the license agreement's one-deployment rule; that is a contract matter, not a technical check.
• A backup from another deployment: huitzo-db db-restore compares the backup's deployment ID with the target database's and refuses a mismatch unless you pass --foreign-deployment <the backup's ID> and type that ID at the prompt. A database that has never booted has no ID yet, so restoring onto a host prepared with --backups-only needs no flag.

Clean-VM check in CI¶

The clean-vm-install workflow repeats this install on a fresh GitHub-hosted Ubuntu 24.04 runner. Its candidate images are built on a self-hosted runner and handed over as a workflow artifact.

A pull request runs it when one of these changes: deploy/customer/, backup.sh, a backend, worker or dashboard Dockerfile, the dashboard's nginx or docker files, or the workflow itself. The full tier always runs it (CI tiers), and covers the migrations, the license trust code and the worker tasks. The full tier runs in the nightly Heavy Gate, on manual dispatch and on ci:heavy pull requests; the Heavy Gate also runs before every staging deploy and customer release. It builds candidate images from the branch with a CI-only license key that only CI jobs hold, assembles the release assets with those digests, and checks: the bad licenses (expired, tampered, dev-signed, a directory, a missing file) stop the install before anything starts; a first run without a license reaches setup mode, creates the admin with the setup token, prints the deployment ID that setup/status reports, listens on 127.0.0.1 only and answers 402 outside the setup routes; a license for another deployment ID is refused by the preflight; a license signed for the VM's own ID then installs the HTTPS profile; the stack is healthy with the HTTPS profile (internal certificates and a hosts entry); port 3000 listens on 127.0.0.1 only; the first admin signs in over HTTPS from the host's non-loopback address and a reload keeps the session; through Caddy, 10 clients with their own addresses sign in 5 times each within a minute with no 429, while a 6th attempt from one client gets 429 even with a forged X-Forwarded-For (per-client buckets); a re-run leaves .env unchanged; a file the backend writes is readable by the worker; a forced certificate renewal keeps the runner pin; beat sends the scheduled-dispatch task within 120 s of up and a user schedule due the next minute creates exactly one execution (Celery beat); an expired-past-grace license makes the license gate refuse the next dispatch tick, so a schedule due afterwards is never claimed and creates none; no container restarts; docker compose restart keeps the data; the backup service is healthy, a worker cannot resolve backup, and the dashboard cannot resolve postgres; and backup.sh (with a throwaway age recipient whose identity is discarded) takes a Hub database backup that db-backup-verify accepts, and captures every volume plus .env and license/license.json. It does not cover Let's Encrypt, your own certificate, --external-tls or registry login, and its cosign is a stand-in that records the identity install.sh asks for (the branch candidates are unsigned). The release run of customer-release.yml covers the published assets: the pre-run check, registry login with a read-only account, real cosign verification of the released digests, setup mode and the release version on /health (CI/CD).

After install¶

• License status: GET /api/v1/license/status (authenticated) or huitzo license status.
• Renewal: /opt/huitzo/install.sh --license new-license.json --no-up runs the license preflight on the new file and, if it is usable, writes it over license/license.json in place. A renewal or reissue always names this deployment's ID. Running services re-read it on their next check; POST /api/v1/license/reload applies it at once (no restart). See License Hard-Gate.
• Token expiry: the GHCR token expires with the license plus its grace, or after one year, whichever is sooner (ADR-045). Huitzo re-issues it alongside license renewal.

Issuance of these files is a Huitzo-side operation: see the License Issuance Runbook.

Support and contacts¶