Backup and restore

What to back up

Preserve a recovery set: the PostgreSQL database, matching deployment configuration and secrets, and the application version. If you run CertOps agents, preserve their local credential, signing pin, custody state, and recovery journals too.

SESSION_SECRET derives the encryption key for database-stored settings as well as signing sessions. Restoring with a different secret can make stored credentials unreadable. Preserve CertOps signing/registration wrapping keys and, for Enterprise SSO, the external SSO_SECRETS_KEYRING; a database dump does not contain those external keys.

Keep secrets in your protected backup/secret store and record which keys match each database backup. Restore the keys and configuration before starting services, then verify stored credential decryption, administrator access, and a real notification.

Docker Compose

The Compose stack runs PostgreSQL as the postgres service (container tokentimer-db), with database and user both named tokentimer by default.

Backup

docker compose exec -T postgres pg_dump -U tokentimer tokentimer > backup.sql

For scheduled backups, add a timestamp and compress:

set -o pipefail
docker compose exec -T postgres pg_dump -U tokentimer tokentimer | gzip > "tokentimer-$(date +%F).sql.gz"

Restore

Restore in an isolated fresh environment, or enter maintenance and stop the API, migration jobs, all workers and external database writers first. Keep PostgreSQL running. Do not restart application writers until the import succeeds and matching configuration/encryption keys are restored.

docker compose exec -T postgres psql -U tokentimer --dbname=tokentimer \
  --set=ON_ERROR_STOP=1 --single-transaction < backup.sql

warning

Restore into an empty database. If the target database already contains data, drop and recreate it first (or restore into a fresh volume), otherwise the restore can fail on conflicts or leave a mixed state.

If you changed DB_USER or DB_NAME in your .env, use those values in the commands above.

Kubernetes (Helm / CloudNativePG)

The Helm chart provisions PostgreSQL as a CloudNativePG Cluster named <release>-pg by default (for a release named tokentimer, the cluster and its pods are tokentimer-pg).

Option A - CNPG-native backups (recommended)

If you enabled the chart's backup support (postgresql.cloudnative.backup.enabled), CloudNativePG handles base backups and WAL archiving to object storage (barman object store with S3 credentials and a retention policy). Prefer this for production: it gives you point-in-time recovery and does not depend on manual dumps.

Trigger an on-demand backup with a CNPG Backup resource, and restore by bootstrapping a new cluster from the object store (see the CloudNativePG recovery documentation for your operator version).

Option B - Manual pg_dump against the cluster pod

For ad-hoc dumps or when CNPG backups are not configured, exec into the current primary pod:

# Use your actual Cluster name if it differs from tokentimer-pg
PRIMARY_POD="$(kubectl get pods -n tokentimer \
  -l "cnpg.io/cluster=tokentimer-pg,cnpg.io/instanceRole=primary" \
  -o jsonpath='{.items[0].metadata.name}')"
test -n "$PRIMARY_POD" || { echo 'No CNPG primary found' >&2; exit 1; }

kubectl exec -n tokentimer "$PRIMARY_POD" -c postgres -- \
  pg_dump --host=/controller/run --username=postgres --dbname=tokentimer > backup.sql

CloudNativePG uses local peer authentication: these commands use the PostgreSQL container's postgres OS/database identity, not -U tokentimer.

Restore into an empty database with its application role already provisioned. Use an isolated new Cluster, or suspend worker CronJobs, stop existing worker/migration Jobs, scale the API to zero, and stop external database writers first. Preserve prior replica/schedule settings for recovery, and leave PostgreSQL and the operator running. Re-select the primary of the restore target immediately before importing:

PRIMARY_POD="$(kubectl get pods -n tokentimer \
  -l "cnpg.io/cluster=tokentimer-pg,cnpg.io/instanceRole=primary" \
  -o jsonpath='{.items[0].metadata.name}')"
test -n "$PRIMARY_POD" || { echo 'No CNPG primary found' >&2; exit 1; }

kubectl exec -i -n tokentimer "$PRIMARY_POD" -c postgres -- \
  psql --host=/controller/run --username=postgres --dbname=tokentimer \
  --set=ON_ERROR_STOP=1 --single-transaction < backup.sql

External PostgreSQL

If you pointed the chart at an external database (postgresql.external), use your existing backup tooling for that instance; preserve the matching deployment configuration and external encryption keys too.

PVCs on uninstall

Do not rely on helm uninstall to preserve the database. The release owns the CloudNativePG Cluster; deleting it can also delete its PVCs and underlying storage, depending on ownership and the volume reclaim policy. Take and verify a separate backup before uninstalling. If you need to preserve cluster storage, follow the CloudNativePG procedures for your operator version and verify the result before deleting the release. Retained volumes are not a substitute for a backup.

Recommended cadence and retention

Environment	Cadence	Retention
Production	Daily automated dump or continuous CNPG WAL archiving	30 days, plus monthly archives per your compliance needs
Staging / lab	Weekly, or before risky changes	7 days

Whatever the schedule, periodically test a restore into a scratch environment. A backup you have never restored is not verified.

Before every upgrade

Always take a backup before major upgrades or database migrations:

# Compose
docker compose exec -T postgres pg_dump -U tokentimer tokentimer > pre-upgrade-backup.sql

# Kubernetes (CNPG)
PRIMARY_POD="$(kubectl get pods -n tokentimer \
  -l "cnpg.io/cluster=tokentimer-pg,cnpg.io/instanceRole=primary" \
  -o jsonpath='{.items[0].metadata.name}')"
test -n "$PRIMARY_POD" || { echo 'No CNPG primary found' >&2; exit 1; }

kubectl exec -n tokentimer "$PRIMARY_POD" -c postgres -- \
  pg_dump --host=/controller/run --username=postgres --dbname=tokentimer > pre-upgrade-backup.sql

Then follow the upgrade steps in the install runbook.