Skip to main content
Version: 0.11

Enable CertOps

What the flag does​

CERTOPS_ENABLED is a platform-wide, fail-closed rollout flag for all CertOps routes: the workspace-scoped routes and the machine-token-authenticated executor routes. When the flag is unset or false, every CertOps route returns 404 NOT_FOUND regardless of role, and the CertOps UI surfaces stay hidden.

Self-hosted TokenTimer has no plan gating for CertOps. Once the flag is on, every workspace on the deployment can use CertOps (subject to normal RBAC). Cloud SaaS adds a separate plan check; that does not apply here.

This lets CertOps code ship dark and lets you enable the feature deliberately, per deployment, when you are ready.

Default behavior when unset​

The resolution order is env > DB > default, fail-closed:

  1. If the CERTOPS_ENABLED env var is set to a recognized boolean string, it wins.
  2. If it is absent, empty, or unrecognized, the code falls back to a system_settings.certops_settings.enabled database value.
  3. If that is also absent, the default is false.

Accepted truthy strings: 1, true, yes, on, enabled. Accepted falsy strings: 0, false, no, off, disabled. Anything else is treated as not set and falls through to the database value.

The code default is false, but both shipped deployments enable CertOps

That resolution order describes the application code. It is not what you get from either packaged deployment, because both supply the variable for you:

  • Docker Compose interpolates a true default into the api, worker-endpoint-check and worker-certops services (CERTOPS_ENABLED: ${CERTOPS_ENABLED:-true}), so leaving it out of .env enables CertOps.
  • Helm ships config.certopsEnabled: true and always renders CERTOPS_ENABLED into the shared ConfigMap, so a stock install is enabled.

To actually disable it, set CERTOPS_ENABLED=false in deploy/compose/.env or config.certopsEnabled=false on Helm. The code default is only reached when the variable is absent from the container entirely, which neither packaged path does.

Where to set it​

Docker Compose​

The Compose file already passes the flag through to the api, worker-endpoint-check and worker-certops services. Set it in your .env:

CERTOPS_ENABLED=true

Then recreate the affected services:

docker compose up -d

Kubernetes​

With the Helm chart, use the dedicated values key rather than an env-var passthrough:

config:
  certopsEnabled: true # default; set to false to disable the whole CertOps surface

The chart renders CERTOPS_ENABLED once into the shared release ConfigMap, which the API, the dashboard, and all six worker CronJobs consume via envFrom. That means one value keeps every process in sync, so the desync failures described below cannot happen. Do not set CERTOPS_ENABLED through the api.env / worker.env passthroughs: those apply per workload, which is exactly how the API and the workers end up disagreeing.

CertOps also needs two encryption keys before any agent can register or be dispatched signed work. The chart auto-generates both and preserves them across helm upgrade; see Agent install and Configuration.

Because the CronJobs pick the value up on their next scheduled run, only the long-running Deployments need a restart:

kubectl rollout restart deployment/tokentimer-api deployment/tokentimer-dashboard -n tokentimer

If you manage the flag through your own ConfigMap instead of the chart, note that Kubernetes does not re-read ConfigMap changes into a running container's environment, so the restart above is mandatory.

warning

Two worker processes need the flag too, not just the API:

  • The endpoint-check worker bridges endpoint SSL observations into the CertOps inventory and calls into the same gated service layer. If its flag is out of sync with the API's, observations silently stop flowing into CertOps.
  • The CertOps maintenance worker (worker-certops) runs the renewal scheduler and the lease/stale-agent/nonce/replay sweeps. If its flag is out of sync, the renewal scheduler treats every workspace as ineligible and counts each certificate as "skipped, paused" without logging an error, so nothing renews automatically while the API and dashboard look perfectly healthy.

Both failures are silent. Set the flag in one place so all three processes read the same value: on Compose that is deploy/compose/.env; on Helm it is config.certopsEnabled, which the chart fans out to every workload for you.

Rollout checklist​

  1. Confirm you are on a TokenTimer version that ships CertOps.
  2. Set CERTOPS_ENABLED=true on the API/backend process.
  3. Set CERTOPS_ENABLED=true on the endpoint-check worker and the CertOps maintenance worker (worker-certops) processes (same value, same time). On Helm, steps 2 and 3 are a single config.certopsEnabled value that reaches every process through the shared ConfigMap.
  4. Confirm the CertOps maintenance worker is actually deployed. On Compose it is the worker-certops service; on Helm it is the certops CronJob (worker.cronjobs.certops.enabled, on by default), named tokentimer-certops for the helm install tokentimer used in the install runbook. Without it, agents never leave active when they go offline and no certificate ever renews automatically. See Worker scheduling.
  5. Set the two CertOps encryption keys if you have not already (CERTOPS_SIGNING_ENCRYPTION_KEY, CERTOPS_REGISTRATION_ENCRYPTION_KEY; the Helm chart generates both for you). These belong on the API process only - the workers never sign jobs or decrypt registration credentials, which is why the Compose worker-certops service deliberately does not carry them. Without them the UI appears but no agent can register and no signed job can be dispatched.
  6. Restart or redeploy the affected processes, since env vars are read into the process environment at container start. On Helm only the API and dashboard Deployments need a rollout restart; CronJobs pick the value up on their next run.
  7. Run the verification steps below.
  8. Record the change in your ops log; the flag is your kill switch for the entire CertOps surface.

Verification steps​

  1. Route check (enabled): call any CertOps route with valid credentials. You should see a real response (success payload or a normal auth/validation error), not a blanket 404 NOT_FOUND.
  2. Dashboard check: log in and open a workspace. You should see CertOps surfaces appear: the CertOps enrichment on certificate tokens, the public PEM card in Import tokens, and the Certificate operations page reachable from the Control Center certificate-operations panel footer link and from Workspace Preferences (last section).
  3. Worker bridge check: with an HTTPS endpoint monitor configured, wait for the next endpoint check pass. You should see the observed certificate recorded in the managed-certificate inventory (visible in the token detail CertOps panel).
  4. Sweep worker check: tail the CertOps maintenance worker's logs for one tick (default every minute). You should see a sweep summary line naming the sweeps (lease reaper, stale agents, nonce, registration replay, renewal scheduler). Silence here means the worker is not deployed or not scheduled, which is a silent failure (see the warning above).
  5. Fail-closed check (optional but recommended): flip the flag back to false in a staging environment and restart. You should see every CertOps route return 404 NOT_FOUND again, with the UI surfaces hidden.

Rollback​

There is no migration or data cleanup involved in rolling back. Set CERTOPS_ENABLED to false (or remove the key, which falls back to the database value, then to false), restart the backend and worker, and all CertOps routes immediately return 404 again.

note

No jobs, machine tokens, or evidence records are deleted on rollback. They simply become inaccessible until the flag is re-enabled.

The system_settings.certops_settings database override exists for cases where you want a runtime kill switch without a redeploy, but the env var always takes priority when set to a recognized value. Do not rely on the database value while the env var is also set; the env var wins.

Fail-closed boot requirements​

Enabling CERTOPS_ENABLED turns on the routes, but two Ed25519/AES wrap keys must also be present in the API process environment before certain operations will succeed. Both are fail-closed: the operations that need them throw a specific ..._ENCRYPTION_KEY_MISSING error rather than silently degrading, and neither key is ever logged or returned in a response.

Env varFormatRequired for
CERTOPS_SIGNING_ENCRYPTION_KEY64 hex chars (32 bytes)Generating the control-plane Ed25519 signing key and signing dispatched jobs. Reading an existing public key does not need it.
CERTOPS_REGISTRATION_ENCRYPTION_KEY64 hex chars (32 bytes)Encrypting agent registration replay credentials at rest during agent bootstrap.

Generate either with:

openssl rand -hex 32

Set both alongside CERTOPS_ENABLED before you plan to onboard your first agent; a workspace that only uses external executors (no native agent, no signed dispatch) can run without them, but job signing and agent registration will fail with a clear _MISSING error code the moment they are needed. There is no separate rollout step beyond setting the two env vars once, since both are process-level configuration, not per-workspace settings.

Kill switch vs the platform flag​

CERTOPS_ENABLED and the workspace kill switch are two independent gates, not layers of the same control:

  • CERTOPS_ENABLED (this flag) is deployment-wide, env/DB-backed, and requires a process restart to change. It is the switch for whether CertOps exists on your instance at all.
  • Workspace kill switch (certops.kill_switch.manage permission, toggled from Certificate operations → the panel at the top of every tab, showing an Active/Paused badge with a Pause/Resume button for workspace admins) is per-workspace, takes effect immediately with no restart, and pauses new job creation and dispatch for one workspace while leaving inventory, status, audit, and evidence reporting readable. Approving or rejecting an already-pending-approval job is deliberately still allowed while paused, since that is exactly the kind of decision an operator needs during an incident.

Use the platform flag to decide whether CertOps ships on your deployment at all; use the workspace kill switch as your day-to-day pause button when something looks wrong in one workspace.

Troubleshooting​

SymptomLikely causeFix
All CertOps routes return 404 NOT_FOUND for every workspaceCERTOPS_ENABLED unset or false platform-wideSet CERTOPS_ENABLED=true on the backend (and worker), restart both
Flag flipped in config but behavior has not changedProcesses have not restarted, so they still hold the old environmentRestart/redeploy the backend and worker; env is read at container start
Executor (machine token) requests return 404 but the dashboard UI worksMachine-token routes and workspace routes share the same flag, so if only executor calls fail the flag is not the problemCheck the executor route path (/api/v1/certops/...) and the token's scopes
Endpoint observations are not appearing in CertOps inventoryEndpoint-check worker's CERTOPS_ENABLED is out of sync with the backend'sSet the flag identically on both, restart the worker
Agents that are clearly down still show as active in the fleet panelThe CertOps maintenance worker is not running, so the stale-agent sweep never demotes themDeploy/start worker-certops; see Worker scheduling
No certificate renews automatically, but manual jobs work fineCertOps maintenance worker missing, its renewal-scheduler sweep disabled, or its CERTOPS_ENABLED out of sync with the API'sConfirm the worker runs, CERTOPS_SWEEP_RENEWAL_SCHEDULER_ENABLED is not false, and the flag matches the API
Jobs stay in claimed/running forever after an agent diesLease reaper sweep not running (worker missing or CERTOPS_SWEEP_LEASE_REAPER_ENABLED=false)Start the worker / re-enable the sweep, then see Reconciling interrupted jobs
Flag set but value seems ignoredValue not in the recognized truthy/falsy sets, so it falls through to DB/defaultUse true or false exactly (or another recognized string)

Next steps​