Skip to main content
Version: 0.15

Renew certificates automatically

Goal​

Confirm that a managed certificate will renew without you, and diagnose the case where it silently will not.

This runbook exists because of a specific and expensive failure shape: a certificate that reads active in your inventory, has a real expiry date, shows no failed jobs, and is quietly never renewed by anything. Nothing in that picture looks wrong until the certificate expires.

Start with the renewal badge​

Renewals tab​

Every certificate in your inventory carries a renewal badge, computed from the same inputs the scheduler uses. See The renewal badge for each label. No auto-renewal is the dangerous state: an agent holds the key but there is no usable profile, so the certificate expires. Monitored only is expected. Auto-renewal off is a choice, and the certificate still expires. The Renewals tab (/certops/renewals) lists certificates that already have a profile, in expiry order.

The five conditions​

TokenTimer schedules a renewal for a certificate only when all five hold.

ConditionHow to checkIf it fails
A linked renewal profile with a complete snapshotThe badge reads "Auto-renews". The scheduler counts failures as skipped_incomplete_profileSee Step 1
Agent-deployable key custody (agent-local, proxy-agent-local, or os-store-managed)The certificate detail shows its key custody modeObserved-only certificates can never auto-renew. Renew them out of band, or bring them under agent management
Inside the renewal windowExpiry within 30 days by default (CERTOPS_RENEWAL_THRESHOLD_DAYS), or the profile's own lead time (renew_before_days)Nothing to do; it is not due yet
A renewable inventory statusAnything other than revoked or decommissionedRetired certificates are excluded by design
Automatic renewal switched onThe Renewal automation page shows the profile as On. The scheduler counts the rest as skipped_auto_renew_disabledSee Step 1b

Two further conditions are temporary rather than structural. A certificate is skipped for the current pass and retried on the next when the workspace is paused or CertOps is disabled deployment-wide, and when the per-CA in-flight cap (CERTOPS_RENEWAL_PER_CA_CAP) is already full.

Confirm the maintenance worker is running​

Nothing below matters if the sweep never executes. The renewal scheduler lives in the CertOps maintenance worker, and its absence is silent: the API keeps working and the deployment looks healthy.

  • Docker Compose: the worker-certops service must be running.
  • Helm: the certops CronJob must be enabled and its recent runs must be succeeding.

Both must also see the same CERTOPS_ENABLED value as the API. If the API has CertOps on and the worker does not, the sweep treats every certificate as "workspace paused" and logs nothing at error level. See the warning in Worker scheduling.

If you scrape metrics, certops_maintenance_sweeps_total{sweep="renewal-scheduler"} should be increasing.

Step 1 - Check the certificate has a renewal profile​

Where the profile comes from​

This is the answer most of the time.

TokenTimer will not dispatch a renewal it cannot fully specify: CA endpoint, ACME command profile, DNS provider and zone, deployment paths, SAN policy, and key parameters all have to be present and valid. A certificate with no linked profile, or one whose snapshot fails validation, is skipped on every scheduling pass, forever, without producing a job or an alert.

Where a profile comes from depends on how the certificate arrived:

How the certificate arrivedProfile
Issued by TokenTimer (an issue job)Derived automatically when the issuance succeeded, named Derived: <common name>
Imported as a PEMNone. Set up renewal is intended to fix this once an agent has custody, but currently cannot; see the caveat below
Discovered on an agent filesystemNone yet, but Set up renewal can arm one: the scan that found the certificate recorded its deployment path
Observed via an endpoint or domain monitorNone, and it cannot renew anyway (no key custody)
Reported by the Kubernetes controllerRenewal is cert-manager's job, not the scheduler's

A profile exists because an issuance produced it, or because you ran a real renewal against a known deployment path. There is no field-by-field create form: every field a renewal needs was proven correct by a real ACME order against a real host, and a hand-written profile is a set of values nobody has tested until the renewal runs unattended against a rate-limited CA.

Set up renewal​

Two things can trigger a derivation: an issue job, and Set up renewal on the Certificates tab (/certops/certificates). For a certificate that is already active, agent-deployable, and has no profile, its row carries a Set up renewal button that asks for the four inputs the scheduler needs (ACME command profile, CA endpoint, DNS provider, DNS zone) and would run a real renew job against the certificate's recorded deployment path. A profile is derived only if that job succeeds, exactly like an issuance.

Still unreachable for an imported certificate

Set up renewal requires a deployment path to already be recorded on the certificate. A certificate discovered on an agent's filesystem has one, recorded by the scan that found it, so the route works for that source. An imported PEM does not - nothing records a deployment path for an import - so it still fails with CERTOPS_RENEWAL_SETUP_NO_DEPLOYED_PATH. Re-issuing through TokenTimer is the only way today to get a profile onto an imported certificate.

So the fix for a certificate with no profile is: for an agent-discovered certificate, Set up renewal; for an imported one, re-issue it through TokenTimer instead.

If a certificate you issued has no profile, derivation declined because the issue payload was missing a field the profile needs: caEndpoint, commandRef, dnsProvider, dnsZone, or certPath. That is deliberate. The certificate is still promoted and usable, because losing a real issuance over renewal metadata would be the worse outcome. Re-issue with a complete payload. The CERTOPS_RENEWAL_PROFILE_DERIVATION_DECLINED audit event records which field was missing (see Audit events). Look for certops-renewal-profile-derivation-failed in the API logs to see the same field.

A derived profile is editable, within limits

Its renewal lead time (renew_before_days) and its on/off switch are yours to change on the Renewals tab (/certops/renewals). Changing the lead time affects only the certificates using that profile.

Deployment details are not editable​

Re-issue if the CA, command, DNS, or path must change. A request to change those fields is refused with 422 CERTOPS_PROFILE_FIELD_IMMUTABLE.

Step 1b - Check renewal was not switched off​

Automatic renewal switch​

A profile carries an automatic-renewal switch. When it is off (status of disabled, or archived), the profile is complete and the certificate is eligible, but TokenTimer skips it on purpose and the badge reads Auto-renewal off.

This skip looks identical to a healthy certificate in every view except the badge, so check it before assuming a scheduling fault:

  1. Open the Renewals tab of Certificate operations (/certops/renewals). A switched-off profile shows an orange Off badge, and the upcoming-renewals list carries a standing count of certificates that will not renew.
  2. Confirm somebody meant it. CERTOPS_RENEWAL_PROFILE_UPDATED in the audit log records who changed it, when, and what changed (a switch appears as autoRenewEnabled in changes). A profile with no update event was switched off nowhere: check CERTOPS_RENEWAL_PROFILE_DERIVED instead, which is written when TokenTimer generated the profile itself and shows exactly what it granted.
  3. If it was not intended, switch it back on. It takes effect on the next scheduling pass.

The skipped_auto_renew_disabled scheduler outcome counts these fleet-wide, which is how you notice a certificate that was switched off months ago and forgotten.

Scheduler outcome metrics​

certops_renewal_scheduler_certificates is labelled by outcome and reflects the last sweep. It exists precisely so a fleet that renews nothing is distinguishable from a fleet with nothing due.

certops_renewal_scheduler_certificates{outcome="scanned"}
certops_renewal_scheduler_certificates{outcome="created"}
certops_renewal_scheduler_certificates{outcome="replayed"}
certops_renewal_scheduler_certificates{outcome="skipped_paused"}
certops_renewal_scheduler_certificates{outcome="skipped_ca_cap"}
certops_renewal_scheduler_certificates{outcome="skipped_incomplete_profile"}
certops_renewal_scheduler_certificates{outcome="skipped_not_agent_deployable"}
certops_renewal_scheduler_certificates{outcome="skipped_auto_renew_disabled"}
certops_renewal_scheduler_certificates{outcome="errors"}

How to read it:

  • scanned is 0: nothing is due. Either your certificates are all outside the window, or not_after is not populated.
  • skipped_incomplete_profile is non-zero: certificates are due and being refused. Go to Step 1. This is the series to alert on.
  • skipped_not_agent_deployable is non-zero: expected if you monitor third-party certificates. Confirm it matches the number of observed-only certificates you know about.
  • skipped_auto_renew_disabled is non-zero: somebody switched renewal off. Deliberate, so not a fault, but confirm it is still the intent. Go to Step 1b.
  • skipped_ca_cap is persistently non-zero: renewals are starved by the cap. Raise CERTOPS_RENEWAL_PER_CA_CAP or spread expiry dates.
  • replayed is non-zero: a job for that expiry window already exists. Normal on repeat sweeps.
  • errors is non-zero: check the worker logs for certops-renewal-scheduler-cert-failure.

For the full metric list see CertOps worker metrics.

Step 2 - Check the job got claimed​

A scheduled job is not a completed renewal. Renew jobs are created open-claim: no agent is nominated, and the job is matched at claim time against each agent's declared target selectors, DNS providers, and command profiles. The exception is a certificate discovered on a specific agent's filesystem, which pins the job to that agent.

If the job sits at pending:

  1. Is it pinned to an offline agent? Check the Agent fleet panel. A pinned job waits rather than moving to a healthy host, because a different host is not a valid substitute.
  2. Does any online agent declare what the job requires? Compare the job's required target selector, DNS provider, and command profile against the agent declarations.
  3. Is it waiting on approval? A job at pending_approval needs an explicit decision and will never time out into running. See Approval gates.
  4. Is the certificate still provisioning? A renew job against a provisioning certificate is only offered to agents that declare evidence-claim-binding-v1. See When a certificate stays at provisioning.

Step 3 - Confirm failures reach you​

A renewal that fails should tell you. The certificate's linked token has to resolve to a contact group with a deliverable channel (email contacts or a webhook), or your workspace needs a default one.

On this instance the outbox drain sweep must also be enabled (CERTOPS_SWEEP_OUTBOX_DRAIN_ENABLED). It is what turns a recorded failure into a delivered alert. Disabled, alerts accumulate undelivered rather than being lost, but you hear nothing in the meantime.

See Certificate renewal failures for the full routing rules, and note that a failed issuance does not raise this alert: watch for certificates stuck at provisioning instead.

Verification​

You have a certificate that will renew on its own when:

  1. Its renewal badge reads Auto-renews, with a renewal date you recognise.
  2. It appears in Upcoming renewals on the Renewals tab with automatic renewal On. Any other label (Off, No profile, Incomplete, No key access, No expiry) names the reason it will not renew, and only Off is fixable from that page.
  3. An agent is online that matches the certificate's target selector, DNS provider, and command profile.
  4. Its token resolves to a contact group that would reach you if the renewal failed.
  5. certops_maintenance_sweeps_total{sweep="renewal-scheduler"} is increasing.
  6. certops_renewal_scheduler_certificates{outcome="skipped_incomplete_profile"} is 0.

To prove it end to end without waiting for expiry, run a manual renew job against the certificate with dryRun: true. That exercises profile resolution, agent matching, and the trust chain with no filesystem side effects. A dry run that completes means the same job would run for real.

Renewal profile endpoints​

All paths are under /api/v1/workspaces/{id}/certops. Reading needs manager or above; writing needs workspace admin (certops.renewal_profile.manage).

Reads sit at manager rather than viewer because a profile body carries deployment topology: certificate and key paths, the reload unit, file ownership and modes, ACME command reference, CA account reference, and DNS zone. A viewer keeps full visibility of what expires and when, via Certificates and the renewal badge, without learning where a given key lives on a host.

EndpointPurpose
GET .../profilesList renewal profiles with the certificate count each covers
GET .../profiles/{profileId}One profile, including which fields are editable
PATCH .../profiles/{profileId}Update the safe subset: on/off, lead time, SAN policy, key parameters, verification, preferred chain
GET .../renewals/upcomingEvery renewable certificate, soonest expiry first, with the renewal window, the last attempt, and blockedReason when the scheduler will not act on it
POST .../certificates/{certId}/renewal-setupSet up renewal: runs a real renew job against a discovered path and arms profile derivation on success (dry run arms nothing)
DELETE .../certificates/{certId}/profileDetach: unlinks the certificate from its profile; the profile row and any other certificate using it are untouched
POST .../renewal-setup-intents/{outboxId}/retryRetry: re-arms a failed setup intent. Refused for a skipped one, since that was a decision, not a failure

There is no direct create and no delete on profiles itself: you cannot type profile field values into an API request and have one appear. renewal-setup derives a profile from a job outcome the same way an issuance does.