Renew certificates automatically

Goal

Confirm that a managed certificate will renew without you, and diagnose the case where it silently will not.

This runbook exists because of a specific and expensive failure shape: a certificate that reads active in the inventory, has a real expiry date, shows no failed jobs, and is quietly never renewed by anything. Nothing in that picture looks wrong until the certificate expires.

Start with the renewal badge

Every certificate in the inventory carries a renewal badge. It is computed server-side from the scheduler's own inputs, using the same completeness check the sweep admits on, so it cannot promise automatic renewal for a certificate the sweep would refuse. Read it before working through anything below.

Badge	What it means	What to do
Auto-renews	The scheduler will pick this certificate up. The badge also shows the date renewal becomes due and the lead time in days	Nothing
Auto-renewal off	Agent-deployable and configured, but renewal was switched off on its profile. This certificate will expire.	Step 2b if that was not intended
No auto-renewal	Agent-deployable, but no usable renewal profile. This certificate will expire.	Step 2
Monitored only	No agent holds the key (observed via an endpoint or domain monitor), so nothing can renew it here. Expected, not a fault	Renew it wherever it is actually managed
Renewal not applicable	Retired, or renewal belongs to another system such as cert-manager	Nothing
Renewal unknown	The state could not be determined. Treat as manual until confirmed	Check the API is reachable and up to date, then re-read

"No auto-renewal" is the dangerous state and the reason this badge exists. It is deliberately kept distinct from "Monitored only", because one is a certificate you believe is managed and is not, and the other is working as designed. "Auto-renewal off" is a third case: somebody chose this, so it is not a fault, but the certificate still expires.

The five conditions

The renewal scheduler creates a renew job for a certificate only when all five hold.

Condition	How to check	If it fails
A linked renewal profile with a complete snapshot	The badge reads "Auto-renews". The scheduler counts failures as skipped_incomplete_profile	See Step 2
Agent-deployable key custody (key_mode of agent-local or proxy-agent-local)	The certificate detail shows its key custody mode	Observed-only certificates can never auto-renew. Renew them out of band, or bring them under agent management
Inside the renewal window	not_after within CERTOPS_RENEWAL_THRESHOLD_DAYS (default 30), or the profile's own renew_before_days	Nothing to do; it is not due yet
A renewable inventory status	Anything other than revoked or decommissioned	Retired certificates are excluded by design
Automatic renewal switched on for its profile	The Renewal automation page shows the profile as On. The scheduler counts the rest as skipped_auto_renew_disabled	See Step 2b

Two further conditions are temporary rather than structural. A certificate is skipped for the current sweep and retried on the next one when the workspace is paused or CertOps is disabled deployment-wide, and when the per-CA in-flight cap (CERTOPS_RENEWAL_PER_CA_CAP) is already full.

Step 1 - Confirm the maintenance worker is running

Nothing below matters if the sweep never executes. The renewal scheduler lives in the CertOps maintenance worker, and its absence is silent: the API keeps working and the deployment looks healthy.

• Docker Compose: the worker-certops service must be running.
• Helm: the certops CronJob must be enabled and its recent runs must be succeeding.

Both must also see the same CERTOPS_ENABLED value as the API. If the API has CertOps on and the worker does not, the sweep treats every certificate as "workspace paused" and logs nothing at error level. See the warning in Worker scheduling.

If you scrape metrics, certops_maintenance_sweeps_total{sweep="renewal-scheduler"} should be increasing.

Step 2 - Check the certificate has a renewal profile

This is the answer most of the time.

The scheduler will not dispatch a renewal it cannot fully specify: CA endpoint, ACME command profile, DNS provider and zone, deployment paths, SAN policy, and key parameters all have to be present and valid. A certificate with no linked profile, or one whose snapshot fails validation, is skipped on every single sweep, forever, without producing a job or an alert.

Where a profile comes from depends on how the certificate arrived:

How the certificate arrived	Profile
Issued by TokenTimer (an issue job)	Derived automatically when the issuance succeeded, named Derived: <common name>
Imported as a PEM	None. Set up renewal is intended to fix this once an agent has custody, but currently cannot; see the caveat below
Discovered on an agent filesystem	None yet, but Set up renewal can arm one: the scan that found the certificate recorded its deployment path
Observed via an endpoint or domain monitor	None, and it cannot renew anyway (no key custody)
Reported by the Kubernetes controller	Renewal is cert-manager's job, not the scheduler's

A profile exists because an issuance produced it, or because you ran a real renewal against a known deployment path. There is no field-by-field profile create form: the API and dashboard both derive a profile from a job that actually succeeded, never from typed-in values nobody has run against a live host.

Two things can trigger that derivation:

• An issue job, described in the rest of this step.
• Set up renewal, on the Certificates tab (/certops/certificates): for a certificate that is already active, agent-deployable, and has no profile, its row carries a Set up renewal button. It asks for the four inputs the scheduler needs (ACME command profile, CA endpoint, DNS provider, DNS zone), then would run a real renew job against the certificate's recorded deployment path. A profile is derived only if that job succeeds, exactly like an issuance.

Still unreachable for an imported certificate

The button confirms successfully for a filesystem-discovered certificate: agent filesystem discovery writes managed_certificates.deployed_cert_path from the path it found the certificate at, which is exactly what this route requires. An imported PEM has no such path written anywhere - there is no "take custody" action for an import that would - so confirming there, dry run or real, still fails with CERTOPS_RENEWAL_SETUP_NO_DEPLOYED_PATH. Re-issuing through CertOps is the only way today to get a profile onto an imported certificate.

So the fix for a certificate with no profile is: for a filesystem-discovered certificate, use Set up renewal; for an imported one, re-issue it through CertOps instead of describing a profile by hand.

If a certificate you issued has no profile, derivation declined because the issue payload was missing a field the profile needs: caEndpoint, commandRef, dnsProvider, dnsZone, or certPath. That is deliberate. The certificate is still promoted and usable, because losing a real issuance over renewal metadata would be the worse outcome. Look for certops-renewal-profile-derivation-failed in the API logs to see which field was missing, then re-issue with a complete payload.

A derived profile is editable, within limits

Its renewal lead time (renew_before_days) and its on/off switch are yours to change on the Renewal automation page. Changing the lead time affects only the certificates using that profile, which is the supported way to give one certificate a longer runway than the rest of the fleet.

Its deployment details are not editable, and a request to change them is refused with 422 CERTOPS_PROFILE_FIELD_IMMUTABLE. See Why you cannot edit a profile's deployment details.

Step 2b - Check renewal was not switched off

A profile carries an automatic-renewal switch. When it is off (status of disabled, or archived), the profile is complete and the certificate is eligible, but the scheduler skips it on purpose and the badge reads Auto-renewal off.

This is the one skip that looks identical to a healthy certificate in every view except the badge, so check it before assuming a scheduling fault:

1. Open the Renewals tab of Certificate operations (/certops/renewals). A switched-off profile shows an orange Off badge, and the upcoming-renewals list carries a standing count of certificates that will not renew.
2. Confirm somebody meant it. CERTOPS_RENEWAL_PROFILE_UPDATED in the audit log records who changed it, when, and what changed (a switch appears as autoRenewEnabled in changes). A profile with no update event was switched off nowhere: check CERTOPS_RENEWAL_PROFILE_DERIVED instead, which is written when TokenTimer generated the profile itself and shows exactly what it granted.
3. If it was not intended, switch it back on. It takes effect on the next sweep.

The skipped_auto_renew_disabled outcome metric counts these fleet-wide, which is how you notice a certificate that was switched off months ago and forgotten.

Step 3 - Read the scheduler outcome metrics

certops_renewal_scheduler_certificates is labelled by outcome and reflects the last sweep. It exists precisely so a fleet that renews nothing is distinguishable from a fleet with nothing due.

certops_renewal_scheduler_certificates{outcome="scanned"}
certops_renewal_scheduler_certificates{outcome="created"}
certops_renewal_scheduler_certificates{outcome="replayed"}
certops_renewal_scheduler_certificates{outcome="skipped_paused"}
certops_renewal_scheduler_certificates{outcome="skipped_ca_cap"}
certops_renewal_scheduler_certificates{outcome="skipped_incomplete_profile"}
certops_renewal_scheduler_certificates{outcome="skipped_not_agent_deployable"}
certops_renewal_scheduler_certificates{outcome="skipped_auto_renew_disabled"}
certops_renewal_scheduler_certificates{outcome="errors"}

How to read it:

• scanned is 0: nothing is due. Either your certificates are all outside the window, or not_after is not populated.
• skipped_incomplete_profile is non-zero: certificates are due and being refused. Go to Step 2. This is the series to alert on.
• skipped_not_agent_deployable is non-zero: expected if you monitor third-party certificates. Confirm it matches the number of observed-only certificates you know about.
• skipped_auto_renew_disabled is non-zero: somebody switched renewal off. Deliberate, so not a fault, but confirm it is still the intent. Go to Step 2b.
• skipped_ca_cap is persistently non-zero: renewals are starved by the cap. Raise CERTOPS_RENEWAL_PER_CA_CAP or spread expiry dates.
• replayed is non-zero: a job for that expiry window already exists. Normal on repeat sweeps.
• errors is non-zero: check the worker logs for certops-renewal-scheduler-cert-failure.

For the full metric list see CertOps worker metrics.

Step 4 - Check the job got claimed

A created job is not a completed renewal. Renew jobs are created open-claim: no agent is nominated, and the job is matched at claim time against each agent's declared target selectors, DNS providers, and command profiles. The exception is a certificate discovered on a specific agent's filesystem, which pins the job to that agent.

If the job sits at pending:

1. Is it pinned to an offline agent? Check the Agent fleet panel. A pinned job waits rather than moving to a healthy host, because a different host is not a valid substitute.
2. Does any online agent declare what the job requires? Compare the job's required target selector, DNS provider, and command profile against the agent declarations.
3. Is it waiting on approval? A job at pending_approval needs an explicit decision and will never time out into running. See Approval gates.
4. Is the certificate still provisioning? A renew job against a provisioning certificate is only offered to agents that declare evidence-claim-binding-v1. Upgrade the agent. See When a certificate stays at provisioning.

Step 5 - Confirm failures reach you

A renewal that fails should tell you. Two things have to be true:

• The outbox drain sweep is enabled (CERTOPS_SWEEP_OUTBOX_DRAIN_ENABLED). It is what turns a recorded failure into a delivered alert. Disabled, alerts accumulate undelivered rather than being lost, but you hear nothing in the meantime.
• The certificate's linked token resolves to a contact group with a deliverable channel, or your workspace has a default one.

See Certificate renewal failures for the full routing rules, and note that a failed issuance does not raise this alert: watch for certificates stuck at provisioning instead.

Verification

You have a certificate that will renew on its own when:

1. Its renewal badge reads Auto-renews, with a renewal date you recognise.
2. It appears in Upcoming renewals on the Renewal automation page with automatic renewal On. Any other label there (Off, No profile, Incomplete, No key access, No expiry) names the reason it will not renew, and only Off is fixable from that page.
3. certops_maintenance_sweeps_total{sweep="renewal-scheduler"} is increasing.
4. certops_renewal_scheduler_certificates{outcome="skipped_incomplete_profile"} is 0.
5. An agent is online that matches the certificate's target selector, DNS provider, and command profile.

To prove it end to end without waiting for expiry, run a manual renew job against the certificate with dryRun: true. That exercises profile resolution, agent matching, and the trust chain with no filesystem side effects. A dry run that completes means the same job would run for real.

Renewal profile endpoints

All paths are under /api/v1/workspaces/:id/certops. Reading needs manager or above; writing needs workspace admin (certops.renewal_profile.manage).

Reads sit at manager rather than viewer because a profile body carries deployment topology: the certificate and key paths on disk, the reload unit, file ownership and modes, the ACME command reference, the CA account reference, and the DNS zone. That is closer to the agent and machine-token endpoints than to the certificate inventory, which exposes only expiry metadata. A viewer keeps full visibility of what expires and when, via Certificates and the renewal badge, without learning where a given key lives on a host.

Endpoint	Purpose
GET .../profiles	List renewal profiles with the certificate count each covers
GET .../profiles/:profileId	One profile, including which fields are editable
PATCH .../profiles/:profileId	Update the safe subset: on/off, lead time, SAN policy, key parameters, verification, preferred chain
GET .../renewals/upcoming	Every renewable certificate, soonest expiry first, with the renewal window, the last attempt, and blockedReason when the scheduler will not act on it
POST .../certificates/:certId/renewal-setup	Set up renewal: runs a real renew job against a discovered path and arms profile derivation on success (dry run arms nothing). Refused per the precondition table above
DELETE .../certificates/:certId/profile	Detach: unlinks the certificate from its profile; the profile row and any other certificate using it are untouched
POST .../renewal-setup-intents/:outboxId/retry	Retry: re-arms a failed setup intent. Refused for a skipped one, since that was a decision, not a failure

There is deliberately no direct create and no delete on profiles itself: you cannot type profile field values into an API request and have one appear. renewal-setup is not an exception to that; it derives a profile from a job outcome the same way an issuance does. See Step 2 for why.

Related pages

• Certificate Automation - the scheduling model in full.
• Configuration Reference - renewal and sweep variables.
• Certificate renewal failures - alerting on failures.
• Reconciling interrupted jobs - when a renewal stopped reporting midway.