Skip to main content
Version: 0.11

Certificate renewal failure alerts

Overview​

When a CertOps renew job reaches a terminal failure state, TokenTimer raises a cert_renewal_failed alert through the same delivery pipeline as your expiry and endpoint-health alerts: the alert queue, your configured delivery window, and the contact group attached to the certificate's token. Four independent paths can reach that terminal state, and all four raise this alert:

  • The agent reports failed, blocked, or rejected by its own policy.
  • The control-plane lease reaper marks the job orphaned_unknown_effect after the agent went silent.
  • A forced agent retirement fences the job: a renew job the retired agent still held is either cancelled outright or, if it had reached running, marked orphaned_unknown_effect for the same reason the lease reaper uses that status - the real-world effect is unproven.
  • An external executor (your own tooling over the machine-token event API, cert-manager, or another controller) reports failed or rejected for a renew job.

This is a safety net on top of scheduled renewals: if automation, an agent, or your own executor fails to complete a renewal, you find out through the channels you already use for expiry reminders, without having to watch the job timeline yourself.

The alert intent is recorded durably​

The transaction that decides a renewal job has terminally failed also records the intent to alert in a control-plane outbox (certops_outbox), in that same transaction. Nothing else happens at that moment: resolving contacts, picking channels, and inserting into the alert queue all happen afterwards, in the outbox-drain sweep of the CertOps maintenance worker.

That split exists because of a real failure mode. Previously the alert was queued inline on a best-effort basis, wrapped so that an alerting problem could never abort result ingestion. The cost was that any error in contact resolution lost the intent silently: one log line, no retry, no record. You saw a failed job and never got a notification, with nothing to tell you a notification had been attempted.

What this means in practice:

  • A failed renewal that is recorded is guaranteed to have an alert intent recorded with it. The two commit together or not at all.
  • Delivery is retried. A drain attempt that throws is retried with exponential backoff up to 5 attempts, then parked as failed for an operator instead of disappearing.
  • A structural skip is final, and it says why. If there is genuinely nothing to alert (no linked token, no contact group with a deliverable channel), the row goes terminal as skipped with the reason preserved. Retrying could not change the answer.
  • If the maintenance worker is not running, no renewal-failure alert is ever delivered. Intents accumulate as pending outbox rows instead of being lost, so you recover every one of them once the worker starts, but until then the alert channel is silent while jobs keep failing. If renewals are failing on the dashboard and no alert arrives, check that the certops worker target is actually scheduled before looking at contact groups. See the scheduling table in Maintenance sweeps.

When it fires​

All of these conditions must hold, or nothing is queued:

  • Operation is renew: deploy, reload, revoke, and no-op jobs never raise this alert, even if they fail. A failed issue job does not raise it either, which is deliberate and explained in Why a failed issuance is not a renewal failure.
  • The transition was not a deliberate human decision: a job an approver rejected, or one an operator cancelled by hand, never alerts. The person who did it was looking at the job at the time, and emailing them about their own action is noise. An agent rejecting a job on its own local policy grounds is a different thing and does alert, because nobody chose that outcome. A job cancelled by a forced agent retirement is different again: an operator chose to retire the agent, but not to abandon that specific renewal mid-flight, so this alerts.
  • The status is a real failure: failed, blocked, and agent-side rejected alert. succeeded and dry_run_complete never do, and neither do the non-terminal statuses (pending, approved, claimed, running). A dry run changes nothing on the host by construction, so there is no failure to act on. cancelled alerts only when something other than a manual operator cancellation caused it, which is why a forced retirement's cancellation is treated as alert-worthy while a manual one is not.
  • orphaned_unknown_effect always alerts (for a renew job): the outcome when an agent went silent, or was forcibly retired, after it may already have changed something on a host. This is the one case where doing nothing risks a half-deployed certificate, so it is classified as high priority. See Reconciling interrupted jobs.
  • Reached by an agent-executed job, a forced agent retirement, or an external executor: the intent is recorded by agent result ingestion, the control-plane lease reaper, the force-retirement fencing sweep, and executor event ingestion. All four write the same durable outbox intent through the same policy classifier, so none of them is a second-class alerting path.
  • Linked to a managed certificate: the job's subject must be a managed_certificate with subject id set, and that managed certificate must be linked to a cert-category token. A break-glass job with no subject, or a certificate that was imported without a linked token, has nothing to anchor the alert to and is skipped as no_managed_certificate_subject or no_linked_token.
  • A workspace admin exists: the alert is addressed to a workspace admin, resolved the same way as other CertOps notifications. A workspace with no admin member skips the alert (no_recipient).
  • At least one deliverable channel: the resolved contact group must have email contacts or a configured webhook, or the row is skipped as no_channels. There is no separate "must configure this" step beyond your normal contact group setup.

None of these conditions being unmet is an error: the renewal failure is still fully visible on the job timeline and in the audit log either way. The alert is an additional notification, not the source of truth for job status.

Why a failed issuance is not a renewal failure​

A failed renew alerts. A failed issue does not. That asymmetry is intentional, and it is worth understanding because it changes what you have to watch.

A renewal failing means a certificate that already exists, is already serving traffic, and has a real expiry date did not get replaced. If nobody acts, that certificate expires and something breaks. That is exactly what an alert is for.

An issuance failing means a certificate you asked for was never created. Nothing is serving it, nothing is expiring, and no deadline is approaching. There is also nothing to route an alert to in the usual sense: the certificate row exists at provisioning, but it has no verified expiry and no linked token yet, which is the anchor renewal alerts are addressed through. What you get instead is the failed job, its evidence, and a provisioning certificate sitting visibly in your inventory, which is the record you actually work from when retrying.

So watch provisioning certificates for issuance problems and alerts for renewal problems. Once an issued certificate reaches active, its later renewals alert exactly like any other certificate's.

Idempotency​

Exactly one alert is queued per job, keyed by cert_renewal_failed:<jobId>. Two layers enforce that: the outbox row itself is unique per job and event type, so a retried deciding transaction records one intent, and the alert insert is existence-checked on the alert key, so a redelivered outbox event cannot produce a second notification.

If you retry the underlying renewal and it creates a new job that also fails, that is a distinct alert (a new jobId, a new key) - which is intentional, since a second failure on a fresh attempt is new information worth telling you about.

Channels and routing​

Contact groups are resolved from the linked token of the failed job's managed certificate, in this order:

  1. The linked token's own contact group, if one is assigned to that token.
  2. Otherwise the workspace default contact group.
  3. If the token points at a contact group that no longer exists, resolution falls back to the workspace default rather than giving up. A contact group deleted after it was assigned to a token therefore degrades to the default instead of silently dropping the alert.
  4. Email is included when the resolved group has email contacts and workspace email alerts are enabled.
  5. Webhooks are included when the resolved group references a workspace webhook (Slack, Teams, Discord, PagerDuty, or generic) that exists in your webhook configuration. A group naming a webhook you have since removed contributes no webhook channel.
  6. WhatsApp is not supported for this alert type. Renewal failures only deliver over email and webhooks, even if the contact group has WhatsApp contacts configured for other alert types.

If steps 1 to 3 produce no group, or the group produces no channel, nothing is queued and the outbox row records no_channels. Assign a contact group with at least one email contact or webhook to the certificate's token, or set a workspace default.

Delivery still respects your workspace delivery window: a failure detected outside the window is deferred and delivered once the window opens, exactly like any other alert.

What you get in the alert​

The alert carries the certificate identity (via the linked token) and the frozen error code from the job, so you can tell at a glance which certificate failed and why, without necessarily opening the dashboard. For the full detail, deployment attempts, and any redacted command output, open the job on Certificate operations and review its timeline and evidence.

info

Zero-custody applies here too: the alert queue and the outbox both store only ids and the frozen error code, never job payload contents or anything from evidence. The outbox rejects any payload field outside its per-event allowlist and rescans for key material before persisting.

If a failure produced no alert​

Work down this list in order; the first three are far more common than a contact-group problem.

  1. Is the certops maintenance worker scheduled and running? No worker means no drain sweep and no delivery. See Maintenance sweeps.
  2. Was the operation actually a renew? A failed issue job does not alert. See Why a failed issuance is not a renewal failure.
  3. Did a human end the job with a direct decision? Approver rejections and manual operator cancellations never alert. A job cancelled as a side effect of force-retiring its agent is not in this category and does alert.
  4. Is the certificate linked to a token? An imported or provisioning certificate with no linked token has no alert anchor.
  5. Does the resolved contact group have a deliverable channel? Email contacts, or a webhook that still exists in your workspace webhook configuration.