Skip to main content
Version: next

Certificate renewal failure alerts

Overview​

When an agent-executed CertOps renew job reaches a terminal failure state (failed, or orphaned_unknown_effect via the lease reaper), TokenTimer raises a cert_renewal_failed alert through the same delivery pipeline as your expiry and endpoint-health alerts: the alert queue, your configured delivery window, and the contact group attached to the certificate's token.

This is a safety net on top of scheduled renewals: if automation or an agent-executed renewal fails, you find out through the channels you already use for expiry reminders, without having to watch the job timeline yourself. Renewal jobs reported by an external executor over the machine-token event API do not raise this alert; that path is not yet wired to renewal-failure notifications.

When it fires​

All of these conditions must hold, or nothing is queued:

  • Operation is renew: deploy, reload, revoke, and no-op jobs never raise this alert, even if they fail.
  • Reached by an agent-executed job: the alert is queued from agent result ingestion and from the control-plane lease reaper (when an agent stops responding mid-renewal). Renew jobs reported over the executor machine-token event API do not queue this alert today.
  • Terminal failed or orphaned_unknown_effect status: a job that is still claimed/running, or that ends in rejected, blocked, or cancelled, does not trigger it. The lease reaper's orphaned_unknown_effect outcome (side effects unknown after an unresponsive agent) raises the same alert as an ordinary failed result, since both mean the renewal did not succeed and needs attention.
  • Linked to a managed certificate: the job's subject must be a managed_certificate with subject id set, and that managed certificate must be linked to a cert-category token. A break-glass job with no subject, or a certificate that was imported without a linked token, has nothing to anchor the alert to and is skipped.
  • A workspace admin exists: the alert is addressed to a workspace admin, resolved the same way as other CertOps notifications. A workspace with no admin member skips the alert.
  • At least one deliverable channel: the linked token's contact group (or the workspace default, if none is set on the token) must have email contacts or a configured webhook. If neither is present, nothing is queued; there is no separate "must configure this" step beyond your normal contact group setup.

None of these conditions being unmet is an error: the renewal failure is still fully visible on the job timeline and in the audit log either way. The alert is an additional notification, not the source of truth for job status.

Idempotency​

Exactly one alert is queued per job, keyed by cert_renewal_failed:<jobId>. A retry wave that resubmits the same failure outcome for the same job (for example a duplicate result delivery from an agent) never creates a second alert for that job. If you retry the underlying renewal and it creates a new job that also fails, that is a distinct alert (a new jobId, a new key) - which is intentional, since a second failure on a fresh attempt is new information worth telling you about.

Channels and routing​

Renewal failure alerts follow the same contact-group resolution as your other certificate alerts:

  1. The certificate's linked token's contact group, if one is assigned.
  2. Otherwise, the workspace default contact group.
  3. Email is included when the resolved group has email contacts and workspace email alerts are enabled.
  4. Webhooks are included when the resolved group references a workspace webhook (Slack, Teams, Discord, PagerDuty, or generic) that exists in your webhook configuration.
  5. WhatsApp is not supported for this alert type. Renewal failures only deliver over email and webhooks, even if the contact group has WhatsApp contacts configured for other alert types.

Delivery still respects your workspace delivery window: a failure detected outside the window is deferred and delivered once the window opens, exactly like any other alert.

What you get in the alert​

The alert carries the certificate identity (via the linked token) and the frozen error code from the job, so you can tell at a glance which certificate failed and why, without necessarily opening the dashboard. For the full detail, deployment attempts, and any redacted command output, open the job on Certificate operations and review its timeline and evidence.

info

Zero-custody applies here too: the alert queue stores only ids and the error code, never job payload contents or anything from evidence.

  • Alerts & thresholds - delivery windows, contact groups, and channel setup shared by every alert type.
  • Certificate Automation - how renewal jobs are scheduled and what makes one eligible.
  • Approval gates - if a renewal requires approval and is never decided, it stays at pending_approval and does not fail or alert; approve or reject it explicitly.