Skip to main content

Certificate renewal failure alerts

Overview​

When a CertOps renew job reaches a terminal failure state, TokenTimer raises a cert_renewal_failed alert through the same delivery pipeline as your expiry and endpoint-health alerts: the alert queue, your configured delivery window, and the contact group attached to the certificate's token. Four independent paths can reach that terminal state, and all four raise this alert:

  • The agent reports failed, blocked, or rejected by its own policy.
  • The control-plane lease reaper marks the job orphaned_unknown_effect after the agent went silent.
  • A forced agent retirement fences the job: a renew job the retired agent still held is either cancelled outright or, if it had reached running, marked orphaned_unknown_effect for the same reason the lease reaper uses that status - the real-world effect is unproven.
  • An external executor (your own tooling over the machine-token event API, cert-manager, or another controller) reports failed or rejected for a renew job.

This is a safety net on top of scheduled renewals: if automation, an agent, or your own executor fails to complete a renewal, you find out through the channels you already use for expiry reminders, without having to watch the job timeline yourself.

The alert intent is recorded durably​

The transaction that decides a renewal job has terminally failed also records the intent to alert in the same transaction. Resolving your contact groups, picking channels, and inserting into the alert queue all happen afterwards, in a separate retried background sweep.

That split matters to you for one reason: an alert can no longer be lost because contact resolution had a bad moment. Previously delivery was attempted inline on a best-effort basis, and any error there dropped the notification silently, so a failed renewal could pass unannounced. Now the intent is committed alongside the failure, retried with backoff, and if it genuinely cannot be delivered (no linked token, no contact group with a deliverable channel) that outcome is recorded with a reason rather than vanishing.

When it fires​

All of these conditions must hold, or nothing is queued:

  • Operation is renew: deploy, reload, revoke, and no-op jobs never raise this alert, even if they fail. A failed issue job does not raise it either, which is deliberate and explained in Why a failed issuance is not a renewal failure.
  • The transition was not a deliberate human decision: a job an approver rejected, or one an operator cancelled by hand, never alerts. The person who did it was looking at the job at the time. An agent rejecting a job on its own local policy grounds is a different thing and does alert, because nobody chose that outcome. A job cancelled by a forced agent retirement is different again: an operator chose to retire the agent, but not to abandon that specific renewal mid-flight, so this alerts.
  • The status is a real failure: failed, blocked, and agent-side rejected alert. succeeded and dry_run_complete never do, and neither do the non-terminal statuses (pending, approved, claimed, running). A dry run changes nothing on the host by construction, so there is no failure to act on. cancelled alerts only when something other than a manual operator cancellation caused it, which is why a forced retirement's cancellation is treated as alert-worthy while a manual one is not.
  • orphaned_unknown_effect always alerts (for a renew job): the outcome when an agent went silent, or was forcibly retired, after it may already have changed something on a host. This is the one case where doing nothing risks a half-deployed certificate, so it is treated as high priority. See Reconciling interrupted jobs.
  • Reached by an agent-executed job, a forced agent retirement, or an external executor: the intent is recorded by agent result ingestion, the control-plane lease reaper, the force-retirement fencing sweep, and executor event ingestion. All four write the same durable intent through the same policy classifier, so none of them is a second-class alerting path.
  • Linked to a managed certificate: the job's subject must be a managed_certificate with subject id set, and that managed certificate must be linked to a cert-category token. A break-glass job with no subject, or a certificate that was imported without a linked token, has nothing to anchor the alert to and is skipped.
  • A workspace admin exists: the alert is addressed to a workspace admin, resolved the same way as other CertOps notifications. A workspace with no admin member skips the alert.
  • At least one deliverable channel: the resolved contact group must have email contacts or a configured webhook. If neither is present, nothing is queued; there is no separate "must configure this" step beyond your normal contact group setup.

None of these conditions being unmet is an error: the renewal failure is still fully visible on the job timeline and in the audit log either way. The alert is an additional notification, not the source of truth for job status.

Why a failed issuance is not a renewal failure​

A failed renew alerts. A failed issue does not. That asymmetry is intentional, and it changes what you have to watch.

A renewal failing means a certificate that already exists, is already serving traffic, and has a real expiry date did not get replaced. If nobody acts, it expires and something breaks. That is exactly what an alert is for.

An issuance failing means a certificate you asked for was never created. Nothing is serving it, nothing is expiring, and no deadline is approaching. There is also nothing to route an alert to in the usual sense: the certificate row exists at provisioning, but it has no verified expiry and no linked token yet, and the linked token is the anchor renewal alerts are addressed through. What you get instead is the failed job, its evidence, and a provisioning certificate sitting visibly in your inventory, which is the record you work from when retrying.

So watch provisioning certificates for issuance problems and alerts for renewal problems. Once an issued certificate reaches active, its later renewals alert exactly like any other certificate's.

Idempotency​

Exactly one alert is queued per job, keyed by cert_renewal_failed:<jobId>. Two layers enforce that: the recorded intent is unique per job, and the alert insert is existence-checked on the alert key, so a redelivered intent cannot produce a second notification.

If you retry the underlying renewal and it creates a new job that also fails, that is a distinct alert (a new jobId, a new key) - which is intentional, since a second failure on a fresh attempt is new information worth telling you about.

Channels and routing​

Contact groups are resolved from the linked token of the failed job's managed certificate, in this order:

  1. The linked token's own contact group, if one is assigned to that token.
  2. Otherwise the workspace default contact group.
  3. If the token points at a contact group that no longer exists, resolution falls back to the workspace default rather than giving up. A contact group deleted after it was assigned to a token therefore degrades to the default instead of silently dropping the alert.
  4. Email is included when the resolved group has email contacts and workspace email alerts are enabled.
  5. Webhooks (Pro/Team) are included when the resolved group references a workspace webhook (Slack, Teams, Discord, PagerDuty, or generic) that exists in your webhook configuration. A group naming a webhook you have since removed contributes no webhook channel.
  6. WhatsApp is not supported for this alert type. Renewal failures only deliver over email and webhooks, even if the contact group has WhatsApp contacts configured for other alert types.

If steps 1 to 3 produce no group, or the group produces no channel, nothing is queued. Assign a contact group with at least one email contact or webhook to the certificate's token, or set a workspace default.

Delivery still respects your workspace delivery window: a failure detected outside the window is deferred and delivered once the window opens, exactly like any other alert.

What you get in the alert​

The alert carries the certificate identity (via the linked token) and the frozen error code from the job, so you can tell at a glance which certificate failed and why, without necessarily opening the dashboard. For the full detail, deployment attempts, and any redacted command output, open the job on Certificate operations and review its timeline and evidence.

info

Zero-custody applies here too: the alert queue and the recorded intent both store only ids and the frozen error code, never job payload contents or anything from evidence.

If a failure produced no alert​

Work down this list in order; the first two are far more common than a contact-group problem.

  1. Was the operation actually a renew? A failed issue job does not alert. See Why a failed issuance is not a renewal failure.
  2. Did a human end the job with a direct decision? Approver rejections and manual operator cancellations never alert. A job cancelled as a side effect of force-retiring its agent is not in this category and does alert.
  3. Is the certificate linked to a token? An imported or provisioning certificate with no linked token has no alert anchor.
  4. Does the resolved contact group have a deliverable channel? Email contacts, or a webhook that still exists in your workspace webhook configuration.

Plan availability​

This alert is part of CertOps and is available on the Pro and Team plans. It reuses your existing alert channel configuration; no separate setup is required beyond having a deliverable contact group on the certificate's token.