Reconciling interrupted jobs

Goal

Understand and clear jobs that stopped reporting midway, in particular the orphaned_unknown_effect status, without risking a duplicate deployment or a second certificate order.

Why this state exists

An agent holds a lease on a job while it works, renewing the lease as it goes. If the agent dies, loses the network, or is killed mid-run, the lease eventually expires and the control plane has to decide what to do with the job. That decision cannot be "just retry", because a retry is only safe when nothing has happened yet.

TokenTimer distinguishes the two cases from evidence, not from guesswork:

Situation	What is proven	What happens
Lease never renewed, job still claimed	The agent never got far enough to do anything observable	Automatic requeue with backoff, up to the job's attempt limit
Lease was renewed, or job reached running	The agent may have ordered a certificate, written a key, deployed a file, or reloaded a service	Never silently retried, flagged for a human

In the second case the job is set to orphaned_unknown_effect with needs_operator_reconciliation, error_code: effects_unknown, and a reconciliation reason of lease_expired_after_side_effect_window_agent_unresponsive. When the interrupted job was a renewal, this also raises a cert_renewal_failed alert, so an orphaned renewal reaches you through your normal alerting channels rather than sitting unnoticed on the jobs list.

note

This is not an ordinary failure. A failed job means "we know it did not work". orphaned_unknown_effect means "we do not know how far it got", which is a different and more careful statement. Treat it as an investigation, not as a retry button.

Before flagging anything, the control plane gives a live agent room to finish: while the agent is still heartbeating and the hard grace window has not elapsed, the job is simply left alone so a late result can still land.

Timing

Setting	Default	Meaning
CERTOPS_JOB_LEASE_SECONDS	900 (15 min)	How long a claim is valid before it must be renewed
CERTOPS_LEASE_HARD_GRACE_MS	3600000 (1 h)	Extra time a still-heartbeating agent gets before its job is judged
CERTOPS_AGENT_OFFLINE_AFTER_MS	600000 (10 min)	Heartbeat silence after which an agent counts as offline for this decision

So a job that was actively running does not get flagged the moment its lease lapses. In the worst case a genuinely dead agent's job is flagged after roughly the lease plus the hard grace.

Step 1 - Find the affected jobs

Open Certificate operations at /certops/operations and look for jobs whose status is orphaned_unknown_effect. The job log carries a job.failed entry from the lease-reaper sweep with the outcome, the attempt count, and the previous status, which tells you how far the job had progressed.

Step 2 - Determine what actually happened

Check the host, not the dashboard. The dashboard only knows what was reported, and the whole point of this state is that reporting stopped. On the agent host:

1. Read the agent log around the job's timestamps (journalctl -u tokentimer-agent).
2. Check the agent's job journal: the agent records what it was doing so an interrupted run can be understood after the fact.
3. Inspect the certificate on disk. Compare the current certificate's serial, fingerprint, and notAfter against what the job intended:

openssl x509 -in /path/to/cert.pem -noout -serial -fingerprint -sha256 -dates

4. If the job included an ACME order, check the CA side too. A renewal that completed but never reported still consumed rate budget at the CA, and the certificate may already be issued.

Step 3 - Reconcile

Pick the branch that matches what you found.

The work completed. The certificate on disk is the new one and the service is serving it. Nothing needs to run. Re-import or re-scan the certificate so the inventory reflects reality, and close out the job. The next scheduled observation will also pick up the new expiry.

The work never started. No new key, no new certificate, no changed deployment target. Create a fresh job for the same certificate (manual job or bulk renew) and let it run normally.

The work partially completed. This is the case that needs judgement. Common shapes:

• Key and CSR written, but no certificate issued: safe to re-run. The agent reuses or regenerates the key locally and orders again.
• Certificate issued but not deployed: do not re-order, or you will burn CA rate limit for nothing. Deploy the existing certificate with a deploy job.
• Deployed but not reloaded: the service is still serving the old certificate. Run a reload job, or reload the service yourself.

warning

Watch CA rate limits when reconciling several certificates at once. Let's Encrypt in particular counts duplicate orders, and a batch of "just retry everything" can exhaust the limit for a domain set. CERTOPS_RENEWAL_PER_CA_CAP bounds the scheduler, but it does not bound jobs you create by hand.

Step 4 - Prevent recurrence

A single orphaned job after a host reboot is normal. A pattern is a signal:

Pattern	Likely cause	Action
Same host repeatedly	Agent process being killed (OOM, deploy tooling, competing config management)	Check journalctl -u tokentimer-agent for restarts and the host's memory pressure
Many hosts at once	Network path to the control plane, or the control plane being unreachable	Agents are outbound-only; check egress, TLS interception, and proxy timeouts
Long-running renewals only	Jobs legitimately taking longer than the lease	Raise CERTOPS_JOB_LEASE_SECONDS rather than leaving jobs to be reaped
Clock complaints alongside	Host clock drift	Fix NTP on the agent host; signed jobs enforce a validity window

Related

• Automation and executors - how jobs are planned, routed, and dispatched.
• Approval gates - holding jobs for human sign-off before they run.
• Certificate renewal failures - the alert raised when a renewal reaches terminal failure.
• Connect an external CertOps executor - the machine-token path for your own tooling.