Reconciling interrupted jobs

Goal

Understand and clear jobs that stopped reporting midway, in particular the orphaned_unknown_effect status, without risking a duplicate deployment or a second certificate order.

Why this state exists

Your agent holds a lease on a job while it works, renewing the lease as it goes. If the agent process dies, loses connectivity, or is killed mid-run, the lease eventually expires and TokenTimer has to decide what to do with the job. That decision cannot be "just retry", because a retry is only safe when nothing has happened yet.

TokenTimer distinguishes the two cases from evidence, not from guesswork:

Situation	What is proven	What happens
Lease never renewed, job still claimed	The agent never got far enough to do anything observable	Automatic requeue with backoff, up to the job's attempt limit
Lease was renewed, or job reached running	The agent may have ordered a certificate, written a key, deployed a file, or reloaded a service	Never silently retried, flagged for you

In the second case the job is set to orphaned_unknown_effect and marked as needing reconciliation, with an error code of effects_unknown. When the interrupted job was a renewal, this also raises a cert_renewal_failed alert, so an orphaned renewal reaches you through your normal alerting channels rather than sitting unnoticed on the jobs list.

note

This is not an ordinary failure. A failed job means "we know it did not work". orphaned_unknown_effect means "we do not know how far it got", which is a different and more careful statement. Treat it as an investigation, not as a retry button.

Because your agent runs on your own hosts and TokenTimer never connects to it, TokenTimer cannot inspect the host to find out what happened. That is why this state asks a human to look. Before flagging anything, TokenTimer gives a live agent room to finish: while your agent is still heartbeating and the grace window has not elapsed, the job is left alone so a late result can still land. A job that was actively running is therefore not flagged the instant its lease lapses.

Step 1 - Find the affected jobs

Open Certificate operations at /certops/operations and look for jobs whose status is orphaned_unknown_effect. The job log carries an entry from the lease-reaper sweep with the outcome, the attempt count, and the previous status, which tells you how far the job had progressed.

Step 2 - Determine what actually happened

Check the host, not the dashboard. The dashboard only knows what was reported, and the whole point of this state is that reporting stopped. On the agent host:

1. Read the agent log around the job's timestamps (journalctl -u tokentimer-agent).
2. Check the agent's job journal: the agent records what it was doing so an interrupted run can be understood after the fact.
3. Inspect the certificate on disk. Compare the current certificate's serial, fingerprint, and notAfter against what the job intended:

openssl x509 -in /path/to/cert.pem -noout -serial -fingerprint -sha256 -dates

4. If the job included an ACME order, check the CA side too. A renewal that completed but never reported still consumed rate budget at the CA, and the certificate may already be issued.

Step 3 - Reconcile

Pick the branch that matches what you found.

The work completed. The certificate on disk is the new one and the service is serving it. Nothing needs to run. Re-import or re-scan the certificate so your inventory reflects reality, and close out the job. The next scheduled observation will also pick up the new expiry.

The work never started. No new key, no new certificate, no changed deployment target. Create a fresh job for the same certificate (manual job or bulk renew) and let it run normally.

The work partially completed. This is the case that needs judgement. Common shapes:

• Key and CSR written, but no certificate issued: safe to re-run. The agent reuses or regenerates the key locally and orders again.
• Certificate issued but not deployed: do not re-order, or you will burn CA rate limit for nothing. Deploy the existing certificate with a deploy job.
• Deployed but not reloaded: the service is still serving the old certificate. Run a reload job, or reload the service yourself.

warning

Watch CA rate limits when reconciling several certificates at once. Let's Encrypt in particular counts duplicate orders, and a batch of "just retry everything" can exhaust the limit for a domain set.

Step 4 - Prevent recurrence

A single orphaned job after a host reboot is normal. A pattern is a signal:

Pattern	Likely cause	Action
Same host repeatedly	Agent process being killed (OOM, deploy tooling, competing config management)	Check journalctl -u tokentimer-agent for restarts and the host's memory pressure
Many hosts at once	Network path from your hosts to TokenTimer	Agents are outbound-only; check egress, TLS interception, and proxy timeouts
Long-running renewals only	Jobs legitimately taking longer than the lease window	Contact support so the lease window can be reviewed for your workspace
Clock complaints alongside	Host clock drift	Fix NTP on the agent host; signed jobs enforce a validity window

Related

• Automation and executors - how jobs are planned, routed, and dispatched.
• Approval gates - holding jobs for human sign-off before they run.
• Certificate renewal failures - the alert raised when a renewal reaches terminal failure.
• Connect an external CertOps executor - the machine-token path for your own tooling.