Rotating the job-signing key

Goal

Replace the control-plane Ed25519 job-signing key without interrupting your agent fleet, and know how to force the rotation through when an agent cannot acknowledge.

Background

Every job dispatched to an agent is signed with a control-plane Ed25519 private key. Agents verify that signature against a pinned public key and refuse anything that does not match, which is what stops a forged or replayed job from executing.

That pinning is exactly why the key cannot simply be swapped. If the control plane started signing with a new key while agents still trusted only the old one, every agent would reject every job. Rotation is therefore two-phase and overlapping:

Phase	Active key	Previous key	Agent behaviour
Before	ttsk_A signs	none	Agents pinned to ttsk_A
After begin	ttsk_B signs	ttsk_A is retiring, still verifies	Agents learn about ttsk_B on heartbeat, re-pin, and acknowledge
After complete	ttsk_B signs	ttsk_A is retired	Agents pinned to ttsk_B

The retiring key keeps working during the overlap, so jobs already in flight are never orphaned by the rotation itself.

Deployment-wide, not per-workspace

There is one job-signing key per deployment, not one per workspace, so rotation is an operator action and is intentionally not exposed on any workspace API route or dashboard page. It is a command you run on a host with database access and the CertOps environment loaded.

Prerequisites

• CERTOPS_SIGNING_ENCRYPTION_KEY must be set and valid (64 hex characters). Starting a rotation generates a new private key and has to wrap it, so it fails closed without this value.
• Database access from wherever you run the command, with the same environment the API uses.
• Ideally a healthy fleet: agents that are offline during the whole overlap cannot acknowledge, and will need to re-pin before they can run jobs again.

Step 1 - Check the current state

pnpm certops:rotate-signing-key status

You should see the active key id, whether a rotation is already in progress, and how much of the fleet has acknowledged the active key:

Active signing key:   ttsk_...
Retiring signing key: none (no rotation in progress)
Fleet acknowledgement: 4/4 active agents have pinned the active key

Add --json for machine-readable output if you are scripting this.

Step 2 - Begin the rotation

pnpm certops:rotate-signing-key begin

This generates a new key, makes it active immediately, and moves the previous key to retiring. New jobs are signed with the new key from this moment on, while the retiring key continues to verify.

Only one rotation may be in flight at a time; starting a second is refused rather than leaving two retiring keys behind.

Step 3 - Wait for the fleet to acknowledge

Agents pick up the new key through their normal heartbeat, re-pin it, and acknowledge. Watch progress with:

pnpm certops:rotate-signing-key status

You should see the acknowledgement count climb to match the number of active agents. How long this takes depends on your heartbeat interval, so allow at least a couple of intervals before treating a lagging agent as stuck.

If an agent never acknowledges, check that it is running and reaching the control plane (journalctl -u tokentimer-agent). An agent that is simply offline will acknowledge whenever it comes back, as long as you have not completed the rotation yet.

Step 4 - Complete the rotation

pnpm certops:rotate-signing-key complete

This retires the old key. It refuses to run while any active agent has not acknowledged, reporting the count instead, so you cannot accidentally lock a host out of running jobs.

Forcing an incomplete rotation

When an agent is gone for good, or you are responding to a suspected key compromise and cannot wait, force it:

pnpm certops:rotate-signing-key complete --force --reason "suspected key compromise, ticket SEC-1234"

--reason is required with --force and is recorded for the audit trail.

What forcing actually does

Any agent still pinned to the retired key will reject dispatched jobs as an integrity failure until it re-pins. Those hosts stop renewing certificates until you bring them back. Force when the security benefit outweighs that, and expect to fix the stragglers afterwards.

Responding to a suspected compromise

If you believe the signing key material leaked, the key alone does not let an attacker execute anything: an agent also enforces its own default-deny local policy, an issuedAt/expiresAt validity window, and a replay cache. It does mean forged jobs could pass signature verification, so treat it as urgent.

1. Rotate immediately (begin, then complete --force --reason ... if you cannot wait for acknowledgement).
2. Consider rotating CERTOPS_SIGNING_ENCRYPTION_KEY too if the wrap key itself may have leaked. Rotate the signing key first, then the wrap key, so you are never left with an unreadable active key.
3. Review the audit log for CERTOPS_SIGNING_KEY_ROTATION_STARTED and CERTOPS_SIGNING_KEY_ROTATION_COMPLETED, plus job dispatch activity in the exposure window.
4. Use the kill switch to pause CertOps for affected workspaces while you investigate; it stops new jobs and dispatch without destroying history.

Audit trail

Action	When
CERTOPS_SIGNING_KEY_ROTATION_STARTED	A new key became active and the previous one entered retiring
CERTOPS_SIGNING_KEY_ROTATION_COMPLETED	The retiring key was retired, including whether it was forced and why

Troubleshooting

Symptom	Likely cause	Fix
CERTOPS_SIGNING_ENCRYPTION_KEY_MISSING	The wrap key is unset or not 64 hex characters	Set it in the same environment the API uses, then retry
"A signing-key rotation is already in progress"	A previous begin was never completed	Run status, then complete (or complete --force)
"No rotation is in progress" on complete	Nothing is retiring	Run status; if you meant to start one, run begin
Acknowledgement count stuck below the fleet size	One or more agents offline or unable to reach the control plane	Fix the agent, or force with a reason if it is gone
Agents fail jobs with an integrity error after forcing	They are still pinned to the retired key	Restart the agent so it re-pins, or re-register it

Related

• Configuration reference - the CertOps environment variables, including both encryption keys.
• Enable CertOps - the fail-closed boot requirements and rollout order.
• Reconciling interrupted jobs - when a job stops reporting midway.