Rotating the job-signing key

Goal

Replace the control-plane Ed25519 job-signing key without interrupting your agent fleet, and know how to force the rotation through when an agent cannot acknowledge.

Background

Every job dispatched to an agent is signed with a control-plane Ed25519 private key. Agents verify that signature against a pinned public key and refuse anything that does not match, which is what stops a forged or replayed job from executing.

That pinning is exactly why the key cannot simply be swapped. If the control plane started signing with a new key while agents still trusted only the old one, every agent would reject every job. Rotation is therefore two-phase and overlapping:

Phase	Active key	Previous key	Agent behaviour
Before	ttsk_A signs	none	Agents pinned to ttsk_A
After begin	ttsk_B signs	ttsk_A is retiring, still verifies	Agents learn about ttsk_B on heartbeat, re-pin, and acknowledge
After complete	ttsk_B signs	ttsk_A is retired	Agents pinned to ttsk_B

The retiring key keeps working during the overlap, so jobs already in flight are never orphaned by the rotation itself.

Deployment-wide, not per-workspace

There is one job-signing key per deployment, not one per workspace, so rotation is an operator action and is intentionally not exposed on any workspace API route or dashboard page. It is a command you run on a host with database access and the CertOps environment loaded.

Prerequisites

• CERTOPS_SIGNING_ENCRYPTION_KEY must be set and valid (64 hex characters). Starting a rotation generates a new private key and has to wrap it, so it fails closed without this value.

• A protected source checkout of the matching Core release, with its dependencies installed and the API's database settings and encryption keys loaded. Published API images do not contain the repository-root rotation script. Run the following commands from that checkout; see the repository development guide for dependency setup.

• Database access from that operator host. Keep the matching CERTOPS_SIGNING_ENCRYPTION_KEY available throughout the rotation; do not replace it while wrapped signing records still depend on it.

• Ideally a healthy fleet: agents that are offline during the whole overlap cannot acknowledge, and will need to re-pin before they can run jobs again.

Step 1 - Check the current state

pnpm certops:rotate-signing-key status

You should see the active key id, whether a rotation is already in progress, and how much of the fleet has acknowledged the active key:

Active signing key:   ttsk_...
Retiring signing key: none (no rotation in progress)
Fleet acknowledgement: 4/4 active agents have pinned the active key

Add --json for machine-readable output if you are scripting this.

Step 2 - Begin the rotation

pnpm certops:rotate-signing-key begin

This generates a new key, makes it active immediately, and moves the previous key to retiring. New jobs are signed with the new key from this moment on, while the retiring key continues to verify.

Only one rotation may be in flight at a time; starting a second is refused rather than leaving two retiring keys behind.

Step 3 - Wait for the fleet to acknowledge

Agents pick up the new key through their normal heartbeat, re-pin it, and acknowledge. Watch progress with:

pnpm certops:rotate-signing-key status

You should see the acknowledgement count climb to match the number of active agents. How long this takes depends on your heartbeat interval, so allow at least a couple of intervals before treating a lagging agent as stuck.

If an agent never acknowledges, check that it is running and reaching the control plane (journalctl -u tokentimer-agent). An agent that is simply offline will acknowledge whenever it comes back, as long as you have not completed the rotation yet.

Step 4 - Complete the rotation

pnpm certops:rotate-signing-key complete

This retires the old key. It refuses to run while any active agent has not acknowledged, reporting the count instead, so you cannot accidentally lock a host out of running jobs.

Forcing an incomplete rotation

When an agent is gone for good, or you are responding to a suspected key compromise and cannot wait, force it:

pnpm certops:rotate-signing-key complete --force --reason "suspected key compromise, ticket SEC-1234"

--reason is required with --force and is recorded for the audit trail.

What forcing actually does

Any agent still pinned to the retired key will reject dispatched jobs as an integrity failure until it re-pins. Those hosts stop renewing certificates until you bring them back. Force when the security benefit outweighs that, and expect to fix the stragglers afterwards.

Responding to a suspected compromise

If you believe the signing key material leaked, forged jobs could pass signature verification and execute within the agent's allowed local scope. Local policy, validity windows, and replay checks still apply, but do not make a signing-key compromise harmless. Treat it as urgent.

1. Rotate immediately (begin, then complete --force --reason ... if you cannot wait for acknowledgement).
2. If the wrap key may have leaked, plan a separate encrypted-record migration. begin and complete rotate the signing key, not the wrap key, and do not re-encrypt stored records. Replacing CERTOPS_SIGNING_ENCRYPTION_KEY alone makes existing signing keys unreadable. Retain the old wrap key securely until every dependent record has been migrated and recovery verified.
3. Review the audit log for CERTOPS_SIGNING_KEY_ROTATION_STARTED and CERTOPS_SIGNING_KEY_ROTATION_COMPLETED, plus job dispatch activity in the exposure window.
4. Use the kill switch to pause CertOps for affected workspaces while you investigate; it stops new jobs and dispatch without destroying history.

Audit trail

Action	When
CERTOPS_SIGNING_KEY_ROTATION_STARTED	A new key became active and the previous one entered retiring
CERTOPS_SIGNING_KEY_ROTATION_COMPLETED	The retiring key was retired, including whether it was forced and why

Troubleshooting

Symptom	Likely cause	Fix
CERTOPS_SIGNING_ENCRYPTION_KEY_MISSING	The wrap key is unset or not 64 hex characters	Set it in the same environment the API uses, then retry
"A signing-key rotation is already in progress"	A previous begin was never completed	Run status, then complete (or complete --force)
"No rotation is in progress" on complete	Nothing is retiring	Run status; if you meant to start one, run begin
Acknowledgement count stuck below the fleet size	One or more agents offline or unable to reach the control plane	Fix the agent, or force with a reason if it is gone
Agents fail jobs with an integrity error after forcing	They are still pinned to the retired key	Restart the agent so it re-pins, or re-register it

Related

• Configuration reference - the CertOps environment variables, including both encryption keys.
• Enable CertOps - the fail-closed boot requirements and rollout order.
• Reconciling interrupted jobs - when a job stops reporting midway.