Certificate Automation

Overview

CertOps automation renews certificates end to end on your own instance. The TokenTimer control plane schedules renewal work before certificates expire, routes each job to an eligible executor in your environment, and records issuance, deployment, reload, verification, and evidence on an audited timeline. Approval gates and a workspace kill switch keep humans in control of what runs.

The control plane plans, signs, and audits; execution happens where your certificates live. TokenTimer never connects into your hosts and never touches private keys: keys are generated and used on the agent host (or inside your cluster with cert-manager), and any payload containing key material is rejected outright.

Three execution modes are available and can be mixed freely:

• TokenTimer agent: an outbound-only daemon on your hosts that verifies signed jobs, runs ACME renewals with DNS-01, deploys atomically, reloads services, and verifies the served certificate. See The TokenTimer agent.
• cert-manager controller: a Kubernetes controller that observes cert-manager certificates and, in opt-in provisioning mode, creates authorized Certificate resources. See cert-manager mode.
• External executors: your own renewal scripts, certbot/acme.sh hooks, and CI pipelines authenticate with a machine API token and report against executor jobs. See Connecting external executors.

Certificate operations is one entry in the navigation sidebar for managers and admins, labelled CertOps. It opens onto a row of tabs rather than a single page: Jobs (/certops/jobs, the default), Certificates (/certops/certificates), Renewals (/certops/renewals), Agents (/certops/agents), and Settings (/certops/settings, machine API tokens). The old /certops/operations URL still resolves; it now redirects to the Jobs tab, so bookmarked links and published runbooks keep working. See Certificates for the managed inventory these tabs attach to.

Enabling automation

Agents, machine tokens, scheduling, and the executor API ship with CertOps. Enable CertOps on your install, then restart the API and worker processes:

CERTOPS_ENABLED=true

When CertOps is disabled, token management, agent routes, and executor endpoints return 404 and none of the Certificate operations tabs are reachable. There is no plan gating on self-hosted installs; all workspaces on your instance can use automation once CertOps is enabled.

For the full list of environment variables, see the Configuration Reference. See also Enable CertOps for the rollout runbook.

Scheduled renewals and approvals

Renewal work is planned by the control plane, not by cron jobs you have to remember:

• Renewal scheduler: a maintenance worker sweep creates renewal jobs when a managed certificate approaches expiry. A job is only scheduled when the certificate has a complete, validated renewal profile (see below), so automation never creates unexecutable work.
• Renewal profiles: a profile binds the operational inputs a renewal needs: the CA endpoint, the command profile (certbot or acme.sh), and the target. The profile snapshot is resolved at schedule time and bound into the signed job payload before approval, so what was approved is exactly what runs. Certificates that TokenTimer issued get their profile derived automatically from the issuance that created them; see Where the renewal profile comes from.
• Eligible-agent routing: jobs are routed to agents whose declared capabilities and target selectors match. Agents claim work over an outbound-only protocol with lease renewal. A claim is exclusive: the claim query row-locks candidates and skips rows another agent already holds, so two agents can never receive the same job. Lease expiry is deliberately conservative: a job whose lease never renewed has proven no side effects and is safely requeued, but a job that was already running is never silently retried. It is held while the agent may still report in, then flagged for manual reconciliation. See Reconciling interrupted jobs.
• Certificate ownership pinning: for a certificate an agent discovered on its own filesystem, renew/deploy/reload/revoke jobs are automatically pinned to that agent (assignedAgentId), so no other agent in the fleet can claim work for a certificate it has no custody of. See How a job is bound to one agent.
• Signed dispatch: every dispatched job is signed with a control-plane Ed25519 key. Agents verify the signature against a pinned key, enforce an issuedAt/expiresAt validity window, and keep a persisted replay cache, so a forged or replayed job never executes.
• Approval gates: jobs can require explicit approval before dispatch. A requester cannot approve their own job, the approval is bound to the exact job payload hash, and every decision is written to the audit log inside the approval transaction. See Approval gates for the full non-requester and invalidation rules.
• Per-CA caps and bulk renew: CERTOPS_RENEWAL_PER_CA_CAP bounds in-flight renewals per CA endpoint per workspace. It is not just a scheduler throttle: manual jobs and bulk renew reserve capacity against the same counter and fail with 409 CERTOPS_RENEWAL_PER_CA_CAP_EXCEEDED when it is full, so no creation path can flood a CA. POST .../certops/jobs/bulk-renew pushes many certificates through the same validated creation path with per-item results.
• Kill switch: a workspace-level pause blocks new CertOps jobs and dispatch instantly, without deleting queued work or history. Inventory, status, audit, and evidence reporting keep working while paused. The deployment-wide CERTOPS_ENABLED flag remains a separate gate. Toggle it from Certificate operations: the panel at the top of every tab shows the current Active/Paused badge, with a Pause/Resume button visible to workspace admins (certops.kill_switch.manage).
• Renewal failure alerting: failed renewals raise a cert_renewal_failed alert through your normal alerting channels. See Certificate renewal failures for when it fires and how it routes.

When a renewal actually gets scheduled

Five things have to be true. If a certificate is not renewing on its own, work down this list in order; it is ordered by how often each one is the answer.

1. The certificate has a linked renewal profile with a complete snapshot. No profile means no job, ever. See Where the renewal profile comes from.
2. Its key custody is agent-deployable: key_mode of agent-local or proxy-agent-local. A certificate that was only observed (via an HTTPS endpoint monitor, a domain monitor, or the Kubernetes controller in observe mode) has no key TokenTimer's agents can rotate, so it is monitored and alerted on but never auto-renewed. See Not every certificate in inventory can be renewed for you.
3. It is inside the renewal window: not_after falls within the threshold (see below).
4. Its inventory status is renewable: revoked and decommissioned certificates are excluded outright. Anything else, including provisioning, is in scope.
5. Automatic renewal is switched on for its profile. A profile whose status is disabled or archived is skipped deliberately. See Switching automatic renewal off.

On top of that, a certificate is skipped for this sweep (and retried on the next one) when the workspace is paused, when CertOps is disabled deployment-wide, or when the per-CA cap is already full. A certificate is also skipped when a non-terminal renew job for it already exists, which is what stops a sweep every few minutes from stacking up duplicate renewals.

You do not have to evaluate that list by hand. GET /api/v1/workspaces/{id}/certops/certificates returns a renewal object per certificate, and the dashboard renders it as a badge:

{
  "renewal": {
    "schemaVersion": 1,
    "state": "auto",
    "keyMode": "agent-local",
    "profileId": "…",
    "profileName": "Derived: api.example.com",
    "renewBeforeDays": 30,
    "renewsFrom": "2026-09-01T00:00:00.000Z",
    "detail": "Renewal is attempted automatically from 30 days before expiry."
  }
}

state is one of auto, disabled, not-configured, not-eligible, or not-applicable. Completeness is answered by the same function the sweep admits on, so auto cannot be reported for a certificate the sweep would refuse. renewsFrom is when the sweep starts picking the certificate up (not_after minus the effective lead time), not a promise of the exact renewal moment. An unknown or missing renewal object is rendered as a caution rather than an all-clear, so a stale client cannot show a false reassurance. See The renewal badge.

not-configured is the state worth acting on: the certificate is agent-deployable, so you probably believe it is managed, and it will expire. disabled means somebody switched automatic renewal off deliberately, which is a supported choice but still means the certificate expires unless you renew it yourself. not-eligible means observed-only custody and is working as designed.

Seeing and changing what will renew

Automatic renewal is on by default for everything TokenTimer issued, because the profile is derived for you. The Renewals tab of Certificate operations (/certops/renewals) is where you see and change that:

• Upcoming renewals lists every renewable certificate in the workspace, soonest expiry first, with the date the scheduler starts attempting a renewal and the outcome of the last attempt. It deliberately includes certificates the scheduler will not act on, each labelled with why, because those are the ones that will silently expire:

  Label	Meaning	What to do
  On	The scheduler will renew this certificate.	Nothing.
  Off	Automatic renewal was switched off for its profile.	A deliberate choice. Switch it back on below, or renew by hand.
  No profile	The certificate has no renewal profile, so the scheduler will never see it. Profiles are created automatically at issuance, so a missing one means that step did not complete.	Re-issue the certificate. Toggling nothing will help.
  Incomplete	A profile exists but the scheduler cannot execute it, so it will never produce a renewal job.	Re-issue the certificate to rebuild a working profile.
  No key access	No agent holds this certificate's private key, so no agent can renew it. Typical of a certificate that was only observed from the outside, for example by an endpoint or domain monitor.	Issue the certificate through CertOps so an agent owns its key. A profile alone cannot fix this.
  No expiry	No expiry date is recorded, so the scheduler cannot tell when the certificate is due.	Re-run discovery or re-issue.

  An empty table therefore means the workspace has no renewable certificates at all, not merely none that are covered. Restricting this list to certificates that do renew would make the page report an all-clear precisely when nothing renews, which is the one answer it must never give by accident.

  "On" describes the profile's switch, not whether the workspace can currently act

  On/Off here (and the Auto-renews/Auto-renewal off renewal badge) read the certificate's own renewal profile only. They do not reflect the workspace kill switch or the deployment-wide CERTOPS_ENABLED flag. A certificate can show On while its workspace is paused: the profile itself is genuinely switched on and would fire the moment the workspace resumes, but nothing renews in the meantime. Check the Active/Paused badge at the top of every Certificate operations tab separately; this page and the kill switch are independent signals by design; see Kill switch vs the platform flag.

• Renewal profiles lists the profiles in the workspace with the certificate count each one covers, and offers exactly two controls: an automatic-renewal switch and the renewal lead time.

Reading either panel needs manager or above. Both controls need workspace admin (certops.renewal_profile.manage).

Reads sit at manager rather than viewer because a profile body carries deployment topology: certificate and key paths, the reload unit, file ownership and modes, the ACME command reference, the CA account reference, and the DNS zone. That makes it closer to the agent and machine-token surfaces than to the certificate inventory, which exposes only expiry metadata. A viewer keeps full visibility of what expires and when, through Certificates and the renewal badge, without learning where a given key lives on a host.

Why you cannot edit a profile's deployment details

A derived profile records an issuance that provably worked on a real host: that CA endpoint, that ACME command profile, that DNS provider and zone, that certificate path. Those fields are read-only through the API and the dashboard, and an attempt to change them is refused with 422 CERTOPS_PROFILE_FIELD_IMMUTABLE naming the offending fields.

That is not a missing feature. Repointing where a live certificate gets written, or which command writes it, is a re-issuance against a real host, not a settings change, and a renewal is the wrong moment to discover that the new value was wrong. Re-issue with the payload you want instead; derivation records the new reality.

The editable subset is the part that cannot change what executes on a host: the SAN policy, key algorithm and size, key rotation policy, verification settings, preferred chain, the renewal lead time, and the on/off switch. Every write is validated by validateRenewalProfile, the same gate the scheduler admits on, so the API cannot store a profile the scheduler would later refuse.

Re-issuing the same hostname used to reset these edits; fixed

A derived profile is identified by its name, Derived: <common name> (<certificate id>). The certificate id suffix means two certificates sharing a common name get separate profile rows: issuing a second certificate for a common name you already have no longer touches the first certificate's profile or its edits. (Earlier releases keyed the profile on name alone, so a re-issuance rewrote a shared row and could silently drop edits or leave a second certificate with no profile at all; that is fixed. A profile's certificate count is now always 1 for a derived profile.)

Switching automatic renewal off

Switching a profile off sets its status to disabled, and the scheduler skips every certificate linked to it. Expiry monitoring and alerting keep running, so you still hear about the certificate; only the renewal stops.

This is the one control here that can cause an outage, so it is the one that asks for confirmation, states how many certificates the profile covers, and shows an orange badge and a standing warning on the schedule afterwards. For a derived profile that count is now always 1; a profile you authored yourself can still cover several certificates, and switching it off affects all of them. Switching back on takes effect on the next sweep and needs no confirmation, because the safe direction should not carry friction.

archived is a third profile status that also stops renewal, but it is not settable through the API. An archived profile is shown without controls.

The renewal window

The threshold is resolved per certificate, with a per-profile override taking precedence over the deployment default:

Level	Where	Default
Deployment default	CERTOPS_RENEWAL_THRESHOLD_DAYS	30 days
Per-profile override	renew_before_days on the certificate's profile	unset, so the deployment default applies

Set the override on a profile when one certificate needs a different lead time than the rest of your fleet, for example a certificate behind a slow change-approval process. An invalid or non-positive CERTOPS_RENEWAL_THRESHOLD_DAYS falls back to 30 rather than disabling renewals, so a typo in your environment cannot silently switch automation off.

Do not set the lead time close to or above the certificate's actual lifetime

The scheduler re-evaluates every certificate on every sweep. If renew_before_days is wider than the CA's typical certificate lifetime (about 90 days for Let's Encrypt), a certificate is inside its renewal window from the moment it is issued and stays there after every renewal, because each renewal only extends not_after by another lifetime. The scheduler has no per-certificate cooldown, only a per-CA cap on how many renewals a single sweep will create workspace-wide (perCaCap), so a misconfigured profile like this renews again on every sweep interval indefinitely rather than settling once the certificate is genuinely due. Against a rate-limited CA this risks exhausting your issuance quota. Keep the lead time well under the CA's actual lifetime; there is currently no server-side warning or clamp if you do not.

Where the renewal profile comes from

This is the single most common reason a fleet looks healthy and renews nothing, so it is worth being explicit.

The scheduler refuses to create a renew job unless the certificate links to a profile carrying a complete, validated renewal snapshot: CA endpoint, ACME command profile, DNS provider and zone, deployment paths, SAN policy, and key parameters. That gate is intentional. Dispatching a half-specified renewal would hand an agent a job it cannot run, against a real rate-limited CA.

For certificates TokenTimer issued, that profile is now derived for you at the moment issuance succeeds. When an issue job is reconciled and the certificate is promoted to active, the control plane builds a renewal profile from the payload that just succeeded and links the certificate to it. A successful issuance is the one moment where every field is both known and proven to work: the agent completed a real ACME order with exactly that CA endpoint, DNS provider, zone, command profile, and deployment path.

What that means for you:

• You do not author a profile for anything TokenTimer issued. It appears on the Renewal automation page named Derived: <common name>, where you can change its renewal lead time or switch renewal off. Its deployment details are deliberately read-only; see Why you cannot edit a profile's deployment details.
• The SAN set is pinned to what the CA actually issued, not to what the job requested. If the CA normalised or dropped a name, the renewal reproduces the certificate that is deployed rather than the one that was asked for.
• A profile you wrote yourself is never overwritten. If the certificate already links to a profile, derivation leaves it alone.
• Derivation can decline, and the issuance still succeeds. If the issue payload lacked a field the profile needs (CA endpoint, command profile, DNS provider and zone, or cert path), the certificate is still promoted to active and the derivation is skipped with a logged reason. You get a working certificate that will not auto-renew, which is strictly better than losing the issuance. Watch the skip metric below, or look for the certificate on the Renewal automation page: a certificate with no profile does not appear there at all, and its renewal badge reads not-configured. Re-issue with a complete payload to get a profile derived.
• Certificates that arrived some other way still need a profile. An imported PEM or an endpoint-monitor observation has no issuance payload to derive from and no recorded deployment path, so Set up renewal cannot arm a renewal for either: re-issue such a certificate through CertOps instead. A certificate discovered on an agent filesystem is different: its deployment path is recorded from the scan that found it, so Set up renewal (see Bringing an existing certificate under automatic renewal) can adopt it without re-issuing. Observed-only certificates cannot be renewed at all either way.

Bringing an existing certificate under automatic renewal

Not available for an imported or endpoint-monitor certificate

Set up renewal requires managed_certificates.deployed_cert_path to be set (CERTOPS_RENEWAL_SETUP_NO_DEPLOYED_PATH if it is not). Agent filesystem discovery writes this column from the path it found the certificate at, so the route is reachable for that source. Nothing writes it for an imported PEM or an endpoint-monitor observation, so Set up renewal always refuses for those two - the row's renewal.state reads not-configured and the button appears (its own precondition is only "no linked profile yet"), but confirming it, dry run or real, always ends in the refusal below. The only working way today to get a profile onto an imported or observed-only certificate is to re-issue it through CertOps. Re-issuing replaces the certificate's identity (a new managed_certificates row), which is a bigger step than adopting the existing one in place, so treat this as a known limitation rather than a workaround you are missing.

Re-issuance is not the only intended path to a derived profile. On the Certificates tab (/certops/certificates), a certificate that is active, agent-deployable, and has no linked profile yet gets a Set up renewal button in its row.

Clicking it opens a form for the four inputs a renewal needs that TokenTimer cannot infer on its own: the ACME command profile, the CA endpoint, the DNS provider, and the DNS zone. Confirming does not save those as settings; it runs a real renew job against the certificate's recorded deployment path, pinned to the agent that recorded it, and derives a profile only if that job succeeds - exactly the same way as a successful issue job. A dry run (if you trigger one through the API instead of the button) proves the same path with no filesystem side effects, but deliberately arms no profile-derivation intent, so a dry run can never adopt a certificate into automatic renewal on its own. This is the working path for a certificate discovered on an agent's filesystem; per the caveat above, an imported or endpoint-monitor certificate never has a recorded deployment path, so for those two this flow only ever reaches the CA/agent for a certificate that already has one - which, being agent-issued or agent-discovered, would refuse for the opposite reason (CERTOPS_RENEWAL_SETUP_ALREADY_CONFIGURED) if a profile is already linked.

Set up renewal is refused, not silently unavailable, when:

Condition not met	Error code	What to do
Certificate already has a profile	CERTOPS_RENEWAL_SETUP_ALREADY_CONFIGURED	Use Detach first if you want to replace it, then set up renewal again
Not agent-deployable (key_mode unset or externally managed)	CERTOPS_CERTIFICATE_NOT_AGENT_DEPLOYABLE	No agent holds the key; this certificate cannot auto-renew here at all
No discovered deployment path recorded	CERTOPS_RENEWAL_SETUP_NO_DEPLOYED_PATH	Permanent today for an imported or endpoint-monitor certificate; see the caveat above. Not an issue for a filesystem-discovered certificate, which always has one. Re-issue through CertOps instead for the other two sources
Deployed to more than one location	CERTOPS_RENEWAL_SETUP_MULTI_LOCATION	Not supported yet; the schema records one deployment path per certificate

While the triggering renewal job is in flight, the certificate's row shows a Setting up automatic renewal badge rather than the Set up renewal button again. If the job fails, the badge reads Setup failed with a Retry action that re-arms the same derivation intent without re-running the whole form. A profile that was deliberately skipped (for example, the issue payload was incomplete) shows Setup skipped instead, and is not retryable from here, matching Derivation can decline, and the issuance still succeeds above.

Detach (on the same row, once a profile is linked) removes only this certificate's link to its profile; the profile row itself, and any other certificate sharing it, is untouched. Use it before re-running Set up renewal, or before manually pointing the certificate at a different profile through re-issuance.

Renew jobs are not pre-pinned to an agent

A scheduled renew job does not require you to nominate an agent. It is created open-claim and matched at claim time by the agent's declared target selectors, DNS providers, and command profiles. Any agent in the fleet whose declarations match can pick it up.

The one exception is custody: if the certificate was discovered on a specific agent's own filesystem, the job is pinned to that agent, because renewing a path only means something on the host where that path exists. See How a job is bound to one agent.

If a renewal sits at pending and never gets claimed, the cause is almost always a matching problem rather than a scheduling one: no online agent declares the required target selector, DNS provider, or command profile, or the job is pinned to an agent that is offline.

Seeing what the scheduler decided

A sweep that creates zero jobs is ambiguous on its own: it could mean nothing was due, or it could mean every certificate in your fleet was refused. The worker exports the difference as certops_renewal_scheduler_certificates, a gauge labelled by outcome, reflecting the last sweep:

outcome	Meaning	What to do
scanned	Certificates that came back as due in this sweep	Baseline for the others
created	New renew jobs created	Nothing
replayed	An existing job matched the idempotency key, so no new job was created	Nothing; this is a rerun inside the same expiry window
skipped_paused	Workspace paused, or CertOps disabled deployment-wide	Resume the workspace when you are ready
skipped_ca_cap	Per-CA in-flight cap was full	Usually self-clearing; raise CERTOPS_RENEWAL_PER_CA_CAP if renewals are chronically starved
skipped_incomplete_profile	No linked profile, or its snapshot failed validation	Act on this. These certificates will expire unrenewed. See Where the renewal profile comes from
skipped_not_agent_deployable	Observed-only custody, so no agent can rotate the key	Expected for monitored certificates; renew them out of band
skipped_auto_renew_disabled	Somebody switched automatic renewal off on the profile (disabled or archived)	Expected if intentional. These certificates will expire unrenewed, so confirm it was deliberate. See Switching automatic renewal off
errors	Unexpected per-certificate failures	Check the worker logs

A non-zero skipped_incomplete_profile with a zero created is the exact signature of the failure this release fixed: certificates approaching expiry that automation is quietly refusing to touch. Alert on it.

Alert on skipped_auto_renew_disabled differently: it is a deliberate choice rather than a fault, so a non-zero value is only worth investigating if nobody remembers making it. The Renewal automation page names which profiles they are, and the audit log records who switched each one off.

certops_renewal_jobs_created remains available as the plain count of jobs created by the last sweep.

Bulk renew

POST /api/v1/workspaces/{id}/certops/jobs/bulk-renew queues a renew job for up to 100 managed certificates in one request. Each certificate goes through exactly the same creation path as a single manual job (manager role, kill switch, approval gate, payload validation, per-CA cap), so per-certificate behavior is identical to creating them one at a time.

It is a partial-failure envelope, not all-or-nothing. The response is 200 with a per-item outcome for every certificate, even when some or all items failed. Only whole-request shape problems (malformed body, more than 100 ids, non-UUID or duplicate ids) return 400. Always read the per-item results; a 200 does not mean every certificate was queued.

Dry run first. Pass dryRun: true to preflight each certificate without writing anything: existence, renewable inventory status, payload validation, and whether a non-terminal renew job is already in flight (reported as activeJobId, which is how you catch accidental double-renewals before committing).

Idempotency keys, and the re-run trap

Every item is created with a derived per-certificate idempotency key, so a retried batch returns the already-created jobs (marked replayed: true) instead of enqueueing duplicates:

• With a request-level idempotencyKey: the per-item key is bulk-renew:<your key>:<certificateId>.
• Without one: the server derives bulk-renew:auto:<certificateId>.

That fallback is the trap. Because the derived key depends only on the certificate id, it is the same on every later bulk renew of that certificate, forever. Idempotency keys are not released when a job reaches a terminal status. So if you bulk renew a set of certificates, cancel the jobs, then bulk renew the same set again with different options (adding requiresApproval: true, for example), the second call does not create new jobs: it matches the old keys and either replays the originals or reports a conflict. It reads as "bulk renew silently did nothing".

Pass an explicit, fresh idempotencyKey whenever you intend a genuinely new attempt rather than a retry of the same one. Reserve the omitted form for what it was designed for: safely retrying a request you are not sure was received. A scheme like reconcile-2026-07-24-a works well because it makes the intent auditable too.

The TokenTimer agent

The agent is a zero-dependency Node.js daemon (Node 22 or later) that runs on the hosts where certificates are deployed. It is outbound-only: it opens HTTPS connections to your TokenTimer instance and never accepts inbound connections. No listening port, no firewall holes.

Installation

1. On the Agents tab of Certificate operations, click Deploy an agent to open the install wizard and create a bootstrap token. It is shown exactly once and consumed at first start.
2. Copy the generated install command and run it on the target host. The installer (install-agent.sh) verifies the runtime, creates a dedicated system user and install directory, stores the bootstrap token in a 0600-mode file, and installs a hardened systemd unit (ProtectSystem=strict). --dry-run prints every action without touching the system.
3. Watch the Agent fleet panel flip the agent to registered on its first heartbeat. The fleet panel shows status, heartbeat, clock drift, and compatibility, and lets you retire an agent (a retired agent exits cleanly on its next heartbeat).

If your instance uses a private CA for its own HTTPS endpoint, point the agent at a PEM bundle with the caBundlePath config field (or TOKENTIMER_AGENT_CA_BUNDLE).

For install options, write-path and reload permissions, upgrades, and troubleshooting, see Install and upgrade an agent.

Reading agent health in the fleet panel

The fleet panel shows two different things that both look like "is this agent up", and knowing which is which saves a lot of confusion:

• Status is the stored value in the database (active, offline, retired). It is only changed by the stale-agent sweep in the CertOps maintenance worker, which runs periodically.
• Liveness is computed fresh on every read, from the agent's last heartbeat against CERTOPS_AGENT_OFFLINE_AFTER_MS (default 10 minutes). It is live, stale, or retired.

Liveness exists precisely so the panel does not lie to you between sweeps: an agent that crashed thirty seconds ago still has status: active in the database, but reads as stale immediately. Trust liveness for "is it responding right now"; status is the durable record.

Two practical consequences:

• If every agent shows stale but active, and the state never resolves, the CertOps maintenance worker is probably not running. That is a deployment problem, not an agent problem. See the warning in Worker scheduling.
• Neither state retires an agent. Going offline is observational only: the agent keeps its credential and its assigned jobs, and picks up where it left off when it comes back. Only an explicit retire invalidates the credential.

Clock drift and NTP are shown alongside. Drift matters because signed job dispatch enforces an issuedAt/expiresAt window, so a badly skewed host can reject valid work. A dash rather than a value in the clock drift column is normal for observe-only agents, which do not measure it.

Agent-local policy and permissions

The agent enforces its own default-deny policy, configured in its local config.json: allowlists for command profiles, filesystem paths, CA endpoints, and DNS zones/providers. Local policy always wins over control-plane intent; a job that exceeds it is rejected and the rejection is reported as evidence. Key-export requests are rejected unconditionally, with no config knob to permit them.

A dry-run mode (execution.dryRun, the default) exercises the full trust chain and reports the steps a job would run without any filesystem or exec side effects, so you can validate a rollout safely before enabling real execution.

What a renewal job does

Entirely on your host, the agent:

1. generates (or reuses) the private key and builds the CSR locally; keys are written 0600 and never leave the host,
2. drives certbot or acme.sh in CSR mode against the allowlisted CA endpoint,
3. solves DNS-01 challenges with native solvers (see below),
4. deploys atomically with a timestamped backup and automatic rollback on failure - the full chain (leaf plus intermediates) by default, or leaf and chain as separate files when the job sets chainPath (see First-time issuance),
5. validate-then-reloads nginx, Apache, or HAProxy (a failing validation means the reload never runs),
6. verifies the deployment by fingerprinting the installed certificate, with an optional live TLS probe against the serving endpoint,
7. reports each step as evidence on the job timeline.

Deployment targets are allowlisted absolute paths on the agent host; the deploy module re-checks the realpath-resolved destination immediately before writing so a symlink cannot escape the allowlisted roots.

DNS-01 providers

Native DNS-01 solvers ship for 11 providers: Cloudflare, Route 53, Azure DNS, Google Cloud DNS, RFC 2136, acme-dns, OVHcloud, Hetzner, Infomaniak, Exoscale, and PowerDNS. certbot/acme.sh call back into the agent's DNS hook, which polls for propagation before validation and locks records across concurrent challenges.

Provider credentials live in 0600-mode files on the agent host, referenced from the agent's config.json; they are never passed on argv, never sent to the control plane, and a zoneProviderMap routes zones to providers on multi-provider hosts. The control plane stores only references and allowlists.

Per-provider credential fields, propagation tuning, zone routing, and provider-specific caveats are documented in DNS-01 providers.

PKI and private CA integrations

The ACME adapters work against any ACME-compatible CA endpoint on the agent's allowlist, including internal PKI such as step-ca or vendor ACME gateways. Because the CSR is generated locally and only public material moves, internal PKI workflows keep full key custody. Private CA bundles are supported both for the CA connection (via the ACME tool's own trust configuration) and for the agent-to-control-plane channel (caBundlePath).

cert-manager mode

For Kubernetes estates, the CertOps cert-manager controller runs in your cluster and is deployed via the TokenTimer Helm chart. It is disabled by default and defaults to observe mode:

• Observe (read-only RBAC): watches cert-manager Certificate/CertificateRequest resources and reports public-only observations (status first; tls.key is never read). Renewals stay entirely with cert-manager; TokenTimer gets inventory, rotation history, and renewal status.
• Provision (explicit opt-in, additive RBAC of Certificate create/patch only): the controller creates narrowly authorized, cluster-bound Certificate resources on TokenTimer's instruction, re-authorized before every mutation. cert-manager then issues and manages the key inside the cluster. The controller never writes Secrets and never deletes Kubernetes resources.

The controller is outbound-only, like the agent: your cluster calls TokenTimer; TokenTimer never connects to your Kubernetes API and does not accept uploaded kubeconfigs. For the Helm values that deploy it, see Path C - CertOps Kubernetes controller.

Connecting external executors

External tooling remains a fully supported execution mode: anything that can send an HTTPS request can report certificate work against a job. Scheduled renewal jobs are created by TokenTimer automation; you can also create a job manually for break-glass work, hook testing, or work performed entirely by your own tooling:

1. Have a job to report against: use a scheduled job from the jobs list, or create one as a workspace manager: on the Jobs tab of Certificate operations, click Create manual job (or call POST /api/v1/workspaces/{id}/certops/jobs). Choose an operation (issue, renew, deploy, reload, revoke, or noop). Optionally set subjectType + subjectId together (for example a managed certificate id); an issue job takes neither, because it creates the managed certificate itself (see First-time issuance). Copy the job ID (shown truncated and copyable on each job row).

   This flow does not need payload at all: an external executor does the real work itself and only reports the outcome, so nothing here has to describe a host. If you instead want a CertOps agent to execute a manual renew/deploy/reload job against an existing certificate, supply the full payload yourself (commandRef, caEndpoint, certPath, dnsZone, dnsProvider, target, sans, and so on, the same shape as the issue payload). Pointing a manual job at a certificate that already has a linked renewal profile does not pull that profile's fields in for you; only the scheduler sweep and Bulk renew derive payload from a linked profile automatically.

   Do not submit the renewalProfile snapshot as the payload

   A certificate's stored renewal profile is visible as a renewalProfile object (in public_metadata and in the profile API), and it is tempting to copy that object into payload when hand-building a manual renew job. The API accepts it and creates the job, but the agent reads flattened top-level fields (target, commandRef, and so on), not a nested renewalProfile, so the job fails agent-side with an opaque "renew job has no target.reference"-style error instead of a clear validation error at creation time. Either omit payload entirely and use Bulk renew (which derives and flattens the profile for you), or build the full flattened payload shown above yourself. This is the same distinction the paragraph above already makes; it is called out again here because the failure mode it produces is confusing enough to be worth its own warning.

2. Create a machine API token with at least certops:events:write (add certops:evidence:write if the executor uploads evidence). Store the ttx_ value once; it is shown only at creation.

3. Hand the executor the token, your workspace ID, and the job ID (environment variables or a secret store).

4. Run the work locally (renew/deploy/reload on your hosts). The executor posts lifecycle events against that jobId.

5. Review status, timeline, and evidence on the Jobs tab of Certificate operations. Search the audit log with the job ID if you need the create/report trail.

Machine API tokens

Machine API tokens are scoped API keys for non-human callers. They authenticate scripts and executors against the CertOps API without a browser session. Each token is bound to one workspace and carries an explicit set of scopes.

• Format: tokens look like ttx_<id>_<secret>. The ttx_<id> prefix identifies the token in lists; the secret part is never shown again.
• Hashed at rest: only a SHA-256 hash of the token is stored. The raw value cannot be recovered.
• Show once: the full token is displayed a single time at creation. Copy it immediately into your secret store.
• Optional expiry: set an expiry in days at creation. Expired tokens are refused automatically.
• Last used: the token list shows when each token last authenticated, so you can spot stale or unused credentials.

Create and revoke tokens from the Settings tab of Certificate operations (/certops/settings): Create token opens a modal for the name, scopes, and optional expiry, and the token list below it shows existing tokens with revoke controls. Revocation is immediate and cannot be undone. Send the token as a bearer credential:

Authorization: Bearer ttx_abc123_...

Token scopes

Scopes limit what a token can do. Grant only what the executor needs.

• certops:read read certificate inventory data
• certops:events:write report executor events for existing jobs (status updates only; creating jobs requires a session as workspace manager, not a machine token)
• certops:jobs:read read executor jobs and their timelines
• certops:evidence:write attach evidence to jobs
• certops:observations:write, certops:provision:execute controller-only scopes for the cert-manager controller integration; either one requires the token to be bound to exactly one controllerClusterId at creation, which is immutable afterward. See cert-manager mode and the RBAC modes table.

A typical renewal hook needs certops:events:write and, if it uploads command output, certops:evidence:write. Read scopes are only needed by tools that query state.

Token security

Treat machine API tokens like passwords.

• Never share a token between executors or teams. Create one token per integration so you can revoke them independently.
• Never commit tokens to source control or bake them into images. Inject them from a secret manager or environment variable at runtime.
• Prefer expiry: set an expiry and rotate tokens on a schedule instead of keeping long-lived credentials.
• Revoke on suspicion: if a token may have leaked, revoke it immediately and create a replacement. Check the last-used timestamp for unexpected activity.

The full token value is shown only once at creation. If you lose it, revoke the token and create a new one; the raw value cannot be recovered from the stored hash.

Executor jobs

An executor job is a recorded unit of certificate work: a first-time issuance, a renewal, a deployment, a service reload, a revocation, or a no-op check. Jobs are created by TokenTimer's renewal scheduler, by a workspace manager, or through the bulk-renew endpoint; executors (agents or your own tooling) perform the work and report against the job. TokenTimer builds a timeline of log entries and evidence for each job.

Where jobs come from

Jobs enter the system in three ways, and each job records its origin in a source field:

• Scheduled automation (automation): the renewal scheduler creates renewal jobs when a managed certificate with a validated renewal profile approaches expiry, subject to approval gates and per-CA caps.
• Manual creation (api): a workspace manager creates a job for break-glass work, hook testing, or work performed by external tooling.
• System (system): jobs created by TokenTimer's control plane outside the renewal sweep.

Machine-token executors cannot create jobs; they only report against an existing jobId. To create a job manually:

• Dashboard: Certificate operations → Jobs tab → Create manual job. Job IDs are listed there (truncated, copyable).
• API: POST /api/v1/workspaces/{id}/certops/jobs (session authentication; workspace manager role; source forced to api).

To look up existing jobs for executor hooks, use that page or GET /api/v1/workspaces/{id}/certops/jobs. Posting executor events for an unknown jobId returns 404 CERTOPS_JOB_NOT_FOUND rather than creating it.

The Jobs tab of Certificate operations lists executor jobs for the workspace with status, operation, and timing. Workspace managers can open Create manual job from there, including when the list is empty. Selecting a job opens its timeline: every reported event in order, plus attached evidence with redaction markers where secrets were removed.

Jobs move through statuses such as pending, claimed, running, and terminal states succeeded, failed, rejected, blocked, cancelled, or orphaned_unknown_effect. Jobs that require approval start at pending_approval and move to approved before becoming claimable. Transitions are monotonic: a job never moves backward, and a terminal status is final. Events that would regress a job are recorded in the log without changing the job status.

orphaned_unknown_effect is the one status that is not a plain success or failure: it means an agent went silent after it may already have changed something on a host, so TokenTimer refuses to guess. See Reconciling interrupted jobs.

First-time issuance with an issue job

Every other operation assumes the certificate already exists in inventory. issue is the one that asks for a brand-new certificate: use it when TokenTimer has no managed certificate for the identity yet and you want the result to land in the inventory rather than only on disk.

The control plane creates the managed certificate straight away, at status provisioning, links the job to it, and dispatches the job to a capable agent. The certificate is therefore visible on the certificate surfaces from the moment the job is accepted, before any ACME work has happened.

The request contract differs from a renewal in a few specific ways:

• No subjectType or subjectId: there is no certificate to point at yet, so neither may be sent. The server creates the managed certificate and links the job to it.
• No payload.certificateId: the server assigns it.
• idempotencyKey is required, not optional: a retried request must never end up creating a second certificate. It must be 1 to 128 characters of letters, digits, ., _, : or -. Reposting the identical request returns the original job with the same subjectId, and no second certificate or ACME order is created. Reusing the key with a different body is rejected as a conflict (HTTP 409), because one key means one certificate.
• payload.certPath must be an absolute file path, for example /etc/ssl/tokentimer/www.example.com.pem. A relative path is rejected because the agent would resolve it against an unspecified working directory, and a trailing slash is rejected because it names a directory rather than a file. Both are rejected at request time on purpose: agent-side they only fail after the ACME order has been placed, which burns a real rate-limited order and leaves the certificate stuck at provisioning. This field does not apply to a windows-iis target; see below.
• No renewalProfile: it is not valid on an issue job. You do not need one, because a profile is derived for you when the issuance succeeds. See Where the renewal profile comes from.
• Everything else is what a renew job takes: commandRef, caEndpoint, certPath, dnsZone, dnsProvider, target, and sans, plus optional acmeKind, keyRotation, reloadService, verifyHost, verifyPort, and chainPath.

By default the agent deploys the full chain (leaf plus intermediates) to certPath, which is what most web servers expect. Set payload.chainPath to a second absolute file path if your deployment target wants the leaf and the intermediate chain in separate files (some HAProxy and non-nginx configurations do); the agent then writes only the leaf to certPath and the intermediates to chainPath. Unlike certPath, chainPath is not validated at request time - a malformed value is only caught agent-side, after the ACME order has already succeeded, in the same way certPath used to fail before its own validation was added.

Concurrent retries with the same key are serialized on the identity, so two clients racing the same request both receive the same job rather than one of them getting a 500.

curl -sS -b cookies.txt -X POST \
  'https://your-tokentimer.example/api/v1/workspaces/WORKSPACE_ID/certops/jobs' \
  -H 'Content-Type: application/json' \
  -d '{
    "operation": "issue",
    "idempotencyKey": "issue-www-example-com-1",
    "payload": {
      "target": { "type": "domain", "reference": "www.example.com" },
      "sans": ["www.example.com"],
      "commandRef": "certbot-csr",
      "caEndpoint": "https://acme-v02.api.letsencrypt.org/directory",
      "dnsZone": "example.com",
      "dnsProvider": "cloudflare",
      "certPath": "/etc/ssl/tokentimer/www.example.com.pem"
    }
  }'

The agent does exactly what it does on a renewal: an ACME order with the configured command profile and DNS-01 provider, an atomic deploy to certPath, an optional service reload, then verification. The control plane dispatches the work to agents as a renew action, so nothing changes in the agent's own execution path.

target.type: "windows-iis": deploying to a Windows certificate store and IIS binding

A domain target (above) deploys a certificate to a file. A windows-iis target deploys instead to a Windows machine certificate store plus an IIS site binding, with the private key generated and held entirely inside Windows CNG (Cryptography API: Next Generation) — it is never written to disk as a file, .pfx, or otherwise. This is the target type a Windows agent's windows-cert-store-v1/iis-binding-v1 capabilities exist to serve; see Install an agent (Windows) for the Windows-specific installer and the Windows/IIS worked example for a full traced walkthrough.

The shape differs from a domain target in exactly these ways:

• target.store is the Windows certificate store name the certificate is enrolled into, almost always "My" (the machine's Personal store, LocalMachine\My). A different store (for example WebHosting, a convention some IIS deployments use) is supported: the agent enrolls into My first (the only store certreq itself can target) and then mirrors the result into the requested store, removing the My copy afterward. This mirror path is not yet verified against a real Windows host the way the default My path is; the default store is the better-proven choice until that verification lands.
• target.binding replaces certPath entirely: { "site": "<IIS site name>", "port": <int>, "sniHost": "<optional SNI hostname>" }. site and port are required; sniHost is optional and, when present, binds the certificate to that specific hostname on a shared listener (netsh http add sslcert hostnameport=...) rather than to every hostname on that IP:port. This does not, however, guarantee every client receives that certificate: Windows itself always gives a non-SNI (ipport=) binding on the same port precedence over an SNI (hostnameport=) binding, for any client connecting to an address that also has its own ipport= binding, whether that address is a wildcard (0.0.0.0 / [::]) or one specific IP. This is http.sys's own dispatch rule; neither binding's own configuration can override it. The agent checks for a known conflicting binding immediately after deploying an SNI binding and reports one, when found, as a non-fatal precedenceWarning in the deploy evidence: review that warning before assuming the SNI-bound certificate is what every client actually receives.
• No payload.certPath, payload.chainPath, or payload.keyPath at all. Sending any of them alongside a windows-iis target is rejected: there is no filesystem destination to validate.
• No payload.keyMode to set yourself. The control plane derives keyMode: "os-store-managed" automatically from target.type, both for the issue job's own dispatch and for the renewal profile it derives afterward.

{
  "operation": "issue",
  "idempotencyKey": "issue-iis-example-1",
  "payload": {
    "target": {
      "type": "windows-iis",
      "reference": "www.example.com",
      "store": "My",
      "binding": { "site": "Default Web Site", "port": 443, "sniHost": "www.example.com" }
    },
    "sans": ["www.example.com"],
    "commandRef": "acmesh-signcsr",
    "caEndpoint": "https://acme-v02.api.letsencrypt.org/directory",
    "dnsZone": "example.com",
    "dnsProvider": "cloudflare"
  }
}

The agent runs the same ACME order as any other issuance (real CA, real DNS-01 challenge), then generates the key and completes enrollment entirely inside CNG (certreq -new / -accept), binds the resulting certificate to the configured IIS site/port/SNI host (netsh http add sslcert), and verifies the live TLS handshake against the actual binding before reporting success. No iisreset or blanket reload is ever issued; only the one configured binding changes. A certificate this target type supersedes is retained (not deleted) for a configurable grace window, so a bad rollback always has something to roll back to.

Issue jobs are only offered to agents that declare evidence-claim-binding-v1

Promoting a newly issued certificate to active depends on evidence the agent bound to the exact job claim it was executing. An older agent does not bind evidence that way, so an issue job it ran could never be reconciled and the certificate would sit at provisioning forever with a real certificate deployed on the host.

Rather than let that happen, the claim query only offers a job to an agent that declares the evidence-claim-binding-v1 capability when either:

• the job's operation is issue, or
• the job is a renew against a certificate that is still provisioning (which is how you retry a failed issuance).

What you see if your fleet is behind: the issue job stays at pending and is never claimed, with no error anywhere. Upgrade the agents on the hosts that should run issuance. Ordinary renewals of already-active certificates are unaffected and keep being claimed by older agents exactly as before.

On success, the certificate moves from provisioning to active, and its fingerprint, validity dates, serial, subject, and SANs are filled in from the evidence the agent reported for that job. There is no wait for a discovery scan. A renewal profile is also derived at this point so the certificate auto-renews from here on; see Where the renewal profile comes from.

On failure, the certificate stays at provisioning with the failed job and its evidence attached. Nothing is cleaned up automatically. Fix the cause and run an ordinary renew job against the certificate that now exists, or retire the provisioning certificate. Until then it behaves like any other non-terminal certificate: it can be retired (revoked or decommissioned) and it counts as a tracked certificate in the workspace.

A failed issuance does not raise a renewal-failure alert

Only failed renew jobs raise cert_renewal_failed. A failed issuance has no existing certificate at risk of expiring, so there is no deadline to warn you about. Watch your inventory for certificates stuck at provisioning, not your alert channel, when tracking issuance problems. See Why a failed issuance is not a renewal failure.

When a certificate stays at provisioning

Promotion to active is deliberately strict. It happens only on verify-step evidence that is bound to the current job claim and that carries both a fingerprint and an expiry. Each of those guards a specific way the inventory could end up lying to you:

• Verify-step only, because the agent reports success twice: once when the CA returns the certificate, and once after the deployed file has been read back and fingerprinted. Only the second describes what is actually on the host.
• Claim-bound only, because a job can be attempted more than once and evidence from an earlier attempt outlives it. Without the binding, attempt 2 could be promoted using attempt 1's fingerprint.
• Fingerprint and expiry mandatory, because a certificate with no expiry cannot be renewed on schedule or alerted on. It would look healthy and be silently unmanaged.

When a guard fails, the row stays provisioning and records a reconciliation_reason you can read:

reconciliation_reason	What happened	What to do
no_claim_bound_verify_evidence	The job reported success but attached no verify-step evidence bound to that claim. Usually an agent that predates evidence-claim-binding-v1, or a job whose verify step did not run	Upgrade the agent, then run a renew job against the certificate. Check whether verifyHost was configured
verify_evidence_missing_fingerprint	Verify evidence arrived but carried no SHA-256 fingerprint, so the control plane cannot record which certificate is deployed	Re-run a renew job. If it recurs, inspect the job's evidence on the timeline and check the agent version
verify_evidence_missing_expiry	Verify evidence arrived with no parseable validity end, so the certificate could never be renewed or alerted on	Re-run a renew job and check the deployed file is a valid certificate at the path the agent verified

A provisioning certificate is not a broken certificate: the material may well be live on the host. It means TokenTimer does not yet have proof of what is there, so it refuses to treat it as managed. Re-running a renew job against it is the normal fix, and that retry reconciles through the same path.

The active-renewal path uses the same three guards and the same fix, but the failure mode is worse if it goes unnoticed: this refresh runs against a certificate that is already active, and its whole job is to update not_after and mirror the new fingerprint onto the linked token after a successful renewal. A guard failure here now sets reconciliation_reason on the certificate and writes a CERTOPS_CERTIFICATE_RENEWAL_UNRECONCILED audit event instead of returning silently, and the certificate shows the same Unreconciled badge described for issuance above. Before this was added, not_after stayed unchanged on a silent failure, which meant the scheduler's (certificateId, notAfter) idempotency key never advanced and the same "successful" renewal could be re-derived and repeated against the same CA on the next sweep.

How a job is bound to one agent

Two independent mechanisms decide which agent runs a job, and both matter if you run more than one agent.

1. Claim exclusivity. Claiming is transactional and row-locked with FOR UPDATE SKIP LOCKED. If several agents poll at the same instant, each gets a different job and never the same one. Once claimed, the job carries a lease held by that agent; nobody else can claim it while the lease is alive.

2. Assignment (assignedAgentId). Exclusivity alone does not mean the right agent claimed the job. A job with no assignment and no target selector is claimable by any online agent that declares support for the operation, including one with no relationship to the certificate. So for renew, deploy, reload, and revoke jobs against a certificate an agent discovered on its own filesystem, TokenTimer sets assignedAgentId to that discovering agent at creation time. The job is then pinned: only that agent can ever claim it.

This matters because certificate work is host-specific. Renewing /etc/nginx/ssl/site.pem only means something on the host where that file exists; another agent claiming the job would fail on a missing path, and you would see a renewal failure alert for what was really a routing mistake.

Consequences worth knowing:

• If the owning agent is offline, its jobs wait. They stay pending rather than being picked up by a healthy agent, because a different host is not a valid substitute. Check the Agent fleet panel first when a job seems stuck at pending.
• If you replaced the host (rebuilt it, re-registered the agent, migrated the service), jobs pinned to the retired agent will never run. Pass an explicit assignedAgentId for the new agent, or let the new agent rediscover the certificate so future jobs pin correctly.
• An explicit assignedAgentId always wins over the automatic one, which is the supported way to hand a certificate over to a different agent.
• Certificates with no agent custody are rejected at creation instead of being assigned at all. See Not every certificate in inventory can be renewed for you.
• Target selectors still apply on top of assignment. They are the coarser tool (route by role or environment); assignment is the exact one.

Reporting events

Executors report work by posting executor events with a machine API token. Each event carries a caller-chosen eventId, a jobId, the workspace ID, a status, an event type, and a timestamp.

• Event types: job.accepted, job.started, job.progress, job.completed, job.failed, job.rejected, and evidence.attached.
• Idempotent by eventId: sending the same eventId with the same payload again is a safe no-op, so executors can retry freely. Reusing an eventId with a different payload is rejected as a conflict.
• Events target an existing job: every event must reference a jobId that already exists (created by a workspace manager or by TokenTimer automation). Machine tokens cannot create jobs. Posting to an unknown jobId is rejected with CERTOPS_JOB_NOT_FOUND.
• Unknown fields are rejected: extra top-level fields in the request body are rejected rather than silently dropped, so a typo in a field name surfaces immediately instead of being ignored.
• Inline evidence: an event may include up to 16 evidence items, and public metadata name/value pairs (secret-looking names are refused).

Requests need the certops:events:write scope. Evidence-bearing requests also need certops:evidence:write.

Evidence and redaction

Evidence entries attach proof of work to a job: command output, ACME logs, deployment confirmations, or certificate fingerprints. Evidence appears on the job timeline in the dashboard.

Evidence output is scanned before storage. Values that look like generic secrets (passwords, API keys, bearer tokens) are replaced with a redaction placeholder, and the entry is marked with redactionApplied so the timeline shows that redaction occurred. Redaction is best effort; do not rely on it as your only safeguard.

Evidence output is capped at 64 KiB before redaction runs; oversized output is rejected outright with CERTOPS_EVIDENCE_OUTPUT_TOO_LARGE rather than truncated. The stored outputSizeBytes and outputSha256 both describe the redacted output that was actually persisted, not your original submission, since redaction can shrink or grow the byte length.

Private keys are different: they are never redacted and stored, they are rejected outright. Any event, evidence, or metadata payload containing private key material (PEM key blocks, key-material field names, PFX or JKS blobs) fails with HTTP 422 and PRIVATE_KEY_MATERIAL_REJECTED. Nothing from the request is stored.

Never pipe raw certbot or openssl output that may include key files into evidence. Filter your logs so only public material and status output is uploaded.

Executor example

A minimal executor posts one event when work completes. This example reports a successful renewal from a certbot deploy-hook. The jobId must already exist in TokenTimer (create one with Create manual job or POST /api/v1/workspaces/{id}/certops/jobs, or pick one from the jobs list). Look it up beforehand and pass it to the hook. Do not invent a UUID locally; unknown job ids are rejected with 404 CERTOPS_JOB_NOT_FOUND:

#!/bin/sh
# /etc/letsencrypt/renewal-hooks/deploy/tokentimer-report.sh
# TOKENTIMER_TOKEN comes from your secret store, never hardcode it.
# TOKENTIMER_JOB_ID is an existing job id from Create manual job /
# POST .../certops/jobs, or from the jobs list (GET .../jobs, scope certops:jobs:read).

JOB_ID="${TOKENTIMER_JOB_ID:?set TOKENTIMER_JOB_ID to an existing job id}"
EVENT_ID="$JOB_ID-completed"

curl -s -X POST \
  'https://your-tokentimer.example/api/v1/certops/executor/events' \
  -H "Authorization: Bearer $TOKENTIMER_TOKEN" \
  -H 'Content-Type: application/json' \
  -d "{
    \"schemaVersion\": 1,
    \"eventId\": \"$EVENT_ID\",
    \"jobId\": \"$JOB_ID\",
    \"workspaceId\": \"WORKSPACE_ID\",
    \"executorId\": \"certbot-$(hostname)\",
    \"status\": \"succeeded\",
    \"eventType\": \"job.completed\",
    \"occurredAt\": \"$(date -u +%Y-%m-%dT%H:%M:%SZ)\",
    \"message\": \"certbot renewed $RENEWED_DOMAINS\"
  }"

For a longer-running job, post job.started when work begins and job.completed or job.failed when it ends, reusing the same jobId with distinct eventId values. Retries can resend the exact same request safely. The same pattern works for acme.sh hooks, Ansible handlers, or any script that can send an HTTPS request.

API reference

Executor endpoints authenticate with Authorization: Bearer ttx_... and are rate limited per token:

• POST /api/v1/certops/executor/events ingest an executor event for an existing job
• POST /api/v1/certops/jobs/:jobId/events append an event to an existing job
• POST /api/v1/certops/jobs/:jobId/evidence attach evidence to a job

Each machine token shares a single rate-limit bucket (120 requests / 60s by default) across all three routes above, not a separate budget per route; exceeding it returns HTTP 429 CERTOPS_MACHINE_RATE_LIMITED. Common error codes: 401 CERTOPS_API_TOKEN_UNAUTHORIZED (bad/revoked/expired token), 403 CERTOPS_API_TOKEN_SCOPE_DENIED (missing scope), 404 CERTOPS_JOB_NOT_FOUND (unknown jobId), 409 CERTOPS_EXECUTOR_EVENT_CONFLICT (same eventId replayed with a different payload), 422 PRIVATE_KEY_MATERIAL_REJECTED, and 413 CERTOPS_EVIDENCE_OUTPUT_TOO_LARGE.

Workspace-scoped endpoints under /api/v1/workspaces/:id/certops serve the dashboard and session-authenticated callers:

• GET .../jobs list executor jobs
• POST .../jobs create an executor job manually (workspace manager; body: { operation, subjectType?, subjectId?, payload? }; source is forced to api; an issue job takes no subject and requires idempotencyKey, see First-time issuance)
• GET .../jobs/:jobId job detail
• GET .../jobs/:jobId/log job timeline entries
• GET .../jobs/:jobId/evidence evidence attached to the job
• GET .../tokens list machine API tokens
• POST .../tokens create a token (raw ttx_ value returned once)
• POST .../tokens/:tokenId/revoke revoke a token

See Create a manual CertOps job for the create-job request schema, the API reference for authentication, request bodies, and response shapes, and Certificates for the inventory endpoints.