Certificate Automation

Overview

CertOps automation renews certificates end to end. TokenTimer Cloud schedules renewal work before certificates expire, routes each job to an eligible executor in your environment, and records what happened: issuance, deployment, reload, verification, and evidence. Approval gates and a workspace kill switch keep humans in control of what runs.

The control plane plans, signs, and audits; execution happens where your certificates live. TokenTimer never connects into your network and never touches private keys: keys are generated and used on your hosts (or inside your cluster with cert-manager), and any payload containing key material is rejected outright.

Three execution modes are available, and they can be mixed freely in one workspace:

• TokenTimer agent: an outbound-only daemon on your hosts that verifies signed jobs, runs ACME renewals with DNS-01, deploys atomically, reloads services, and verifies the served certificate. See Deploying an agent.
• cert-manager controller: a Kubernetes controller that observes cert-manager certificates and, in opt-in provisioning mode, creates authorized Certificate resources. Key custody stays with cert-manager. It is self-hosted: you run TokenTimer's open-source controller image against your own cluster, and it shares the same per-workspace plan gate and count caps as agents, tracked as an independent counter (see Plan limits). See CertOps with ACME and cert-manager.
• External executors: your own renewal scripts, certbot/acme.sh hooks, and CI pipelines authenticate with a machine API token and report against executor jobs. See Connecting external executors.

Certificate operations is one entry in the navigation sidebar, reached from the Control Center footer link or from Workspace Preferences when CertOps is enabled. It opens onto a row of tabs: Jobs (/certops/jobs, the default; /certops/operations redirects here), Certificates (/certops/certificates), Renewals (/certops/renewals), Agents (/certops/agents), and Settings (/certops/settings, machine API tokens). The managed-certificate inventory and per-row Set up renewal / Detach / Retire actions live on the Certificates tab.

Scheduled renewals and approvals

Renewal work is planned by TokenTimer, not by cron jobs you have to remember:

• Renewal scheduler: when a managed certificate approaches expiry, the scheduler creates a renewal job automatically. A job is only scheduled when the certificate has a complete, validated renewal profile (CA endpoint, command profile, target), so automation never creates unexecutable work. For certificates TokenTimer issued, that profile is derived for you; see Where the renewal profile comes from.
• Eligible-agent routing: each job is routed to an agent whose declared capabilities and target selectors match the work. Agents claim jobs over an outbound-only protocol; the control plane never connects to your hosts. A claim is exclusive, so two agents can never receive the same job, and jobs for a certificate an agent discovered are pinned to that agent (see How a job is bound to one agent). If an agent stops reporting mid-job, TokenTimer only auto-retries when no side effects were possible, and otherwise flags the job for review (see Reconciling interrupted jobs).
• Signed dispatch: every dispatched job is signed with a control-plane Ed25519 key. Agents verify the signature against a pinned key, enforce a validity window, and keep a replay cache, so a forged or replayed job never executes.
• Approval gates: jobs can require an explicit approval before dispatch. A requester cannot approve their own job, the approval is bound to the exact job payload hash, and every decision lands in the audit log. See Approval gates for the full non-requester and invalidation rules.
• Per-CA caps and bulk renew: an in-flight cap bounds renewals per CA endpoint per workspace across every origin (scheduler, manual, or bulk renew). Bulk renew (below) goes through the same cap as a single manual job.
• Metered renewal-job quota: renewal jobs also draw against a monthly, per-workspace quota that resets on the first day of the month (UTC), on top of the per-CA cap above. See Plan limits for current numbers. Renewal profiles themselves are never metered; only the renewal job counts.
• Kill switch: a workspace-level pause blocks new CertOps jobs and dispatch instantly, without deleting queued work or history. Inventory, status, audit, and evidence reporting keep working while paused. A deliberate exception covers recovery actions: retrying a stuck renewal setup, detaching a renewal profile, and approving or rejecting an already-queued job all stay reachable while paused, so you can clean up or unblock in-flight work during an incident without having to resume first. Only actions that would create new work (new manual jobs, new renewal scheduling) are blocked. Toggle it from Certificate operations: the panel at the top of every tab shows the current Active/Paused badge, with a Pause/Resume button visible to workspace admins (certops.kill_switch.manage).
• Renewal failure alerting: failed renewals raise a cert_renewal_failed alert through your normal TokenTimer alerting channels. See Certificate renewal failures for when it fires and how it routes.

When a renewal actually gets scheduled

Five things have to be true. If a certificate is not renewing on its own, work down this list in order; it is ordered by how often each one is the answer.

1. The certificate has a linked renewal profile with a complete snapshot. No profile means no job, ever. See Where the renewal profile comes from.
2. Its key custody is agent-deployable: an agent must actually hold the key. A certificate that was only observed (via an HTTPS endpoint monitor or a domain monitor) has no key TokenTimer can rotate, so it is monitored and alerted on but never auto-renewed. See Not every certificate in inventory can be renewed for you.
3. It is inside the renewal window: by default, expiry within the next 30 days. A profile can set its own lead time with renew_before_days when one certificate needs longer than the rest of your fleet, for example one behind a slow change-approval process. Keep this well under the CA's actual certificate lifetime (about 90 days for Let's Encrypt): a lead time close to or above the lifetime keeps the certificate permanently inside its renewal window, so it renews again on every scheduler sweep instead of settling once it is genuinely due.
4. Its inventory status is renewable: revoked and decommissioned certificates are excluded outright.
5. Automatic renewal is switched on for its profile. A profile whose renewal has been switched off is skipped deliberately. See Switching automatic renewal off.

A certificate is also skipped for the current pass (and retried on the next) when the workspace is paused, when the per-CA cap is full, or when a renew job for it is already in flight. That last one is what stops repeated scheduling passes from stacking duplicate renewals.

You do not have to evaluate that list by hand: every certificate in the inventory carries a renewal badge, computed from the same inputs the scheduler uses, so it cannot read "Auto-renews" for a certificate that would in fact be skipped. When it reads Auto-renews it also shows the date renewal becomes due and the lead time in days. No auto-renewal is the state to act on: an agent holds the key, so you probably believe the certificate is managed, and it will expire. Auto-renewal off means somebody switched it off on purpose, which is supported but still ends in expiry. See The renewal badge.

The same data is on the API as a renewal object per certificate on GET /api/v1/workspaces/{id}/certops/certificates, with state one of auto, disabled, not-configured, not-eligible, or not-applicable, plus renewBeforeDays and renewsFrom.

Seeing and changing what will renew

Automatic renewal is on by default for everything TokenTimer issued, because the profile is derived for you. The Renewals tab of Certificate operations (/certops/renewals) is where you see and change that:

• Upcoming renewals lists every renewable certificate in the workspace, soonest expiry first, with the date renewal starts being attempted and the outcome of the last attempt. It deliberately includes certificates renewal will not cover, each labelled with why:

  Label	Meaning	What to do
  On	This certificate will be renewed automatically.	Nothing.
  Off	Automatic renewal was switched off for its profile.	A deliberate choice. Switch it back on below, or renew by hand.
  No profile	No renewal profile exists, so renewal will never pick this certificate up. Profiles are created automatically at issuance, so a missing one means that step did not complete.	Re-issue the certificate.
  Incomplete	A profile exists but cannot be executed, so it will never produce a renewal.	Re-issue the certificate to rebuild a working profile.
  No key access	No agent holds this certificate's private key, so nothing can renew it. Typical of a certificate that was only observed from the outside, for example by an endpoint or domain monitor.	Issue the certificate through CertOps so an agent owns its key. A profile alone cannot fix this.
  No expiry	No expiry date is recorded, so renewal cannot tell when this certificate is due.	Re-run discovery or re-issue.

  An empty table therefore means the workspace has no renewable certificates at all, not merely none that are covered. Restricting the list to certificates that do renew would make this page report an all-clear precisely when nothing renews.

  "On" describes the profile's switch, not whether the workspace can currently act

  On/Off here (and the Auto-renews/Auto-renewal off renewal badge) read the certificate's own renewal profile only. They do not reflect the workspace kill switch. A certificate can show On while its workspace is paused: the profile itself is genuinely switched on and would fire the moment the workspace resumes, but nothing renews in the meantime. Check the Active/Paused badge at the top of every Certificate operations tab separately; this page and the kill switch are independent signals by design.

• Renewal profiles lists the profiles in your workspace with the number of certificates each covers, and offers exactly two controls: an automatic-renewal switch and the renewal lead time.

Reading either panel needs manager or above; both controls need workspace admin. A profile body carries deployment topology (certificate and key paths, the reload unit, file ownership and modes, the ACME command reference, the CA account reference, the DNS zone), so it is gated like the agent and machine-token surfaces rather than like the certificate inventory. A viewer keeps full visibility of what expires and when, through Certificates and the renewal badge, without learning where a given key lives on a host.

Why you cannot edit a profile's deployment details

A derived profile records an issuance that provably worked: that CA endpoint, that ACME command profile, that DNS provider and zone, that certificate path. Those fields are read-only, and a request to change them is refused.

That is deliberate. Repointing where a live certificate is written, or which command writes it, is a re-issuance rather than a settings change, and an unattended renewal against a rate-limited CA is the worst possible time to discover the new value was wrong. Re-issue with the payload you want instead.

The editable subset is the part that cannot change what runs on a host: SAN policy, key algorithm and size, key rotation policy, verification settings, preferred chain, the renewal lead time, and the on/off switch.

Re-issuing the same hostname used to reset these edits; fixed

A derived profile is identified by its name, Derived: <common name> (<certificate id>). The certificate id suffix means two certificates sharing a common name get separate profile rows: issuing a second certificate for a common name you already have no longer touches the first certificate's profile or its edits. (Earlier releases keyed the profile on name alone, so a re-issuance rewrote a shared row and could silently drop edits or leave a second certificate with no profile at all; that is fixed.)

Switching automatic renewal off

Switching a profile off makes the scheduler skip every certificate linked to it. Expiry monitoring and alerting keep running, so you still hear about the certificate; only the renewal stops.

This is the one control that can cause an outage, so it asks for confirmation, states how many certificates the profile covers, and leaves a standing warning on the schedule afterwards. For a derived profile that count is now always 1, since a derived profile is scoped to the certificate it came from; a profile you authored yourself can still cover several certificates, and switching it off affects all of them. Switching back on takes effect on the next scheduling pass and needs no confirmation. Every change is recorded in the audit log with who made it.

Where the renewal profile comes from

This is the single most common reason a fleet looks healthy and renews nothing, so it is worth being explicit.

The scheduler refuses to create a renew job unless the certificate links to a profile carrying a complete, validated renewal snapshot: CA endpoint, ACME command profile, DNS provider and zone, deployment paths, SAN policy, and key parameters. That gate is intentional. Dispatching a half-specified renewal would hand an agent a job it cannot run, against a real rate-limited CA.

For certificates TokenTimer issued, that profile is now derived for you at the moment issuance succeeds. When an issue job is reconciled and the certificate is promoted to active, TokenTimer builds a renewal profile from the payload that just succeeded and links the certificate to it. A successful issuance is the one moment where every field is both known and proven to work: the agent completed a real ACME order with exactly that CA endpoint, DNS provider, zone, command profile, and deployment path.

What that means for you:

• You do not author a profile for anything TokenTimer issued. It appears on the Renewal automation page named Derived: <common name>, where you can change its renewal lead time or switch renewal off. Its deployment details are deliberately read-only; see Why you cannot edit a profile's deployment details.
• The SAN set is pinned to what the CA actually issued, not to what the job requested. If the CA normalised or dropped a name, the renewal reproduces the certificate that is deployed rather than the one that was asked for.
• A profile you wrote yourself is never overwritten. If the certificate already links to a profile, derivation leaves it alone.
• Derivation can decline, and the issuance still succeeds. If the issue payload lacked a field the profile needs (CA endpoint, command profile, DNS provider and zone, or cert path), the certificate is still promoted to active and no profile is linked. You get a working certificate that will not auto-renew, which is better than losing the issuance. Such a certificate does not appear on the Renewal automation page at all, and its renewal badge reads No auto-renewal. Re-issue with a complete payload to get a profile derived.
• Certificates that arrived some other way still need a profile. An imported PEM or an endpoint-monitor observation has no issuance payload to derive from and no recorded deployment path, so Set up renewal cannot arm a renewal for either: re-issue such a certificate through TokenTimer instead. A certificate discovered on an agent filesystem is different: its deployment path is recorded from the scan that found it, so Set up renewal (see Bringing an existing certificate under automatic renewal) can adopt it without re-issuing. Observed-only certificates cannot be renewed at all either way.

Bringing an existing certificate under automatic renewal

Not available for an imported or endpoint-monitor certificate

Set up renewal requires a recorded deployment path (CERTOPS_RENEWAL_SETUP_NO_DEPLOYED_PATH if there is none). An agent-filesystem discovery records the path it found the certificate at, so the route is reachable for that source. An imported PEM or an endpoint-monitor observation records no deployment path at all - there is no code path that writes one for either - so Set up renewal always refuses for them today. The button still appears for these two (its own precondition is only "no linked profile yet") and the row still reads No auto-renewal, but confirming - dry run or real - always ends in the refusal below. The only working way today to get a profile onto an imported or observed-only certificate is to re-issue it through TokenTimer, which creates a new certificate identity rather than adopting the existing one in place.

Re-issuance is not the only intended path to a derived profile. On the Certificates tab (/certops/certificates), a certificate that is active, agent-deployable, and has no linked profile yet gets a Set up renewal button in its row.

Clicking it opens a form for the four inputs a renewal needs that TokenTimer cannot infer on its own: the ACME command profile, the CA endpoint, the DNS provider, and the DNS zone. Confirming does not save those as settings; it runs a real renew job against the certificate's recorded deployment path, pinned to the agent that recorded it, and derives a profile only if that job succeeds - exactly the same way as a successful issue job. This is the working path for a certificate discovered on an agent's filesystem; per the caveat above, it is not reachable yet for an imported or endpoint-monitor certificate, since neither ever has a recorded deployment path to renew against.

Set up renewal is refused, not silently unavailable, when the certificate already has a profile, is not agent-deployable, has no discovered deployment path (imported and endpoint-monitor certificates today; not filesystem-discovered ones), or is deployed to more than one location (not supported today; the schema records one deployment path per certificate).

While the triggering renewal job is in flight, the certificate's row shows a Setting up automatic renewal badge instead of the button. If the job fails, the badge reads Setup failed with a Retry action that re-arms the same derivation intent without redoing the form. A profile that was deliberately skipped shows Setup skipped instead and is not retryable from here.

Detach (on the same row, once a profile is linked) removes only this certificate's link to its profile; the profile row itself, and any other certificate sharing it, is untouched.

Renew jobs are not pre-pinned to an agent

A scheduled renew job does not require you to nominate an agent. It is created open-claim and matched at claim time by the agent's declared target selectors, DNS providers, and command profiles. Any agent in your fleet whose declarations match can pick it up.

The one exception is custody: if the certificate was discovered on a specific agent's own filesystem, the job is pinned to that agent, because renewing a path only means something on the host where that path exists. See How a job is bound to one agent.

If a renewal sits at pending and is never claimed, the cause is almost always a matching problem rather than a scheduling one: no online agent declares the required target selector, DNS provider, or command profile, or the job is pinned to an agent that is offline.

Bulk renew

POST /api/v1/workspaces/{id}/certops/jobs/bulk-renew queues a renew job for up to 100 managed certificates in one request. Each certificate goes through exactly the same creation path as a single manual job (manager role, kill switch, approval gate, payload validation, per-CA cap), so per-certificate behavior is identical to creating them one at a time.

It is a partial-failure envelope, not all-or-nothing. The response is 200 with a per-item outcome for every certificate, even when some or all items failed. Only whole-request shape problems (malformed body, more than 100 ids, non-UUID or duplicate ids) return 400. Always read the per-item results; a 200 does not mean every certificate was queued.

Dry run first. Pass dryRun: true to preflight each certificate without writing anything: existence, renewable inventory status, payload validation, and whether a non-terminal renew job is already in flight (reported as activeJobId, which is how you catch accidental double-renewals before committing).

Idempotency keys, and the re-run trap

Every item is created with a derived per-certificate idempotency key, so a retried batch returns the already-created jobs (marked replayed: true) instead of enqueueing duplicates:

• With a request-level idempotencyKey: the per-item key is bulk-renew:<your key>:<certificateId>.
• Without one: the server derives bulk-renew:auto:<certificateId>.

That fallback is the trap. Because the derived key depends only on the certificate id, it is the same on every later bulk renew of that certificate, forever. Idempotency keys are not released when a job reaches a terminal status. So if you bulk renew a set of certificates, cancel the jobs, then bulk renew the same set again with different options (adding requiresApproval: true, for example), the second call does not create new jobs: it matches the old keys and either replays the originals or reports a conflict. It reads as "bulk renew silently did nothing".

Pass an explicit, fresh idempotencyKey whenever you intend a genuinely new attempt rather than a retry of the same one. Reserve the omitted form for what it was designed for: safely retrying a request you are not sure was received. A scheme like reconcile-2026-07-24-a works well because it makes the intent auditable too.

Deploying an agent

The TokenTimer agent is the execution plane for hosts: a zero-dependency Node.js daemon that is outbound-only (it opens HTTPS connections to TokenTimer Cloud and never accepts inbound connections).

1. On the Agents tab of Certificate operations, click Deploy an agent to open the install wizard and create a bootstrap token (shown exactly once).
2. Copy the generated install command and run it on the target host. The installer creates a dedicated system user and a hardened systemd unit.
3. Watch the Agent fleet panel flip the agent to registered on its first heartbeat. The fleet panel shows status, heartbeat, and compatibility, and lets you retire an agent.

For install options, write-path and reload permissions, upgrades, and troubleshooting, see Install and upgrade an agent. Agent count is capped per plan (see Plan limits) and tracked independently of the self-hosted controller count and the managed-certificate slot count.

Agents do not transfer between workspaces

An agent's registration is scoped to the workspace it registered under and cannot be moved in place. To put an agent under a different workspace, retire it in the old workspace and install a fresh agent with a new bootstrap token in the target workspace.

Reading agent health in the fleet panel

The fleet panel shows two things that both look like "is this agent up", and knowing which is which saves confusion:

• Status is the durable record (active, offline, retired), updated by a periodic sweep.
• Liveness is computed fresh on every read from the last heartbeat: live, stale, or retired.

Liveness exists so the panel does not lie to you between sweeps. An agent that crashed thirty seconds ago still reads active in the durable record but shows as stale immediately. Trust liveness for "is it responding right now".

Note that going offline is observational only: the agent keeps its credential and any jobs assigned to it, and resumes when it comes back. Only an explicit retire invalidates the credential. Clock drift is shown alongside because signed dispatch enforces a validity window, so a badly skewed host can reject valid work; a dash instead of a value is normal for observe-only agents, which do not measure it.

What the agent does on a renewal job, entirely on your host:

• generates (or reuses) the private key and builds the CSR locally; keys never leave the host,
• drives certbot or acme.sh in CSR mode against your allowlisted CA endpoint (public ACME CAs or internal, ACME-compatible PKI, including private CA bundles),
• solves DNS-01 challenges with native solvers for 11 providers (Cloudflare, Route 53, Azure DNS, Google Cloud DNS, RFC 2136, acme-dns, OVHcloud, Hetzner, Infomaniak, Exoscale, PowerDNS); DNS credentials live in files on the agent host and never leave it (see DNS-01 providers),
• deploys atomically with backup and automatic rollback, then validate-then-reloads nginx, Apache, or HAProxy,
• verifies the deployment by fingerprinting the installed certificate, with an optional live TLS probe against the endpoint,
• reports each step as evidence on the job timeline.

The agent enforces its own local policy: allowlists for commands, paths, CA endpoints, and DNS zones/providers that the control plane cannot override, and key-export requests are rejected unconditionally. A dry-run mode exercises the full trust chain and reports the planned steps without side effects, so you can validate a rollout safely.

Connecting external executors

External tooling remains a fully supported execution mode: anything that can send an HTTPS request can report certificate work against a job. Scheduled renewal jobs are created by TokenTimer automation; you can also create a job manually for break-glass work, hook testing, or work performed entirely by your own tooling:

1. Have a job to report against: use a scheduled job from the jobs list, or create one as a workspace manager: on the Jobs tab of Certificate operations, click Create manual job (or call POST /api/v1/workspaces/{id}/certops/jobs). Choose an operation (issue, renew, deploy, reload, revoke, or noop). Optionally set subjectType + subjectId together (for example a managed certificate id); an issue job takes neither, because it creates the managed certificate itself (see First-time issuance). Copy the job ID (shown truncated and copyable on each job row).

   This flow does not need payload at all: an external executor does the real work itself and only reports the outcome, so nothing here has to describe a host. If you instead want a TokenTimer agent to execute a manual deploy/reload job against an existing certificate, supply the full payload yourself (commandRef, caEndpoint, certPath, dnsZone, dnsProvider, target, sans, and so on, the same shape as the issue payload). A manual renew job is different: the server always resolves every execution field from the certificate's own stored renewal profile, so payload on a renew job accepts only an optional reason string; naming any other field fails the request with CERTOPS_RENEWAL_OVERRIDE_INVALID before a job is even created. Pointing a manual deploy/reload job at a certificate that already has a linked renewal profile does not pull that profile's fields in for you; only the scheduler sweep and Bulk renew derive payload from a linked profile automatically (and the same restriction applies there: bulk renew's shared payload also accepts only reason).

2. Create a machine API token with at least certops:events:write (add certops:evidence:write if the executor uploads evidence). Store the ttx_ value once; it is shown only at creation.

3. Hand the executor the token, your workspace ID, and the job ID (environment variables or a secret store).

4. Run the work locally (renew/deploy/reload on your hosts). The executor posts lifecycle events against that jobId.

5. Review status, timeline, and evidence on the Jobs tab of Certificate operations. Search the audit log with the job ID if you need the create/report trail.

Plan availability

Pro and up

On TokenTimer Cloud, machine API tokens and the executor API are available on the Pro and Team plans, like the rest of CertOps. See Plan limits.

• Free: token management and the executor API return a plan-required response (HTTP 402 PLAN_FEATURE_REQUIRED).
• Pro / Team: full access to machine tokens, manual job creation, executor events, jobs, and evidence.
• Frozen workspaces: if a workspace is frozen (for example after a subscription lapse), executor API calls are blocked with HTTP 403 WORKSPACE_FROZEN until the workspace is reactivated. This check runs before the plan check, so a frozen free workspace also sees WORKSPACE_FROZEN rather than PLAN_FEATURE_REQUIRED.
• Rate limiting: each machine token shares a single rate-limit bucket (120 requests per 60-second window by default) across all executor routes, not a separate budget per route. Exceeding it returns HTTP 429 CERTOPS_MACHINE_RATE_LIMITED with a Retry-After header; executors should back off and retry.

Self-hosted agents and self-hosted cert-manager controllers follow the same Pro/Team plan gate as machine API tokens, each with its own per-plan count cap; see Plan limits for current numbers.

info

Executor event ingestion is idempotent by eventId, so a retried request after a rate limit or network error will not create duplicate timeline entries. Retries never create jobs.

Machine API tokens

Machine API tokens are scoped API keys for non-human callers. They authenticate scripts and executors against the CertOps API without a browser session. Each token is bound to one workspace and carries an explicit set of scopes.

• Format: tokens look like ttx_<id>_<secret>. The ttx_<id> prefix identifies the token in lists; the secret part is never shown again.
• Hashed at rest: TokenTimer stores only a SHA-256 hash of the token. The raw value cannot be recovered, by you or by TokenTimer.
• Show once: the full token is displayed a single time at creation. Copy it immediately into your secret store.
• Optional expiry: set an expiry in days at creation. Expired tokens are refused automatically.
• Last used: the token list shows when each token last authenticated, so you can spot stale or unused credentials.

Creating and revoking tokens

On the Settings tab of Certificate operations, Create token opens a modal for workspace admins to set a name, scopes, and optional expiry, and the token list below it shows existing tokens with revoke controls. Revoking a token disables it immediately; revocation cannot be undone. If an executor still needs access, create a new token and rotate it into your configuration.

Using a token

Send the token as a bearer credential on executor API requests:

Authorization: Bearer ttx_abc123_...

Token scopes

Scopes limit what a token can do. Grant only what the executor needs.

• certops:read — read certificate inventory data.

• certops:events:write — report executor events for existing jobs (status updates only; creating jobs requires a session as workspace manager, not a machine token).

• certops:jobs:read — read executor jobs and their timelines.

• certops:evidence:write — attach evidence to jobs.

• certops:observations:write, certops:provision:execute — controller-only scopes for the cert-manager controller integration; either one requires the token to be bound to exactly one cluster (controllerClusterId) at creation, which is immutable afterward. A bound cluster counts toward the self-hosted controller cap in Plan limits, tracked independently of agent count. See CertOps with ACME and cert-manager.

A typical renewal hook needs certops:events:write and, if it uploads command output, certops:evidence:write. Read scopes are only needed by tools that query state.

Token security

Treat machine API tokens like passwords.

• Never share a token between executors or teams. Create one token per integration so you can revoke them independently.
• Never commit tokens to source control or bake them into images. Inject them from a secret manager or environment variable at runtime.
• Prefer expiry: set an expiry and rotate tokens on a schedule instead of keeping long-lived credentials.
• Revoke on suspicion: if a token may have leaked, revoke it immediately and create a replacement. Check the last-used timestamp for unexpected activity.

warning

The full token value is shown only once at creation. If you lose it, revoke the token and create a new one; TokenTimer cannot recover it.

Executor jobs

An executor job is a recorded unit of certificate work: a first-time issuance, a renewal, a deployment, a service reload, a revocation, or a no-op check. Jobs are created by TokenTimer's renewal scheduler, by a workspace manager, or through the bulk-renew endpoint; executors (agents or your own tooling) perform the work and report against the job. TokenTimer builds a timeline of log entries and evidence for each job.

Where jobs come from

Jobs enter the system in three ways, and each job records its origin in a source field:

• Scheduled automation (automation): the renewal scheduler creates renewal jobs when a managed certificate with a validated renewal profile approaches expiry, subject to approval gates and per-CA caps.
• Manual creation (api): a workspace manager creates a job for break-glass work, hook testing, or work performed by external tooling.
• System (system): jobs created by TokenTimer's control plane outside the renewal sweep.

Machine-token executors cannot create jobs; they only report against an existing jobId. To create a job manually:

• Dashboard: on the Jobs tab of Certificate operations, use Create manual job. Job IDs are listed there (truncated, with copy-to-clipboard) alongside status and timeline. When the list is empty, the tab offers the same control to create the first job.
• API: POST /api/v1/workspaces/{id}/certops/jobs (session authentication; workspace manager role required). Required body field: operation. Optional: subjectType + subjectId together, payload, idempotencyKey. The server sets source to api and ignores any client-supplied source. An issue job is the exception: it takes no subject and idempotencyKey is required (see First-time issuance).

To find existing jobId values your executors should report against, use the jobs list on that page or GET /api/v1/workspaces/{id}/certops/jobs (session authentication, or a machine token with certops:jobs:read).

Manual create vs executor reporting

Workspace managers create jobs with Create manual job or POST /api/v1/workspaces/{id}/certops/jobs (session auth; source is forced to api). Machine-token executor routes only report against an existing jobId; they never create one. An unknown jobId returns 404 CERTOPS_JOB_NOT_FOUND.

Lifecycle actions

• issue: request a brand-new certificate that TokenTimer does not track yet (see First-time issuance).
• renew: obtain a new certificate for an identity.
• deploy: install a certificate on a target.
• reload: reload a service so it serves the new certificate.
• revoke: revoke a certificate with the CA.
• no-op: a check that determined no action was needed.

First-time issuance with an issue job

Every other operation assumes the certificate already exists in your inventory. issue is the one that asks for a brand-new certificate: use it when TokenTimer has no managed certificate for the identity yet and you want the result to land in the inventory rather than only on disk.

TokenTimer creates the managed certificate straight away, at status provisioning, links the job to it, and dispatches the job to a capable agent. The certificate is therefore visible on the certificate surfaces from the moment the job is accepted, before any ACME work has happened.

The request contract differs from a renewal in a few specific ways:

• No subjectType or subjectId: there is no certificate to point at yet, so neither may be sent. TokenTimer creates the managed certificate and links the job to it.
• No payload.certificateId: TokenTimer assigns it.
• idempotencyKey is required, not optional: a retried request must never end up creating a second certificate. It must be 1 to 128 characters of letters, digits, ., _, : or -. Reposting the identical request returns the original job with the same subjectId, and no second certificate or ACME order is created. Reusing the key with a different body is rejected as a conflict (HTTP 409), because one key means one certificate.
• payload.certPath must be an absolute file path, for example /etc/ssl/tokentimer/www.example.com.pem. A relative path is rejected because the agent would resolve it against an unspecified working directory, and a trailing slash is rejected because it names a directory rather than a file. Both are rejected up front on purpose: agent-side they only fail after the ACME order has been placed, which burns a real rate-limited order and leaves the certificate stuck at provisioning. This field does not apply to a windows-iis target; see below.
• No renewalProfile: it is not valid on an issue job. You do not need one, because a profile is derived for you when the issuance succeeds. See Where the renewal profile comes from.
• Everything else is what a renew job takes: commandRef, caEndpoint, certPath, dnsZone, dnsProvider, target, and sans, plus optional acmeKind, keyRotation, reloadService, verifyHost, verifyPort, and chainPath.

By default the agent deploys the full chain (leaf plus intermediates) to certPath, which is what most web servers expect. Set payload.chainPath to a second absolute file path if your deployment target wants the leaf and the intermediate chain in separate files; the agent then writes only the leaf to certPath and the intermediates to chainPath. Unlike certPath, chainPath is not validated up front - a malformed value is only caught agent-side, after the ACME order has already succeeded.

Concurrent retries with the same key are serialized, so two clients racing the same request both receive the same job rather than one of them getting an error.

curl -sS -b cookies.txt -X POST \
  'https://tokentimer.ch/api/v1/workspaces/WORKSPACE_ID/certops/jobs' \
  -H 'Content-Type: application/json' \
  -d '{
    "operation": "issue",
    "idempotencyKey": "issue-www-example-com-1",
    "payload": {
      "target": { "type": "domain", "reference": "www.example.com" },
      "sans": ["www.example.com"],
      "commandRef": "certbot-csr",
      "caEndpoint": "https://acme-v02.api.letsencrypt.org/directory",
      "dnsZone": "example.com",
      "dnsProvider": "cloudflare",
      "certPath": "/etc/ssl/tokentimer/www.example.com.pem"
    }
  }'

The agent does exactly what it does on a renewal: an ACME order with the configured command profile and DNS-01 provider, an atomic deploy to certPath, an optional service reload, then verification. TokenTimer dispatches the work to agents as a renew action, so nothing changes in the agent's own execution path.

target.type: "windows-iis": deploying to a Windows certificate store and IIS binding

A domain target (above) deploys a certificate to a file. A windows-iis target deploys instead to a Windows machine certificate store plus an IIS site binding, with the private key generated and held entirely inside Windows CNG (Cryptography API: Next Generation) — it is never written to disk as a file, .pfx, or otherwise. This is the target type a Windows agent's windows-cert-store-v1/iis-binding-v1 capabilities exist to serve; see Install an agent (Windows) for the Windows-specific installer and the Windows/IIS worked example for a full traced walkthrough.

The shape differs from a domain target in exactly these ways:

• target.store is the Windows certificate store name the certificate is enrolled into, almost always "My" (the machine's Personal store, LocalMachine\My). A different store (for example WebHosting, a convention some IIS deployments use) is supported: the agent enrolls into My first (the only store certreq itself can target) and then mirrors the result into the requested store, removing the My copy afterward. This mirror path is not yet verified against a real Windows host the way the default My path is; the default store is the better-proven choice until that verification lands.
• target.binding replaces certPath entirely: { "site": "<IIS site name>", "port": <int>, "sniHost": "<optional SNI hostname>" }. site and port are required; sniHost is optional and, when present, binds the certificate to that specific hostname on a shared listener (netsh http add sslcert hostnameport=...) rather than to every hostname on that IP:port. This does not, however, guarantee every client receives that certificate: Windows itself always gives a non-SNI (ipport=) binding on the same port precedence over an SNI (hostnameport=) binding, for any client connecting to an address that also has its own ipport= binding, whether that address is a wildcard (0.0.0.0 / [::]) or one specific IP. This is http.sys's own dispatch rule; neither binding's own configuration can override it. The agent checks for a known conflicting binding immediately after deploying an SNI binding and reports one, when found, as a non-fatal precedenceWarning in the deploy evidence: review that warning before assuming the SNI-bound certificate is what every client actually receives.
• No payload.certPath, payload.chainPath, or payload.keyPath at all. Sending any of them alongside a windows-iis target is rejected: there is no filesystem destination to validate.
• No payload.keyMode to set yourself. TokenTimer derives keyMode: "os-store-managed" automatically from target.type, both for the issue job's own dispatch and for the renewal profile it derives afterward.

{
  "operation": "issue",
  "idempotencyKey": "issue-iis-example-1",
  "payload": {
    "target": {
      "type": "windows-iis",
      "reference": "www.example.com",
      "store": "My",
      "binding": { "site": "Default Web Site", "port": 443, "sniHost": "www.example.com" }
    },
    "sans": ["www.example.com"],
    "commandRef": "acmesh-signcsr",
    "caEndpoint": "https://acme-v02.api.letsencrypt.org/directory",
    "dnsZone": "example.com",
    "dnsProvider": "cloudflare"
  }
}

The agent runs the same ACME order as any other issuance (real CA, real DNS-01 challenge), then generates the key and completes enrollment entirely inside CNG (certreq -new / -accept), binds the resulting certificate to the configured IIS site/port/SNI host (netsh http add sslcert), and verifies the live TLS handshake against the actual binding before reporting success. No iisreset or blanket reload is ever issued; only the one configured binding changes. A certificate this target type supersedes is retained (not deleted) for a configurable grace window, so a bad rollback always has something to roll back to.

Issue jobs are only offered to agents that support claim-bound evidence

Promoting a newly issued certificate to active depends on evidence the agent bound to the exact job claim it was executing. An older agent does not bind evidence that way, so an issue job it ran could never be reconciled and the certificate would sit at provisioning forever with a real certificate deployed on the host.

Rather than let that happen, a job is only offered to an agent declaring the evidence-claim-binding-v1 capability when either:

• the job's operation is issue, or
• the job is a renew against a certificate that is still provisioning (which is how you retry a failed issuance).

What you see if your fleet is behind: the issue job stays at pending and is never claimed, with no error anywhere. Upgrade the agents on the hosts that should run issuance. Ordinary renewals of already-active certificates are unaffected and keep being claimed by older agents exactly as before.

On success, the certificate moves from provisioning to active, and its fingerprint, validity dates, serial, subject, and SANs are filled in from the evidence the agent reported for that job. There is no wait for a discovery scan. A renewal profile is also derived at this point so the certificate auto-renews from here on; see Where the renewal profile comes from.

On failure, the certificate stays at provisioning with the failed job and its evidence attached. Nothing is cleaned up automatically. Fix the cause and run an ordinary renew job against the certificate that now exists, or retire the provisioning certificate. Until then it behaves like any other non-terminal certificate: it can be retired (revoked or decommissioned) and it consumes a managed certificate slot (see Plans and limits).

A failed issuance does not raise a renewal-failure alert

Only failed renew jobs raise cert_renewal_failed. A failed issuance has no existing certificate at risk of expiring, so there is no deadline to warn you about. Watch your inventory for certificates stuck at provisioning, not your alert channel, when tracking issuance problems. See Why a failed issuance is not a renewal failure.

When a certificate stays at provisioning

Promotion to active is deliberately strict. It happens only on verify-step evidence that is bound to the current job claim and that carries both a fingerprint and an expiry. Each of those guards a specific way your inventory could end up lying to you:

• Verify-step only, because the agent reports success twice: once when the CA returns the certificate, and once after the deployed file has been read back and fingerprinted. Only the second describes what is actually on the host.
• Claim-bound only, because a job can be attempted more than once and evidence from an earlier attempt outlives it. Without the binding, attempt 2 could be promoted using attempt 1's fingerprint.
• Fingerprint and expiry mandatory, because a certificate with no expiry cannot be renewed on schedule or alerted on. It would look healthy and be silently unmanaged.

When a guard fails, the row stays provisioning and records a reason you can read:

Reason	What happened	What to do
no_claim_bound_verify_evidence	The job reported success but attached no verify-step evidence bound to that claim. Usually an agent that predates evidence-claim-binding-v1, or a job whose verify step did not run	Upgrade the agent, then run a renew job against the certificate. Check whether verifyHost was configured
verify_evidence_missing_fingerprint	Verify evidence arrived but carried no SHA-256 fingerprint, so TokenTimer cannot record which certificate is deployed	Re-run a renew job. If it recurs, inspect the job's evidence on the timeline and check the agent version
verify_evidence_missing_expiry	Verify evidence arrived with no parseable validity end, so the certificate could never be renewed or alerted on	Re-run a renew job and check the deployed file is a valid certificate at the path the agent verified

A provisioning certificate is not a broken certificate: the material may well be live on the host. It means TokenTimer does not yet have proof of what is there, so it refuses to treat it as managed. Re-running a renew job against it is the normal fix, and that retry reconciles through the same path.

A certificate can also reach active and still not auto-renew, when promotion succeeds but renewal-profile derivation declines. That case reads differently: the row's reconciliationReason is set to renewal_profile_<reason> (for example renewal_profile_derivation_failed for an incomplete issue payload), distinct from the provisioning-stuck cases in the table above, and from the CERTOPS_RENEWAL_PROFILE_DERIVATION_DECLINED audit event that records the same decline. Cloud does not yet surface reconciliationReason as a dashboard badge; check it on the certificate via the API (GET .../certops/certificates or the detail route) or via the audit event above.

The same three guards apply again later, on every renewal of an already-active certificate, where the job is to update notAfter and mirror the new fingerprint onto the linked token. A guard failure there sets reconciliationReason and writes a CERTOPS_CERTIFICATE_RENEWAL_UNRECONCILED audit event, the same as above. This matters more than it sounds: the renewal scheduler's idempotency key is (certificateId, notAfter), so if notAfter never advances, the next scheduling sweep can re-derive the same key and repeat the "successful" renewal against the same CA.

Status transitions

Jobs move through statuses such as pending, claimed, running, and terminal states succeeded, failed, rejected, blocked, cancelled, or orphaned_unknown_effect. Jobs that require approval start at pending_approval and move to approved before becoming claimable. Transitions are monotonic: a job never moves backward, and a terminal status is final. Events that would regress a job are recorded in the log without changing the job status.

orphaned_unknown_effect is the one status that is neither success nor plain failure: an agent went silent after it may already have changed something on a host, so TokenTimer refuses to guess and asks you to look. See Reconciling interrupted jobs.

How a job is bound to one agent

Two independent mechanisms decide which agent runs a job, and both matter once you run more than one agent.

1. Claim exclusivity. Claiming is transactional and row-locked, so if several agents poll at the same instant each gets a different job and never the same one. Once claimed, the job carries a lease held by that agent and nobody else can claim it while the lease is alive.

2. Assignment. Exclusivity alone does not mean the right agent claimed the job. A job with no assignment and no target selector is claimable by any online agent that supports the operation, including one with no relationship to the certificate. So for renew, deploy, reload, and revoke jobs against a certificate an agent discovered on its own filesystem, TokenTimer pins the job to that discovering agent at creation time. Only that agent can ever claim it.

This matters because certificate work is host-specific. Renewing /etc/nginx/ssl/site.pem only means something on the host where that file exists; another agent claiming the job would fail on a missing path, and you would get a renewal failure alert for what was really a routing mistake.

Consequences worth knowing:

• If the owning agent is offline, its jobs wait at pending rather than being picked up by a healthy agent, because a different host is not a valid substitute. Check the Agent fleet panel first when a job seems stuck.
• If you replaced the host (rebuilt it, re-registered the agent, migrated the service), jobs pinned to the old agent will never run. Assign the new agent explicitly, or let it rediscover the certificate so future jobs pin correctly.
• An explicit agent assignment always wins over the automatic one, which is the supported way to hand a certificate to a different agent.
• Certificates with no agent custody are rejected at creation instead of being assigned at all. See Not every certificate in inventory can be renewed for you.
• Target selectors still apply on top of assignment. They are the coarser tool (route by role or environment); assignment is the exact one.

Reviewing jobs in the dashboard

The Jobs tab of Certificate operations lists executor jobs for the workspace with status, operation, and timing. Each row shows a truncated, copyable job ID. Workspace managers can open Create manual job from there (including when the list is empty). Selecting a job opens its timeline: every reported event in order, plus attached evidence with redaction markers where secrets were removed.

Reporting events

Executors report work by posting executor events with a machine API token. Each event carries a caller-chosen eventId, a jobId, the workspace ID, a status, an event type, and a timestamp.

• Event types: job.accepted, job.started, job.progress, job.completed, job.failed, job.rejected, and evidence.attached.
• Idempotent by eventId: sending the same eventId with the same payload again is a safe no-op, so executors can retry freely. Reusing an eventId with a different payload is rejected as a conflict.
• Events target an existing job: every event must reference a jobId that already exists (created by a workspace manager or by TokenTimer automation). Machine tokens cannot create jobs. Posting to an unknown jobId is rejected with CERTOPS_JOB_NOT_FOUND.
• Unknown fields are rejected: extra top-level fields in the request body are rejected rather than silently dropped, so a typo in a field name surfaces immediately instead of being ignored.
• Inline evidence: an event may include up to 16 evidence items, and public metadata name/value pairs (secret-looking names are refused).

Requests need the certops:events:write scope. Evidence-bearing requests also need certops:evidence:write.

Evidence and redaction

Evidence entries attach proof of work to a job: command output, ACME logs, deployment confirmations, or certificate fingerprints. Evidence appears on the job timeline in the dashboard.

Automatic secret redaction

Evidence output is scanned before storage. Values that look like generic secrets (passwords, API keys, bearer tokens) are replaced with a redaction placeholder, and the entry is marked with redactionApplied so the timeline shows that redaction occurred. Redaction is best effort; do not rely on it as your only safeguard.

Evidence output is capped at 64 KiB before redaction runs; oversized output is rejected outright with CERTOPS_EVIDENCE_OUTPUT_TOO_LARGE rather than truncated. The stored outputSizeBytes and outputSha256 both describe the redacted output that was actually persisted, not your original submission, since redaction can shrink or grow the byte length.

Strict no-private-key rule

Private keys are different: they are never redacted and stored, they are rejected outright. Any event, evidence, or metadata payload containing private key material (PEM key blocks, key-material field names, PFX or JKS blobs) fails with HTTP 422 and PRIVATE_KEY_MATERIAL_REJECTED. Nothing from the request is stored.

warning

Never pipe raw certbot or openssl output that may include key files into evidence. Filter your logs so only public material and status output is uploaded.

Executor example

A minimal executor posts one event when work completes. This example reports a successful renewal from a certbot deploy-hook. The jobId must already exist in TokenTimer (create one with Create manual job or POST /api/v1/workspaces/{id}/certops/jobs, or pick one from the jobs list). Look it up beforehand and pass it to the hook. Do not invent a UUID locally; unknown job ids are rejected with 404 CERTOPS_JOB_NOT_FOUND:

#!/bin/sh
# /etc/letsencrypt/renewal-hooks/deploy/tokentimer-report.sh
# TOKENTIMER_TOKEN comes from your secret store, never hardcode it.
# TOKENTIMER_JOB_ID is an existing job id from Create manual job /
# POST .../certops/jobs, or from the jobs list (GET .../jobs, scope certops:jobs:read).

JOB_ID="${TOKENTIMER_JOB_ID:?set TOKENTIMER_JOB_ID to an existing job id}"
EVENT_ID="$JOB_ID-completed"

curl -s -X POST \
  'https://tokentimer.ch/api/v1/certops/executor/events' \
  -H "Authorization: Bearer $TOKENTIMER_TOKEN" \
  -H 'Content-Type: application/json' \
  -d "{
    \"schemaVersion\": 1,
    \"eventId\": \"$EVENT_ID\",
    \"jobId\": \"$JOB_ID\",
    \"workspaceId\": \"WORKSPACE_ID\",
    \"executorId\": \"certbot-$(hostname)\",
    \"status\": \"succeeded\",
    \"eventType\": \"job.completed\",
    \"occurredAt\": \"$(date -u +%Y-%m-%dT%H:%M:%SZ)\",
    \"message\": \"certbot renewed $RENEWED_DOMAINS\"
  }"

For a longer-running job, post job.started when work begins and job.completed or job.failed when it ends, reusing the same jobId with distinct eventId values. Retries can resend the exact same request safely.

The same pattern works for acme.sh hooks, Ansible handlers, or any script that can send an HTTPS request.

API reference

Executor API (machine token auth)

Authenticated with Authorization: Bearer ttx_.... Pro/Team plans only; blocked for frozen workspaces; rate limited per token.

• POST /api/v1/certops/executor/events — ingest an executor event for an existing job (scope certops:events:write).
• POST /api/v1/certops/jobs/{jobId}/events — append an event to an existing job (scope certops:events:write).
• POST /api/v1/certops/jobs/{jobId}/evidence — attach evidence to a job (scope certops:evidence:write).

Each machine token shares a single rate-limit bucket (120 requests / 60s by default) across all three executor routes above, not a separate budget per route; exceeding it returns HTTP 429 CERTOPS_MACHINE_RATE_LIMITED. Common error codes: 401 CERTOPS_API_TOKEN_UNAUTHORIZED (bad/revoked/expired token), 403 CERTOPS_API_TOKEN_SCOPE_DENIED (missing scope), 404 CERTOPS_JOB_NOT_FOUND (unknown jobId), 409 CERTOPS_EXECUTOR_EVENT_CONFLICT (same eventId replayed with a different payload), 422 PRIVATE_KEY_MATERIAL_REJECTED, and 413 CERTOPS_EVIDENCE_OUTPUT_TOO_LARGE.

Workspace API (dashboard session auth)

Workspace-scoped endpoints under /api/v1/workspaces/{id}/certops, used by the dashboard and available to session-authenticated callers:

• GET /jobs — list executor jobs (limit, offset, optional status filter).
• POST /jobs — create an executor job manually (workspace manager; body: { operation, subjectType?, subjectId?, payload? }; source is forced to api; an issue job takes no subject and requires idempotencyKey, see First-time issuance).
• GET /jobs/{jobId} — fetch one job.
• GET /jobs/{jobId}/log — list the job timeline entries.
• GET /jobs/{jobId}/evidence — list evidence attached to the job.
• GET /tokens — list machine API tokens (prefix, scopes, status, expiry, last used).
• POST /tokens — create a token; the response includes the raw ttx_ token once.
• POST /tokens/{tokenId}/revoke — revoke a token immediately.

See Create a manual CertOps job for the create-job request schema, the API reference for authentication and pagination conventions, and Certificates for the inventory endpoints.