Skip to main content

Certificate Automation

Overview​

CertOps automation renews certificates end to end. TokenTimer Cloud schedules renewal work before certificates expire, routes each job to an eligible executor in your environment, and records what happened: issuance, deployment, reload, verification, and evidence. Approval gates and a workspace kill switch keep humans in control of what runs.

The control plane plans, signs, and audits; execution happens where your certificates live. TokenTimer never connects into your network and never touches private keys: keys are generated and used on your hosts (or inside your cluster with cert-manager), and any payload containing key material is rejected outright.

Three execution modes are available, and they can be mixed freely in one workspace:

  • TokenTimer agent: an outbound-only daemon on your hosts that verifies signed jobs, runs ACME renewals with DNS-01, deploys atomically, reloads services, and verifies the served certificate. See Deploying an agent.
  • cert-manager controller: a Kubernetes controller that observes cert-manager certificates and, in opt-in provisioning mode, creates authorized Certificate resources. Key custody stays with cert-manager. See CertOps with ACME and cert-manager.
  • External executors: your own renewal scripts, certbot/acme.sh hooks, and CI pipelines authenticate with a machine API token and report against executor jobs. See Connecting external executors.

You manage agents, machine API tokens, and jobs on the Certificate operations page (/certops/operations in the dashboard), reached from the Control Center footer link or from Workspace Preferences when CertOps is enabled. See Certificates for the managed inventory these jobs attach to.

Scheduled renewals and approvals​

Renewal work is planned by TokenTimer, not by cron jobs you have to remember:

  • Renewal scheduler: when a managed certificate approaches expiry, the scheduler creates a renewal job automatically. A job is only scheduled when the certificate has a complete, validated renewal profile (CA endpoint, command profile, target), so automation never creates unexecutable work.
  • Eligible-agent routing: each job is routed to an agent whose declared capabilities and target selectors match the work. Agents claim jobs over an outbound-only protocol; the control plane never connects to your hosts. If an agent stops reporting mid-job, TokenTimer only auto-retries when no side effects were possible, and otherwise flags the job for review (see Reconciling interrupted jobs).
  • Signed dispatch: every dispatched job is signed with a control-plane Ed25519 key. Agents verify the signature against a pinned key, enforce a validity window, and keep a replay cache, so a forged or replayed job never executes.
  • Approval gates: jobs can require an explicit approval before dispatch. A requester cannot approve their own job, the approval is bound to the exact job payload hash, and every decision lands in the audit log. See Approval gates for the full non-requester and invalidation rules.
  • Per-CA caps and bulk renew: sweep-time caps bound the number of in-flight renewals per CA endpoint so the scheduler cannot flood one CA, and a bulk-renew endpoint pushes many certificates through the same validated creation path.
  • Kill switch: a workspace-level pause blocks new CertOps jobs and dispatch instantly, without deleting queued work or history. Inventory, status, audit, and evidence reporting keep working while paused.
  • Renewal failure alerting: failed renewals raise a cert_renewal_failed alert through your normal TokenTimer alerting channels. See Certificate renewal failures for when it fires and how it routes.

Deploying an agent​

The TokenTimer agent is the execution plane for hosts: a zero-dependency Node.js daemon that is outbound-only (it opens HTTPS connections to TokenTimer Cloud and never accepts inbound connections).

  1. On Certificate operations, open the Deploy an agent panel and create a bootstrap token (shown exactly once).
  2. Copy the generated install command and run it on the target host. The installer creates a dedicated system user and a hardened systemd unit.
  3. Watch the Agent fleet panel flip the agent to registered on its first heartbeat. The fleet panel shows status, heartbeat, and compatibility, and lets you retire an agent.

For install options, write-path and reload permissions, upgrades, and troubleshooting, see Install and upgrade an agent.

What the agent does on a renewal job, entirely on your host:

  • generates (or reuses) the private key and builds the CSR locally; keys never leave the host,
  • drives certbot or acme.sh in CSR mode against your allowlisted CA endpoint (public ACME CAs or internal, ACME-compatible PKI, including private CA bundles),
  • solves DNS-01 challenges with native solvers for 11 providers (Cloudflare, Route 53, Azure DNS, Google Cloud DNS, RFC 2136, acme-dns, OVHcloud, Hetzner, Infomaniak, Exoscale, PowerDNS); DNS credentials live in files on the agent host and never leave it (see DNS-01 providers),
  • deploys atomically with backup and automatic rollback, then validate-then-reloads nginx, Apache, or HAProxy,
  • verifies the deployment by fingerprinting the installed certificate, with an optional live TLS probe against the endpoint,
  • reports each step as evidence on the job timeline.

The agent enforces its own local policy: allowlists for commands, paths, CA endpoints, and DNS zones/providers that the control plane cannot override, and key-export requests are rejected unconditionally. A dry-run mode exercises the full trust chain and reports the planned steps without side effects, so you can validate a rollout safely.

Connecting external executors​

External tooling remains a fully supported execution mode: anything that can send an HTTPS request can report certificate work against a job. Scheduled renewal jobs are created by TokenTimer automation; you can also create a job manually for break-glass work, hook testing, or work performed entirely by your own tooling:

  1. Have a job to report against: use a scheduled job from the jobs list, or create one as a workspace manager: on Certificate operations, click Create manual job (or call POST /api/v1/workspaces/{id}/certops/jobs). Choose an operation (renew, deploy, reload, revoke, or noop). Optionally set subjectType + subjectId together (for example a managed certificate id). Copy the job ID (shown truncated and copyable on each job row).
  2. Create a machine API token with at least certops:events:write (add certops:evidence:write if the executor uploads evidence). Store the ttx_ value once; it is shown only at creation.
  3. Hand the executor the token, your workspace ID, and the job ID (environment variables or a secret store).
  4. Run the work locally (renew/deploy/reload on your hosts). The executor posts lifecycle events against that jobId.
  5. Review status, timeline, and evidence on Certificate operations. Search the audit log with the job ID if you need the create/report trail.

Plan availability​

Pro and up

On TokenTimer Cloud, machine API tokens and the executor API are available on the Pro and Team plans, like the rest of CertOps. See Plan limits.

  • Free: token management and the executor API return a plan-required response (HTTP 402 PLAN_FEATURE_REQUIRED).
  • Pro / Team: full access to machine tokens, manual job creation, executor events, jobs, and evidence.
  • Frozen workspaces: if a workspace is frozen (for example after a subscription lapse), executor API calls are blocked with HTTP 403 WORKSPACE_FROZEN until the workspace is reactivated. This check runs before the plan check, so a frozen free workspace also sees WORKSPACE_FROZEN rather than PLAN_FEATURE_REQUIRED.
  • Rate limiting: each machine token shares a single rate-limit bucket (120 requests per 60-second window by default) across all executor routes, not a separate budget per route. Exceeding it returns HTTP 429 CERTOPS_MACHINE_RATE_LIMITED with a Retry-After header; executors should back off and retry.
info

Executor event ingestion is idempotent by eventId, so a retried request after a rate limit or network error will not create duplicate timeline entries. Retries never create jobs.

Machine API tokens​

Machine API tokens are scoped API keys for non-human callers. They authenticate scripts and executors against the CertOps API without a browser session. Each token is bound to one workspace and carries an explicit set of scopes.

  • Format: tokens look like ttx_<id>_<secret>. The ttx_<id> prefix identifies the token in lists; the secret part is never shown again.
  • Hashed at rest: TokenTimer stores only a SHA-256 hash of the token. The raw value cannot be recovered, by you or by TokenTimer.
  • Show once: the full token is displayed a single time at creation. Copy it immediately into your secret store.
  • Optional expiry: set an expiry in days at creation. Expired tokens are refused automatically.
  • Last used: the token list shows when each token last authenticated, so you can spot stale or unused credentials.

Creating and revoking tokens​

On the Certificate operations page, the API tokens panel lets workspace admins create a token with a name, scopes, and optional expiry. Revoking a token disables it immediately; revocation cannot be undone. If an executor still needs access, create a new token and rotate it into your configuration.

Using a token​

Send the token as a bearer credential on executor API requests:

Authorization: Bearer ttx_abc123_...

Token scopes​

Scopes limit what a token can do. Grant only what the executor needs.

  • certops:read — read certificate inventory data.
  • certops:events:write — report executor events for existing jobs (status updates only; creating jobs requires a session as workspace manager, not a machine token).
  • certops:jobs:read — read executor jobs and their timelines.
  • certops:evidence:write — attach evidence to jobs.

A typical renewal hook needs certops:events:write and, if it uploads command output, certops:evidence:write. Read scopes are only needed by tools that query state.

Token security​

Treat machine API tokens like passwords.

  • Never share a token between executors or teams. Create one token per integration so you can revoke them independently.
  • Never commit tokens to source control or bake them into images. Inject them from a secret manager or environment variable at runtime.
  • Prefer expiry: set an expiry and rotate tokens on a schedule instead of keeping long-lived credentials.
  • Revoke on suspicion: if a token may have leaked, revoke it immediately and create a replacement. Check the last-used timestamp for unexpected activity.
warning

The full token value is shown only once at creation. If you lose it, revoke the token and create a new one; TokenTimer cannot recover it.

Executor jobs​

An executor job is a recorded unit of certificate work: a renewal, a deployment, a service reload, a revocation, or a no-op check. Jobs are created by TokenTimer's renewal scheduler, by a workspace manager, or through the bulk-renew endpoint; executors (agents or your own tooling) perform the work and report against the job. TokenTimer builds a timeline of log entries and evidence for each job.

Where jobs come from​

Jobs enter the system in three ways, and each job records its origin in a source field:

  • Scheduled automation (automation): the renewal scheduler creates renewal jobs when a managed certificate with a validated renewal profile approaches expiry, subject to approval gates and per-CA caps.
  • Manual creation (api): a workspace manager creates a job for break-glass work, hook testing, or work performed by external tooling.
  • System (system): jobs created by TokenTimer's control plane outside the renewal sweep.

Machine-token executors cannot create jobs; they only report against an existing jobId. To create a job manually:

  • Dashboard: on the Certificate operations page, use Create manual job. Job IDs are listed there (truncated, with copy-to-clipboard) alongside status and timeline. When the list is empty, the page offers the same control to create the first job.
  • API: POST /api/v1/workspaces/{id}/certops/jobs (session authentication; workspace manager role required). Required body field: operation. Optional: subjectType + subjectId together, payload, idempotencyKey. The server sets source to api and ignores any client-supplied source.

To find existing jobId values your executors should report against, use the jobs list on that page or GET /api/v1/workspaces/{id}/certops/jobs (session authentication, or a machine token with certops:jobs:read).

Manual create vs executor reporting

Workspace managers create jobs with Create manual job or POST /api/v1/workspaces/{id}/certops/jobs (session auth; source is forced to api). Machine-token executor routes only report against an existing jobId; they never create one. An unknown jobId returns 404 CERTOPS_JOB_NOT_FOUND.

Lifecycle actions​

  • renew: obtain a new certificate for an identity.
  • deploy: install a certificate on a target.
  • reload: reload a service so it serves the new certificate.
  • revoke: revoke a certificate with the CA.
  • no-op: a check that determined no action was needed.

Status transitions​

Jobs move through statuses such as pending, claimed, running, and terminal states succeeded, failed, rejected, blocked, or cancelled. Jobs that require approval start at pending_approval and move to approved before becoming claimable. Transitions are monotonic: a job never moves backward, and a terminal status is final. Events that would regress a job are recorded in the log without changing the job status.

Reviewing jobs in the dashboard​

The Certificate operations page lists executor jobs for the workspace with status, operation, and timing. Each row shows a truncated, copyable job ID. Workspace managers can open Create manual job from the jobs panel (including when the list is empty). Selecting a job opens its timeline: every reported event in order, plus attached evidence with redaction markers where secrets were removed.

Reporting events​

Executors report work by posting executor events with a machine API token. Each event carries a caller-chosen eventId, a jobId, the workspace ID, a status, an event type, and a timestamp.

  • Event types: job.accepted, job.started, job.progress, job.completed, job.failed, job.rejected, and evidence.attached.
  • Idempotent by eventId: sending the same eventId with the same payload again is a safe no-op, so executors can retry freely. Reusing an eventId with a different payload is rejected as a conflict.
  • Events target an existing job: every event must reference a jobId that already exists (created by a workspace manager or by TokenTimer automation). Machine tokens cannot create jobs. Posting to an unknown jobId is rejected with CERTOPS_JOB_NOT_FOUND.
  • Unknown fields are rejected: extra top-level fields in the request body are rejected rather than silently dropped, so a typo in a field name surfaces immediately instead of being ignored.
  • Inline evidence: an event may include up to 16 evidence items, and public metadata name/value pairs (secret-looking names are refused).

Requests need the certops:events:write scope. Evidence-bearing requests also need certops:evidence:write.

Evidence and redaction​

Evidence entries attach proof of work to a job: command output, ACME logs, deployment confirmations, or certificate fingerprints. Evidence appears on the job timeline in the dashboard.

Automatic secret redaction​

Evidence output is scanned before storage. Values that look like generic secrets (passwords, API keys, bearer tokens) are replaced with a redaction placeholder, and the entry is marked with redactionApplied so the timeline shows that redaction occurred. Redaction is best effort; do not rely on it as your only safeguard.

Evidence output is capped at 64 KiB before redaction runs; oversized output is rejected outright with CERTOPS_EVIDENCE_OUTPUT_TOO_LARGE rather than truncated. The stored outputSizeBytes and outputSha256 both describe the redacted output that was actually persisted, not your original submission, since redaction can shrink or grow the byte length.

Strict no-private-key rule​

Private keys are different: they are never redacted and stored, they are rejected outright. Any event, evidence, or metadata payload containing private key material (PEM key blocks, key-material field names, PFX or JKS blobs) fails with HTTP 422 and PRIVATE_KEY_MATERIAL_REJECTED. Nothing from the request is stored.

warning

Never pipe raw certbot or openssl output that may include key files into evidence. Filter your logs so only public material and status output is uploaded.

Executor example​

A minimal executor posts one event when work completes. This example reports a successful renewal from a certbot deploy-hook. The jobId must already exist in TokenTimer (create one with Create manual job or POST /api/v1/workspaces/{id}/certops/jobs, or pick one from the jobs list). Look it up beforehand and pass it to the hook. Do not invent a UUID locally; unknown job ids are rejected with 404 CERTOPS_JOB_NOT_FOUND:

#!/bin/sh
# /etc/letsencrypt/renewal-hooks/deploy/tokentimer-report.sh
# TOKENTIMER_TOKEN comes from your secret store, never hardcode it.
# TOKENTIMER_JOB_ID is an existing job id from Create manual job /
# POST .../certops/jobs, or from the jobs list (GET .../jobs, scope certops:jobs:read).

JOB_ID="${TOKENTIMER_JOB_ID:?set TOKENTIMER_JOB_ID to an existing job id}"
EVENT_ID="$JOB_ID-completed"

curl -s -X POST \
  'https://tokentimer.ch/api/v1/certops/executor/events' \
  -H "Authorization: Bearer $TOKENTIMER_TOKEN" \
  -H 'Content-Type: application/json' \
  -d "{
    \"schemaVersion\": 1,
    \"eventId\": \"$EVENT_ID\",
    \"jobId\": \"$JOB_ID\",
    \"workspaceId\": \"WORKSPACE_ID\",
    \"executorId\": \"certbot-$(hostname)\",
    \"status\": \"succeeded\",
    \"eventType\": \"job.completed\",
    \"occurredAt\": \"$(date -u +%Y-%m-%dT%H:%M:%SZ)\",
    \"message\": \"certbot renewed $RENEWED_DOMAINS\"
  }"

For a longer-running job, post job.started when work begins and job.completed or job.failed when it ends, reusing the same jobId with distinct eventId values. Retries can resend the exact same request safely.

The same pattern works for acme.sh hooks, Ansible handlers, or any script that can send an HTTPS request.

API reference​

Executor API (machine token auth)​

Authenticated with Authorization: Bearer ttx_.... Pro/Team plans only; blocked for frozen workspaces; rate limited per token.

  • POST /api/v1/certops/executor/events — ingest an executor event for an existing job (scope certops:events:write).
  • POST /api/v1/certops/jobs/{jobId}/events — append an event to an existing job (scope certops:events:write).
  • POST /api/v1/certops/jobs/{jobId}/evidence — attach evidence to a job (scope certops:evidence:write).

Each machine token shares a single rate-limit bucket (120 requests / 60s by default) across all three executor routes above, not a separate budget per route; exceeding it returns HTTP 429 CERTOPS_MACHINE_RATE_LIMITED. Common error codes: 401 CERTOPS_API_TOKEN_UNAUTHORIZED (bad/revoked/expired token), 403 CERTOPS_API_TOKEN_SCOPE_DENIED (missing scope), 404 CERTOPS_JOB_NOT_FOUND (unknown jobId), 409 CERTOPS_EXECUTOR_EVENT_CONFLICT (same eventId replayed with a different payload), 422 PRIVATE_KEY_MATERIAL_REJECTED, and 413 CERTOPS_EVIDENCE_OUTPUT_TOO_LARGE.

Workspace API (dashboard session auth)​

Workspace-scoped endpoints under /api/v1/workspaces/{id}/certops, used by the dashboard and available to session-authenticated callers:

  • GET /jobs — list executor jobs (limit, offset, optional status filter).
  • POST /jobs — create an executor job manually (workspace manager; body: { operation, subjectType?, subjectId?, payload? }; source is forced to api).
  • GET /jobs/{jobId} — fetch one job.
  • GET /jobs/{jobId}/log — list the job timeline entries.
  • GET /jobs/{jobId}/evidence — list evidence attached to the job.
  • GET /tokens — list machine API tokens (prefix, scopes, status, expiry, last used).
  • POST /tokens — create a token; the response includes the raw ttx_ token once.
  • POST /tokens/{tokenId}/revoke — revoke a token immediately.

See Create a manual CertOps job for the create-job request schema, the API reference for authentication and pagination conventions, and Certificates for the inventory endpoints.