Certificate Automation

Overview

CertOps automation lets external tools report certificate work back to your TokenTimer instance. Renewal scripts, certbot or acme.sh hooks, and CI pipelines authenticate with a machine API token and report against executor jobs: what was renewed, deployed, reloaded, or revoked, when, and with what outcome. A workspace manager creates each job first; the executor only posts events for an existing jobId.

TokenTimer stays an observer and system of record. It never runs renewals, never connects to your hosts, and never touches private keys. Executors do the work where the certificates live; TokenTimer keeps the timeline, status, and evidence.

You manage tokens and review jobs on the Certificate operations page (/certops/operations in the dashboard), reached from the Control Center footer link or from Workspace Preferences when CertOps is enabled. See Certificates for the managed inventory these jobs attach to.

Enabling automation

Machine tokens and the executor API ship with CertOps. Enable CertOps on your install, then restart the API and worker processes:

CERTOPS_ENABLED=true

When CertOps is disabled, token management and executor endpoints return 404 and the Certificate operations page is not reachable. There is no plan gating on self-hosted installs; all workspaces on your instance can use automation once CertOps is enabled.

For the full list of environment variables, see the Configuration Reference.

Typical workflow

1. Create a job as a workspace manager: on Certificate operations, click Create manual job (or call POST /api/v1/workspaces/{id}/certops/jobs). Choose an operation (renew, deploy, reload, revoke, or noop). Optionally set subjectType + subjectId together (for example a managed certificate id). Copy the returned job ID (also shown truncated and copyable on each job row).
2. Create a machine API token with at least certops:events:write (add certops:evidence:write if the executor uploads evidence). Store the ttx_ value once; it is shown only at creation.
3. Hand the executor the token, your workspace ID, and the job ID (environment variables or a secret store).
4. Run the work locally (renew/deploy/reload on your hosts). The executor posts lifecycle events against that jobId.
5. Review status, timeline, and evidence on Certificate operations. Search the audit log with the job ID if you need the create/report trail.

Until automated scheduling ships, step 1 is how every jobId enters the system for break-glass and hook testing.

Machine API tokens

Machine API tokens are scoped API keys for non-human callers. They authenticate scripts and executors against the CertOps API without a browser session. Each token is bound to one workspace and carries an explicit set of scopes.

• Format: tokens look like ttx_<id>_<secret>. The ttx_<id> prefix identifies the token in lists; the secret part is never shown again.
• Hashed at rest: only a SHA-256 hash of the token is stored. The raw value cannot be recovered.
• Show once: the full token is displayed a single time at creation. Copy it immediately into your secret store.
• Optional expiry: set an expiry in days at creation. Expired tokens are refused automatically.
• Last used: the token list shows when each token last authenticated, so you can spot stale or unused credentials.

Create and revoke tokens from the API tokens panel on the Certificate operations page. Revocation is immediate and cannot be undone. Send the token as a bearer credential:

Authorization: Bearer ttx_abc123_...

Token scopes

Scopes limit what a token can do. Grant only what the executor needs.

• certops:read read certificate inventory data
• certops:events:write report executor events for existing jobs (status updates only; creating jobs requires a session as workspace manager, not a machine token)
• certops:jobs:read read executor jobs and their timelines
• certops:evidence:write attach evidence to jobs

A typical renewal hook needs certops:events:write and, if it uploads command output, certops:evidence:write. Read scopes are only needed by tools that query state.

Token security

Treat machine API tokens like passwords.

• Never share a token between executors or teams. Create one token per integration so you can revoke them independently.
• Never commit tokens to source control or bake them into images. Inject them from a secret manager or environment variable at runtime.
• Prefer expiry: set an expiry and rotate tokens on a schedule instead of keeping long-lived credentials.
• Revoke on suspicion: if a token may have leaked, revoke it immediately and create a replacement. Check the last-used timestamp for unexpected activity.

The full token value is shown only once at creation. If you lose it, revoke the token and create a new one; the raw value cannot be recovered from the stored hash.

Executor jobs

An executor job is a recorded unit of certificate work performed outside TokenTimer: a renewal, a deployment, a service reload, a revocation, or a no-op check. A workspace manager (or, later, TokenTimer automation) creates the job record; external executors report against that existing job by posting events. TokenTimer builds a timeline of log entries and evidence for each job.

Where jobs come from

Jobs are created by a workspace manager or by TokenTimer's control plane. Machine-token executors cannot create jobs; they only report against an existing jobId. Each job records its origin in a source field (for example api for manual creation, or system / automation).

In the current release, automated job planning (for example scheduling a renewal job when a managed certificate approaches expiry) is not yet active, so a fresh workspace starts with an empty jobs list. Create a jobId manually when you need one:

• Dashboard: Certificate operations → Create manual job. Job IDs are listed there (truncated, copyable).
• API: POST /api/v1/workspaces/{id}/certops/jobs (session authentication; workspace manager role; source forced to api).

To look up existing jobs for executor hooks, use that page or GET /api/v1/workspaces/{id}/certops/jobs. Posting executor events for an unknown jobId returns 404 CERTOPS_JOB_NOT_FOUND rather than creating it.

The Certificate operations page lists executor jobs for the workspace with status, operation, and timing. Workspace managers can open Create manual job from the jobs panel, including when the list is empty. Selecting a job opens its timeline: every reported event in order, plus attached evidence with redaction markers where secrets were removed.

Jobs move through statuses such as accepted, claimed, running, and terminal states succeeded, failed, rejected, blocked, or cancelled. Transitions are monotonic: a job never moves backward, and a terminal status is final. Events that would regress a job are recorded in the log without changing the job status.

Reporting events

Executors report work by posting executor events with a machine API token. Each event carries a caller-chosen eventId, a jobId, the workspace ID, a status, an event type, and a timestamp.

• Event types: job.accepted, job.started, job.progress, job.completed, job.failed, job.rejected, and evidence.attached.
• Idempotent by eventId: sending the same eventId with the same payload again is a safe no-op, so executors can retry freely. Reusing an eventId with a different payload is rejected as a conflict.
• Events target an existing job: every event must reference a jobId that already exists (created by a workspace manager or by TokenTimer automation). Machine tokens cannot create jobs. Posting to an unknown jobId is rejected with CERTOPS_JOB_NOT_FOUND.
• Unknown fields are rejected: extra top-level fields in the request body are rejected rather than silently dropped, so a typo in a field name surfaces immediately instead of being ignored.
• Inline evidence: an event may include up to 16 evidence items, and public metadata name/value pairs (secret-looking names are refused).

Requests need the certops:events:write scope. Evidence-bearing requests also need certops:evidence:write.

Evidence and redaction

Evidence entries attach proof of work to a job: command output, ACME logs, deployment confirmations, or certificate fingerprints. Evidence appears on the job timeline in the dashboard.

Evidence output is scanned before storage. Values that look like generic secrets (passwords, API keys, bearer tokens) are replaced with a redaction placeholder, and the entry is marked with redactionApplied so the timeline shows that redaction occurred. Redaction is best effort; do not rely on it as your only safeguard.

Evidence output is capped at 64 KiB before redaction runs; oversized output is rejected outright with CERTOPS_EVIDENCE_OUTPUT_TOO_LARGE rather than truncated. The stored outputSizeBytes and outputSha256 both describe the redacted output that was actually persisted, not your original submission, since redaction can shrink or grow the byte length.

Private keys are different: they are never redacted and stored, they are rejected outright. Any event, evidence, or metadata payload containing private key material (PEM key blocks, key-material field names, PFX or JKS blobs) fails with HTTP 422 and PRIVATE_KEY_MATERIAL_REJECTED. Nothing from the request is stored.

Never pipe raw certbot or openssl output that may include key files into evidence. Filter your logs so only public material and status output is uploaded.

Executor example

A minimal executor posts one event when work completes. This example reports a successful renewal from a certbot deploy-hook. The jobId must already exist in TokenTimer (create one with Create manual job or POST /api/v1/workspaces/{id}/certops/jobs, or pick one from the jobs list). Look it up beforehand and pass it to the hook. Do not invent a UUID locally; unknown job ids are rejected with 404 CERTOPS_JOB_NOT_FOUND:

#!/bin/sh
# /etc/letsencrypt/renewal-hooks/deploy/tokentimer-report.sh
# TOKENTIMER_TOKEN comes from your secret store, never hardcode it.
# TOKENTIMER_JOB_ID is an existing job id from Create manual job /
# POST .../certops/jobs, or from the jobs list (GET .../jobs, scope certops:jobs:read).

JOB_ID="${TOKENTIMER_JOB_ID:?set TOKENTIMER_JOB_ID to an existing job id}"
EVENT_ID="$JOB_ID-completed"

curl -s -X POST \
  'https://your-tokentimer.example/api/v1/certops/executor/events' \
  -H "Authorization: Bearer $TOKENTIMER_TOKEN" \
  -H 'Content-Type: application/json' \
  -d "{
    \"schemaVersion\": 1,
    \"eventId\": \"$EVENT_ID\",
    \"jobId\": \"$JOB_ID\",
    \"workspaceId\": \"WORKSPACE_ID\",
    \"executorId\": \"certbot-$(hostname)\",
    \"status\": \"succeeded\",
    \"eventType\": \"job.completed\",
    \"occurredAt\": \"$(date -u +%Y-%m-%dT%H:%M:%SZ)\",
    \"message\": \"certbot renewed $RENEWED_DOMAINS\"
  }"

For a longer-running job, post job.started when work begins and job.completed or job.failed when it ends, reusing the same jobId with distinct eventId values. Retries can resend the exact same request safely. The same pattern works for acme.sh hooks, Ansible handlers, or any script that can send an HTTPS request.

API reference

Executor endpoints authenticate with Authorization: Bearer ttx_... and are rate limited per token:

• POST /api/v1/certops/executor/events ingest an executor event for an existing job
• POST /api/v1/certops/jobs/:jobId/events append an event to an existing job
• POST /api/v1/certops/jobs/:jobId/evidence attach evidence to a job

Each machine token shares a single rate-limit bucket (120 requests / 60s by default) across all three routes above, not a separate budget per route; exceeding it returns HTTP 429 CERTOPS_MACHINE_RATE_LIMITED. Common error codes: 401 CERTOPS_API_TOKEN_UNAUTHORIZED (bad/revoked/expired token), 403 CERTOPS_API_TOKEN_SCOPE_DENIED (missing scope), 404 CERTOPS_JOB_NOT_FOUND (unknown jobId), 409 CERTOPS_EXECUTOR_EVENT_CONFLICT (same eventId replayed with a different payload), 422 PRIVATE_KEY_MATERIAL_REJECTED, and 413 CERTOPS_EVIDENCE_OUTPUT_TOO_LARGE.

Workspace-scoped endpoints under /api/v1/workspaces/:id/certops serve the dashboard and session-authenticated callers:

• GET .../jobs list executor jobs
• POST .../jobs create an executor job manually (workspace manager; body: { operation, subjectType?, subjectId?, payload? }; source is forced to api)
• GET .../jobs/:jobId job detail
• GET .../jobs/:jobId/log job timeline entries
• GET .../jobs/:jobId/evidence evidence attached to the job
• GET .../tokens list machine API tokens
• POST .../tokens create a token (raw ttx_ value returned once)
• POST .../tokens/:tokenId/revoke revoke a token

See Create a manual CertOps job for the create-job request schema, the API reference for authentication, request bodies, and response shapes, and Certificates for the inventory endpoints.