Skip to main content
Version: 0.18

Install and upgrade an agent on Linux

Goal​

Install the native TokenTimer agent on a host that holds certificates, so renewals, deployments, reloads, and verification run automatically under your control.

Which page do you want?

This page covers the native agent on Linux (systemd). For Windows Server, see Install an agent (Windows). If you only want to report on renewals your own scripts already perform, use Connect an external executor instead.

What gets installed​

The agent is a zero-dependency Node.js daemon. It is outbound-only: it opens HTTPS connections to your TokenTimer instance and never listens on a port, so it needs no inbound firewall rule.

PathPurpose
/opt/tokentimer-agent/appThe agent package, read-only at runtime
/opt/tokentimer-agent/stateConfig dir, mode 0700, owned by the agent user
.../state/config.jsonAgent configuration, mode 0600
.../state/credentialPer-agent credential, written by the agent at registration
.../state/bootstrap.envBootstrap token, deleted automatically after first successful registration
.../state/registration-id.jsonClient-generated registration id used to recover an interrupted registration; cleared once the credential is stored
.../state/outbox/Durable queue of completed job results and evidence not yet acknowledged by the control plane; drained on restart before new work is claimed. Retention-capped at 5000 entries / 7 days
.../state/outbox/dead-letter/Quarantined outbox entries: a delivery failure classified permanent is quarantined immediately, an unclassified (transient) one after 48 hours of retrying. Not drained automatically; retention-capped at 5000 entries / 30 days
.../state/job-journal/Side-effect journal written before the first external mutation. An unresolved entry deliberately blocks automatic re-execution of that job, so a crash mid-issuance cannot silently repeat it
.../state/acme/certbot and acme.sh config, work, and log directories
.../state/trust-receipts/Ownership receipts for distribute-trust/revoke-trust jobs, one per (store, fingerprint). Proves locally which anchors this agent installed, so a revoke-trust job can never remove a certificate it does not hold a confirmed receipt for. See Trust anchors
.../state/trust-work/Scratch directory for staging a trust-anchor PEM before it is moved into the OS store; cleaned up once the job resolves
/etc/systemd/system/tokentimer-agent.service.d/override.confGenerated ReadWritePaths allowlist
/etc/polkit-1/rules.d/50-tokentimer-agent.rulesOptional, only when you allow service reloads

The systemd unit is hardened (ProtectSystem=strict, NoNewPrivileges=true), and the agent runs as a dedicated tokentimer-agent system user rather than root.

Requirements​

RequirementDetail
OSLinux with systemd. Neither installer supports macOS or systemd-less Linux (confirmed on Alpine/OpenRC: install-agent.sh fails at the /etc/systemd/system unit-install step, since that directory does not exist without systemd) - see Installing without systemd below for the manual-run fallback on those hosts. Tested on Ubuntu 22.04, 24.04, and 26.04 LTS and AlmaLinux 9; other systemd-based Debian/Ubuntu LTS and RHEL/Rocky 9+ releases are expected to work the same way (same systemd unit model) but are not independently tested.
RuntimeNode.js 22 or later (>=22.0.0 <25.0.0)
Privilegessudo/root, to run the installer (it creates a system identity, service definition, and directories)
NetworkOutbound HTTPS from the host to your TokenTimer API URL. No inbound access needed.
ACME toolsFor hosts that issue certificates: certbot at /usr/bin/certbot, or acme.sh outside any home directory (the unit sets ProtectHome=true)
Control planeCertOps enabled, plus CERTOPS_SIGNING_ENCRYPTION_KEY and CERTOPS_REGISTRATION_ENCRYPTION_KEY set (see Configuration)

Service reloads require a polkit build with JavaScript .rules support (0.106 or later, including modern integer versions). Ubuntu 22.04's default polkit 0.105 cannot enforce this installer policy; use a supported host such as Ubuntu 24.04 for reload workflows. Discovery and file-only delivery do not require a reload grant. The updated installer checks this prerequisite before changing the host; it does not add a broad sudo fallback.

Installing without systemd​

The installer requires systemd. For a container or another process supervisor, use the checksum-verified agent package and its packaged entry point. Choose one mode; never run a systemd service and a manual process against the same state. This mode does not install a unit or inherit the installer's OS sandbox.

As the dedicated agent account, prepare a private state directory and a config.json containing the real serverUrl, workspaceId, and execution: { "enabled": false, "dryRun": true }. For a private-CA API, set caBundlePath to its trusted PEM. This configures API trust only. If the ACME server uses a private CA, configure the host or ACME client trust store before starting certbot; do not disable TLS verification. Use separate state and credentials for each issuer or consumer.

# Run from the extracted package directory, not a repository checkout.
export TOKENTIMER_AGENT_CONFIG_DIR="$HOME/.local/state/tokentimer-agent"
mkdir -p "$TOKENTIMER_AGENT_CONFIG_DIR"
chmod 700 "$TOKENTIMER_AGENT_CONFIG_DIR"
# Write config.json and local DNS credentials through secure provisioning.
chmod 600 "$TOKENTIMER_AGENT_CONFIG_DIR/config.json"
# Inject TOKENTIMER_AGENT_BOOTSTRAP_TOKEN through your secret manager.
node ./bin/tokentimer-agent.js

Confirm registration and a fresh heartbeat, stop the process, then remove the bootstrap token from its launch environment. Preserve the generated identity and credential. Merge the feature's local policy and restart the same command. Complete the signed dry run before enabling real jobs. Your supervisor must stop and reap the old process before restarting it; two live copies must not share custody state.

For Vault-backed jobs, supervise Vault Agent separately with its own AppRole and protected token sink. A container journey does not qualify systemd sandboxing or host reboot recovery.

Getting the agent package​

The commands below assume ./scripts/install-agent.sh is already on the host. If you do not have a tokentimer-core checkout, download the agent release tarball (contains install-agent.sh plus the agent package) as a GitHub release asset and verify its checksum before running anything from it:

set -e
CORE_VERSION="<version>"   # match the release supported by your deployment
BASE_URL="https://github.com/tokentimerch/tokentimer-core/releases/download/v${CORE_VERSION}"
TARBALL="tokentimer-agent-${CORE_VERSION}.tgz"
# Keep release filenames: the checksum sidecar names this archive.
curl -fsSL -O "${BASE_URL}/${TARBALL}"
curl -fsSL -O "${BASE_URL}/${TARBALL}.sha256"
sha256sum -c "${TARBALL}.sha256"

test ! -e package || { echo "Use an empty staging directory" >&2; exit 1; }
tar -xzf "${TARBALL}"
cd package   # npm-packed tarballs extract into a "package/" directory

install-agent.sh is at ./scripts/install-agent.sh inside that extracted directory; run the commands in Step 1 from there.

Prepare the policy first​

You can write /opt/tokentimer-agent/state/config.json (mode 0600) before the first install: create the folder with sudo install -d -m 0700 /opt/tokentimer-agent/state, save the file, then run the installer. The installer keeps an existing file and only creates one when it is absent. declaredTargetSelectors and declaredCommandProfileNames are read when the agent starts, so having them in place for the first start means the first registration already carries them. On an agent that is already registered, add them, keep the agentId line, and restart. With a loopback http API address, also add "allowInsecureLocalHttp": true yourself.

Step 1 - Run the installer​

Copy the agent package to the host, then run the installer from the package directory. It does not need a token yet: it pauses at a hidden interactive prompt asking for one, so the secret never reaches your shell history or the process list. Create the token in step 2 below and paste it there.

Run it as sudo bash ./scripts/install-agent.sh. This form works with every release tarball. Tarballs published before 0.18.0 ship the script without the executable bit, so sudo ./scripts/install-agent.sh fails with Permission denied there; tarballs built from 0.18.0 ship it executable.

sudo bash ./scripts/install-agent.sh \
  --api-url https://tokentimer.example.com \
  --workspace-id "<your workspace id>"

The installer prompts for the bootstrap token when neither TOKENTIMER_AGENT_BOOTSTRAP_TOKEN nor --bootstrap-token is set. For unattended installs, prefer the environment variable instead of waiting on the prompt:

sudo TOKENTIMER_AGENT_BOOTSTRAP_TOKEN=ttboot_... bash ./scripts/install-agent.sh \
  --api-url https://tokentimer.example.com \
  --workspace-id "<your workspace id>"
Avoid --bootstrap-token on shared hosts

It works, but arguments are visible in process listings while the installer runs. The prompt and the environment variable both avoid that exposure.

Run it with --dry-run first to print every action without touching the system:

sudo bash ./scripts/install-agent.sh --api-url https://tokentimer.example.com \
  --workspace-id "<workspace-id>" --dry-run

Useful options​

OptionUse it for
--write-path PATHAn absolute directory the agent may write certificates into. Repeatable. Never grants all of /etc.
--write-paths-file FILEOne absolute path per line, merged with --write-path
--trust-storeRequire the OS machine trust-store grant. Applied automatically on Debian/Ubuntu and RHEL/AlmaLinux, so you normally do not need to pass it
--no-trust-storeSkip the OS trust-store grant. Use on renew-only hosts that will never receive a trust anchor
--reload-service NAMEAllow systemctl reload for nginx, apache/apache2, httpd, or haproxy via a generated polkit rule. Repeatable. A re-run merges the services you give with the grants already on the host, so adding one service never drops another. Installers older than 0.18.0 rewrite the whole rule from the flags of that run instead (see Re-running the installer)
--replace-reload-servicesMake the --reload-service list authoritative: grants not listed are removed and the installer prints each dropped unit. With no --reload-service it removes the polkit rule. Needs the installer from a 0.18.0 or later package
--ca-bundle PATHPEM bundle when the agent must trust an extra CA (for example a TLS-inspecting egress proxy, or a private CA)
--allow-insecure-local-httpDevelopment only, and only for loopback hosts
--uninstallStop and remove the service, app dir, unit, drop-in, and polkit rule

A typical nginx host:

sudo bash ./scripts/install-agent.sh \
  --api-url https://tokentimer.example.com \
  --workspace-id "<workspace-id>" \
  --write-path /etc/letsencrypt \
  --write-path /etc/nginx/certs \
  --reload-service nginx
ReadWritePaths is not the same as filesystem permissions

--write-path only lets the systemd sandbox reach a directory. The tokentimer-agent user must still be able to write there. Grant that before enabling deploy jobs, for example:

sudo setfacl -m u:tokentimer-agent:rwx /etc/nginx/certs

Without this, deploys fail with a permission error even though the path is allowlisted.

Never pass the OS trust store as --write-path

The installer takes ownership of every --write-path directory. Doing that to /etc/ssl/certs or the RHEL pki anchors breaks CA trust for the whole host. The trust-store grant is a separate mechanism (see below) that deliberately leaves those directories root-owned.

Reload nginx​

nginx -t fails for the unprivileged agent user (it cannot open /run/nginx.pid), so the agent cannot validate nginx itself. Install with --reload-service nginx and use these two command profiles in policy.allowedCommands:

"nginx-validate": { "argv": ["/usr/bin/systemctl", "is-active", "--quiet", "nginx"] },
"nginx-reload": { "argv": ["/usr/bin/systemctl", "reload", "nginx"] }

The validate profile only checks that the unit is running. The stock nginx unit reloads without testing the configuration, so add a one-time drop-in that tests it as root first. It also pauses one second so the new workers serve before a served-certificate check:

sudo mkdir -p /etc/systemd/system/nginx.service.d
sudo tee /etc/systemd/system/nginx.service.d/tokentimer-reload.conf <<'EOF'
[Service]
ExecReload=
ExecReload=/usr/sbin/nginx -t -q -g "daemon on; master_process on;"
ExecReload=/usr/sbin/nginx -g "daemon on; master_process on;" -s reload
ExecReload=/usr/bin/sleep 1
EOF
sudo systemctl daemon-reload

Reload Apache or HAProxy​

A configuration test run as the unprivileged agent user fails as soon as the configuration loads a key or PEM that only root can read (for example Debian's default-ssl site, or a PEM folder under /etc/haproxy). So for Apache and HAProxy too, the validate profile only checks that the unit is running, and the unit's own reload tests the configuration as root and refuses a broken one (a refused reload leaves the old certificate served). Install with --reload-service apache2 or --reload-service haproxy and use these profiles in policy.allowedCommands (the Apache pair, the HAProxy pair, or both):

"apache2-validate": { "argv": ["/usr/bin/systemctl", "is-active", "--quiet", "apache2"] },
"apache2-reload": { "argv": ["/usr/bin/systemctl", "reload", "apache2"] },
"haproxy-validate": { "argv": ["/usr/bin/systemctl", "is-active", "--quiet", "haproxy"] },
"haproxy-reload": { "argv": ["/usr/bin/systemctl", "reload", "haproxy"] }

The issue request's target.keyPath decides where the agent writes the private key. The setup guide uses <certificate file>.key (for example /etc/ssl/tokentimer/site.pem.key), mode 0600, owned by tokentimer-agent; keep it in a folder you listed in --write-path. Point Apache at both files with SSLCertificateFile and SSLCertificateKeyFile. HAProxy loads <crt>.key by itself when the file in bind ... ssl crt holds no key, so bind :443 ssl crt /etc/ssl/tokentimer/site.pem works with site.pem.key beside it (HAProxy 2.8). Enable mod_ssl and write the virtual host or frontend yourself; the agent only replaces the files and reloads the service.

OS trust store (trust anchors)​

Distributing a trust anchor writes into the host's machine trust store. Under the hardened unit, ProtectSystem=strict leaves those directories read-only and tokentimer-agent has no write ACL there by default.

install-agent.sh opens the trust store automatically when Debian/Ubuntu or RHEL/AlmaLinux trust tooling is on PATH, adding the family's directories to ReadWritePaths and granting tokentimer-agent write via setfacl (falling back to group-write when the acl package is missing). Pass --trust-store to require the grant, or --no-trust-store to skip it on renew-only hosts that will never receive a trust anchor.

For the full directory table, the AlmaLinux ACL caveat, and how to recover an already-installed agent that predates this grant, see Diagnose Linux OS trust-store failures.

--write-path is not remembered across reinstalls

Every run of install-agent.sh regenerates the systemd sandbox drop-in (tokentimer-agent.service.d/override.conf) from scratch, from only the --write-path flags given on that invocation. This is deliberate (a stale broad grant should not silently outlive an operator narrowing it), but it means re-running the installer for any reason (troubleshooting, upgrading, changing --api-url) without repeating every --write-path from the original install will silently shrink the sandbox back down to the state directory only.

The failure mode this produces is misleading: the next real deploy job fails with EROFS: read-only file system opening a temp file under your certificate directory, which looks like a filesystem or permissions problem. It is not: it is the sandbox. Confirm the current grant with:

systemctl cat tokentimer-agent | grep ReadWritePaths

If your certificate directory is missing from that line, re-run the installer with the full original set of --write-path flags, then systemctl daemon-reload. Reload grants do not need repeating: a 0.18.0 or later installer keeps them.

The installer creates --write-path directories

The installer creates every --write-path directory and gives it to tokentimer-agent (mode 0750). Files you place there yourself must belong to that user too, or deploys cannot replace them. If you write the systemd drop-in by hand instead, create each directory first: a missing ReadWritePaths= directory stops the unit with status=226/NAMESPACE and Failed to set up mount namespacing.

Polkit is used rather than sudoers deliberately: the unit keeps NoNewPrivileges=true, under which sudo cannot escalate at all.

Re-running the installer and reload grants​

The installer from a 0.18.0 or later package merges the --reload-service units you pass with the grants already in the polkit rule. Running it again with --reload-service haproxy on a host that already reloads nginx and Apache keeps all three. A re-run with no reload flag leaves the existing rule in place.

To make the list exact, add --replace-reload-services. Units you do not list are removed and the installer prints each one it drops. With --replace-reload-services and no --reload-service, the rule is removed. A rule file that the installer did not write is replaced, and the installer warns.

An installer older than 0.18.0 rewrites the rule from the flags of that run only, and deletes it when the run has none. With an older package, list every service on every run, and upgrade the package to get the merge.

Step 2 - Create a bootstrap token​

  1. Open Certificate Management at /certops/agents (the Agents tab).
  2. Click Deploy an agent to open the install wizard and create a bootstrap token.
  3. Copy it and paste it at the installer's prompt from step 1. It is a single-use ttboot_ token, shown exactly once, and it is consumed at the agent's first registration. It expires 24 hours after creation by default (the maximum is 30 days); create a new one if it expires.

Step 3 - Verify registration​

systemctl status tokentimer-agent
journalctl -u tokentimer-agent -f

You should see the agent register and begin heartbeating, and the Agent fleet panel on the Certificate Management page flip it to registered. The fleet panel shows status, last heartbeat, clock drift, an Execution column (Enabled or No capability declared), and a Compatibility column (Compatible / Outdated / Blocked). If the registration wait in Deploy an agent runs past the usual window, the wizard names the likely causes (unreachable API URL, blocked outbound connection, an expired or reused bootstrap token, a mismatched CA bundle) and the per-OS command to read the agent's own logs on the host.

Confirm the bootstrap token file is gone once registration succeeded:

sudo ls -l /opt/tokentimer-agent/state/

bootstrap.env should no longer be present; the agent removes it after the first successful registration and keeps only the per-agent credential. Removal is best-effort, so if the file is still there (for example because of a read-only or oddly-permissioned state dir), delete it yourself once the agent is registered.

Step 4 - Start in dry-run, then enable execution​

A fresh install has execution.enabled: false: it registers, heartbeats, and reports configured certificate discovery, but does not poll for or claim jobs. Prepare local policy in this mode.

To test signed dispatch/reporting, set execution.enabled: true and keep execution.dryRun: true in config.json, restart, then request an approved job with mode: "dry_run". Confirm dry_run_complete and review the result. The local dryRun switch refuses real mutating jobs; it does not turn them into plans. AD CS preflight is a separate real read-only inspection and can run with local dry-run enabled; see the AD CS check sequence.

For an API-created no-op smoke test, list GET /api/v1/workspaces/{workspaceId}/certops/agents, match items[].agentId to the local configuration's agentId, and use that row's items[].id as assignedAgentId. Send POST /api/v1/workspaces/{workspaceId}/certops/jobs with {"operation":"noop","mode":"dry_run","assignedAgentId":"REPLACE_WITH_FLEET_AGENT_ROW_UUID"}; approve it if required by workspace policy, then poll its job endpoint for dry_run_complete.

After the smoke check and a review of the intended operation's local permissions/destinations, allow mutating jobs in config.json:

{
  "execution": {
    "enabled": true,
    "dryRun": false
  }
}

Restart the agent after editing: sudo systemctl restart tokentimer-agent.

Upgrading​

The agent does not self-update, which is deliberate: an execution-plane process that can update itself is an execution-plane process an attacker can update.

  1. Copy the new package to the host.
  2. Re-run install-agent.sh with the same flags. The installer stages the new tree, swaps it in atomically, and restarts the systemd unit, so the new code is what actually runs. The state directory, credential, and keys are preserved, so the agent keeps its identity. The installer still asks for a bootstrap token on every non-dry-run install and checks only its ttboot_ prefix. A registered agent keeps its credential and ignores the token, so paste any ttboot_... value or set TOKENTIMER_AGENT_BOOTSTRAP_TOKEN=ttboot_unused.
  3. Confirm the new version in the Agent fleet panel.

Upgrade the control plane before the agents, confirm registration and heartbeats, then upgrade each agent and verify a signed dry run. Use the agent release that matches your Core version. Agent-side fixes apply only to an upgraded agent: a pre-release 0.18.0 build still quarantines a certificate key after a rolled-back deployment and still marks a Vault distribution binding uncertain after a deploy that wrote nothing (see Troubleshooting). The fleet panel also shows a compatibility result: protocol and version bounds set on the deployment can reject an otherwise matching package.

To force a fleet upgrade, raise CERTOPS_AGENT_MIN_AGENT_VERSION on the control plane; older agents are then blocked at registration and heartbeat with CERTOPS_AGENT_COMPATIBILITY_BLOCKED rather than silently continuing to claim work.

Retiring and uninstalling​

Retire the agent from the Agent fleet panel first, then uninstall on the host. Retiring is not just bookkeeping: it invalidates the agent's credential (its protocol calls then return 410) and lets the agent exit cleanly on its next heartbeat instead of being killed mid-job.

Moving an agent to a different workspace is not supported

There is no in-place transfer of an agent between workspaces. To put an agent under a different workspace, retire it here first, then install a fresh agent and register it in the target workspace with a new bootstrap token.

Retiring an agent that is still working​

Retirement deliberately refuses to yank work out from under a running agent. If the agent currently holds an actively leased job, a plain retire is blocked with 409 CERTOPS_AGENT_RETIRE_BLOCKED, and the response tells you how many jobs are in the way.

The right first move is usually to wait: let the job finish, then retire. If you cannot wait (the host is already gone, or the agent is misbehaving), force it. A force retire requires a reason and is rejected with 400 CERTOPS_AGENT_RETIRE_REASON_INVALID without one. That is enforced by the API, not just the dashboard form, because a forced retire can abandon real work and someone will need to know why later.

A forced retire immediately fences the agent's in-flight work rather than leaving it to time out, and splits it by how far it got:

Job status when forcedBecomesWhy
claimed (dispatched, no execution started)cancelled with error_code: CERTOPS_AGENT_FORCE_RETIREDNothing was done yet, so cancelling is safe and clean
running (execution had started)orphaned_unknown_effect, flagged needs_operator_reconciliationThe agent may already have ordered a certificate, written a key, deployed a file, or reloaded a service. TokenTimer will not guess

The response reports both lists (fenced.cancelledJobIds, fenced.orphanedJobIds), and the CERTOPS_AGENT_RETIRED audit event records force, reason, leasedJobs, and both id lists.

Anything in the orphaned list needs a human. Work through Reconciling interrupted jobs for each one before assuming the certificate is in a known state. If an orphaned job was a renewal, it also raises a cert_renewal_failed alert, so it reaches you through normal alerting rather than sitting unnoticed.

Retiring is idempotent: retiring an already-retired agent succeeds with the original retirement timestamp unchanged, and does not fence anything a second time.

On the host, a retired agent notices at its next heartbeat (which returns 410), logs control plane retired this agent; exiting cleanly, and exits with status 86. The systemd unit sets RestartPreventExitStatus=86, so systemd deliberately does not respawn it into a 410 loop. systemctl status tokentimer-agent then shows the service inactive with exit code 86: that is a successful retirement, not a crash. Compare with a genuine failure, which shows a non-86 status and Restart= kicking in with repeated activating (auto-restart).

Uninstalling​

Once retired, on the host:

sudo bash ./scripts/install-agent.sh --uninstall

This removes the service, app directory, unit, drop-in, and polkit rule. The state directory and system user are preserved so you cannot destroy keys by accident. Remove them deliberately once you are sure:

sudo rm -rf /opt/tokentimer-agent
sudo userdel tokentimer-agent
warning

Uninstalling the agent does not retire it in TokenTimer, and retiring it does not uninstall it. Do both. An uninstalled but un-retired agent keeps a valid credential and simply looks offline; an un-uninstalled but retired agent keeps running locally and gets 410 on every call.

Also remember that certificates that agent discovered stay pinned to it: jobs for them will sit at pending until you assign a new agent or let a new agent rediscover them. See How a job is bound to one agent.

Testing under WSL2​

For a WSL2 test environment, confirm that systemd is enabled and keep the WSL instance running while testing the agent. A systemd service does not keep WSL alive by itself; see Microsoft's WSL systemd guidance. If the agent disappears between test commands, check the WSL instance lifecycle as well as the service logs.

For persistent operation, use a host whose lifecycle keeps the agent running. For a native Windows service, follow Install an agent (Windows).

Troubleshooting​

SymptomLikely causeFix
Service in activating (auto-restart) with status=1/FAILUREStartup failure, most often registration or configRead the real reason with journalctl -u tokentimer-agent -n 50 --no-pager; the unit status alone never says why
Service inactive with exit code 86Not a failure: the agent was retired in TokenTimer and exited cleanlyExpected. Uninstall the host files if you are done, or register a new agent
Installer refuses the API URLURL is not https:Use HTTPS, or --allow-insecure-local-http for a loopback dev host only
Registration fails as unauthorizedBootstrap token already used, expired, or revokedTokens are single-use and expire 24 hours after creation by default (30 days at most); create a new one in the dashboard
Registration fails and CertOps routes 404CertOps disabled on the instanceEnable it on API and worker (see Enable CertOps). The disabled state is deliberately indistinguishable from a missing route
Registration fails referencing an encryption keyCERTOPS_REGISTRATION_ENCRYPTION_KEY unset or malformedSet 64 hex characters and restart the API
Agent blocked at registration or heartbeatVersion outside the accepted windowUpgrade the agent to the current release, or raise CERTOPS_AGENT_MIN_AGENT_VERSION on the control plane
TLS handshake failure to the control planeThe agent does not trust the server certificate (TLS-inspecting egress proxy, or a private CA)Reinstall with --ca-bundle, or set caBundlePath in config.json
Deploy fails with a permission errorPath allowlisted but not writable by the agent userGrant write access (setfacl) to tokentimer-agent
Reload failsService not allowlisted at install timeReinstall with --reload-service NAME
Failed ACME order: the certificate path stays provisioningA first order that fails leaves a provisioning entry with no fingerprint. It cannot be retired (API 428), and it does not need to beRequest the certificate again with the same certPath on the same agent and a new idempotency key. Once every earlier issue job has failed, been cancelled or been rejected, the request reuses that entry. A control plane older than 0.18.0 keeps the path reserved instead: choose a different file name in the same folder
Issue request refused with 409 CERTOPS_CERTIFICATE_PATH_IN_USEAnother certificate on this agent already uses that certPath, or an earlier issue job for it is still running or ended with an unknown outcome (orphaned_unknown_effect). The message names it: Another certificate (<name>, status <status>) already uses this certificate path on this agent: <path>. Retire that certificate in the inventory, or choose a different certPath.Retire the certificate the message names, or choose a different certPath. A provisioning entry that never issued cannot be retired, so when its earlier job ended with an unknown outcome, check the host and choose a different certPath
Job fails with agent custody key is quarantined after a failed promotion or agent custody key does not match the deployed certificateThe agent refuses to reuse a key it cannot prove belongs to the deployed certificate. A 0.18.0 agent restores the previous key itself when a rotation is rolled back cleanly (a failed validate or reload), so only a promotion with an unknown outcome stays quarantined. Pre-release 0.18.0 agent builds quarantined the key after any rolled-back run, and the fix does not repair a host that is already stuckUpgrade the agent to the 0.18.0 release. For a host that is already stuck, or after an unknown-outcome promotion, stop renewals for that certificate, then delete the custody key named in the error (/opt/tokentimer-agent/state/keys/<certificate id>.key.pem on the Linux installer layout) and, when present, its .custody-quarantine marker. The next renewal issues a new key and installs it with the new certificate
Service refuses to start after you replaced config.json: found a stored credential but no agentIdThe replacement file dropped the agentId lineRestore the agentId line from the old file and restart
Clock drift warnings in the fleet panelHost clock skewFix NTP; signed jobs enforce a validity window
Real job is blocked by the local dry-run switchexecution.dryRun: true refuses real jobsFirst validate a signed mode: "dry_run" job, then set execution.dryRun: false to allow real work and restart
Agent registers and heartbeats but never claims jobsexecution.enabled: false (observe-only)Prepare policy, set execution.enabled: true with execution.dryRun: true, restart, and test a signed mode: "dry_run" job
A job stays pending and this agent never claims itThe job is pinned to a different agent (the one that discovered the certificate), or requires a target selector this agent does not declareCheck the job's assigned agent; see How a job is bound to one agent
Renew job creation is refused for a certificate you can see in inventoryThe certificate was only observed, so no agent has key custodyInstall the agent on the host that actually serves it; see which certificates are renewable
Agent looks healthy but no automatic renewals ever happenThe CertOps maintenance worker is not running on the control plane, so no renewal jobs are createdThis is a server-side problem, not an agent one; see Worker scheduling
NTP badge in the fleet panel stays unknowntimedatectl is missing or not usable on the host (containers, some WSL setups, non-systemd Linux)Confirm timedatectl show -p NTPSynchronized --value runs as the tokentimer-agent user; unknown is reported as null, never guessed