Diagnose Linux OS trust-store failures
Goal
Go one level deeper than the OS trust store grant and the trust-anchor job lifecycle: what the installer's ACL grant can and cannot guarantee on Linux, why a distribute-trust or revoke-trust job can report something confusing even when the host is in the right state, and how to recover a stuck installation without guessing.
Read this page when a trust-anchor job on a Linux agent does anything other than cleanly succeed: a non-zero exit from the OS update command, a revoke-trust that refuses to remove an anchor you know is installed, or a /etc/ssl/certs symlink that outlives the certificate it pointed to.
What the ACL grant covers, and what it cannot
The installer's OS trust store grant opens the sandbox (ReadWritePaths) and applies a write ACL so tokentimer-agent can write into the family's anchors directories. That is a filesystem-permission grant, and it is exactly as strong as filesystem permissions can be.
On AlmaLinux and other RHEL-family hosts, update-ca-trust extract's underlying p11-kit step goes one step further: it calls chmod(2) on the extracted directory-hash bundle it just wrote. A chmod call needs ownership of the target, not just write access to it, and a POSIX ACL grant can never confer ownership. So update-ca-trust extract can exit non-zero on a chmod failure at the very end of an otherwise fully successful run, on a host where the ACL grant is correctly in place and the certificate the job was trying to install is, in fact, sitting in the trust store correctly.
On AlmaLinux/RHEL-family hosts specifically, treat a non-zero update-ca-trust extract exit as inconclusive, not as a failure, until you have checked the post-condition below. This is the single most common source of confusion in this part of the job lifecycle.
A genuinely-trusted certificate can still see a non-zero exit
distribute-trust does not trust the update command's exit code by itself. Before deciding the job's outcome, the agent re-observes the real state of the anchors directory (a whole-directory fingerprint scan, not just a check of the one file it wrote) and only then decides what happened:
- If the certificate is genuinely present after the scan, the job reports
installed, with the update command's stderr carried in the result as a non-fatal warning rather than surfaced as a failure. - If the certificate is not present, the job reports the real failure.
This means a job result of installed with a warning attached is a success, and does not need a retry or an operator to intervene. Only treat the job as failed if the post-scan check itself says the certificate is absent.
Two failure categories, and how to tell them apart
A failed distribute-trust/revoke-trust job carries a failureCategory. The two you will see on Linux mean structurally different things:
| Category | What it means | What fixes it |
|---|---|---|
trust_store_not_writable | The sandbox/ACL grant itself is missing or was never applied - the anchors directory is read-only to tokentimer-agent before any update command even runs | Re-run the installer with --trust-store (or the original flag set), then systemctl daemon-reload && systemctl restart tokentimer-agent. See OS trust store |
os_mutation_failed | The grant was present and the agent reached the update command, but the post-scan check confirmed the certificate genuinely did not end up trusted | Investigate the update command's own error (disk space, corrupt existing bundle, p11-kit package issue) - this is a real OS-level failure, not a permissions gap |
Before 0.14.2, the AlmaLinux chmod false-negative described above was also reported as os_mutation_failed, indistinguishable from a genuine failure, which is what made a stuck pending_install receipt (below) look unrecoverable. As of 0.14.2, the AlmaLinux case is no longer misclassified: the re-observation step catches it and reports installed instead.
Revoke hygiene: dangling /etc/ssl/certs symlinks
On Debian/Ubuntu, plain update-ca-certificates after a removal reports 0 added, 0 removed and leaves the just-removed anchor's symlink sitting in /etc/ssl/certs pointing at nothing. Only update-ca-certificates --fresh rebuilds the symlink set from scratch and prunes it.
ls -l /etc/ssl/certs/*.pem shows a broken symlink (typically flagged in red by your shell, or resolvable with ls -lL failing with "No such file or directory") once its target anchor has actually been deleted from /usr/local/share/ca-certificates but the symlink itself was never pruned.
revoke-trust now prefers --fresh when your agent's policy.allowedCommands has the opt-in trust-store:update-ca-certificates-fresh profile configured, and self-heals the symlink as part of the revoke. Configure it once and every future revoke on that agent benefits:
{
"policy": {
"allowedCommands": {
"trust-store:update-ca-certificates-fresh": {
"argv": ["/usr/sbin/update-ca-certificates", "--fresh"]
}
}
}
}
Without that profile configured, revoke-trust falls back to the plain command and reports a non-fatal debian_fresh_command_ref_missing warning on the result. This is a warning rather than a silent failure on purpose: the certificate itself is still correctly revoked either way, only the symlink cleanup is skipped, and the warning tells you that self-heal is available if you want it.
/tmp writability and PrivateTmp
Both update-ca-certificates and update-ca-trust extract call mktemp internally before touching the anchors directory. The shipped systemd unit sets PrivateTmp=true specifically so that call succeeds: without it, ProtectSystem=strict leaves the real /tmp read-only, and the update command fails at its own mktemp call before it ever reaches the anchors directory.
A failure at mktemp looks, from the outside, exactly like "the anchors-directory grant never got applied" - both surface as the update command failing outright. They are not the same problem, and the ACL-grant fix does nothing for a /tmp problem. Confirm which one you actually have with the diagnostic commands below before re-running the installer.
PrivateTmp=true ships in the unit install-agent.sh installs; you only hit this if you are running a hand-written or hand-edited unit that omits it.
Recovering a stuck pending_install receipt
If an earlier distribute-trust on an AlmaLinux host hit the chmod false-negative before 0.14.2, the agent's local receipt for that anchor is stuck at pending_install, and revoke-trust used to refuse to touch it permanently - there was no agent-side recovery path.
The old advice was "just re-run distribute-trust." Do not follow that advice on this specific failure mode: it would hit the identical false negative again, since nothing about the host state changed, and the receipt would stay stuck.
As of 0.14.2, revoke-trust can unwind an old, already-stranded pending_install receipt directly, as long as the anchor is still observably present on the host. Send the same revoke-trust job you would send normally; the agent re-checks the real anchors-directory state first and, finding the certificate genuinely there, finalizes and removes it instead of refusing. The refusal is unchanged, and correct, when the anchor genuinely is not present - that case still means something really did fail and needs investigation, not a receipt that is merely misfiled.
Diagnostic commands
Run these directly on the agent host (as root or via sudo) to confirm which layer is actually broken before changing anything:
# Is /tmp writable under the sandbox? (PrivateTmp=true should be present)
systemctl cat tokentimer-agent | grep ReadWritePaths
# Does the agent user actually have a write ACL on the family directory?
# Debian/Ubuntu:
getfacl /etc/ssl/certs
# RHEL/AlmaLinux:
getfacl /etc/pki/ca-trust/source/anchors
# Which trust-store command profiles does this agent's policy allow?
sudo cat /opt/tokentimer-agent/state/config.json | grep -A2 'trust-store:'
If getfacl shows no tokentimer-agent entry at all (only the traditional owner/group/other bits), the host fell back to group-write with the owner left as root, which happens automatically when the acl package is not installed. That fallback still lets the agent write; if it is also failing, look at group membership (id tokentimer-agent) rather than assuming the ACL grant itself is the problem.
Troubleshooting
| Symptom | Cause | Fix |
|---|---|---|
update-ca-trust extract exits non-zero on AlmaLinux, but the certificate verifies fine with openssl verify | The p11-kit chmod call needs ownership an ACL cannot grant | Nothing to do - as of 0.14.2 the job reports installed with a warning, not a failure |
Job fails with trust_store_not_writable | The sandbox/ACL grant was never applied, or was lost on a re-run without --trust-store | Re-run the installer with the flag set, then reload and restart the unit |
Job fails with os_mutation_failed | The update command genuinely did not result in a trusted certificate | Investigate the update command's own error output; this is a real OS-level failure |
update-ca-certificates/update-ca-trust extract fails at an internal mktemp call | /tmp is read-only under ProtectSystem=strict because PrivateTmp=true is missing from the unit | Reinstall with the shipped unit, or add PrivateTmp=true to a hand-edited one |
Dangling /etc/ssl/certs symlink after a Debian/Ubuntu revoke | Plain update-ca-certificates does not prune symlinks; only --fresh does | Configure the trust-store:update-ca-certificates-fresh profile so future revokes self-heal |
revoke-trust refuses with receipt_pending_install on an anchor you can see is installed | An older, stranded pending_install receipt from before 0.14.2 | Re-run revoke-trust; it now unwinds a stranded receipt whenever the anchor is observably present |
revoke-trust refuses with receipt_pending_install and the anchor is genuinely not installed | The original distribute-trust never actually succeeded | Investigate the original job's result before assuming this is the stranded-receipt case above |
Related
- OS trust store (trust anchors) - the installer's ACL/sandbox grant this page assumes is already in place.
- Distribute and revoke trust anchors - the job lifecycle, ownership receipts, and platform resolution this page troubleshoots.
- Agent configuration - the
trust-store:*command-ref profile names, including the opt-in-freshprofile.