Troubleshooting
This page lists what people and operators see when something is wrong, what causes it, and what to do. The messages are the ones Heliast produces. Where the right action cannot be told from the software alone, the table says to ask your Heliast contact.
Every denial the hook makes is also a ledger row, with the reason, so the ledger is the first place to look. The hook's reply carries the reason in its body and in a response header.
A person is refused a model
| What they see | Cause | What to do |
|---|---|---|
401 unauthenticated | No credential, or the virtual key is unknown, expired, revoked or malformed. The hook gives one answer for all of these on purpose; the specific cause is visible only to an admin | An admin opens Entitlement, then Virtual keys, and checks the key's status. Issue or rotate a key if needed. For a browser session, sign in again |
401 identity provider unavailable: session requests fail closed while the management link is absent | A browser session was used while the sign-in proxy or your identity provider cannot be reached. Requests made with a virtual key are not affected | Check the sign-in proxy and the route to your identity provider |
403 could not determine the requested model | The request names no model | Send "model" in the request body |
403 model "X" is not admitted at a named attestation version | The model is not in the catalogue, or your policy config lacks its digest or attestation version, or the versions differ | Check the model id. An admin checks the catalogue and the policy config (see policy and entitlement) |
403 model "X" is class assessed but is refused: <reason> | An assessed model of your own whose Heliast statement does not verify | See "The hook will not start or load" below, then ask your Heliast contact |
403 host record for "X" is stale: ... the mapping is refused until a human re-reviews it | A hosted-provider mapping is past its review date | An admin re-reviews the host record and renews its review date |
403 subject "S" is not entitled to model "M" | The caller's groups are not granted this model | An admin adds the model to a group under entitlements, or adds the group to the key |
| 403 about a provider (not a configured, enabled provider; workload not permitted; data class may not cross; no mapping to an admitted model) | The reference policy's provider rules | An admin checks the provider configuration: enabled, per_workload, data_classes and models (see environments) |
403 the x-ai-eg-model header does not match the model named in the request body | The routing header and the body name different models | Fix the router or client so both name the same model |
403 policy engine unavailable: ... | The policy engine is down or returned no readable decision. The hook fails closed | Check the policy engine pod. The metric sovereignai_policy_engine_up is 0 and the Critical alert PolicyEngineUnavailable fires |
403 token usage unavailable: the ledger projection could not be read | The hook cannot read the ledger database | Check the database and GATEWAY_DB_DSN |
403 admission state unavailable: failing closed | The catalogue failed to load or reload | Check the catalogue directory (GATEWAY_ADMISSION_DIR) and the hook log line admission state reload failed, failing closed. The hook retries every 60 seconds |
403 denied by policy or policy denied the request without stating a reason | Your own policy denied it without a message | Add a deny reason to the rule |
503 audit unavailable, request refused: no request proceeds unlogged | The audit service did not accept the decision record | See "The audit service" below |
| Refused with no reason from the hook | The router cannot reach the hook, or it timed out. The router fails closed | Check the hook pod and the network policy that admits the router to the hook |
Rate-limit and budget denials come only from policy you write. The reference policy has none, so a limit that you expect and do not see is not enforced. The router's own rate-limit rejection writes no ledger row.
An adapter or reporter is refused
| What you see | Cause | What to do |
|---|---|---|
403 identity assertion refused: the caller is not a trusted gateway adapter (or is repeated, asserted subject is not storable, role is not allowed for this adapter, more than N groups) | The adapter's identity headers are wrong | Check GATEWAY_ADAPTER_TOKENS and GATEWAY_ADAPTER_ROLES, and the subject, groups and role the adapter sends (see use Heliast with a gateway you already run) |
401 unauthenticated: the gateway adapter asserted no subject | The adapter did not send a subject | Fix the adapter configuration |
400 response model mismatch (anomaly): the response report names model "X" but the request was decided for model "Y" | The model that answered is not the one decided. A row marked as denied is still written | Treat it as a security event: check the router's or gateway's model mapping and fallbacks |
400 not bound to an allow decision row, 403 this caller did not decide request_id, 409 a response ... is already recorded | A wrong, forged or repeated usage report | Fix the reporter. Do not retry a 409 |
An admin in the console
| What you see | Cause | What to do |
|---|---|---|
| "Your session ended. Sign in again to carry on" | The session expired | Sign in again. Nothing that was open is saved |
"This section is for administrators", or 403 this needs the admin role | You are signed in without the admin role. Ledger, Entitlement and starting a Compare run are admin-only | Ask an administrator for the role |
403 a request from another origin is refused | The request did not come from the console's own address | Use the console at its own address |
| "The session could not be resolved ... Sign-ins fail closed while the identity provider is unreachable" | The sign-in proxy cannot be reached | Check UI_OAUTH2_PROXY_URL and the sign-in proxy. /readyz shows identity: false |
503 this view is not configured: set UI_GATEWAY_ADMIN_TOKEN (Keys) or UI_AUDIT_QUERY_TOKEN (Ledger, Denials, Budgets) | The console's server has no token | Set the variable on the console's server and reload (run the console) |
| "The audit query API did not answer" or "The hook admin API did not answer", with "check that it is running, then retry" | The console cannot reach the service | Check UI_AUDIT_URL (default 127.0.0.1:9090) or UI_GATEWAY_URL (default 127.0.0.1:8095) and the service |
The same box with "Retry; if it keeps failing, ask an administrator to check the service log", or missing or wrong bearer token | The token the console holds does not match the service's | Make UI_AUDIT_QUERY_TOKEN equal the audit service's AUDIT_QUERY_TOKEN, and UI_GATEWAY_ADMIN_TOKEN equal the hook's GATEWAY_ADMIN_TOKEN |
| Banner "Not reachable from the console: the enforcement hook" or "the audit service" | The console's health check of that service failed | Check the service. What is on screen from before may be out of date |
429 too many requests: try again in N seconds | The console's request limit (UI_RATE_LIMIT, default 600 a minute per client) | Wait, or raise the limit |
| "This view could not be shown" | The page hit an error while drawing | Reload. If it repeats, choose Copy diagnostics in the footer and send it to your administrator |
Keys: subject_id may not start with "gw-", a subject id is required to issue a key | Input rejected | Fix the input |
Keys: the gateway admin token is not configured (503) | GATEWAY_ADMIN_TOKEN is not set on the hook | Set it on the hook |
| Ledger: "Divergence found: the chain does not match" | audit-verify logic found a break at a row | Do not rely on rows from that point. See "The ledger" below |
| Ledger: "Rows are consistent, but no anchor was checked" | No signed anchor has been exported | Not a fault. Rows deleted from the end would not be detected. See audit.anchorInterval |
Compare: a Compare run is already in progress | Only one run goes at a time | Wait, and reload to see its progress |
Compare: the Compare run is not configured: set UI_GATEWAY_URL and UI_GATEWAY_ADMIN_TOKEN | Missing settings | Set both on the console's server |
Compare: the gateway is unreachable at URL or the gateway answered N at URL/healthz; start the stack first | The hook is not answering | Start or fix the hook |
Compare: the gateway refused to issue a virtual key | The admin token is wrong or the hook is down | Check UI_GATEWAY_ADMIN_TOKEN |
Compare: go build fails, or UI_COMPARE_BIN points at ..., which does not exist | No Go toolchain, or a wrong path | Install Go, or set UI_COMPARE_BIN to a prebuilt comparison binary |
| Compare: "Lost contact with the console" | The page could not read the run's progress three times | Reload. The run may still be going |
| Compare: "The Compare results could not be read" | The results directory is not mounted (UI_COMPARE_RESULTS_DIR) | Fix the mount |
Compliance: "could not be loaded", or a file is not present in the ... evidence | The evidence directory is wrong or the file has not been generated | Check the evidence settings, or generate the file with the command the page shows |
| Admission: "No provenance artefacts were read. Check the catalogue directory on the back end" | UI_CATALOGUE_DIR is wrong | Fix the setting |
| Your models: "No client catalogue is configured. Set UI_CLIENT_CATALOGUE_DIR" | Not an error; nothing is configured | Set it if you have your own models |
| Your models: "N file(s) could not be read" | A record file is wrong (no model_id, a class that is not registered or assessed, a duplicate id, or not a regular file) | Fix the record file |
| Your models: "Refused by the hook: ... until a statement signed by Heliast verifies" | The hook refused this assessed model when it loaded the catalogue | See "The hook will not start or load" below |
The hook will not start or load
| What you see | Cause | What to do | |
|---|---|---|---|
GATEWAY_DB_DSN is required | The key store and ledger projection database is not set | Set it | |
refusing to start: GATEWAY_HOOK_TOKEN is not set | The hook would decide for any caller | Set GATEWAY_HOOK_TOKEN (the chart wires it from the Secret key hook-token) | |
refusing to start: AUDIT_INTAKE_TOKEN is not set | The ledger intake token is missing | Set the same token on the hook and on the audit service | |
refusing to start: GATEWAY_ADAPTER_TOKENS or GATEWAY_ADAPTER_ROLES errors | Wrong format, a duplicate name, a token under 32 bytes, tokens that match each other or the hook or admin token | Fix the format: name=token,name=token and `name=role\ | role` |
refusing to serve without admission state followed by a record problem (cannot be read unambiguously, has no catalogue record, has a catalogue record but no gate artefact, the catalogue record and the gate artefact disagree, a class that is not admitted, assessed or registered, two records for one model, a host record with no review date) | The catalogue directory is unreadable or contradicts itself | Fix the files under GATEWAY_ADMISSION_DIR. A reload with the same fault fails the hook closed until a reload succeeds | |
gateway: policy config names model "X" ... refusing to load (no digest or attestation version; not in the catalogue; not admitted; wrong attestation version; a registered model with no digest, or one that names a version) | GATEWAY_POLICY_CONFIG disagrees with the catalogue | Copy the digest and attestation version from the admitted record. A registered model needs a digest and no version | |
read policy config or parse policy config errors | The file is missing or not valid JSON | Fix the file or the path | |
assessed model X is refused: <reason>, for example no statement present, signed by a key that is not a pinned Heliast key, the signature does not verify, the record changed after it was signed, or the weights digest is not the one signed | The signed statement for your own assessed model does not match | Check that GATEWAY_HELIAST_KEYS_DIR holds Heliast's public keys (PEM, ed25519). For any other reason, ask your Heliast contact: the software cannot say whether the statement or the pinned key is stale | |
| Hook pod stays in its init step: "agent-router is not ready" | The hook waits for the router (gateway.requireAgentRouter) | Fix the router first | |
| The hook is healthy but the router's calls to it are all refused | The chart's network policy admits only the router pod to the hook. The response service, adapters and the console's server need their own policy | Add an ingress policy for them (network and ports). This has not been observed in a real cluster |
The hook's /healthz returns 200 even when the policy engine is down; its body and the metric sovereignai_policy_engine_up show the truth, and the chart's readiness probe uses /healthz, so a pod can read Ready while the policy engine is down. Alert on the metric.
The audit service
| What you see | Cause | What to do |
|---|---|---|
refusing to start: AUDIT_INTAKE_TOKEN is not set | Intake token missing | Set it |
SOVEREIGN_AUDIT_DSN is required | The writer database connection is not set | Set the writer role's DSN |
503 the audit query API token is not configured; 401 missing or wrong bearer token | AUDIT_QUERY_TOKEN unset, or the caller's token differs | Set it and align the tokens |
400 after_seq must be a non-negative integer, limit must be a positive integer, order must be asc or desc, until must be after since, the window is over N days | A bad query parameter | Fix the call |
400 body is not a JSON array of events: json: unknown field on intake | A newer hook is talking to an older audit service. The hook then answers 503 | Upgrade the audit service to match the hook |
400 event N cannot be stored | An event that can never be stored | Report it to Heliast. It is not written to the ledger |
Intake 503 audit write-ahead buffer is full; sovereignai_audit_wal_full is 1 | The ledger database was unreachable long enough to fill the buffer | Restore the database; the buffer drains. Watch sovereignai_audit_wal_pending_bytes. The limit is AUDIT_WAL_MAX_BYTES (default 256 MiB) |
Log sink append failed, records retained in the write-ahead buffer; sovereignai_audit_sink_up is 0; the Critical alert AuditWriteFailure | The database is unreachable or rejecting writes | Check the database, the DSN and the writer role |
Log QUARANTINED an event the ledger can never store; sovereignai_audit_quarantined_total rises | An event the ledger cannot store was set aside. The ledger is untouched | Inspect quarantine.jsonl in the write-ahead directory and report the producer |
503 anchor export is not configured | No anchor signing key or directory | Check AUDIT_ANCHOR_DIR, AUDIT_ANCHOR_KEY and AUDIT_ANCHOR_INTERVAL (default 24 hours) |
The ledger
| What you see | Cause | What to do |
|---|---|---|
audit-verify: first divergence at seq N (KIND): REASON, exit 2 | Kinds: chain (a hash does not match), typed_columns (stored columns disagree with the signed payload), state, anchor (an anchored row is missing, or rows after the anchor were deleted) | Treat it as an incident and preserve the database. The software cannot say whether to restore or investigate: ask your Heliast contact |
audit-verify: no anchor found ... and -require-anchor was set; no pinned anchor key is configured; anchor was signed by K, not the pinned key; anchor signature does not verify against the pinned key | Missing input, a wrong pinned key, or a tampered anchor | Pin the right public key, which you get from your Heliast contact |
auditd reconcile reports unreported_allows: N, exit 1 | Allowed requests with no response record: the hook or an adapter stopped, a stream was cut, or a report was refused | Look at the sampled request ids. Some causes are legitimate (an upstream error, a client that gave up). Set audit.reconcile.max to a count you can explain |
sovereignai_audit_completeness_ratio is below 1 and AuditLedgerCompletenessBreach fires | The chain is incomplete or divergent | Run audit-verify, as above |
audit-verify exits 2 for any failure, including a divergence. auditd reconcile exits 0 when the count is at or under the maximum, 1 when over, and 2 when it could not run.
Tools you run
| What you see | Cause | What to do |
|---|---|---|
model-pull: PULL ABORTED: <reason>, nothing written, a model_pull_denied file under <dest>/security-events/ | A digest mismatch, or any check below | Fetch again from a trusted source and look at the mirror |
format "..." is not safetensors; pickle-format checkpoints are rejected outright | Unsafe or wrong weights format | Use safetensors weights |
no signature at URL, signature verification failed against the pinned key, public key ... is neither PEM nor 64 hex characters | A pinned key with a missing or wrong signature, or a bad key | Provide the matching signature or the right key |
files does not pin the manifest path ...: the set is pinned in full or not at all | The weights manifest is incomplete | Regenerate it |
prerequisites not met: stage N NAME failed in an earlier run (or has no completion marker) | An earlier pipeline stage failed or did not finish | Fix that stage and run again from it |
preflight aborted, nothing has been changed: with a driver below 550, CUDA below 12.4, no kubelet, no storage class, or no network policy support | The host does not meet the requirements | Fix the item listed (deploying) |
terraform apply failed; roll back via state, helm upgrade failed; helm rollback is the recovery path | The infrastructure or the release failed | Roll back as the message says |
policy stage aborted: policy unit tests failed; fail-closed until policy is valid | Your policy's tests fail | Fix the policy and run stage 6 again. Heliast stays closed until a valid policy is loaded |
smoke tests aborted: GATEWAY_ADMIN_TOKEN and GATEWAY_HOOK_TOKEN are required in the environment | Missing inputs | Export both tokens |
env-check: FAIL: ... | The provider configuration is invalid (an inline key, a non-https address, 0.0.0.0/0, a duplicate name) | Fix the provider file (environments) |
offline-bundle verify: FAIL: ... (public key fingerprint mismatch, bundle key purpose mismatch, content digest mismatch, artifact ... digest mismatch, bundle carries unlisted files) | A tampered or wrong bundle or key | Do not install it. Fetch again and confirm the fingerprint with your Heliast contact (get and update Heliast) |
Ask Heliast
The software cannot tell you these: what to do after an audit-verify divergence beyond preserving the database, why an assessed model's statement fails to verify, the exact reasons an attestation or an image check reports when it does not verify, and any service terms. Send your administrator's Copy diagnostics text, the failing message, and the request id from the ledger row.