Failure modes
This page says what Heliast does when a part fails and what an operator sees. The rule is to fail closed: when Heliast cannot be sure a request is allowed and recorded, it refuses the request.
Status: these behaviours are written into the code and exercised by tests and on a single development host. They have not yet been observed in a real customer estate. A fail-closed system is an outage from the user's point of view, and Heliast treats it as one.
Summary
| What fails | What Heliast does | What the operator sees |
|---|---|---|
| Policy engine unreachable, or its answer cannot be read | Denies the request | A deny reason in the response and the ledger. The policy engine metric reads 0 and the Critical alert fires |
| Audit ledger unreachable | Requests continue and records are buffered. If the buffer is full, requests are refused | Failure counter rises and the Critical alert fires |
| Audit record cannot be made durable | Refuses the request | A 503 from the hook. No request proceeds unlogged |
| Router or enforcement hook unavailable | Refuses the request | A refused request. The router never forwards without the hook |
| Model catalogue unreadable | Denies every request until it can be read again | A deny reason that names admission state |
| Host record stale or missing | Refuses requests to that route | A deny reason in the response and the ledger |
| Token usage cannot be read | Denies the request | A deny reason naming token usage |
| Weight digest mismatch | Aborts model staging | A security event file, and a non-zero exit |
| Identity provider unreachable | Key-based requests continue. Sign-in sessions fail closed | A stated reason on the denied request |
| Licence expired | Service continues | A daily warning in the logs, and expiry shown in the admin view |
Policy unavailable
The enforcement hook asks the policy engine (OPA) for every decision. If the engine cannot be reached, or returns something the hook cannot parse, the decision is a denial. The reason reaches the user and the ledger. The metric sovereignai_policy_engine_up reads 0, and the Critical alert PolicyEngineUnavailable watches it.
Audit unavailable
The audit service writes each event to a local write-ahead buffer and syncs it to disk before the hook answers. If the ledger database is unreachable, requests continue and the buffer keeps every event and retries. The counter sovereignai_audit_write_failures_total rises and the Critical alert AuditWriteFailure fires. When the database returns, the buffer drains into the ledger. If the buffer itself fills, the hook refuses new requests rather than let them go unlogged.
The decision row is durable before the answer. If the hook cannot get the record accepted it answers 503 and never allows the request. The report the router sends after the forward is different: the answer is already streaming, so a failed write there is logged, and the report is never answered as a success.
Router or hook unavailable
The router calls the hook before every forward, with a setting that refuses the request if the hook is down, slow or in error. That setting must stay off, and the chart documents that turning it on would let an outage bypass the ledger. The hook pod does not start until the router answers ready, so with the router down there is no inference and no fallback path to a model server.
This refusal behaviour is written into the chart. The two tests that prove it against a running router have not yet been run, so treat it as designed, not demonstrated.
Stale or missing host record
A hosted provider route is allowed only if its host record is present and admitted. Each host record carries a review date. Staleness is checked at request time: after that date, the mapping is refused until a person reviews the record again. The refusal names the stale record and the date, and it is written to the ledger. A stale record refuses new governed requests. It does not stop anything already running and it does not contact anyone.
Model catalogue unreadable
The hook loads the admitted model catalogue at start, and will not serve without it. It re-reads the catalogue every 60 seconds by default. If a re-read fails, every request is denied until a re-read succeeds. A policy that names a model the catalogue does not admit is refused when it loads, not at request time.
Failed digest check
At model staging, a digest mismatch, a format that is not safetensors, a missing or wrong signature when a key is pinned, or a set that is only partly present aborts the pull. Nothing is written to the destination. A security event is recorded. See deploying for the order of checks.
Identity provider or management link unavailable
- If the identity provider cannot be reached, requests that present a virtual key continue. A request that relies on a browser session fails closed with a stated reason, and a new sign-in fails closed. This is by design.
- The management link is optional and outbound only. If it is absent, inference, policy and audit continue.
- An expired licence does not stop inference. The licence is checked offline, one warning is logged each day, and expiry is shown in the admin view. Heliast has no kill switch.
These two behaviours are implemented and tested, but they have not been demonstrated in a real estate.
What the operator should watch
AuditWriteFailureandPolicyEngineUnavailable, the two Critical alerts.- Deny reasons in the ledger, which name the cause of each refusal.
- Security event files from model staging.
Limits
- Only these behaviours are described. Other failures, such as a model server crash, are not covered here. Confirm them with your Heliast contact.
- No dashboards are provisioned. Alerts are rules in the in-cluster monitoring stack, and the alert delivery route is yours to configure.
- One metric named in the alert design, the audit completeness ratio, is not yet exposed, so its alert cannot fire.
- A single signed anchor cannot detect deletion of rows after the anchored head.