Use Heliast with LiteLLM

This page sets up the Heliast adapter for the LiteLLM proxy. Read use Heliast with a gateway you already run first.

Tested with LiteLLM 1.105.0, on chat completions, streamed and not streamed. Other call types and versions are listed under Limits.

1. Give the hook an adapter token

Create a secret value for the adapter and set it on the hook:

GATEWAY_ADAPTER_TOKENS=litellm=<a random value of at least 32 bytes>

The same value goes to the adapter in step 3. Keep it out of any repository. To let LiteLLM users have a role other than standard, also set GATEWAY_ADAPTER_ROLES=litellm=standard|premium on the hook.

2. Install the adapter

Your Heliast contact supplies the heliast_litellm package. Place it where the LiteLLM proxy can import it, for example next to your LiteLLM config.

3. Configure LiteLLM

Add the adapter as a callback in your LiteLLM config:

litellm_settings:
  callbacks: heliast_litellm.heliast_hook

general_settings:
  allowed_routes: ["/v1/chat/completions", "/chat/completions", "/v1/completions", "/completions", "/v1/embeddings", "/embeddings"]

The allowed_routes line is not optional. Some LiteLLM routes reach a model without running the adapter. In testing, /v1/messages/count_tokens sent the prompt to the model with no decision and no ledger row. Listing only the routes the adapter decides closes the others.

LiteLLM describes allowed_routes as an Enterprise feature. In version 1.105.0 it is enforced without a licence, and LiteLLM logs an error saying so on every request. A later version may stop enforcing it without a licence, so check after every upgrade that /v1/messages/count_tokens is still refused. If you would rather not depend on it, put a reverse proxy or an ingress rule in front of LiteLLM that passes only those routes.

Set these in the environment of the LiteLLM proxy:

Variable	What it does
`HELIAST_HOOK_URL`	The address of the Heliast hook. Required. Must be https, or on the same machine.
`HELIAST_HOOK_TOKEN`	The adapter token from step 1. Required.
`HELIAST_HOOK_TIMEOUT_SECONDS`	How long to wait for the hook before refusing the request. Default 5.
`HELIAST_DEFAULT_GROUPS`	The group for a key that LiteLLM gives no team. Leave it empty unless you mean every team-less key, including the master key, to be entitled as that group.
`HELIAST_DEFAULT_ROLE`	The role asserted for every LiteLLM caller. Default `standard`. The hook accepts it only if you allowed it for this adapter.
`HELIAST_ALLOW_INSECURE_HOOK`	Set to `1` to allow a plain http hook address that is not on the same machine.
`HELIAST_ON_MODEL_MISMATCH`	`refuse` (default) withholds a non-streamed answer from a model the hook did not decide on. `log` records it and delivers the answer.

4. Map your callers

The adapter tells the hook who LiteLLM authenticated. The hook decides entitlement from that.

The subject is the LiteLLM user id, or the key alias if there is no user, or a short prefix of the hashed key. The ledger shows it as gw-litellm:<subject>.
The group is the LiteLLM team. Your Heliast policy entitles groups to models, so name the group in your policy the same as the LiteLLM team, and give teams names without spaces.
The role is HELIAST_DEFAULT_ROLE for every caller. A LiteLLM key's metadata does not choose the role, because whoever creates a key writes its metadata.

5. Use admitted model names

Clients should ask for the admitted model id. Set the LiteLLM model_name to that id. A name your gateway invents is refused unless a host record maps it to an admitted id.

6. Close the direct routes

Allow connections to your model servers only from the LiteLLM proxy, and connections to the hook only from the LiteLLM proxy and your operators.

7. Check it

Send a chat request for an admitted model. It is answered. The ledger has a decision row and an inference row with token counts, and both carry the same request id.
Send a request for a model that is not admitted. It is refused with the reason, nothing reaches a model, and the ledger has a denial row.
Stop the hook and send a request. It is refused.
Make the ledger unavailable and send a request. It is refused.

What the adapter does to the request

It adds stream_options.include_usage to a streamed chat request, so the last chunk carries the token counts the report needs. It changes nothing else, and it sends the prompt only to the hook, which hashes it and keeps nothing.

Limits

Call types. Chat completions and text completions are decided. Embeddings have a path in the adapter but are not yet tested. Every other call type is refused, including the responses API, images and audio.
Routes that skip the adapter. Measured: every route that carries a model call through the adapter was decided or refused, except /v1/messages/count_tokens, which reached the model. allowed_routes closes it. The LiteLLM pass-through routes (such as /openai/... and /anthropic/...), batches, assistants and files could not be tested here; do not leave them in allowed_routes.
Disconnects. A streamed answer is reported when the stream ends. The report is protected against a cancelled request, but a LiteLLM process that stops mid-stream sends none.
Streaming. A streamed answer cannot be withheld after the fact. If the model that answers is not the one decided on, the mismatch is recorded and the answer is delivered.
Fallbacks. Configure LiteLLM fallbacks only between admitted models your policy entitles.
LiteLLM keys with teams. The mapping in step 4 reads the team from the LiteLLM key. It is tested with the master key and the default group. A key with a team needs the LiteLLM database and has not been tested yet.
Report failures. If the hook cannot write the inference row after the answer, the adapter logs an error and the answer is still delivered. The hook counts the failure on its own metrics.
Versions. Only LiteLLM 1.105.0 is tested. Test an upgrade before you roll it out.