Compare

What Compare is

Compare runs your own tasks across candidate models and scores the answers with checks you define. The result is a measurement on your data. It is not a general ranking of models, and it does not replace the admission verdict. Admission tells you whether a model's licence, provenance and files check out. Compare tells you how a model did on the tasks you gave it. See Reading a verdict for the first.

Heliast never reads your prompt or completion text for this purpose, and it does not host model weights. Compare calls the models you already route to, through the enforcement hook.

The golden set

A golden set is the set of tasks you want a model to do well, with a way to tell whether it did. A golden set lives in a folder with a manifest and one file per task.

The manifest carries:

• a workload id, a name and a version
• a default scorer
• optional run settings: attempts per task, temperature (0 by default) and a token limit
• caveats, which travel into the report

Each task file carries:

• an id
• the input: the request you would send the model
• the expected outcome: your own baseline, such as what production returns today
• the scorer for that task

The set can live in your repository or ours. Compare takes a path.

Build the set from real work, scrubbed of anything you must not share. A small set that matches your workload is more useful than a large generic one.

Scorers

A scorer is the check that decides whether an answer passes. Scorers are executable. There is no model acting as judge. Six kinds exist:

A command scorer runs in a fresh temporary directory with a cleaned environment, a time limit and captured output. It is a process sandbox, not a kernel sandbox. On Linux it can also be run without network access when the system supports it. The report states this limit. Write scorers with your own toolchain, and treat them as code you are choosing to run.

How a run works

1. Compare loads the golden set and checks the candidate models. It refuses to run a model that is not pinned by a weights digest and a recorded revision.
2. For each model, it sends each task through the hook. The request is policy-checked and written to the audit ledger like any other request. The run uses a virtual key. A provider API key is refused and never forwarded.
3. Each answer is scored by the task's scorer. A task may be attempted more than once if the manifest allows. A task counts as completed when an attempt passes.
4. Compare records the measurements per model, and each row carries its environment, its provenance label and its cost basis.
5. Runs are retained. In the console, Run Compare starts a run, and the retained runs can be chosen as a base run (A) and a compared run (B) to see them side by side.

Runs are one model at a time. Separate runs can be combined into one comparison.

What the report shows

• A recommendation and a confidence rating. The recommendation is the result of a fixed rule applied to the figures, and the page says it is the report's own rating. Confidence is low when fewer than two models completed tasks, when a required figure is missing, when any task needs human review, when the winner completed fewer than five tasks, or when the lead is under five percentage points. It is high only when the lead is ten points or more and none of those apply. If no model completed a task, the report makes no recommendation.
• Pass rate. The share of scored tasks that passed. Human-review tasks are excluded.
• A block per task. Green passed, red failed, grey needs a person. Each block also carries a tick, a cross or a question mark, so colour is never the only signal.
• Latency. Time to first token, and the 50th, 95th and 99th percentiles of end-to-end time, one value per completed task.
• Tokens per completed task. Prompt and completion tokens from the server, summed over every attempt.
• GPU memory high-water mark. The highest value sampled during the run, for models served on your own GPUs.
• Cost per completed task. Every attempt is charged. For a model on your own or private-cloud hardware, cost comes from hardware amortisation, with a view showing how it changes with utilisation. For a hosted route it comes from the provider's token price and the date that price was captured. These two bases are not equivalent, and the report says so wherever both appear. A row with no cost basis or no provenance label is refused, not shown.
• Caveats. Everything the report could not establish, stated in plain words.

Provenance label on each row

A row for a local or private-cloud model is labelled verified only if the run hashed the model's weights and they matched the pinned digests in the catalogue. The run checks the directory it is told about and refuses a changed file, a missing file or an extra weights file. It cannot see whether the server loaded those files. A hosted or frontier row is labelled "provenance by host attestation, not verified", always.

Tool-call conformance

If your workload is an agent, how a model handles tool definitions matters. Tool-call conformance is a separate measurement run with compare toolcall. It measures how a model, behind a particular serving stack, handles tool definitions.

• The official suite is a fixed, versioned set of scenarios, identical for every candidate. You can also supply your own tools and scenarios. Your tools file can be OpenAI function JSON or an MCP server's tools/list output saved as JSON. It is read as schemas: no MCP server is contacted and no tool is run.
• Each scenario ends as pass, fail, unsupported or error. Unsupported means the server refused the request shape, which usually means tool calling is not switched on in the server. That says something about the serving setup, not about the model. Error means nothing was measured.
• The checks are executable. They use a subset of JSON Schema and report any keyword they did not check, so a result never claims more validation than was done.
• Results are listed scenario by scenario. There is no combined score and no ranking.
• It does not tell you whether an agent finished a task, and Compare reports no tool-call success rate, agent completion rate or policy-violation count.
• The result describes a model behind a serving stack on a date. The chat template, the tool-call parser and the quantisation all change it. You record these as notes, and they are stated by you and not verified.
• Arguments are kept out of the result unless you ask for them to be retained. Your own runs are kept apart from any set published for general reading.

Limits

• A measurement on your data. A pass rate describes the tasks in your golden set. It is not a ranking of models in general and does not predict performance on other work. A small set gives a wide margin of error.
• Baseline agreement, not ground truth. Where the expected outcome is your production output, a pass means agreement with that baseline.
• No latency or throughput claim from non-production hardware. Latency and cost measured on hardware that is not your production hardware describe that hardware and the harness. They are not a claim about production. Latency on hosted routes reflects the network at run time, not the model alone. The report says so. Concurrency, such as how many users a deployment can serve at once, is not measured by Compare.
• Executable checks only. What a scorer cannot test, it cannot score. Use human tasks for judgement, and note that they are left out of the pass rate.
• Process sandbox. Scorer commands are not isolated at kernel level.
• Pinned models only. A model without a pinned digest and revision does not run. A model with no per-file pins in the catalogue cannot be hash-checked and is refused for local runs.
• Reproducibility. Runs use temperature 0 by default. A reproduce check compares two runs against stated tolerances. Re-run when your data or the candidate set changes.
• Not a decision. Compare gives you evidence. The decision framework is yours.