> ## Documentation Index
> Fetch the complete documentation index at: https://docs.brussle.com/llms.txt
> Use this file to discover all available pages before exploring further.
# Let your outcomes choose
> Score another engine, another wording of the question or another context recipe on the outcomes you already have, see the price first, and switch only when your outcomes prefer it and you confirm.
Before you switch a judgment to another engine, another wording or another recipe, let your outcomes say whether it is better. Brussle answers your labelled answers again with each alternative, corrects each on your outcomes the same way, and compares them with the active version on the same outcomes. You see the price before anything runs, a verdict that says how sure it is, and what your labels could not tell. Nothing switches unless you confirm.
An **alternative** is another definition of the judgment that differs from the active version only in its engine, its [context recipe](/guides/context-recipes), its question or its criteria. Its type, features, `applies_to` and `horizon` stay the same. Thresholds are never part of an alternative: your current thresholds stay when you activate it. It is not the shadow report's **candidate**, the version you are activating.
1. [Read the table](#read-the-table-and-its-quote) and its quote.
2. [Name alternatives](#name-an-alternative), or [draft wordings](#draft-wordings), beyond the automatic ones.
3. [Confirm a run](#confirm-a-run). It is billed as judging.
4. [Read the verdicts](#read-the-verdicts).
5. [Create a version](#create-a-version-from-an-alternative) from the one you choose, and confirm its shadow report.
6. [Read whether the gain held](#read-whether-the-gain-held) on the outcomes that come after.
Alternatives are on Team and Scale. On Developer, `GET .../alternatives` shows [recipe tuning's](/train/recipe-tuning) numbers without the recipes, with `plan_required`, and every other alternatives route returns `plan_required` (402). A version needs a correction, with 100 outcomes and 20 of each kind, before its alternatives can be scored.
## Read the table and its quote
`GET /namespaces/{ns}/judgments/{name}/alternatives` returns the table for the active version. It is `ns.judgments.alternatives(name)` in both SDKs. In the dashboard, the judgment's **Alternatives** tab shows the quote, the table in its two groups and the suggestion, and you name an engine, a wording or a recipe there.
Before a run, the table holds the quote of what is unscored:
```json theme={"theme":{"light":"css-variables","dark":"css-variables"}}
{
"version": 3,
"engine_version": "current+2026-09-24.1",
"status": "not_run",
"reason": "awaiting_confirm",
"automatic": false,
"skip_engines": [],
"labels": {"count": 312, "sampled": false, "outcomes": null},
"corrected_for": 0,
"quote": {
"status": "ready",
"generation": 4,
"labels": 312,
"alternatives": [
{"id": "alt_3f9a2c7e1b04d6a8", "contexts": "stored", "labels": 312, "tokens": 402480, "judgment_units": 780, "new_provider": false},
{"id": "alt_8c1d5e2f90a3b47c", "contexts": "stored", "labels": 312, "tokens": 405600, "judgment_units": 312, "new_provider": false}
],
"judgment_units": 1092,
"tokens": 808080,
"detectable": {"log_loss": 0.05, "auroc": 0.06, "k": 2},
"new_providers": [],
"upstream": []
},
"suggestion": null,
"headline": null,
"next_run_after": null,
"plan_required": null
}
```
The example leaves out `message`, the quote's `cost_usd` fields and the table's rows.
The table has two groups:
* **Engine and named alternatives** (`in_k: true`). They are scored only in a run you confirm, billed as judging. Each `better` verdict in this group is corrected for every alternative of the group scored this month.
* **Recipe tuning's smaller recipes** (`in_k: false`), from the free monthly [recipe tuning](/train/recipe-tuning) run. Their verdict is only `no_worse` or `worse`. They are never billed, and do not count in `corrected_for`.
Each engine this namespace already uses is an automatic alternative, other than the active version's own. An engine is in use when an active judgment version of the namespace uses it, or when it is the namespace's `default_engine`. An org admin can widen this to every engine any namespace of the organization uses (see [turn on suggestions](/guides/discovery#turn-on-suggestions)). To skip an engine, put it in `alternatives.skip_engines` (see [turn on automatic runs](#turn-on-automatic-runs)).
**The quote** prices each unscored engine or named alternative before anything runs:
* **For each alternative:** the labelled answers it would send (`labels`), their `tokens`, `judgment_units` and `cost_usd`, and `contexts` (see [how contexts are made](#how-contexts-are-made)).
* **Totals:** `judgment_units`, `tokens` and `cost_usd` for the run.
* **`detectable`:** the smallest gain in log loss, and in AUROC, that this run can call `better` on these labels with `k` alternatives. A smaller real gain reads `no_worse`. Read it before you pay (see [what the labels can show](#what-the-labels-can-show)).
* **`new_providers`:** engines that would get this namespace's labelled contexts for the first time.
* **`upstream`:** the backfill a recipe needs first, when it selects related documents by another judgment's answers. The run never starts it.
* **`generation`:** which quote this is. It changes when the alternatives, the outcomes or the engines in use change.
Each call the run answers is billed as judging: its size class times its engine's [weight](/pricing#judgments), at the prices of your organization's current tiers. The [pricing page](/pricing#judgments) gives an example.
The quote is priced in the background within seconds (`status: quoting` until then), and again whenever the alternatives, skipped engines or fit change.
| `status` | `reason` | Means |
| - | - | - |
| `not_run` | `awaiting_confirm` | There is something to score, and automatic runs are off. Confirm the quote. |
| `not_run` | `awaiting_run` | Automatic runs are on and the next fit starts the run, or there is nothing to score yet. |
| `not_run` | `too_few_outcomes` | The version has no correction yet. |
| `not_run` | `composite`, `inherited` | No alternatives are scored for a composite. A template's tenants read the template's table. |
| `quoting` | | The quote is being priced. |
| `running` | `budget_paused` or none | This month's run is scoring. With `budget_paused`, it waits for the namespace's budget. |
| `ready` | | `suggestion` names the alternative your outcomes prefer. |
| `none` | `nothing_better` | No alternative was better beyond noise. `message` gives the gain these labels could have shown. |
| `none` | `accuracy_would_drop` | Each recipe that was cheap enough did worse on your outcomes. |
| any | `superseded` | The engine changed during the run. |
## Name an alternative
`POST /namespaces/{ns}/judgments/{name}/alternatives` adds alternatives. Each gives one or more of `engine`, `context`, `question` and `criteria`; the active version gives the rest.
```ts TypeScript theme={"theme":{"light":"css-variables","dark":"css-variables"}}
await ns.judgments.nameAlternatives("material_change", {
alternatives: [
{ engine: { name: "gpt-6-luna", version: "current" } },
{ question: "Does this edit change what either party must do, pay or deliver?" },
{ context: { fields: ["state.heading", "state.text"], previous: { fields: ["state.text"] } } },
],
});
```
```python Python theme={"theme":{"light":"css-variables","dark":"css-variables"}}
ns.judgments.name_alternatives(
"material_change",
alternatives=[
{"engine": {"name": "gpt-6-luna", "version": "current"}},
{"question": "Does this edit change what either party must do, pay or deliver?"},
{"context": {"fields": ["state.heading", "state.text"], "previous": {"fields": ["state.text"]}}},
],
)
```
The response is the table, with the new quote.
* **Each alternative is checked as a new version.** The engine must be `active`, the recipe must compile, and the question must be within the limits. On a judgment with features, the recipe must keep each relation a feature reads. Otherwise `invalid_request`, with a message naming the feature.
* **An alternative that judges exactly like the active version is refused:** "This is the active version."
* **At most 10 named alternatives at a time,** and at most 10 engine and named alternatives scored each month. Remove one with `DELETE .../alternatives/{id}` to name another. You cannot remove an automatic engine alternative; skip its engine instead.
* **The same definition is the same alternative,** with an id of `alt_` and 16 hexadecimal characters. A second request for it adds nothing.
* **Do not send `run` and `confirm` with the names.** That is `conflict`, because you have not read the new quote; the alternatives are still named. With `run: true` alone, the response is the quote.
**An engine this namespace does not use.** When you name one and confirm the run, Brussle sends the stored contexts of your labelled answers to that engine's provider. The quote names it in `new_providers`, and its row has `new_provider: true`. Each [engine's page](/engines/index) says which of your data its provider gets.
### How contexts are made
* **`stored`**: same recipe, for example a new engine or wording. Brussle sends the context each labelled answer read, as stored.
* **`narrowed`**: the recipe reads less. Brussle cuts each stored context to the smaller recipe.
* **`live`**: the recipe reads more, such as a new field or relation, a larger `last_n`, or a relation that selects by another judgment's answers. Brussle compiles each context from today's documents.
A `live` row cannot read what each labelled answer read on the day of its label. Its row says `as_of: "now"`, with this sentence: "Its contexts were built from today's related documents and answers, not from what your labelled answers read; a change in those documents since the label counts against or for it." Compare it with care. Brussle skips a labelled answer whose document you deleted, and counts it in `labels_skipped.deleted`.
A recipe that selects related documents by another judgment's answers needs that judgment active, with an answer on each document it applies to. Otherwise naming it is `invalid_request`, with a message such as "`topic` has no answers on 1,204 documents the recipe reads; backfill it first", and `details` giving the backfill's estimate.
### Draft wordings
`POST /namespaces/{ns}/judgments/{name}/alternatives/drafts` returns up to four other wordings of the question and its criteria. It is `draftWordings(name)` in TypeScript and `draft_wordings(name)` in Python. In the dashboard, it is **Draft wordings**.
```json theme={"theme":{"light":"css-variables","dark":"css-variables"}}
{"drafts": [{"question": "Does this edit change what either party must do, pay or deliver?", "criteria": "Yes only if an obligation, amount or date changes."}]}
```
* **A third-party language model writes the drafts,** the same one that [suggested parts](/guides/composite-judgments) and [discovery](/guides/discovery) use. Brussle sends it only the active version's question, criteria, and options or levels: no document, context, example or outcome.
* **Drafts are off until an org admin turns on Suggestions;** until then the call is `forbidden`. See [turn on suggestions](/guides/discovery#turn-on-suggestions).
* **Each call uses one of the day's `suggest_parts` calls:** 20 for each judgment and 50 for each organization, then `rate_limited` until midnight UTC.
* **Brussle stores and scores no draft.** Read it, edit it if you like, and name it as an alternative with `"drafted": true` beside its `question` and `criteria` to score it.
A drafted wording is not better because a model wrote it. Your outcomes measure it like any other, and its row shows `drafted: true`.
## Confirm a run
`POST /namespaces/{ns}/judgments/{name}/alternatives/run` scores the unscored engine and named alternatives.
1. Send it without `confirm`. The response is the table with the current quote. Nothing runs or is billed.
2. Read the quote: its cost, `detectable` and `new_providers`.
3. Send it again with `confirm: true` and the `quote_generation` you read.
```ts TypeScript theme={"theme":{"light":"css-variables","dark":"css-variables"}}
const table = await ns.judgments.runAlternatives("material_change");
await ns.judgments.runAlternatives("material_change", { confirm: true, quote_generation: table.quote?.generation });
```
```python Python theme={"theme":{"light":"css-variables","dark":"css-variables"}}
table = ns.judgments.run_alternatives("material_change")
ns.judgments.run_alternatives("material_change", confirm=True, quote_generation=table["quote"]["generation"])
```
The confirm starts this month's run and returns 202. In the dashboard, **Run** shows the quote, the budget and each new provider, and asks you to confirm.
* **A confirm is for the current quote.** If the quote changed since you read it, or is being priced again, the confirm is `conflict`. Without `quote_generation`, Brussle confirms the current quote.
* **One run for each version, engine epoch and month.** Naming more alternatives and confirming again extends it, on the same labelled answers. An extension can change a verdict: each is checked again against every alternative scored this month, so `better` can become `no_worse`.
* **`outcomes`** chooses the outcomes to score on, by source and by when observed, as for a [named run](/train/fits#fit-on-outcomes-you-choose). The month's first run fixes them.
* **Billing.** Each answered call is billed as judging, and the namespace's [budget](/behavior#budgets) holds the run; while it does not fit, the run waits with `budget_paused`. A failed call bills nothing, and no call is billed twice. The monthly allowance for free replays does not stop this run.
* **Nothing the run scores is an answer.** Nothing is written to the evaluation log, and the run makes its own fits only to compare: no answer ever reads them.
## Read the verdicts
Brussle scores each alternative and the active version on the same labelled answers: at most 500, those of the correction. Each answer is read with a correction made without that answer, as the nightly fit chooses its correction.
```json theme={"theme":{"light":"css-variables","dark":"css-variables"}}
{
"id": "alt_3f9a2c7e1b04d6a8",
"source": "automatic",
"drafted": false,
"differs": ["engine: jev current → gpt-6-luna current"],
"in_k": true,
"contexts": "stored",
"as_of": null,
"labels": 312,
"labels_skipped": {"deleted": 0, "image_erased": 0, "not_derivable": 0, "failed": 0},
"cost_per_answer": 2.5,
"held_out": {"read_with": "calibrated", "log_loss": 0.391, "auroc": 0.88, "accuracy": 0.86},
"change": {"log_loss": {"mean": -0.061, "interval": {"lower": -0.093, "upper": -0.029}, "p": 0.0001}, "auroc": {"mean": 0.02, "interval": {"lower": -0.01, "upper": 0.05}, "p": 0.09}, "accuracy": 0.03},
"verdict": "better",
"moves_ranking": false
}
```
Some fields are left out.
* **`better`**: lower corrected log loss beyond noise, after the correction for every engine and named alternative scored this month.
* **`moves_ranking`**: ranks your outcomes better (AUROC) beyond noise, after the same correction.
* **`no_worse`**: worse by no more than one standard error or 1%. This verdict has no correction.
* **`worse`**: worse beyond that.
* **`untested`**: too few labelled answers after its skips: fewer than 100, or fewer than 20 of each kind.
* **`unscored`**: not scored yet.
`change` is the alternative minus the active version on the same labelled answers, with its 95% `interval`; for log loss, below 0 is better. `billed` gives what scoring it billed.
**What "corrected for" means.** The best of several equal alternatives still looks better by chance, more so the more you score, so `better` must pass a higher bar for each engine or named alternative scored this month (`corrected_for`). Score only the alternatives you would really choose.
### What the labels can show
With a few hundred labelled answers, only large gains read `better`:
* With one or two alternatives, roughly a gain in log loss of 0.06 at 100 labelled answers and 0.03 at 500. With ten, the gain must be larger.
* An engine that is right five points more often can show at 500 labelled answers.
* A change of recipe or wording that moves the numbers by a few points usually reads `no_worse`.
The quote's `detectable` gives this for your labels before you pay.
**The suggestion.** `suggestion` names the `better` alternative with the lowest log loss; a tie goes to the one that moves the ranking, then the cheaper. If none is `better`, it is recipe tuning's cheaper recipe, with `because: "cheaper"`. If there is neither, it is `null`, and `message` gives the gain these labels could have shown. `headline` says it in one sentence, such as "gpt-6-luna current is better on the calibrated number: held-out log loss 0.39 against 0.45 on 312 labelled answers, corrected for 2 alternatives tried; ranking unchanged within noise."
The best of several looks better on the outcomes it was chosen on than it will later. Trust the number measured after you switch.
## Create a version from an alternative
`POST /namespaces/{ns}/judgments/{name}/alternatives/{id}/version` creates the alternative's definition as a new version, as `POST /judgments` does, and starts its [shadow job](/guides/change-a-judgment#change-a-question-safely-with-a-shadow-report), as `activate` without `force` does. The response is the 202 job. In the dashboard, **Create version** sends you to the shadow report on the **Overview**.
```ts TypeScript theme={"theme":{"light":"css-variables","dark":"css-variables"}}
const job = await ns.judgments.createVersionFromAlternative("material_change", "alt_3f9a2c7e1b04d6a8");
```
```python Python theme={"theme":{"light":"css-variables","dark":"css-variables"}}
job = ns.judgments.create_version_from_alternative("material_change", "alt_3f9a2c7e1b04d6a8")
```
The shadow report gets `alternative` beside `recompute` and `calibration_replay`:
```json theme={"theme":{"light":"css-variables","dark":"css-variables"}}
{"alternative": {"id": "alt_3f9a2c7e1b04d6a8", "verdict": "better", "change": {"log_loss": {"mean": -0.061, "interval": {"lower": -0.093, "upper": -0.029}}, "auroc": null, "accuracy": 0.03}, "k": 2, "drafted": false}}
```
* **`verdict` and `k`**: the verdict, and how many alternatives it was corrected for, when you created the version.
* **`recompute`** prices the backfill of the old answers, and **`calibration_replay`** the [replay](/train/engine-changes) that fits the new version on another engine.
* **Confirm the job to activate the version,** or cancel it to keep the active one.
The new version is corrected as any new version is: from the billed replay on another engine, or from its own outcomes on the same engine. It never reads a fit the alternatives run made.
## Read whether the gain held
On a version created from an alternative, each epoch of the [report](/train/report) has `alternative_selection`:
```json theme={"theme":{"light":"css-variables","dark":"css-variables"}}
{
"alternative_selection": {
"alternative": "alt_3f9a2c7e1b04d6a8",
"drafted": false,
"selected_on": {"fit": "fit_9f3a2c7e1b04d6a8", "version": 3, "labels": 312, "log_loss": 0.391, "auroc": 0.88, "raw_gap": 0.048, "computed_at": "2026-10-14T03:12:40Z"},
"since": {"outcomes": 140, "log_loss": 0.402, "auroc": 0.87, "raw_gap": 0.051, "interval": {"lower": 0.02, "upper": 0.082}},
"held": true
}
}
```
* **`selected_on`**: the numbers it was chosen on: the replaced version's fit and version, the labelled answers, log loss, AUROC, and `raw_gap`, the corrected number's gain in log loss over the engine's raw numbers.
* **`since`**: the same on the outcomes observed after the choice, with the 95% interval of `raw_gap`. `null` under 100 outcomes or 20 of each kind.
* **`held`**: `true` when the gain over raw since the choice is no smaller than the gain it was chosen on, beyond one standard error; `false` when it is smaller by more, and the epoch's `warnings` then has `alternative_gain_not_held`; `too_few` under 100 outcomes or 20 of each kind.
`held` compares gains, not log losses. A shift in your outcomes, such as a new base rate, moves both log losses but not the gain, and the reverse can also happen.
## Turn on automatic runs
By default, a run starts only when you confirm it. To let it start by itself after a correction fit, turn on `alternatives.automatic`, in a `PATCH` of its own:
```json PATCH /v1/namespaces/acme%2Fprod/judgments/material_change theme={"theme":{"light":"css-variables","dark":"css-variables"}}
{"alternatives": {"automatic": true, "skip_engines": []}, "confirm": true}
```
* **Without `confirm`,** the response is `{"alternatives_quote": {...}}`, the quote of a full run, and nothing changes.
* **An automatic run** starts after a fit, at most once a month. It scores at most 10 alternatives on at most 500 labelled answers, billed as judging. `next_run_after` gives the earliest next start.
* **`skip_engines`** lists the engines not to score as automatic alternatives.
In the dashboard, the **Automatic** switch shows the quote of a full run and asks you to confirm. When a fit lands with unscored alternatives and automatic runs are off, Brussle sends `judgment.alternatives_awaiting_confirm`, with the quote. When a run finds an alternative `better`, it sends `judgment.alternative_scored`. See [event types](/guides/events-feed#event-types).
This documentation is built and hosted on [Mintlify](https://mintlify.com), a developer documentation platform.