> ## Documentation Index
> Fetch the complete documentation index at: https://docs.brussle.com/llms.txt
> Use this file to discover all available pages before exploring further.

# Let your outcomes choose

> Score another engine, another wording of the question or another context recipe on the outcomes you already have, see the price first, and switch only when your outcomes prefer it and you confirm.

You can let your outcomes say whether another engine, another wording or another recipe is better before you switch a judgment to it. Brussle answers your labelled answers again with each alternative, corrects each one on your outcomes the same way, and compares them with the active version on the same outcomes. You see the price before anything runs, then a verdict that says how sure it is and what your labels could not tell. Nothing switches unless you confirm.

An **alternative** is another definition of the judgment that differs from the active version only in its engine, its [context recipe](/guides/context-recipes), its question or its criteria, while its type, features, `applies_to` and `horizon` stay the same. Thresholds are never part of an alternative, so your current thresholds stay when you activate one. (An alternative is not the shadow report's **candidate**, which is the version you are activating.)

1. [Read the table](#read-the-table-and-its-quote) and its quote.
2. [Name alternatives](#name-an-alternative), or [draft wordings](#draft-wordings), beyond the automatic ones.
3. [Confirm a run](#confirm-a-run), which is billed as judging.
4. [Read the verdicts](#read-the-verdicts).
5. [Create a version](#create-a-version-from-an-alternative) from the one you choose, and confirm its shadow report.
6. [Read whether the gain held](#read-whether-the-gain-held) on the outcomes that come after.

Alternatives are on Team and Scale. On Developer, `GET .../alternatives` shows [recipe tuning's](/calibration/recipe-tuning) numbers without the recipes, with `plan_required`, and every other alternatives route returns `plan_required` (402). Brussle scores a version's alternatives only once the version has a correction with 100 outcomes, 20 of each kind.

## Read the table and its quote

`GET /namespaces/{ns}/judgments/{name}/alternatives` (`ns.judgments.alternatives(name)` in both SDKs) returns the table for the active version. The judgment's **Alternatives** tab in the dashboard shows the quote, the table in its two groups and the suggestion, and you name an engine, a wording or a recipe there.

Before a run, the table holds the quote of what is still unscored.

```json theme={"theme":{"light":"css-variables","dark":"css-variables"}}
{
  "version": 3,
  "engine_version": "current+2026-09-24.1",
  "status": "not_run",
  "reason": "awaiting_confirm",
  "automatic": false,
  "skip_engines": [],
  "labels": {"count": 312, "sampled": false, "outcomes": null},
  "corrected_for": 0,
  "quote": {
    "status": "ready",
    "generation": 4,
    "labels": 312,
    "alternatives": [
      {"id": "alt_3f9a2c7e1b04d6a8", "contexts": "stored", "labels": 312, "tokens": 402480, "judgment_units": 780, "new_provider": false},
      {"id": "alt_8c1d5e2f90a3b47c", "contexts": "stored", "labels": 312, "tokens": 405600, "judgment_units": 312, "new_provider": false}
    ],
    "judgment_units": 1092,
    "tokens": 808080,
    "detectable": {"log_loss": 0.05, "auroc": 0.06, "k": 2},
    "new_providers": [],
    "upstream": []
  },
  "suggestion": null,
  "headline": null,
  "next_run_after": null,
  "plan_required": null
}
```

A full response also has `message`, the quote's `cost_usd` fields and the table's rows.

The table has two groups:

* **Engine and named alternatives** (`in_k: true`). Brussle scores them only in a run you confirm, which is billed as judging, and corrects each `better` verdict in this group for every alternative of the group scored this month.
* **Recipe tuning's smaller recipes** (`in_k: false`), which come from the free monthly [recipe tuning](/calibration/recipe-tuning) run. Their verdict is only ever `no_worse` or `worse`. They are never billed and never count in `corrected_for`.

Every engine this namespace already uses, other than the active version's own, is an automatic alternative. An engine is in use when an active judgment version of the namespace uses it or when it is the namespace's `default_engine`, and an org admin can widen this to every engine that any namespace of the organization uses (see [turn on suggestions](/guides/discovery#turn-on-suggestions)). To skip an engine, put it in `alternatives.skip_engines` (see [turn on automatic runs](#turn-on-automatic-runs)).

The quote prices each unscored engine or named alternative before anything runs:

* **For each alternative:** the labelled answers it would send (`labels`), their `tokens`, `judgment_units` and `cost_usd`, and `contexts` (see [how contexts are made](#how-contexts-are-made)).
* **Totals:** `judgment_units`, `tokens` and `cost_usd` for the run.
* **`detectable`:** the smallest gain in log loss, and in AUROC, that this run can call `better` on these labels with `k` alternatives. A smaller real gain reads `no_worse`, so read it before you pay (see [what the labels can show](#what-the-labels-can-show)).
* **`new_providers`:** engines that would get this namespace's labelled contexts for the first time.
* **`upstream`:** the backfill a recipe needs first, when it selects related documents by another judgment's answers. The run never starts that backfill.
* **`generation`:** which quote this is. It changes when the alternatives, the outcomes or the engines in use change.

Brussle bills each call the run answers as judging, at its size class times its engine's [weight](/pricing#judgments) and at the prices of your organization's current tiers (the [pricing page](/pricing#judgments) gives an example).

Brussle prices the quote in the background within seconds (`status: quoting` until then), and prices it again whenever the alternatives, skipped engines or fit change.

| `status` | `reason` | Means |
| - | - | - |
| `not_run` | `awaiting_confirm` | There is something to score, and automatic runs are off. Confirm the quote. |
| `not_run` | `awaiting_run` | Automatic runs are on and the next fit starts the run, or there is nothing to score yet. |
| `not_run` | `too_few_outcomes` | The version has no correction yet. |
| `not_run` | `composite`, `inherited` | No alternatives are scored for a composite. A template's tenants read the template's table. |
| `quoting` | | Brussle is pricing the quote. |
| `running` | `budget_paused` or none | This month's run is scoring. With `budget_paused`, it waits for the namespace's budget. |
| `ready` | | `suggestion` names the alternative your outcomes prefer. |
| `none` | `nothing_better` | No alternative was better beyond noise. `message` gives the gain these labels could have shown. |
| `none` | `accuracy_would_drop` | Each recipe that was cheap enough did worse on your outcomes. |
| any | `superseded` | The engine changed during the run. |

## Name an alternative

`POST /namespaces/{ns}/judgments/{name}/alternatives` adds alternatives. Each one sets one or more of `engine`, `context`, `question` and `criteria`, and takes the rest from the active version.

<CodeGroup>
  ```ts TypeScript theme={"theme":{"light":"css-variables","dark":"css-variables"}}
  await ns.judgments.nameAlternatives("material_change", {
    alternatives: [
      { engine: { name: "gpt-6-luna", version: "current" } },
      { question: "Does this edit change what either party must do, pay or deliver?" },
      { context: { fields: ["state.heading", "state.text"], previous: { fields: ["state.text"] } } },
    ],
  });
  ```

  ```python Python theme={"theme":{"light":"css-variables","dark":"css-variables"}}
  ns.judgments.name_alternatives(
      "material_change",
      alternatives=[
          {"engine": {"name": "gpt-6-luna", "version": "current"}},
          {"question": "Does this edit change what either party must do, pay or deliver?"},
          {"context": {"fields": ["state.heading", "state.text"], "previous": {"fields": ["state.text"]}}},
      ],
  )
  ```
</CodeGroup>

The response is the table, with the new quote.

* **Each alternative is checked as a new version.** The engine must be `active`, the recipe must compile and the question must be within the limits. On a judgment with features, the recipe must also keep each relation a feature reads, or the request returns `invalid_request` with a message naming the feature.
* **Brussle refuses an alternative that judges exactly like the active version,** with the message "This is the active version."
* **At most 10 named alternatives at a time,** and at most 10 engine and named alternatives scored each month. To name another, remove one with `DELETE .../alternatives/{id}`. You cannot remove an automatic engine alternative, so skip its engine instead.
* **The same definition is the same alternative,** with an id of `alt_` and 16 hexadecimal characters, so a second request for it adds nothing.
* **Do not send `run` and `confirm` with the names.** Brussle returns `conflict`, since you have not read the new quote. The alternatives are still named. With `run: true` alone, the response is the quote.

When you name an engine this namespace does not use and confirm the run, Brussle sends the stored contexts of your labelled answers to that engine's provider. The quote names that engine in `new_providers`, its row has `new_provider: true`, and each [engine's page](/engines/index) says which of your data its provider gets.

### How contexts are made

* **`stored`**: the recipe is the same (a new engine or wording, for example), so Brussle sends the context each labelled answer read, as stored.
* **`narrowed`**: the recipe reads less, so Brussle cuts each stored context to the smaller recipe.
* **`live`**: the recipe reads more (a new field or relation, a larger `last_n`, or a relation that selects by another judgment's answers), so Brussle compiles each context from today's documents.

A `live` row cannot read what each labelled answer read on the day of its label, so its row says `as_of: "now"`, with this sentence: "Its contexts were built from today's related documents and answers, not from what your labelled answers read; a change in those documents since the label counts against or for it." Compare it with care. Brussle skips a labelled answer whose document you deleted, and counts it in `labels_skipped.deleted`.

A recipe that selects related documents by another judgment's answers needs that judgment active, with an answer on each document it applies to. If it is not, naming the recipe returns `invalid_request`, with a message such as "`topic` has no answers on 1,204 documents the recipe reads; backfill it first", and `details` giving the backfill's estimate.

### Draft wordings

`POST /namespaces/{ns}/judgments/{name}/alternatives/drafts` returns up to four other wordings of the question and its criteria. It is `draftWordings(name)` in TypeScript, `draft_wordings(name)` in Python and **Draft wordings** in the dashboard.

```json theme={"theme":{"light":"css-variables","dark":"css-variables"}}
{"drafts": [{"question": "Does this edit change what either party must do, pay or deliver?", "criteria": "Yes only if an obligation, amount or date changes."}]}
```

* **A third-party language model writes the drafts,** the same one that [suggested parts](/guides/composite-judgments) and [discovery](/guides/discovery) use. Brussle sends it only the active version's question, criteria, and options or levels (no document, context, example or outcome).
* **Drafts are off until an org admin turns on Suggestions,** and until then the call returns `forbidden` (see [turn on suggestions](/guides/discovery#turn-on-suggestions)).
* **Each call uses one of the day's `suggest_parts` calls** (20 for each judgment and 50 for each organization), and after those the call returns `rate_limited` until midnight UTC.
* **Brussle stores and scores no draft.** To score one, read it, edit it if you like, and name it as an alternative with `"drafted": true` beside its `question` and `criteria`.

A drafted wording is not better because a model wrote it, so your outcomes measure it like any other. Its row shows `drafted: true`.

## Confirm a run

`POST /namespaces/{ns}/judgments/{name}/alternatives/run` scores the unscored engine and named alternatives.

1. Send it without `confirm`. The response is the table with the current quote, and nothing runs or is billed.
2. Read the quote: its cost, `detectable` and `new_providers`.
3. Send it again with `confirm: true` and the `quote_generation` you read.

<CodeGroup>
  ```ts TypeScript theme={"theme":{"light":"css-variables","dark":"css-variables"}}
  const table = await ns.judgments.runAlternatives("material_change");
  await ns.judgments.runAlternatives("material_change", { confirm: true, quote_generation: table.quote?.generation });
  ```

  ```python Python theme={"theme":{"light":"css-variables","dark":"css-variables"}}
  table = ns.judgments.run_alternatives("material_change")
  ns.judgments.run_alternatives("material_change", confirm=True, quote_generation=table["quote"]["generation"])
  ```
</CodeGroup>

The confirm starts this month's run and returns 202. **Run** in the dashboard shows the quote, the budget and each new provider, and asks you to confirm.

* **A confirm is for the current quote.** If the quote changed since you read it, or Brussle is pricing it again, the confirm returns `conflict`. Without `quote_generation`, Brussle confirms the current quote.
* **One run for each version, engine epoch and month.** If you name more alternatives and confirm again, Brussle extends the run on the same labelled answers. An extension can change a verdict: Brussle checks each one again against every alternative scored this month, so `better` can become `no_worse`.
* **`outcomes`** chooses the outcomes to score on, by source and by when observed, as for a [named run](/calibration/fits#fit-on-outcomes-you-choose), and the month's first run fixes them.
* **Billing.** Brussle bills each answered call as judging, and the namespace's [budget](/behavior#budgets) holds the run: while the run does not fit, it waits with `budget_paused`. A failed call bills nothing, and no call is billed twice. The monthly allowance for free replays does not stop this run.
* **Nothing the run scores is an answer.** Brussle writes nothing to the evaluation log, and the run makes its own fits only to compare, so no answer ever reads them.

## Read the verdicts

Brussle scores each alternative and the active version on the same labelled answers, which are those of the correction (at most 500). Brussle reads each answer with a correction that did not see that answer's outcome, as the nightly fit's test does.

```json theme={"theme":{"light":"css-variables","dark":"css-variables"}}
{
  "id": "alt_3f9a2c7e1b04d6a8",
  "source": "automatic",
  "drafted": false,
  "differs": ["engine: jev current → gpt-6-luna current"],
  "in_k": true,
  "contexts": "stored",
  "as_of": null,
  "labels": 312,
  "labels_skipped": {"deleted": 0, "image_erased": 0, "not_derivable": 0, "failed": 0},
  "cost_per_answer": 2.5,
  "held_out": {"read_with": "calibrated", "log_loss": 0.391, "auroc": 0.88, "accuracy": 0.86},
  "change": {"log_loss": {"mean": -0.061, "interval": {"lower": -0.093, "upper": -0.029}, "p": 0.0001}, "auroc": {"mean": 0.02, "interval": {"lower": -0.01, "upper": 0.05}, "p": 0.09}, "accuracy": 0.03},
  "verdict": "better",
  "moves_ranking": false
}
```

A full row has more fields.

* **`better`**: lower corrected log loss beyond noise, after the correction for every engine and named alternative scored this month.
* **`moves_ranking`**: ranks your outcomes better (AUROC) beyond noise, after the same correction.
* **`no_worse`**: worse by no more than noise or 1%. This verdict has no correction.
* **`worse`**: worse beyond that.
* **`untested`**: too few labelled answers after its skips (fewer than 100, or fewer than 20 of each kind).
* **`unscored`**: not scored yet.

`change` is the alternative minus the active version on the same labelled answers, with its 95% `interval`, so for log loss a value below 0 is better. `billed` gives what scoring the alternative billed.

`corrected_for` counts the engine and named alternatives scored this month. The best of several equal alternatives still looks better by chance, and more so the more you score, so `better` must pass a higher bar for each one in `corrected_for`. Score only the alternatives you would really choose.

### What the labels can show

With a few hundred labelled answers, only large gains read `better`:

* With up to two alternatives, the gain in log loss must be roughly 0.06 at 100 labelled answers. It must be about 0.03 at 500, and larger with ten alternatives.
* An engine that is right five points more often can show at 500 labelled answers.
* A change of recipe or wording that moves the numbers by a few points usually reads `no_worse`.

The quote's `detectable` gives this for your labels before you pay.

`suggestion` names the `better` alternative with the lowest log loss, and a tie goes to the one that moves the ranking, then to the cheaper one. If none is `better`, the suggestion is recipe tuning's cheaper recipe, with `because: "cheaper"`. If there is neither, `suggestion` is `null` and `message` gives the gain these labels could have shown. `headline` says the same in one sentence, such as "gpt-6-luna current is better on the calibrated number: held-out log loss 0.39 against 0.45 on 312 labelled answers, corrected for 2 alternatives tried; ranking unchanged within noise."

The best of several looks better on the outcomes it was chosen on than it will later, so trust the number measured after you switch.

## Create a version from an alternative

`POST /namespaces/{ns}/judgments/{name}/alternatives/{id}/version` creates the alternative's definition as a new version, as `POST /judgments` does, and starts its [shadow job](/guides/change-a-judgment#change-a-question-safely-with-a-shadow-report), as `activate` without `force` does. The response is the 202 job, and **Create version** in the dashboard sends you to the shadow report on the **Overview**.

<CodeGroup>
  ```ts TypeScript theme={"theme":{"light":"css-variables","dark":"css-variables"}}
  const job = await ns.judgments.createVersionFromAlternative("material_change", "alt_3f9a2c7e1b04d6a8");
  ```

  ```python Python theme={"theme":{"light":"css-variables","dark":"css-variables"}}
  job = ns.judgments.create_version_from_alternative("material_change", "alt_3f9a2c7e1b04d6a8")
  ```
</CodeGroup>

The shadow report gets `alternative` beside `recompute` and `calibration_replay`.

```json theme={"theme":{"light":"css-variables","dark":"css-variables"}}
{"alternative": {"id": "alt_3f9a2c7e1b04d6a8", "verdict": "better", "change": {"log_loss": {"mean": -0.061, "interval": {"lower": -0.093, "upper": -0.029}}, "auroc": null, "accuracy": 0.03}, "k": 2, "drafted": false}}
```

* **`verdict` and `k`**: the verdict, and how many alternatives it was corrected for, when you created the version.
* **`recompute`** prices the backfill of the old answers, and **`calibration_replay`** the [replay](/calibration/engine-changes) that fits the new version on another engine.
* **Confirm the job to activate the version,** or cancel it to keep the active one.

Brussle corrects the new version as it corrects any new version, from the billed replay on another engine or from its own outcomes on the same engine. The new version never reads a fit that the alternatives run made.

## Read whether the gain held

Each epoch of the [report](/calibration/report) has `alternative_selection` on a version created from an alternative.

```json theme={"theme":{"light":"css-variables","dark":"css-variables"}}
{
  "alternative_selection": {
    "alternative": "alt_3f9a2c7e1b04d6a8",
    "drafted": false,
    "selected_on": {"fit": "fit_9f3a2c7e1b04d6a8", "version": 3, "labels": 312, "log_loss": 0.391, "auroc": 0.88, "raw_gap": 0.048, "computed_at": "2026-10-14T03:12:40Z"},
    "since": {"outcomes": 140, "log_loss": 0.402, "auroc": 0.87, "raw_gap": 0.051, "interval": {"lower": 0.02, "upper": 0.082}},
    "held": true
  }
}
```

* **`selected_on`**: the numbers the alternative was chosen on, which are the replaced version's fit and version, the labelled answers, log loss, AUROC and `raw_gap` (the corrected number's gain in log loss over the engine's raw numbers).
* **`since`**: the same on the outcomes observed after the choice, with the 95% interval of `raw_gap`, or `null` under 100 outcomes or 20 of each kind.
* **`held`**: `true` when the gain over raw since the choice is no smaller than the gain it was chosen on, beyond noise. It is `false` when the gain is smaller by more (the epoch's `warnings` then has `alternative_gain_not_held`), and `too_few` under 100 outcomes or 20 of each kind.

`held` compares gains rather than log losses, since a shift in your outcomes (a new base rate, for example) moves both log losses but not the gain. The reverse can also happen.

## Turn on automatic runs

By default, a run starts only when you confirm it. To let it start by itself after a correction fit, turn on `alternatives.automatic` in a `PATCH` of its own.

```json PATCH /v1/namespaces/acme%2Fprod/judgments/material_change theme={"theme":{"light":"css-variables","dark":"css-variables"}}
{"alternatives": {"automatic": true, "skip_engines": []}, "confirm": true}
```

* **Without `confirm`,** the response is `{"alternatives_quote": {...}}`, the quote of a full run, and nothing changes.
* **An automatic run** starts after a fit, at most once a month, and scores at most 10 alternatives on at most 500 labelled answers, billed as judging. `next_run_after` gives the earliest next start.
* **`skip_engines`** lists the engines not to score as automatic alternatives.

The **Automatic** switch in the dashboard shows the quote of a full run and asks you to confirm. Brussle sends `judgment.alternatives_awaiting_confirm`, with the quote, when a fit lands with unscored alternatives and automatic runs are off. When a run finds an alternative `better`, Brussle sends `judgment.alternative_scored` (see [event types](/guides/events-feed#event-types)).


This documentation is built and hosted on [Mintlify](https://mintlify.com), a developer documentation platform.