Let your outcomes choose

You can let your outcomes say whether another engine, another wording or another recipe is better before you switch a judgment to it. Brussle answers your labelled answers again with each alternative, corrects each one on your outcomes the same way, and compares them with the active version on the same outcomes. You see the price before anything runs, then a verdict that says how sure it is and what your labels could not tell. Nothing switches unless you confirm. An alternative is another definition of the judgment that differs from the active version only in its engine, its context recipe, its question or its criteria, while its type, features, applies_to and horizon stay the same. Thresholds are never part of an alternative, so your current thresholds stay when you activate one. (An alternative is not the shadow report’s candidate, which is the version you are activating.)

Read the table and its quote.
Name alternatives, or draft wordings, beyond the automatic ones.
Confirm a run, which is billed as judging.
Read the verdicts.
Create a version from the one you choose, and confirm its shadow report.
Read whether the gain held on the outcomes that come after.

Alternatives are on Team and Scale. On Developer, GET .../alternatives shows recipe tuning’s numbers without the recipes, with plan_required, and every other alternatives route returns plan_required (402). Brussle scores a version’s alternatives only once the version has a correction with 100 outcomes, 20 of each kind.

Read the table and its quote

GET /namespaces/{ns}/judgments/{name}/alternatives (ns.judgments.alternatives(name) in both SDKs) returns the table for the active version. The judgment’s Alternatives tab in the dashboard shows the quote, the table in its two groups and the suggestion, and you name an engine, a wording or a recipe there. Before a run, the table holds the quote of what is still unscored.

{
  "version": 3,
  "engine_version": "current+2026-09-24.1",
  "status": "not_run",
  "reason": "awaiting_confirm",
  "automatic": false,
  "skip_engines": [],
  "labels": {"count": 312, "sampled": false, "outcomes": null},
  "corrected_for": 0,
  "quote": {
    "status": "ready",
    "generation": 4,
    "labels": 312,
    "alternatives": [
      {"id": "alt_3f9a2c7e1b04d6a8", "contexts": "stored", "labels": 312, "tokens": 402480, "judgment_units": 780, "new_provider": false},
      {"id": "alt_8c1d5e2f90a3b47c", "contexts": "stored", "labels": 312, "tokens": 405600, "judgment_units": 312, "new_provider": false}
    ],
    "judgment_units": 1092,
    "tokens": 808080,
    "detectable": {"log_loss": 0.05, "auroc": 0.06, "k": 2},
    "new_providers": [],
    "upstream": []
  },
  "suggestion": null,
  "headline": null,
  "next_run_after": null,
  "plan_required": null
}

A full response also has message, the quote’s cost_usd fields and the table’s rows. The table has two groups:

Engine and named alternatives (in_k: true). Brussle scores them only in a run you confirm, which is billed as judging, and corrects each better verdict in this group for every alternative of the group scored this month.
Recipe tuning’s smaller recipes (in_k: false), which come from the free monthly recipe tuning run. Their verdict is only ever no_worse or worse. They are never billed and never count in corrected_for.

Every engine this namespace already uses, other than the active version’s own, is an automatic alternative. An engine is in use when an active judgment version of the namespace uses it or when it is the namespace’s default_engine, and an org admin can widen this to every engine that any namespace of the organization uses (see turn on suggestions). To skip an engine, put it in alternatives.skip_engines (see turn on automatic runs). The quote prices each unscored engine or named alternative before anything runs:

For each alternative: the labelled answers it would send (labels), their tokens, judgment_units and cost_usd, and contexts (see how contexts are made).
Totals: judgment_units, tokens and cost_usd for the run.
detectable: the smallest gain in log loss, and in AUROC, that this run can call better on these labels with k alternatives. A smaller real gain reads no_worse, so read it before you pay (see what the labels can show).
new_providers: engines that would get this namespace’s labelled contexts for the first time.
upstream: the backfill a recipe needs first, when it selects related documents by another judgment’s answers. The run never starts that backfill.
generation: which quote this is. It changes when the alternatives, the outcomes or the engines in use change.

Brussle bills each call the run answers as judging, at its size class times its engine’s weight and at the prices of your organization’s current tiers (the pricing page gives an example). Brussle prices the quote in the background within seconds (status: quoting until then), and prices it again whenever the alternatives, skipped engines or fit change.

`status`	`reason`	Means
`not_run`	`awaiting_confirm`	There is something to score, and automatic runs are off. Confirm the quote.
`not_run`	`awaiting_run`	Automatic runs are on and the next fit starts the run, or there is nothing to score yet.
`not_run`	`too_few_outcomes`	The version has no correction yet.
`not_run`	`composite`, `inherited`	No alternatives are scored for a composite. A template’s tenants read the template’s table.
`quoting`		Brussle is pricing the quote.
`running`	`budget_paused` or none	This month’s run is scoring. With `budget_paused`, it waits for the namespace’s budget.
`ready`		`suggestion` names the alternative your outcomes prefer.
`none`	`nothing_better`	No alternative was better beyond noise. `message` gives the gain these labels could have shown.
`none`	`accuracy_would_drop`	Each recipe that was cheap enough did worse on your outcomes.
any	`superseded`	The engine changed during the run.

Name an alternative

POST /namespaces/{ns}/judgments/{name}/alternatives adds alternatives. Each one sets one or more of engine, context, question and criteria, and takes the rest from the active version.

await ns.judgments.nameAlternatives("material_change", {
  alternatives: [
    { engine: { name: "gpt-6-luna", version: "current" } },
    { question: "Does this edit change what either party must do, pay or deliver?" },
    { context: { fields: ["state.heading", "state.text"], previous: { fields: ["state.text"] } } },
  ],
});

ns.judgments.name_alternatives(
    "material_change",
    alternatives=[
        {"engine": {"name": "gpt-6-luna", "version": "current"}},
        {"question": "Does this edit change what either party must do, pay or deliver?"},
        {"context": {"fields": ["state.heading", "state.text"], "previous": {"fields": ["state.text"]}}},
    ],
)

The response is the table, with the new quote.

Each alternative is checked as a new version. The engine must be active, the recipe must compile and the question must be within the limits. On a judgment with features, the recipe must also keep each relation a feature reads, or the request returns invalid_request with a message naming the feature.
Brussle refuses an alternative that judges exactly like the active version, with the message “This is the active version.”
At most 10 named alternatives at a time, and at most 10 engine and named alternatives scored each month. To name another, remove one with DELETE .../alternatives/{id}. You cannot remove an automatic engine alternative, so skip its engine instead.
The same definition is the same alternative, with an id of alt_ and 16 hexadecimal characters, so a second request for it adds nothing.
Do not send run and confirm with the names. Brussle returns conflict, since you have not read the new quote. The alternatives are still named. With run: true alone, the response is the quote.

When you name an engine this namespace does not use and confirm the run, Brussle sends the stored contexts of your labelled answers to that engine’s provider. The quote names that engine in new_providers, its row has new_provider: true, and each engine’s page says which of your data its provider gets.

How contexts are made

stored: the recipe is the same (a new engine or wording, for example), so Brussle sends the context each labelled answer read, as stored.
narrowed: the recipe reads less, so Brussle cuts each stored context to the smaller recipe.
live: the recipe reads more (a new field or relation, a larger last_n, or a relation that selects by another judgment’s answers), so Brussle compiles each context from today’s documents.

A live row cannot read what each labelled answer read on the day of its label, so its row says as_of: "now", with this sentence: “Its contexts were built from today’s related documents and answers, not from what your labelled answers read; a change in those documents since the label counts against or for it.” Compare it with care. Brussle skips a labelled answer whose document you deleted, and counts it in labels_skipped.deleted. A recipe that selects related documents by another judgment’s answers needs that judgment active, with an answer on each document it applies to. If it is not, naming the recipe returns invalid_request, with a message such as “topic has no answers on 1,204 documents the recipe reads; backfill it first”, and details giving the backfill’s estimate.

Draft wordings

POST /namespaces/{ns}/judgments/{name}/alternatives/drafts returns up to four other wordings of the question and its criteria. It is draftWordings(name) in TypeScript, draft_wordings(name) in Python and Draft wordings in the dashboard.

{"drafts": [{"question": "Does this edit change what either party must do, pay or deliver?", "criteria": "Yes only if an obligation, amount or date changes."}]}

A third-party language model writes the drafts, the same one that suggested parts and discovery use. Brussle sends it only the active version’s question, criteria, and options or levels (no document, context, example or outcome).
Drafts are off until an org admin turns on Suggestions, and until then the call returns forbidden (see turn on suggestions).
Each call uses one of the day’s suggest_parts calls (20 for each judgment and 50 for each organization), and after those the call returns rate_limited until midnight UTC.
Brussle stores and scores no draft. To score one, read it, edit it if you like, and name it as an alternative with "drafted": true beside its question and criteria.

A drafted wording is not better because a model wrote it, so your outcomes measure it like any other. Its row shows drafted: true.

Confirm a run

POST /namespaces/{ns}/judgments/{name}/alternatives/run scores the unscored engine and named alternatives.

Send it without confirm. The response is the table with the current quote, and nothing runs or is billed.
Read the quote: its cost, detectable and new_providers.
Send it again with confirm: true and the quote_generation you read.

const table = await ns.judgments.runAlternatives("material_change");
await ns.judgments.runAlternatives("material_change", { confirm: true, quote_generation: table.quote?.generation });

table = ns.judgments.run_alternatives("material_change")
ns.judgments.run_alternatives("material_change", confirm=True, quote_generation=table["quote"]["generation"])

The confirm starts this month’s run and returns 202. Run in the dashboard shows the quote, the budget and each new provider, and asks you to confirm.

A confirm is for the current quote. If the quote changed since you read it, or Brussle is pricing it again, the confirm returns conflict. Without quote_generation, Brussle confirms the current quote.
One run for each version, engine epoch and month. If you name more alternatives and confirm again, Brussle extends the run on the same labelled answers. An extension can change a verdict: Brussle checks each one again against every alternative scored this month, so better can become no_worse.
outcomes chooses the outcomes to score on, by source and by when observed, as for a named run, and the month’s first run fixes them.
Billing. Brussle bills each answered call as judging, and the namespace’s budget holds the run: while the run does not fit, it waits with budget_paused. A failed call bills nothing, and no call is billed twice. The monthly allowance for free replays does not stop this run.
Nothing the run scores is an answer. Brussle writes nothing to the evaluation log, and the run makes its own fits only to compare, so no answer ever reads them.

Read the verdicts

Brussle scores each alternative and the active version on the same labelled answers, which are those of the correction (at most 500). Brussle reads each answer with a correction that did not see that answer’s outcome, as the nightly fit’s test does.

{
  "id": "alt_3f9a2c7e1b04d6a8",
  "source": "automatic",
  "drafted": false,
  "differs": ["engine: jev current → gpt-6-luna current"],
  "in_k": true,
  "contexts": "stored",
  "as_of": null,
  "labels": 312,
  "labels_skipped": {"deleted": 0, "image_erased": 0, "not_derivable": 0, "failed": 0},
  "cost_per_answer": 2.5,
  "held_out": {"read_with": "calibrated", "log_loss": 0.391, "auroc": 0.88, "accuracy": 0.86},
  "change": {"log_loss": {"mean": -0.061, "interval": {"lower": -0.093, "upper": -0.029}, "p": 0.0001}, "auroc": {"mean": 0.02, "interval": {"lower": -0.01, "upper": 0.05}, "p": 0.09}, "accuracy": 0.03},
  "verdict": "better",
  "moves_ranking": false
}

A full row has more fields.

better: lower corrected log loss beyond noise, after the correction for every engine and named alternative scored this month.
moves_ranking: ranks your outcomes better (AUROC) beyond noise, after the same correction.
no_worse: worse by no more than noise or 1%. This verdict has no correction.
worse: worse beyond that.
untested: too few labelled answers after its skips (fewer than 100, or fewer than 20 of each kind).
unscored: not scored yet.

change is the alternative minus the active version on the same labelled answers, with its 95% interval, so for log loss a value below 0 is better. billed gives what scoring the alternative billed. corrected_for counts the engine and named alternatives scored this month. The best of several equal alternatives still looks better by chance, and more so the more you score, so better must pass a higher bar for each one in corrected_for. Score only the alternatives you would really choose.

What the labels can show

With a few hundred labelled answers, only large gains read better:

With up to two alternatives, the gain in log loss must be roughly 0.06 at 100 labelled answers. It must be about 0.03 at 500, and larger with ten alternatives.
An engine that is right five points more often can show at 500 labelled answers.
A change of recipe or wording that moves the numbers by a few points usually reads no_worse.

The quote’s detectable gives this for your labels before you pay. suggestion names the better alternative with the lowest log loss, and a tie goes to the one that moves the ranking, then to the cheaper one. If none is better, the suggestion is recipe tuning’s cheaper recipe, with because: "cheaper". If there is neither, suggestion is null and message gives the gain these labels could have shown. headline says the same in one sentence, such as “gpt-6-luna current is better on the calibrated number: held-out log loss 0.39 against 0.45 on 312 labelled answers, corrected for 2 alternatives tried; ranking unchanged within noise.” The best of several looks better on the outcomes it was chosen on than it will later, so trust the number measured after you switch.

Create a version from an alternative

POST /namespaces/{ns}/judgments/{name}/alternatives/{id}/version creates the alternative’s definition as a new version, as POST /judgments does, and starts its shadow job, as activate without force does. The response is the 202 job, and Create version in the dashboard sends you to the shadow report on the Overview.

const job = await ns.judgments.createVersionFromAlternative("material_change", "alt_3f9a2c7e1b04d6a8");

job = ns.judgments.create_version_from_alternative("material_change", "alt_3f9a2c7e1b04d6a8")

The shadow report gets alternative beside recompute and calibration_replay.

{"alternative": {"id": "alt_3f9a2c7e1b04d6a8", "verdict": "better", "change": {"log_loss": {"mean": -0.061, "interval": {"lower": -0.093, "upper": -0.029}}, "auroc": null, "accuracy": 0.03}, "k": 2, "drafted": false}}

verdict and k: the verdict, and how many alternatives it was corrected for, when you created the version.
recompute prices the backfill of the old answers, and calibration_replay the replay that fits the new version on another engine.
Confirm the job to activate the version, or cancel it to keep the active one.

Brussle corrects the new version as it corrects any new version, from the billed replay on another engine or from its own outcomes on the same engine. The new version never reads a fit that the alternatives run made.

Read whether the gain held

Each epoch of the report has alternative_selection on a version created from an alternative.

{
  "alternative_selection": {
    "alternative": "alt_3f9a2c7e1b04d6a8",
    "drafted": false,
    "selected_on": {"fit": "fit_9f3a2c7e1b04d6a8", "version": 3, "labels": 312, "log_loss": 0.391, "auroc": 0.88, "raw_gap": 0.048, "computed_at": "2026-10-14T03:12:40Z"},
    "since": {"outcomes": 140, "log_loss": 0.402, "auroc": 0.87, "raw_gap": 0.051, "interval": {"lower": 0.02, "upper": 0.082}},
    "held": true
  }
}

selected_on: the numbers the alternative was chosen on, which are the replaced version’s fit and version, the labelled answers, log loss, AUROC and raw_gap (the corrected number’s gain in log loss over the engine’s raw numbers).
since: the same on the outcomes observed after the choice, with the 95% interval of raw_gap, or null under 100 outcomes or 20 of each kind.
held: true when the gain over raw since the choice is no smaller than the gain it was chosen on, beyond noise. It is false when the gain is smaller by more (the epoch’s warnings then has alternative_gain_not_held), and too_few under 100 outcomes or 20 of each kind.

held compares gains rather than log losses, since a shift in your outcomes (a new base rate, for example) moves both log losses but not the gain. The reverse can also happen.

Turn on automatic runs

By default, a run starts only when you confirm it. To let it start by itself after a correction fit, turn on alternatives.automatic in a PATCH of its own.

PATCH /v1/namespaces/acme%2Fprod/judgments/material_change

{"alternatives": {"automatic": true, "skip_engines": []}, "confirm": true}

Without confirm, the response is {"alternatives_quote": {...}}, the quote of a full run, and nothing changes.
An automatic run starts after a fit, at most once a month, and scores at most 10 alternatives on at most 500 labelled answers, billed as judging. next_run_after gives the earliest next start.
skip_engines lists the engines not to score as automatic alternatives.

The Automatic switch in the dashboard shows the quote of a full run and asks you to confirm. Brussle sends judgment.alternatives_awaiting_confirm, with the quote, when a fit lands with unscored alternatives and automatic runs are off. When a run finds an alternative better, Brussle sends judgment.alternative_scored (see event types).

Start here

Calibration

Decisions

Keep answers current

Read other records

Your data

Platforms

Before you build

Let your outcomes choose

Read the table and its quote

Name an alternative

How contexts are made

Draft wordings

Confirm a run

Read the verdicts

What the labels can show

Create a version from an alternative

Read whether the gain held

Turn on automatic runs

​Read the table and its quote

​Name an alternative

​How contexts are made

​Draft wordings

​Confirm a run

​Read the verdicts

​What the labels can show

​Create a version from an alternative

​Read whether the gain held

​Turn on automatic runs

Read the table and its quote

Name an alternative

How contexts are made

Draft wordings

Confirm a run

Read the verdicts

What the labels can show

Create a version from an alternative

Read whether the gain held

Turn on automatic runs