Let your outcomes choose

Before you switch a judgment to another engine, another wording or another recipe, let your outcomes say whether it is better. Brussle answers your labelled answers again with each alternative, corrects each on your outcomes the same way, and compares them with the active version on the same outcomes. You see the price before anything runs, a verdict that says how sure it is, and what your labels could not tell. Nothing switches unless you confirm. An alternative is another definition of the judgment that differs from the active version only in its engine, its context recipe, its question or its criteria. Its type, features, applies_to and horizon stay the same. Thresholds are never part of an alternative: your current thresholds stay when you activate it. It is not the shadow report’s candidate, the version you are activating.

Read the table and its quote.
Name alternatives, or draft wordings, beyond the automatic ones.
Confirm a run. It is billed as judging.
Read the verdicts.
Create a version from the one you choose, and confirm its shadow report.
Read whether the gain held on the outcomes that come after.

Alternatives are on Team and Scale. On Developer, GET .../alternatives shows recipe tuning’s numbers without the recipes, with plan_required, and every other alternatives route returns plan_required (402). A version needs a correction, with 100 outcomes and 20 of each kind, before its alternatives can be scored.

Read the table and its quote

GET /namespaces/{ns}/judgments/{name}/alternatives returns the table for the active version. It is ns.judgments.alternatives(name) in both SDKs. In the dashboard, the judgment’s Alternatives tab shows the quote, the table in its two groups and the suggestion, and you name an engine, a wording or a recipe there. Before a run, the table holds the quote of what is unscored:

{
  "version": 3,
  "engine_version": "current+2026-09-24.1",
  "status": "not_run",
  "reason": "awaiting_confirm",
  "automatic": false,
  "skip_engines": [],
  "labels": {"count": 312, "sampled": false, "outcomes": null},
  "corrected_for": 0,
  "quote": {
    "status": "ready",
    "generation": 4,
    "labels": 312,
    "alternatives": [
      {"id": "alt_3f9a2c7e1b04d6a8", "contexts": "stored", "labels": 312, "tokens": 402480, "judgment_units": 780, "new_provider": false},
      {"id": "alt_8c1d5e2f90a3b47c", "contexts": "stored", "labels": 312, "tokens": 405600, "judgment_units": 312, "new_provider": false}
    ],
    "judgment_units": 1092,
    "tokens": 808080,
    "detectable": {"log_loss": 0.05, "auroc": 0.06, "k": 2},
    "new_providers": [],
    "upstream": []
  },
  "suggestion": null,
  "headline": null,
  "next_run_after": null,
  "plan_required": null
}

The example leaves out message, the quote’s cost_usd fields and the table’s rows. The table has two groups:

Engine and named alternatives (in_k: true). They are scored only in a run you confirm, billed as judging. Each better verdict in this group is corrected for every alternative of the group scored this month.
Recipe tuning’s smaller recipes (in_k: false), from the free monthly recipe tuning run. Their verdict is only no_worse or worse. They are never billed, and do not count in corrected_for.

Each engine this namespace already uses is an automatic alternative, other than the active version’s own. An engine is in use when an active judgment version of the namespace uses it, or when it is the namespace’s default_engine. An org admin can widen this to every engine any namespace of the organization uses (see turn on suggestions). To skip an engine, put it in alternatives.skip_engines (see turn on automatic runs). The quote prices each unscored engine or named alternative before anything runs:

For each alternative: the labelled answers it would send (labels), their tokens, judgment_units and cost_usd, and contexts (see how contexts are made).
Totals: judgment_units, tokens and cost_usd for the run.
detectable: the smallest gain in log loss, and in AUROC, that this run can call better on these labels with k alternatives. A smaller real gain reads no_worse. Read it before you pay (see what the labels can show).
new_providers: engines that would get this namespace’s labelled contexts for the first time.
upstream: the backfill a recipe needs first, when it selects related documents by another judgment’s answers. The run never starts it.
generation: which quote this is. It changes when the alternatives, the outcomes or the engines in use change.

Each call the run answers is billed as judging: its size class times its engine’s weight, at the prices of your organization’s current tiers. The pricing page gives an example. The quote is priced in the background within seconds (status: quoting until then), and again whenever the alternatives, skipped engines or fit change.

`status`	`reason`	Means
`not_run`	`awaiting_confirm`	There is something to score, and automatic runs are off. Confirm the quote.
`not_run`	`awaiting_run`	Automatic runs are on and the next fit starts the run, or there is nothing to score yet.
`not_run`	`too_few_outcomes`	The version has no correction yet.
`not_run`	`composite`, `inherited`	No alternatives are scored for a composite. A template’s tenants read the template’s table.
`quoting`		The quote is being priced.
`running`	`budget_paused` or none	This month’s run is scoring. With `budget_paused`, it waits for the namespace’s budget.
`ready`		`suggestion` names the alternative your outcomes prefer.
`none`	`nothing_better`	No alternative was better beyond noise. `message` gives the gain these labels could have shown.
`none`	`accuracy_would_drop`	Each recipe that was cheap enough did worse on your outcomes.
any	`superseded`	The engine changed during the run.

Name an alternative

POST /namespaces/{ns}/judgments/{name}/alternatives adds alternatives. Each gives one or more of engine, context, question and criteria; the active version gives the rest.

await ns.judgments.nameAlternatives("material_change", {
  alternatives: [
    { engine: { name: "gpt-6-luna", version: "current" } },
    { question: "Does this edit change what either party must do, pay or deliver?" },
    { context: { fields: ["state.heading", "state.text"], previous: { fields: ["state.text"] } } },
  ],
});

ns.judgments.name_alternatives(
    "material_change",
    alternatives=[
        {"engine": {"name": "gpt-6-luna", "version": "current"}},
        {"question": "Does this edit change what either party must do, pay or deliver?"},
        {"context": {"fields": ["state.heading", "state.text"], "previous": {"fields": ["state.text"]}}},
    ],
)

The response is the table, with the new quote.

Each alternative is checked as a new version. The engine must be active, the recipe must compile, and the question must be within the limits. On a judgment with features, the recipe must keep each relation a feature reads. Otherwise invalid_request, with a message naming the feature.
An alternative that judges exactly like the active version is refused: “This is the active version.”
At most 10 named alternatives at a time, and at most 10 engine and named alternatives scored each month. Remove one with DELETE .../alternatives/{id} to name another. You cannot remove an automatic engine alternative; skip its engine instead.
The same definition is the same alternative, with an id of alt_ and 16 hexadecimal characters. A second request for it adds nothing.
Do not send run and confirm with the names. That is conflict, because you have not read the new quote; the alternatives are still named. With run: true alone, the response is the quote.

An engine this namespace does not use. When you name one and confirm the run, Brussle sends the stored contexts of your labelled answers to that engine’s provider. The quote names it in new_providers, and its row has new_provider: true. Each engine’s page says which of your data its provider gets.

How contexts are made

stored: same recipe, for example a new engine or wording. Brussle sends the context each labelled answer read, as stored.
narrowed: the recipe reads less. Brussle cuts each stored context to the smaller recipe.
live: the recipe reads more, such as a new field or relation, a larger last_n, or a relation that selects by another judgment’s answers. Brussle compiles each context from today’s documents.

A live row cannot read what each labelled answer read on the day of its label. Its row says as_of: "now", with this sentence: “Its contexts were built from today’s related documents and answers, not from what your labelled answers read; a change in those documents since the label counts against or for it.” Compare it with care. Brussle skips a labelled answer whose document you deleted, and counts it in labels_skipped.deleted. A recipe that selects related documents by another judgment’s answers needs that judgment active, with an answer on each document it applies to. Otherwise naming it is invalid_request, with a message such as “topic has no answers on 1,204 documents the recipe reads; backfill it first”, and details giving the backfill’s estimate.

Draft wordings

POST /namespaces/{ns}/judgments/{name}/alternatives/drafts returns up to four other wordings of the question and its criteria. It is draftWordings(name) in TypeScript and draft_wordings(name) in Python. In the dashboard, it is Draft wordings.

{"drafts": [{"question": "Does this edit change what either party must do, pay or deliver?", "criteria": "Yes only if an obligation, amount or date changes."}]}

A third-party language model writes the drafts, the same one that suggested parts and discovery use. Brussle sends it only the active version’s question, criteria, and options or levels: no document, context, example or outcome.
Drafts are off until an org admin turns on Suggestions; until then the call is forbidden. See turn on suggestions.
Each call uses one of the day’s suggest_parts calls: 20 for each judgment and 50 for each organization, then rate_limited until midnight UTC.
Brussle stores and scores no draft. Read it, edit it if you like, and name it as an alternative with "drafted": true beside its question and criteria to score it.

A drafted wording is not better because a model wrote it. Your outcomes measure it like any other, and its row shows drafted: true.

Confirm a run

POST /namespaces/{ns}/judgments/{name}/alternatives/run scores the unscored engine and named alternatives.

Send it without confirm. The response is the table with the current quote. Nothing runs or is billed.
Read the quote: its cost, detectable and new_providers.
Send it again with confirm: true and the quote_generation you read.

const table = await ns.judgments.runAlternatives("material_change");
await ns.judgments.runAlternatives("material_change", { confirm: true, quote_generation: table.quote?.generation });

table = ns.judgments.run_alternatives("material_change")
ns.judgments.run_alternatives("material_change", confirm=True, quote_generation=table["quote"]["generation"])

The confirm starts this month’s run and returns 202. In the dashboard, Run shows the quote, the budget and each new provider, and asks you to confirm.

A confirm is for the current quote. If the quote changed since you read it, or is being priced again, the confirm is conflict. Without quote_generation, Brussle confirms the current quote.
One run for each version, engine epoch and month. Naming more alternatives and confirming again extends it, on the same labelled answers. An extension can change a verdict: each is checked again against every alternative scored this month, so better can become no_worse.
outcomes chooses the outcomes to score on, by source and by when observed, as for a named run. The month’s first run fixes them.
Billing. Each answered call is billed as judging, and the namespace’s budget holds the run; while it does not fit, the run waits with budget_paused. A failed call bills nothing, and no call is billed twice. The monthly allowance for free replays does not stop this run.
Nothing the run scores is an answer. Nothing is written to the evaluation log, and the run makes its own fits only to compare: no answer ever reads them.

Read the verdicts

Brussle scores each alternative and the active version on the same labelled answers: at most 500, those of the correction. Each answer is read with a correction made without that answer, as the nightly fit chooses its correction.

{
  "id": "alt_3f9a2c7e1b04d6a8",
  "source": "automatic",
  "drafted": false,
  "differs": ["engine: jev current → gpt-6-luna current"],
  "in_k": true,
  "contexts": "stored",
  "as_of": null,
  "labels": 312,
  "labels_skipped": {"deleted": 0, "image_erased": 0, "not_derivable": 0, "failed": 0},
  "cost_per_answer": 2.5,
  "held_out": {"read_with": "calibrated", "log_loss": 0.391, "auroc": 0.88, "accuracy": 0.86},
  "change": {"log_loss": {"mean": -0.061, "interval": {"lower": -0.093, "upper": -0.029}, "p": 0.0001}, "auroc": {"mean": 0.02, "interval": {"lower": -0.01, "upper": 0.05}, "p": 0.09}, "accuracy": 0.03},
  "verdict": "better",
  "moves_ranking": false
}

Some fields are left out.

better: lower corrected log loss beyond noise, after the correction for every engine and named alternative scored this month.
moves_ranking: ranks your outcomes better (AUROC) beyond noise, after the same correction.
no_worse: worse by no more than one standard error or 1%. This verdict has no correction.
worse: worse beyond that.
untested: too few labelled answers after its skips: fewer than 100, or fewer than 20 of each kind.
unscored: not scored yet.

change is the alternative minus the active version on the same labelled answers, with its 95% interval; for log loss, below 0 is better. billed gives what scoring it billed. What “corrected for” means. The best of several equal alternatives still looks better by chance, more so the more you score, so better must pass a higher bar for each engine or named alternative scored this month (corrected_for). Score only the alternatives you would really choose.

What the labels can show

With a few hundred labelled answers, only large gains read better:

With one or two alternatives, roughly a gain in log loss of 0.06 at 100 labelled answers and 0.03 at 500. With ten, the gain must be larger.
An engine that is right five points more often can show at 500 labelled answers.
A change of recipe or wording that moves the numbers by a few points usually reads no_worse.

The quote’s detectable gives this for your labels before you pay. The suggestion. suggestion names the better alternative with the lowest log loss; a tie goes to the one that moves the ranking, then the cheaper. If none is better, it is recipe tuning’s cheaper recipe, with because: "cheaper". If there is neither, it is null, and message gives the gain these labels could have shown. headline says it in one sentence, such as “gpt-6-luna current is better on the calibrated number: held-out log loss 0.39 against 0.45 on 312 labelled answers, corrected for 2 alternatives tried; ranking unchanged within noise.” The best of several looks better on the outcomes it was chosen on than it will later. Trust the number measured after you switch.

Create a version from an alternative

POST /namespaces/{ns}/judgments/{name}/alternatives/{id}/version creates the alternative’s definition as a new version, as POST /judgments does, and starts its shadow job, as activate without force does. The response is the 202 job. In the dashboard, Create version sends you to the shadow report on the Overview.

const job = await ns.judgments.createVersionFromAlternative("material_change", "alt_3f9a2c7e1b04d6a8");

job = ns.judgments.create_version_from_alternative("material_change", "alt_3f9a2c7e1b04d6a8")

The shadow report gets alternative beside recompute and calibration_replay:

{"alternative": {"id": "alt_3f9a2c7e1b04d6a8", "verdict": "better", "change": {"log_loss": {"mean": -0.061, "interval": {"lower": -0.093, "upper": -0.029}}, "auroc": null, "accuracy": 0.03}, "k": 2, "drafted": false}}

verdict and k: the verdict, and how many alternatives it was corrected for, when you created the version.
recompute prices the backfill of the old answers, and calibration_replay the replay that fits the new version on another engine.
Confirm the job to activate the version, or cancel it to keep the active one.

The new version is corrected as any new version is: from the billed replay on another engine, or from its own outcomes on the same engine. It never reads a fit the alternatives run made.

Read whether the gain held

On a version created from an alternative, each epoch of the report has alternative_selection:

{
  "alternative_selection": {
    "alternative": "alt_3f9a2c7e1b04d6a8",
    "drafted": false,
    "selected_on": {"fit": "fit_9f3a2c7e1b04d6a8", "version": 3, "labels": 312, "log_loss": 0.391, "auroc": 0.88, "raw_gap": 0.048, "computed_at": "2026-10-14T03:12:40Z"},
    "since": {"outcomes": 140, "log_loss": 0.402, "auroc": 0.87, "raw_gap": 0.051, "interval": {"lower": 0.02, "upper": 0.082}},
    "held": true
  }
}

selected_on: the numbers it was chosen on: the replaced version’s fit and version, the labelled answers, log loss, AUROC, and raw_gap, the corrected number’s gain in log loss over the engine’s raw numbers.
since: the same on the outcomes observed after the choice, with the 95% interval of raw_gap. null under 100 outcomes or 20 of each kind.
held: true when the gain over raw since the choice is no smaller than the gain it was chosen on, beyond one standard error; false when it is smaller by more, and the epoch’s warnings then has alternative_gain_not_held; too_few under 100 outcomes or 20 of each kind.

held compares gains, not log losses. A shift in your outcomes, such as a new base rate, moves both log losses but not the gain, and the reverse can also happen.

Turn on automatic runs

By default, a run starts only when you confirm it. To let it start by itself after a correction fit, turn on alternatives.automatic, in a PATCH of its own:

PATCH /v1/namespaces/acme%2Fprod/judgments/material_change

{"alternatives": {"automatic": true, "skip_engines": []}, "confirm": true}

Without confirm, the response is {"alternatives_quote": {...}}, the quote of a full run, and nothing changes.
An automatic run starts after a fit, at most once a month. It scores at most 10 alternatives on at most 500 labelled answers, billed as judging. next_run_after gives the earliest next start.
skip_engines lists the engines not to score as automatic alternatives.

In the dashboard, the Automatic switch shows the quote of a full run and asks you to confirm. When a fit lands with unscored alternatives and automatic runs are off, Brussle sends judgment.alternatives_awaiting_confirm, with the quote. When a run finds an alternative better, it sends judgment.alternative_scored. See event types.

Start here

Train on outcomes

Decisions

Keep answers current

Read other records

Your data

Platforms

Before you build

Let your outcomes choose

Read the table and its quote

Name an alternative

How contexts are made

Draft wordings

Confirm a run

Read the verdicts

What the labels can show

Create a version from an alternative

Read whether the gain held

Turn on automatic runs

​Read the table and its quote

​Name an alternative

​How contexts are made

​Draft wordings

​Confirm a run

​Read the verdicts

​What the labels can show

​Create a version from an alternative

​Read whether the gain held

​Turn on automatic runs

Read the table and its quote

Name an alternative

How contexts are made

Draft wordings

Confirm a run

Read the verdicts

What the labels can show

Create a version from an alternative

Read whether the gain held

Turn on automatic runs