When a new fit replaces the current one

​

When fits run

• The first run starts within a few minutes of the first outcome you post, or of the judgment’s first outcome rules. A run makes a fit only for an epoch that meets the minimums, so the first run usually makes none.
• Until there is a fit, Brussle runs again every hour, for the current epoch and for a new epoch with no fit yet. The first fit comes within the hour after your outcomes meet the minimums.
• Once there is a fit, Brussle makes a new fit each night, on all the outcomes so far, rule outcomes included. It replaces the current fit only under the rule below.
• When Brussle re-judges answers to correct them, the next run starts within a few minutes after the correction ends. See when answers are corrected.

​

Fit now

• You can ask once an hour for each judgment. Within that hour, the call returns rate_limited (429), and Retry-After gives the seconds until the next call is accepted. Named runs share this limit.
• If a run is already waiting or running, that run serves the call.
• It runs on every plan, and it is never billed.

​

The rule

1. It is clearly better on the outcomes that arrived since the current fit was made. The current fit never saw them, so they are a fair test of both. The challenger is scored on each of them without that outcome, and must beat the current fit by more than two standard errors of the mean difference in log loss.
2. It moves the number by at least a point: 0.01 on average over all the judgment’s outcomes. For a choice or score, this is the probability that moves between the options.

• An outcome arrives when you post it, not when it was observed. An implicit negative arrives when its window closes. After the engine changes, a replayed outcome arrives when it is replayed. An outcome whose answer was judged again with changed inputs counts as new too.
• The current fit is withdrawn when the engine’s raw numbers beat it by the same margin on the outcomes since. Answers then read the night’s fit, or the raw numbers if that did not beat them.
• A new start takes the night’s fit at once: the first fit, a new engine epoch, a new version, and a change from fitting on all outcomes to fitting on labelling-queue labels alone, or back.
• Below 100 outcomes, while the current fit or the night’s rests on fewer than 100, each night’s fit replaces the current one, under the margin below 100 rather than this test.
• When a run cannot fit again, because outcomes were deleted and the epoch fell below the minimums, the current fit stays.
• It applies to every judgment: to calibrated on a bool, a choice and a score, to p of a judgment with features, and to a template tenant’s own fit and its own weights, on that tenant’s own outcomes. The exception is a choice among candidates with features: each run’s fit replaces the current one, as a first fit does.
• On a steady process, the rule changes the fit by chance about once in three to five years for each version and epoch. A real change in your outcomes, such as a new base rate, shows in the outcomes since, and the new fit replaces the current one once it is clearly better.

​

Read the current fit and the challenger

• current: the fit answers read now. fit is its id, or null when it did not beat the raw numbers. run holds it. pinned is true for a pinned fit, and pin_suspended is older_preprocessing when the pin is suspended. fitted_at is when the run that made it ran; a fit kept since an earlier run keeps that date, so it can be weeks older than the newest run. since measures it against the raw numbers on the outcomes that arrived after it was made: outcomes, fallback_log_loss (the raw numbers’), log_loss (the current fit’s) and the standard_error of the difference. since is null while those outcomes hold fewer than 20 of each kind, and on the night the fit was made. current is null while the epoch has no fit of its own.
• challenger: the newest run’s fit when it did not replace the current one; null when it did, and when there was no current fit. It has its own fit, run, fitted_at, outcomes and held_out, and versus_current: the challenger against the current fit on the outcomes since. gain is the current fit’s log loss minus the challenger’s, so a positive gain favours the challenger. shift is how far the challenger would move the number on average over every outcome. gain and standard_error are null with fewer than 20 of each kind since. On a choice or score with features, moved_picks is the share of outcomes whose most probable option would change.

​

Why a new fit was held

• too_few_outcomes_since: fewer than 20 of each kind arrived since the current fit was made. Post more outcomes, or wait for them.
• not_clearly_better: on the outcomes since, the new fit did not beat the current one by more than two standard errors (versus_current.gain and standard_error). The difference is within noise.
• too_small_a_move: the new fit is better, but would move the number by less than 0.01 on average (versus_current.shift). On a judgment with features, each move re-judges the judgments that read p, billed, so a move this small is not worth it.
• pinned: a pin holds the fit.

​

Promote the newest fit

const answer = await ns.judgments.fitCalibration("material_change", { promote: true });
if (!("queued_at" in answer)) {
  // An estimate: other judgments read this one. Read `answer.downstream`, then confirm.
  await ns.judgments.fitCalibration("material_change", { promote: true, confirm: true });
}

answer = ns.judgments.fit_calibration("material_change", promote=True)
if "queued_at" not in answer:
    # An estimate: other judgments read this one. Read answer["downstream"], then confirm.
    ns.judgments.fit_calibration("material_change", promote=True, confirm=True)

• The run makes its own new fit and applies it. It fits all the outcomes so far, as a nightly run does, and that fit becomes current for each version and epoch that is not pinned, tenants included. A promote never moves a pin.
• With readers, confirm first. On a bool with features that other judgments read, a new current fit judges their documents again, billed. Without confirm, the call returns the downstream estimate (200), made against last night’s fits, and queues nothing. With confirm: true, the run is queued, and the answer carries the estimate too.
• A confirmed promote is not held by the hourly limit. You accepted its cost, so it is queued even within an hour of Fit now. Without readers, a promote is queued at once and counts as Fit now’s request for the hour.
• Templates. On a namespace that inherits the judgment, a promote is invalid_request; promote on the template. Fit now without a body still queues the template’s run.

​

Pin a fit

{"calibration": {"pins": [
  {"run": "01JA2J0P2R4T6W8Y0A2C4E6G9B", "version": 3, "engine": "jev", "engine_version": "current+2026-09-24.1", "fit": "fit_9f3a2c7e1b04d6a8"}
]}}

const pins = [{ run: "01JA2J0P2R4T6W8Y0A2C4E6G9B", version: 3, engine: "jev", engine_version: "current+2026-09-24.1", fit: "fit_9f3a2c7e1b04d6a8" }];
await ns.judgments.update("material_change", { calibration: { pins } });

pins = [{"run": "01JA2J0P2R4T6W8Y0A2C4E6G9B", "version": 3, "engine": "jev", "engine_version": "current+2026-09-24.1", "fit": "fit_9f3a2c7e1b04d6a8"}]
ns.judgments.update("material_change", calibration={"pins": pins})

• Send calibration alone in its PATCH /namespaces/{ns}/judgments/{name}, without other settings, or the request is invalid_request.
• The list replaces all the pins. Include the pins you want to keep. A pin sent again from the same run stays as it is. Two pins for one version and epoch is invalid_request.
• The pin writes a run (cause: pin) that holds the pinned fit, and every later run holds it too. The response has the judgment, with calibration: {"mode": "pinned", "pins": [...]}, and each pin has pinned_at. mode is auto when no pin applies.
• Answers of that version and epoch read the pinned fit, also when a later fit would do better. Every other version and epoch stays auto. Each run still makes a fit and lists it as the challenger; when it clearly beats the pinned fit on the outcomes since, the report warns pinned_fit_beaten.
• A pin is a setting, like thresholds. It makes no version, and a new version starts in auto.
• On any judgment without features, a pin changes only calibrated. Readers use the raw numbers, so nothing is judged again or billed, and there is nothing to confirm.

const result = await ns.judgments.update("material_change", { calibration: { pins } });
if ("downstream" in result && !("judgment" in result)) {
  // Nothing changed. Show each reader's cost, then confirm.
  console.log(result.downstream);
  await ns.judgments.update("material_change", { calibration: { pins }, confirm: true });
}

result = ns.judgments.update("material_change", calibration={"pins": pins})
if "downstream" in result and "judgment" not in result:
    # Nothing changed. Show each reader's cost, then confirm.
    print(result["downstream"])
    ns.judgments.update("material_change", calibration={"pins": pins}, confirm=True)

​

A pin and a new epoch

​

A suspended pin

​

Lift a pin

await ns.judgments.update("material_change", { calibration: { pins: [] } });

ns.judgments.update("material_change", calibration={"pins": []})

• The next run comes at once. Brussle brings it forward, usually within a few minutes. It does not count toward Fit now’s limit, and the list of fits shows it as cause: nightly.
• Until that run ends, answers read the pinned fit and still show "pinned": true. GET on the judgment already shows mode: auto.
• A lift keeps the number until a clearly better fit arrives. The formerly pinned fit stays current, now in auto, and that run’s fit replaces it only when it is clearly better on the outcomes since the pinned fit was made, usually many by then. To take tonight’s fit anyway, promote. On a bool with features, when the fit p reads changes, readers are re-judged and billed, with no confirm.

​

Events when the fit changes

• judgment.fit_applied reports how many documents crossed each of the judgment’s thresholds. When p goes back to the engine’s, the event is judgment.fit_withdrawn. Both carry fit, run and pinned.
• The event gives each reader’s re-judges: engine_judgments, which are billed, and reuses, re-judges where only a feature the reader keeps moved, which use their engine answer again at no charge.
• The counts come from those re-judges, so the event arrives after they have started.