Skip to main content
The calibration report measures a judgment’s numbers against your outcomes, before and after the correction. It tells you how true the numbers are, how well they rank your documents, why an epoch has no correction yet, and how much the correction has lowered the error since it started.

​
Get the report

GET /namespaces/{ns}/judgments/{name}/calibration returns the report for the active version. It is ns.judgments.calibration(name) in both SDKs. In the dashboard, it is the Calibration section of the judgment’s Overview, under the definition. Brussle makes the report when you ask for it, from all the outcomes and answers up to your request. Its raw and calibrated metrics include outcomes you posted a minute ago and answers corrected after the fit. The fit itself, with held_out, held_out_metrics and lift, changes only at the next fit (see when fits run). A report can take longer to make than the limit for a request, typically the first read for a judgment with a long history. You then get unavailable (503) with Retry-After. Brussle keeps working on it. Retry after that time, as the SDKs do, and the retry gets the report.
The values are illustrative. The example leaves out each epoch’s ten-bin reliability curve and band, and all of lift.history but the first point. In the example, the correction cuts log loss and calibration error, and auroc stays the same. That is what a correction without features does: it makes the numbers true and keeps the order of the answers. accuracy rises from 0.801 to 0.837 only because some corrected numbers move to the other side of 0.5, not because the engine got better at telling the answers apart.

​
Read the top of the report

  • outcomes and outcomes_by_source: the outcomes that count, by source: posted, rule (made by outcome rules), queue (from the labelling queue) and implicit (implicit negatives). Their sum is outcomes.
  • rule_outcomes_paused: rule outcomes left out because your plan did not include outcome rules when they were observed, after a downgrade to Developer. Brussle keeps them, outside outcomes. It stays 0 while your plan includes rules.
  • unmatched_outcomes: posted and rule outcomes that join no answer, by reason. See outcomes that join nothing.
  • censored_predictions: with implicit negatives on, the windows with no outcome that do not count as false. See implicit negatives.
  • mode: pinned when a pin applies to a version and epoch of the judgment, auto otherwise.

​
Read each epoch

The report lists one entry for each engine epoch of the active version, newest first. A new version starts with no fit and an empty history.
  • outcomes, outcomes_by_source, rule_outcomes_paused: as at the top, for the outcomes joined to this epoch’s answers. implicit_negatives counts the implicit ones. Outcomes replayed into the epoch after the engine changed are counted in replay.outcomes instead, so each outcome counts once.
  • fitted_on: queue when the epoch’s queue labels alone meet the fit minimums (50, with 15 of each kind). The fit then uses only them, each weighted for how common its part of the range is in your traffic. Otherwise all: each outcome counts once.
  • fit_source and replay: whether the fit uses outcomes replayed after the engine changed. See when the engine changes.
  • stage and fitted_at: how settled the fit is, and when the run that made the current fit ran (the same date as current.fitted_at). null when nothing is fitted; early from 50 outcomes; full from 1,000; features on a bool whose fit uses its features. See when you get a correction.
  • not_fitted: why the epoch has no correction. See why an epoch has no correction.
  • prior: what the fit starts from (identity, the engine’s own numbers, or pool on a template’s tenant) and weight, your outcomes’ share. null when nothing is fitted, and from 1,000 outcomes. See below 100 outcomes.
  • held_out: the test a fit must pass before answers read it, recorded when the fit was made. It gives the log loss of the raw answers and of fits made without the outcomes they score. The fit applies only when calibrated_log_loss is lower. Also raw_ece and calibrated_ece, the standard_error of the difference, and on fits made since 2026-10-13 raw_brier and calibrated_brier, weighted as the log losses are. holdout says how outcomes were held out: five_groups of documents, or below 100 outcomes per_document. gain_standard_errors is the win over raw in standard errors; below 100 outcomes a fit applies only above 1. null when nothing is fitted. A template tenant’s own test does not give these values.
  • held_out_metrics: each way of reading the number, scored on the same held-out outcomes. See how well answers rank.
  • coverage: the lowest and highest raw value among the outcomes the fit uses (the queue labels, when fitted_on is queue). The raw value is p for a bool, and the probability of the most probable option or level for a choice or score.
  • warnings: see warnings.
  • raw and calibrated: the same metrics before and after the correction, on every outcome so far and the answer it joins now. calibrated applies the current fit, so it can include outcomes and answers the fit did not use. With raw_better, it shows what the fit would have given. calibrated is null while nothing is fitted. For a composite, raw measures the combined p and calibrated is null.
    • accuracy: a bool answer counts as true when p is at least 0.5; a choice or score must have the outcome as its most probable option or level. When the event is rare, accuracy says little; read held_out_metrics.
    • expected_calibration_error: how far the stated probabilities are from how often things came true. Lower is better.
    • log_loss: lower is better.
    • reliability: 10 bins of equal width, such as {"lower": 0.8, "upper": 0.9, "count": 137, "mean_predicted": 0.845, "observed": 0.883}. A well calibrated judgment has observed near mean_predicted in each bin. A choice or score answer goes in a bin by the probability of its most probable option or level. Empty bins have null means.
    • mean_level_distance, on a score only: the mean number of levels between the most probable level and the observed one.
  • band: see the band.
  • current and challenger: the fit answers read now, and the night’s fit when it did not replace it. See read the current fit and the challenger.
  • tenant and tenants: on a template, what a tenant reads and why, and how many tenants read each. null for a namespace’s own judgment.
  • alternative_selection: on a version made from an alternative, whether its gain held. See read whether the gain held.

​
Why an epoch has no correction

not_fitted is null when the epoch has a correction of its own. Otherwise its reason is one of: When more than one applies, reason names the first in this order: a kind of outcome fully missing (all true is too_few_negatives, all false is too_few_positives, one value only is too_few_values); then fewer than 50 outcomes; then fewer than 15 of a kind. 40 outcomes that are all true read too_few_negatives, not too_few_outcomes. message says what to post, such as “Only positive outcomes so far: post outcomes for documents where it did not happen.”

​
Warnings

Each warning has a code and a message.
  • above_thresholds and high_band: the outcomes look like a reviewed sample, and the fit is reliable only where they are. See label more than the reviewed documents.
  • inputs_changed_since_fit: answers the fit used were judged again since, with different inputs. See when answers are corrected.
  • pinned_fit_beaten: the night’s fit clearly beats the pinned fit on the outcomes since the pin’s fit was made.
  • pin_suspended: a pin with features no longer applies. See pin a fit.
  • alternative_gain_not_held: a version made from an alternative has lost the gain it was chosen on. See read whether the gain held.

​
The band

band is the fit’s corrected number and its 95% interval at 21 raw values from 0.00 to 1.00, such as {"raw": 0.5, "calibrated": 0.41, "lower": 0.36, "upper": 0.46}. Where it is wide, the fit is less sure of its numbers. For a judgment with features, Brussle draws it at each feature’s typical value. On a tenant, it is what the tenant’s answers read. The Calibration section shades it around the reliability curve. band is for a bool and a choice among candidates (on whether a match exists). It is null for a choice or score over fixed options, for a composite, when nothing is fitted, and for a fit made before fits recorded their intervals.

​
How well answers rank

Accuracy says little when the event is rare. If the event happens to 4% of documents, a judgment that answers “no” every time is 96% accurate and finds none of them. When you act on the highest p first, such as a review queue sorted by p, the order is what counts. For a bool, each epoch’s held_out_metrics scores each way of reading p on the same held-out outcomes as the fit’s test, with labelling-queue labels weighted to your traffic. For each way it gives:
  • log_loss, expected_calibration_error and brier: how far the probabilities are from what happened. Lower is better.
  • auroc: the chance that an answer where the event happened has a higher p than one where it did not. 0.5 is chance; 1 is a perfect order.
  • precision_at_5 and precision_at_10: the share of events among the 5% and 10% of answers with the highest p. null when 5% or 10% of each held-out group is less than one answer.
Compare each precision with base_rate, the share of events among all the answers. A precision of 0.30 at the top 5%, with a base rate of 0.04, means 30% of the documents at the top of your queue had the event, against 4% of all documents. The ways of reading p:
  • raw: the engine’s own p.
  • scalar: p corrected on the engine’s answer alone. It ranks exactly as raw does, unless the fit gives the same p to answers the engine told apart; then it ranks a little closer to chance.
  • features: p corrected with the judgment’s features and the engine’s answer. Features can change the order. It also gives engine_weight and per_feature (see read each feature).
  • features_only: p from the features alone. Answers never read it. Compare it with features to see what the engine adds to your own numbers.
Brussle holds out outcomes in groups and makes one fit without each group. It measures the order within each group and averages the groups by their weight, so a group with one kind of outcome adds nothing to auroc. log_loss and expected_calibration_error are over all the outcomes. Answers with the same p count as a tie: half for auroc, and by their share of events at the edge of the top 5% or 10%. features and features_only are null without features. On a choice or score with features, the same four readings score the whole distribution, with accuracy (and on a score, mean_level_distance) in place of the ranking. For a choice among candidates, see features on a choice among candidates. held_out_metrics is null for a choice or score without features, for a composite, while an epoch has no fit, and on a template tenant with its own fit.

​
Read the lift

lift tells how much the correction has lowered the judgment’s held-out log loss since its first fit in the current epoch. The Calibration section opens with it in one sentence: in the example, the correction lowered the held-out log loss by 20%, on 1,432 outcomes. On Team and Scale the section also charts the log loss after each fit: the raw log loss beside what answers read, what each new fit would score, and the outcomes each rests on. On Developer it shows the sentence as a preview. The API returns lift on every plan.
  • headline: from the newest point of the current epoch. error_reduction is the part of the raw answers’ held-out log loss that the correction removes, (raw_log_loss − applied_log_loss) / raw_log_loss, on the same outcomes for both: 0.204 is 20%. interval is its 95% interval; when lower is 0 or less, the decrease is not clear yet. More outcomes make it narrower. applied is false when the fit did not beat the raw numbers, and then error_reduction is 0. since is the epoch’s first fit; outcomes is what the newest fit uses.
  • history: one point for each fit whose numbers changed, oldest first, at most one a day for each epoch, kept for 400 days. Each point has fitted_at, outcomes, outcomes_by_source, fitted_on, source (replay for a fit made from a replay), stage, applied, and the held-out log loss and calibration error, raw and corrected, with the Brier score from 2026-10-13. A point is added when the log loss or calibration error changes; the Brier score adds none of its own. role is absent for what answers read, and challenger for what a new fit would score on a night the current fit was kept. While a fit is kept, the line of what answers read stays flat; the chart draws challengers dotted, and where the lines meet a new fit replaced the current one. Every number is measured on outcomes the fit did not see. Every run, with its fits, stays in the list of fits for the life of the namespace.
  • withheld: why there is no headline yet, in the form of not_fitted: too few outcomes in the current epoch, a fit held back below 100 outcomes (within_noise, readers_need_full), or awaiting_fit before the first point. There is never a headline below the fit minimums.
  • model_changed: present after the engine changed. It gives the earlier epoch (from), the current one (to) and the day the current one started (on). A headline never mixes epochs; the new epoch starts its own from its first fit, and the chart keeps the old epoch’s points beside the new.
lift is null for a composite, whose p comes from its combiner, and on a template’s tenant (the template’s report has the pool’s lift). Log loss measures how true the numbers are. It does not count right answers, and it does not say how well answers rank: read auroc and precision_at_5 against base_rate for that. Precision and recall at your named thresholds are not part of the lift either. Without features, thresholds read the engine’s numbers, which the correction never changes, so they measure the engine. The recommender gives precision and recall.