Skip to main content
The calibration report measures a judgment’s numbers against your outcomes, before and after the correction. It tells you how true the numbers are, how well they rank your documents, why an epoch has no correction yet, and how much the correction has lowered the error since it started.

​
Get the report

GET /namespaces/{ns}/judgments/{name}/calibration returns the report for the active version, and both SDKs call it ns.judgments.calibration(name). The dashboard shows it as the Calibration section of the judgment’s Overview, under the definition. Brussle makes the report when you ask for it, from all the outcomes and answers up to your request, so its raw and calibrated metrics include outcomes you posted a minute ago and answers corrected after the fit. The fit itself, with held_out, held_out_metrics and lift, changes only at the next fit (see when fits run). A report can take longer to make than the limit for a request, usually on the first read for a judgment with a long history. You then get unavailable (503) with Retry-After while Brussle keeps working on it. Retry after that time, as the SDKs do, and the retry gets the report.
The values are illustrative, and the example leaves out each epoch’s ten-bin reliability curve and band, and all of lift.history but the first point. A correction without features makes the numbers true and keeps the order of the answers, so in the example it cuts log loss and calibration error while auroc stays the same. accuracy rises from 0.801 to 0.837 only because some corrected numbers move to the other side of 0.5. The engine got no better at telling the answers apart.

​
Read the top of the report

  • outcomes and outcomes_by_source: the outcomes that count, by source: posted, rule (made by outcome rules), queue (from the labelling queue) and implicit (implicit negatives). Their sum is outcomes.
  • rule_outcomes_paused: rule outcomes left out because your plan did not include outcome rules when they were observed, after a downgrade to Developer. Brussle keeps them outside outcomes, and the count stays 0 while your plan includes rules.
  • unmatched_outcomes: posted and rule outcomes that join no answer, by reason. See outcomes that join nothing.
  • censored_predictions: with implicit negatives on, the windows with no outcome that do not count as false. See implicit negatives.
  • mode: pinned when a pin applies to a version and epoch of the judgment, auto otherwise.

​
Read each epoch

The report lists one entry for each engine epoch of the active version, newest first. A new version starts with no fit and an empty history.
  • outcomes, outcomes_by_source, rule_outcomes_paused: as at the top, for the outcomes joined to this epoch’s answers. implicit_negatives counts the implicit ones. Brussle counts outcomes replayed into the epoch after the engine changed in replay.outcomes instead, so each outcome counts once.
  • fitted_on: queue when the epoch’s queue labels alone meet the fit minimums (50, with 15 of each kind). The fit then uses only them, each weighted for how common its part of the range is in your traffic. Otherwise it is all, and each outcome counts once.
  • fit_source and replay: whether the fit uses outcomes replayed after the engine changed. See when the engine changes.
  • stage and fitted_at: how settled the fit is, and when the run that made the current fit ran (the same date as current.fitted_at). stage is null when nothing is fitted, early from 50 outcomes, full from 1,000, and features on a bool whose fit uses its features. See when you get a correction.
  • not_fitted: why the epoch has no correction. See why an epoch has no correction.
  • prior: what the fit starts from (identity, the engine’s own numbers, or pool on a template’s tenant) and weight, your outcomes’ share. It is null when nothing is fitted, and from 1,000 outcomes. See below 100 outcomes.
  • held_out: the test a fit must pass before answers read it, recorded when the fit was made. It gives the log loss of the raw answers and of fits scored on outcomes they did not see, and the fit applies only when calibrated_log_loss is lower. It also gives raw_ece, calibrated_ece and the standard_error of the difference, and on fits made since 2026-10-13 raw_brier and calibrated_brier, weighted as the log losses are. holdout says how outcomes were held out (five_groups of documents, or per_document below 100 outcomes). gain_standard_errors is the win over raw in standard errors, and below 100 outcomes a fit applies only above 1. held_out is null when nothing is fitted, and a template tenant’s own test does not give these values.
  • held_out_metrics: each way of reading the number, scored on the same held-out outcomes. See how well answers rank.
  • coverage: the lowest and highest raw value among the outcomes the fit uses (the queue labels, when fitted_on is queue). The raw value is p for a bool, and the probability of the most probable option or level for a choice or score. labels_on_later_revisions counts the labelled examples that joined an answer made after the document already had one: the document changed, was judged again, and the label was dated after the change. A label dated at discovery rather than at decision time joins this way. See date each label when the answer was acted on.
  • effective_sample_size: what the outcomes the fit uses count for once each is weighted: the outcome count with fitted_on of all, and less with queue labels weighted to skewed traffic, where a few heavy labels carry most of the weight. The held-out numbers are as sure as this count, not the label count. See what the even half costs.
  • features_margin: on a bool with features, once a fit with them exists: its held-out gain over the fit on the engine’s answer alone, the standard_error of that gain, and labels_needed, an estimate of the further labels, drawn as these were, before the gain is clear by one standard error (0 when it already is; null when the gain is zero or negative, since more of the same labels do not turn that round). null elsewhere.
  • warnings: see warnings.
  • raw and calibrated: the same metrics before and after the correction, on every outcome so far and the answer it joins now. calibrated applies the current fit, so it can include outcomes and answers the fit did not use. With raw_better it shows what the fit would have given, and it is null while nothing is fitted. On a composite, raw measures the combined p and calibrated is null.
    • accuracy: a bool answer counts as true when p is at least 0.5, and a choice or score must have the outcome as its most probable option or level. Accuracy says little when the event is rare, so read held_out_metrics then.
    • expected_calibration_error: how far the stated probabilities are from how often things came true. Lower is better.
    • log_loss: lower is better.
    • reliability: 10 bins of equal width, such as {"lower": 0.8, "upper": 0.9, "count": 137, "mean_predicted": 0.845, "observed": 0.883}. A well calibrated judgment has observed near mean_predicted in each bin. A choice or score answer goes in a bin by the probability of its most probable option or level. Empty bins have null means.
    • mean_level_distance, on a score only: the mean number of levels between the most probable level and the observed one.
  • band: see the band.
  • current and challenger: the fit answers read now, and the night’s fit when it did not replace it. See read the current fit and the challenger.
  • tenant and tenants: on a template, what a tenant reads and why, and how many tenants read each. null for a namespace’s own judgment.
  • alternative_selection: on a version made from an alternative, whether its gain held. See read whether the gain held.

​
Why an epoch has no correction

not_fitted is null when the epoch has a correction of its own. Otherwise its reason is one of: When more than one applies, reason names the first in this order: a kind of outcome fully missing (all true is too_few_negatives, all false is too_few_positives, one value only is too_few_values); then fewer than 50 outcomes; then fewer than 15 of a kind. So 40 outcomes that are all true read too_few_negatives, not too_few_outcomes. message says what to post, such as “Only positive outcomes so far: post outcomes for documents where it did not happen.”

​
Warnings

Each warning has a code and a message.
  • above_thresholds and high_band: the outcomes look like a reviewed sample, and the fit is reliable only where they are. See label more than the reviewed documents.
  • coverage_truncated: more than 20% of the live answers have a raw p outside the range the labels cover. It gives uncovered, the shares below and above the labelled range, and live, the percentiles the range’s bounds sit at among the live answers. A fit on these labels, and a threshold set from them, act on answers the labels say nothing about. The threshold recommender carries it too. Label a random sample, or use the labelling queue.
  • reads_like_the_outcome: the engine’s answer (engine) or a feature, named in features, predicts the outcomes almost perfectly on its own. A number that predicts the outcome this well is usually written once the outcome is known: a document changed to show what happened, or an outcome dated after the document changed, so the label measures an answer that had already read the outcome. held_out_metrics.raw.warning carries it for the engine’s answer, and each feature’s per_feature entry for a feature. See a number that reads like the outcome.
  • inputs_changed_since_fit: answers the fit used were judged again since, with different inputs. See when answers are corrected.
  • pinned_fit_beaten: the night’s fit clearly beats the pinned fit on the outcomes since the pin’s fit was made.
  • pin_suspended: a pin with features no longer applies. See pin a fit.
  • alternative_gain_not_held: a version made from an alternative has lost the gain it was chosen on. See read whether the gain held.

​
The band

band is the fit’s corrected number and its 95% interval at 21 raw values, such as {"raw": 0.5, "calibrated": 0.41, "lower": 0.36, "upper": 0.46}. The raw values run from 0.00 to 1.00, and where the band is wide the fit is less sure of its numbers. Brussle draws the band at each feature’s typical value for a judgment with features. On a tenant, the band is what the tenant’s answers read. The Calibration section shades it around the reliability curve. band is for a bool and a choice among candidates (on whether a match exists). It is null for a choice or score over fixed options, for a composite, when nothing is fitted, and for a fit made before fits recorded their intervals.

​
How well answers rank

Accuracy says little when the event is rare. If the event happens to 4% of documents, a judgment that answers “no” every time is 96% accurate and finds none of them. When you act on the highest p first (a review queue sorted by p, say), the order is what counts. Each epoch’s held_out_metrics on a bool scores each way of reading p on the same held-out outcomes as the fit’s test, with labelling-queue labels weighted to your traffic. For each way it gives:
  • log_loss, expected_calibration_error and brier: how far the probabilities are from what happened. Lower is better.
  • auroc: the chance that an answer where the event happened has a higher p than one where it did not. 0.5 is chance; 1 is a perfect order.
  • precision_at_5 and precision_at_10: the share of events among the 5% and 10% of answers with the highest p. null when there are too few scored answers for a top 5% or 10%.
Compare each precision with base_rate, the share of events among all the answers. Say the precision at the top 5% is 0.30 and the base rate is 0.04. Then 30% of the documents at the top of your queue had the event, against 4% of all documents. The ways of reading p:
  • raw: the engine’s own p. Its warning is reads_like_the_outcome when the engine’s answer alone predicts the outcomes almost perfectly, as a feature’s reading warns (see warnings). This reading, the features_margin and the names in the reads_like_the_outcome warning come from the newest run’s fit, which read the labels last, even while an earlier fit stays current under the promotion rule.
  • scalar: p corrected on the engine’s answer alone. It ranks exactly as raw does, unless the fit gives the same p to answers the engine told apart, and then it ranks a little closer to chance.
  • features: p corrected with the judgment’s features and the engine’s answer, so features can change the order. It also gives engine_weight and per_feature (see read each feature).
  • features_only: p from the features alone, which answers never read. Compare it with features to see what the engine adds to your own numbers.
Each reading is scored on outcomes that its fit did not see. auroc compares only answers that the same fit scored, so it can leave some outcomes out, while log_loss and expected_calibration_error are over all the outcomes. Answers with the same p count as a tie (half for auroc, and by their share of events at the edge of the top 5% or 10%). features and features_only are null without features. The same four readings score the whole distribution on a choice or score with features, with accuracy (and on a score, mean_level_distance) in place of the ranking. For a choice among candidates, see features on a choice among candidates. held_out_metrics is null for a choice or score without features, for a composite, while an epoch has no fit, and on a template tenant with its own fit.

​
Read the lift

lift tells how much the correction has lowered the judgment’s held-out log loss since its first fit in the current epoch. The Calibration section opens with it in one sentence (in the example, that the correction lowered the held-out log loss by 20%, on 1,432 outcomes). On Team and Scale the section also charts the log loss after each fit, with the raw log loss beside what answers read, what each new fit would score, and the outcomes each rests on. On Developer the section shows the sentence as a preview, and the API returns lift on every plan.
  • headline: from the newest point of the current epoch. error_reduction is the part of the raw answers’ held-out log loss that the correction removes, (raw_log_loss − applied_log_loss) / raw_log_loss, on the same outcomes for both (0.204 is 20%). interval is its 95% interval, which more outcomes make narrower. When lower is 0 or less, the decrease is not clear yet. applied is false when the fit did not beat the raw numbers, and error_reduction is then 0. since is the epoch’s first fit, and outcomes is what the newest fit uses.
  • history: one point for each fit whose numbers changed, oldest first, at most one a day for each epoch, kept for 400 days. Each point has fitted_at, outcomes, outcomes_by_source, fitted_on, source (replay for a fit made from a replay), stage, applied, and the held-out log loss and calibration error, raw and corrected, with the Brier score from 2026-10-13. Brussle adds a point when the log loss or calibration error changes, and the Brier score adds none of its own. role is absent for what answers read, and challenger for what a new fit would score on a night the current fit was kept. While a fit is kept the line of what answers read stays flat, the chart draws challengers dotted, and where the lines meet a new fit replaced the current one. Every number is measured on outcomes the fit did not see, and every run, with its fits, stays in the list of fits for the life of the namespace.
  • withheld: why there is no headline yet, in the form of not_fitted: too few outcomes in the current epoch, a fit held back below 100 outcomes (within_noise, readers_need_full), or awaiting_fit before the first point. There is never a headline below the fit minimums.
  • model_changed: present after the engine changed. It gives the earlier epoch (from), the current one (to) and the day the current one started (on). A headline never mixes epochs, so the new epoch starts its own from its first fit, and the chart keeps the old epoch’s points beside the new.
lift is null for a composite, whose p comes from its combiner, and on a template’s tenant (the template’s report has the pool’s lift). Log loss measures how true the numbers are. It does not count right answers or say how well answers rank (read auroc and precision_at_5 against base_rate for that). Precision and recall at your named thresholds are not part of the lift either. Without features, thresholds read the engine’s numbers, which the correction never changes, so they measure the engine. The recommender gives precision and recall.