> ## Documentation Index > Fetch the complete documentation index at: https://docs.brussle.com/llms.txt > Use this file to discover all available pages before exploring further. # Read the calibration report > The calibration report measures a judgment's numbers against your outcomes, before and after the correction, for each engine epoch: how true the numbers are, how well they rank, and what the correction removed over time. The calibration report measures a judgment's numbers against your outcomes, before and after the [correction](/train/correction). It tells you how true the numbers are, how well they rank your documents, why an epoch has no correction yet, and how much the correction has lowered the error since it started. ## Get the report `GET /namespaces/{ns}/judgments/{name}/calibration` returns the report for the active version. It is `ns.judgments.calibration(name)` in both SDKs. In the dashboard, it is the **Calibration** section of the judgment's **Overview**, under the definition. Brussle makes the report when you ask for it, from all the outcomes and answers up to your request. Its `raw` and `calibrated` metrics include outcomes you posted a minute ago and answers corrected after the fit. The fit itself, with `held_out`, `held_out_metrics` and `lift`, changes only at the next fit (see [when fits run](/train/promotion#when-fits-run)). A report can take longer to make than the [limit for a request](/limits#requests), typically the first read for a judgment with a long history. You then get `unavailable` (503) with `Retry-After`. Brussle keeps working on it. Retry after that time, as the SDKs do, and the retry gets the report. ```json theme={"theme":{"light":"css-variables","dark":"css-variables"}} { "judgment": "material_change", "type": "bool", "version": 3, "outcomes": 1469, "outcomes_by_source": {"posted": 400, "rule": 1069, "queue": 0, "implicit": 0}, "rule_outcomes_paused": 0, "unmatched_outcomes": {"no_evaluation": 12, "before_first_evaluation": 0, "after_horizon": 0, "same_window": 0, "after_positive": 0, "does_not_fit": 0}, "censored_predictions": {"window_open": 0, "document_deleted": 0, "after_positive": 0}, "mode": "auto", "epochs": [ { "engine": "jev", "engine_version": "current+2026-09-24.1", "outcomes": 1432, "outcomes_by_source": {"posted": 390, "rule": 1042, "queue": 0, "implicit": 0}, "fitted_on": "all", "fit_source": "outcomes", "replay": null, "rule_outcomes_paused": 0, "implicit_negatives": 0, "stage": "full", "fitted_at": "2026-09-30T03:00:00Z", "from_previous_epoch": false, "not_fitted": null, "held_out": {"raw_log_loss": 0.441, "calibrated_log_loss": 0.351, "raw_brier": 0.141, "calibrated_brier": 0.112}, "held_out_metrics": { "base_rate": 0.21, "raw": {"log_loss": 0.441, "expected_calibration_error": 0.152, "brier": 0.141, "auroc": 0.74, "precision_at_5": 0.71, "precision_at_10": 0.63}, "scalar": {"log_loss": 0.351, "expected_calibration_error": 0.012, "brier": 0.112, "auroc": 0.74, "precision_at_5": 0.71, "precision_at_10": 0.62}, "features": null, "features_only": null }, "coverage": {"lower": 0.01, "upper": 0.99}, "warnings": [], "raw": {"accuracy": 0.801, "expected_calibration_error": 0.159, "log_loss": 0.441}, "calibrated": {"accuracy": 0.837, "expected_calibration_error": 0.008, "log_loss": 0.344}, "tenant": null, "tenants": null, "current": { "fit": "fit_9f3a2c7e1b04d6a8", "run": "01JA2J0P2R4T6W8Y0A2C4E6G9B", "pinned": false, "fitted_at": "2026-09-30T03:00:00Z", "since": {"outcomes": 37, "fallback": "raw", "fallback_log_loss": 0.412, "log_loss": 0.301, "standard_error": 0.021} }, "challenger": { "fit": "fit_2b7c41d09e6a3f58", "run": "01JA2J0P2R4T6W8Y0A2C4E6G9B", "fitted_at": "2026-10-02T03:00:00Z", "outcomes": 1432, "held_out": {"raw_log_loss": 0.405, "calibrated_log_loss": 0.298, "standard_error": 0.009}, "versus_current": {"outcomes": 37, "gain": 0.004, "standard_error": 0.011, "shift": 0.003}, "held": "not_clearly_better" } } ], "lift": { "headline": { "metric": "log_loss", "baseline": "raw", "engine": "jev", "engine_version": "current+2026-09-24.1", "since": "2026-09-25T03:00:00Z", "as_of": "2026-10-02T03:00:00Z", "outcomes": 1432, "applied": true, "raw_log_loss": 0.441, "applied_log_loss": 0.351, "error_reduction": 0.204, "interval": {"lower": 0.162, "upper": 0.246} }, "withheld": null, "model_changed": null, "history": [ {"fitted_at": "2026-09-25T03:00:00Z", "version": 3, "engine": "jev", "engine_version": "current+2026-09-24.1", "outcomes": 212, "outcomes_by_source": {"posted": 212, "rule": 0, "queue": 0, "implicit": 0}, "fitted_on": "all", "source": "outcomes", "stage": "early", "applied": true, "raw_log_loss": 0.452, "calibrated_log_loss": 0.401, "raw_ece": 0.151, "calibrated_ece": 0.041, "standard_error": 0.012, "raw_brier": 0.146, "calibrated_brier": 0.128} ] } } ``` The values are illustrative. The example leaves out each epoch's ten-bin `reliability` curve and `band`, and all of `lift.history` but the first point. In the example, the correction cuts log loss and calibration error, and `auroc` stays the same. That is what a correction without features does: it makes the numbers true and keeps the order of the answers. `accuracy` rises from 0.801 to 0.837 only because some corrected numbers move to the other side of 0.5, not because the engine got better at telling the answers apart. ## Read the top of the report * **`outcomes`** and **`outcomes_by_source`**: the outcomes that count, by source: `posted`, `rule` (made by [outcome rules](/train/outcomes#record-outcomes-from-your-own-writes)), `queue` (from the [labelling queue](/train/thresholds#label-a-random-sample)) and `implicit` ([implicit negatives](/train/horizons#turn-on-implicit-negatives)). Their sum is `outcomes`. * **`rule_outcomes_paused`**: rule outcomes left out because your plan did not include outcome rules when they were observed, after a downgrade to Developer. Brussle keeps them, outside `outcomes`. It stays 0 while your plan includes rules. * **`unmatched_outcomes`**: posted and rule outcomes that join no answer, by reason. See [outcomes that join nothing](/train/outcomes#outcomes-that-join-nothing). * **`censored_predictions`**: with implicit negatives on, the windows with no outcome that do not count as `false`. See [implicit negatives](/train/horizons#turn-on-implicit-negatives). * **`mode`**: `pinned` when a [pin](/train/promotion#pin-a-fit) applies to a version and epoch of the judgment, `auto` otherwise. ## Read each epoch The report lists one entry for each engine [epoch](/train/engine-changes) of the active version, newest first. A new version starts with no fit and an empty history. * **`outcomes`**, **`outcomes_by_source`**, **`rule_outcomes_paused`**: as at the top, for the outcomes joined to this epoch's answers. **`implicit_negatives`** counts the implicit ones. Outcomes replayed into the epoch after the engine changed are counted in `replay.outcomes` instead, so each outcome counts once. * **`fitted_on`**: `queue` when the epoch's queue labels alone meet the fit minimums (50, with 15 of each kind). The fit then uses only them, each weighted for how common its part of the range is in your traffic. Otherwise `all`: each outcome counts once. * **`fit_source`** and **`replay`**: whether the fit uses outcomes replayed after the engine changed. See [when the engine changes](/train/engine-changes#read-the-replay-in-the-report). * **`stage`** and **`fitted_at`**: how settled the fit is, and when the run that made the current fit ran (the same date as `current.fitted_at`). `null` when nothing is fitted; `early` from 50 outcomes; `full` from 1,000; `features` on a `bool` whose fit uses its [features](/train/features). See [when you get a correction](/train/correction#when-you-get-a-correction). * **`not_fitted`**: why the epoch has no correction. See [why an epoch has no correction](#why-an-epoch-has-no-correction). * **`prior`**: what the fit starts from (`identity`, the engine's own numbers, or `pool` on a template's tenant) and `weight`, your outcomes' share. `null` when nothing is fitted, and from 1,000 outcomes. See [below 100 outcomes](/train/correction#below-100-outcomes). * **`held_out`**: the test a fit must pass before answers read it, recorded when the fit was made. It gives the log loss of the raw answers and of fits made without the outcomes they score. The fit applies only when `calibrated_log_loss` is lower. Also `raw_ece` and `calibrated_ece`, the `standard_error` of the difference, and on fits made since 2026-10-13 `raw_brier` and `calibrated_brier`, weighted as the log losses are. `holdout` says how outcomes were held out: `five_groups` of documents, or below 100 outcomes `per_document`. `gain_standard_errors` is the win over raw in standard errors; below 100 outcomes a fit applies only above 1. `null` when nothing is fitted. A template tenant's own test does not give these values. * **`held_out_metrics`**: each way of reading the number, scored on the same held-out outcomes. See [how well answers rank](#how-well-answers-rank). * **`coverage`**: the lowest and highest raw value among the outcomes the fit uses (the queue labels, when `fitted_on` is `queue`). The raw value is `p` for a `bool`, and the probability of the most probable option or level for a `choice` or `score`. * **`warnings`**: see [warnings](#warnings). * **`raw`** and **`calibrated`**: the same metrics before and after the correction, on every outcome so far and the answer it joins now. `calibrated` applies the current fit, so it can include outcomes and answers the fit did not use. With `raw_better`, it shows what the fit would have given. `calibrated` is `null` while nothing is fitted. For a composite, `raw` measures the combined `p` and `calibrated` is `null`. * **`accuracy`**: a `bool` answer counts as `true` when `p` is at least 0.5; a `choice` or `score` must have the outcome as its most probable option or level. When the event is rare, accuracy says little; read `held_out_metrics`. * **`expected_calibration_error`**: how far the stated probabilities are from how often things came true. Lower is better. * **`log_loss`**: lower is better. * **`reliability`**: 10 bins of equal width, such as `{"lower": 0.8, "upper": 0.9, "count": 137, "mean_predicted": 0.845, "observed": 0.883}`. A well calibrated judgment has `observed` near `mean_predicted` in each bin. A `choice` or `score` answer goes in a bin by the probability of its most probable option or level. Empty bins have `null` means. * **`mean_level_distance`**, on a `score` only: the mean number of levels between the most probable level and the observed one. * **`band`**: see [the band](#the-band). * **`current`** and **`challenger`**: the fit answers read now, and the night's fit when it did not replace it. See [read the current fit and the challenger](/train/promotion#read-the-current-fit-and-the-challenger). * **`tenant`** and **`tenants`**: on a [template](/train/tenants#read-the-report-on-a-tenant), what a tenant reads and why, and how many tenants read each. `null` for a namespace's own judgment. * **`alternative_selection`**: on a version made from an alternative, whether its gain held. See [read whether the gain held](/train/alternatives#read-whether-the-gain-held). ## Why an epoch has no correction `not_fitted` is `null` when the epoch has a correction of its own. Otherwise its `reason` is one of: | `reason` | Means | | - | - | | `too_few_outcomes` | Fewer than 50 outcomes | | `too_few_negatives` | Fewer than 15 `false` outcomes (`bool`) | | `too_few_positives` | Fewer than 15 `true` outcomes (`bool`) | | `too_few_values` | A `choice` or `score` has fewer than two values with 15 outcomes each | | `awaiting_fit` | There are enough outcomes. The next fit, within the hour, calibrates the epoch. To fit at once, select **Fit now** or call `POST /namespaces/{ns}/judgments/{name}/calibration/fit`. | | `raw_better` | The fit was not better than the raw numbers on held-out outcomes, so answers stay raw. You do not have to do anything. | | `within_noise` | Below 100 outcomes, the fit beat the raw numbers by no more than one standard error (`gain_standard_errors`). Answers stay raw until it is sure. | | `readers_need_full` | A judgment with features below 100 outcomes. Its fit is shown, and applies from 100, because thresholds and readers act on its `p`. | When more than one applies, `reason` names the first in this order: a kind of outcome fully missing (all `true` is `too_few_negatives`, all `false` is `too_few_positives`, one value only is `too_few_values`); then fewer than 50 outcomes; then fewer than 15 of a kind. 40 outcomes that are all `true` read `too_few_negatives`, not `too_few_outcomes`. `message` says what to post, such as "Only positive outcomes so far: post outcomes for documents where it did not happen." ## Warnings Each warning has a `code` and a `message`. * **`above_thresholds`** and **`high_band`**: the outcomes look like a reviewed sample, and the fit is reliable only where they are. See [label more than the reviewed documents](/train/outcomes#label-more-than-the-reviewed-documents). * **`inputs_changed_since_fit`**: answers the fit used were judged again since, with different inputs. See [when answers are corrected](/train/correction#when-answers-are-corrected). * **`pinned_fit_beaten`**: the night's fit clearly beats the [pinned fit](/train/promotion#pin-a-fit) on the outcomes since the pin's fit was made. * **`pin_suspended`**: a pin with features no longer applies. See [pin a fit](/train/promotion#pin-a-fit). * **`alternative_gain_not_held`**: a version made from an alternative has lost the gain it was chosen on. See [read whether the gain held](/train/alternatives#read-whether-the-gain-held). ## The band `band` is the fit's corrected number and its 95% interval at 21 raw values from 0.00 to 1.00, such as `{"raw": 0.5, "calibrated": 0.41, "lower": 0.36, "upper": 0.46}`. Where it is wide, the fit is less sure of its numbers. For a judgment with features, Brussle draws it at each feature's typical value. On a tenant, it is what the tenant's answers read. The **Calibration** section shades it around the reliability curve. `band` is for a `bool` and a choice among candidates (on whether a match exists). It is `null` for a `choice` or `score` over fixed options, for a composite, when nothing is fitted, and for a fit made before fits recorded their intervals. ## How well answers rank Accuracy says little when the event is rare. If the event happens to 4% of documents, a judgment that answers "no" every time is 96% accurate and finds none of them. When you act on the highest `p` first, such as a review queue sorted by `p`, the order is what counts. For a `bool`, each epoch's `held_out_metrics` scores each way of reading `p` on the same held-out outcomes as the fit's test, with labelling-queue labels weighted to your traffic. For each way it gives: * **`log_loss`**, **`expected_calibration_error`** and **`brier`**: how far the probabilities are from what happened. Lower is better. * **`auroc`**: the chance that an answer where the event happened has a higher `p` than one where it did not. 0.5 is chance; 1 is a perfect order. * **`precision_at_5`** and **`precision_at_10`**: the share of events among the 5% and 10% of answers with the highest `p`. `null` when 5% or 10% of each held-out group is less than one answer. Compare each precision with **`base_rate`**, the share of events among all the answers. A precision of 0.30 at the top 5%, with a base rate of 0.04, means 30% of the documents at the top of your queue had the event, against 4% of all documents. The ways of reading `p`: * **`raw`**: the engine's own `p`. * **`scalar`**: `p` corrected on the engine's answer alone. It ranks exactly as `raw` does, unless the fit gives the same `p` to answers the engine told apart; then it ranks a little closer to chance. * **`features`**: `p` corrected with the judgment's [features](/train/features) and the engine's answer. Features can change the order. It also gives `engine_weight` and `per_feature` (see [read each feature](/train/features#read-each-feature)). * **`features_only`**: `p` from the features alone. Answers never read it. Compare it with `features` to see what the engine adds to your own numbers. Brussle holds out outcomes in groups and makes one fit without each group. It measures the order within each group and averages the groups by their weight, so a group with one kind of outcome adds nothing to `auroc`. `log_loss` and `expected_calibration_error` are over all the outcomes. Answers with the same `p` count as a tie: half for `auroc`, and by their share of events at the edge of the top 5% or 10%. `features` and `features_only` are `null` without features. On a `choice` or `score` with features, the same four readings score the whole distribution, with `accuracy` (and on a `score`, `mean_level_distance`) in place of the ranking. For a choice among candidates, see [features on a choice among candidates](/train/features#features-on-a-choice-among-candidates). `held_out_metrics` is `null` for a `choice` or `score` without features, for a composite, while an epoch has no fit, and on a template tenant with its own fit. ## Read the lift `lift` tells how much the correction has lowered the judgment's held-out log loss since its first fit in the current epoch. The **Calibration** section opens with it in one sentence: in the example, the correction lowered the held-out log loss by 20%, on 1,432 outcomes. On Team and Scale the section also charts the log loss after each fit: the raw log loss beside what answers read, what each new fit would score, and the outcomes each rests on. On Developer it shows the sentence as a preview. The API returns `lift` on every plan. * **`headline`**: from the newest point of the current epoch. `error_reduction` is the part of the raw answers' held-out log loss that the correction removes, `(raw_log_loss − applied_log_loss) / raw_log_loss`, on the same outcomes for both: 0.204 is 20%. `interval` is its 95% interval; when `lower` is 0 or less, the decrease is not clear yet. More outcomes make it narrower. `applied` is `false` when the fit did not beat the raw numbers, and then `error_reduction` is 0. `since` is the epoch's first fit; `outcomes` is what the newest fit uses. * **`history`**: one point for each fit whose numbers changed, oldest first, at most one a day for each epoch, kept for 400 days. Each point has `fitted_at`, `outcomes`, `outcomes_by_source`, `fitted_on`, `source` (`replay` for a fit made from a [replay](/train/engine-changes)), `stage`, `applied`, and the held-out log loss and calibration error, raw and corrected, with the Brier score from 2026-10-13. A point is added when the log loss or calibration error changes; the Brier score adds none of its own. `role` is absent for what answers read, and `challenger` for what a new fit would score on a night the current fit was kept. While a fit is kept, the line of what answers read stays flat; the chart draws challengers dotted, and where the lines meet a new fit replaced the current one. Every number is measured on outcomes the fit did not see. Every run, with its fits, stays in the [list of fits](/train/fits) for the life of the namespace. * **`withheld`**: why there is no headline yet, in the form of `not_fitted`: too few outcomes in the current epoch, a fit held back below 100 outcomes (`within_noise`, `readers_need_full`), or `awaiting_fit` before the first point. There is never a headline below the fit minimums. * **`model_changed`**: present after the engine changed. It gives the earlier epoch (`from`), the current one (`to`) and the day the current one started (`on`). A headline never mixes epochs; the new epoch starts its own from its first fit, and the chart keeps the old epoch's points beside the new. `lift` is `null` for a [composite](/guides/composite-judgments), whose `p` comes from its combiner, and on a template's tenant (the template's report has the pool's lift). Log loss measures how true the numbers are. It does not count right answers, and it does not say how well answers rank: read `auroc` and `precision_at_5` against `base_rate` for that. Precision and recall at your named thresholds are not part of the lift either. Without features, thresholds read the engine's numbers, which the correction never changes, so they measure the engine. The [recommender](/train/thresholds) gives precision and recall. This documentation is built and hosted on [Mintlify](https://mintlify.com), a developer documentation platform.