> ## Documentation Index
> Fetch the complete documentation index at: https://docs.brussle.com/llms.txt
> Use this file to discover all available pages before exploring further.

# Backtest on history

> Replay the records you already have one period at a time, label each answer with what happened next, and measure the judgment and its correction before you use it live.

A backtest measures a judgment and its correction on your history, before you use it live. You replay the records you already have one period at a time, and label each period's answers with what happened in the next one. Since your history already holds the outcome, you do not wait for it.

1. Use a `bool` judgment with the default `horizon` of `0s`. Ask a narrow question about what the record shows, and keep counts as `features`.
2. Read related records with `last_n`, not `window`.
3. Write one period of history, and wait until each of its answers exists.
4. Post one labelled example for each judged document the period touched, saying what happened in the next period, with `observed_at` now.
5. Repeat for the next period. When the replay is done, let the fit run and read the report.
6. Compare `features` with `features_only` in the report, since the difference is what the engine adds to your counts.

## Labelled examples, not a horizon

A [labelled example](/calibration/outcomes#post-labelled-examples) joins the answer that was current at its `observed_at`, so a label dated after a period's answers exist and before the next period's writes joins the answer for that period. That is the backtest.

You cannot backtest a judgment with a `horizon`, because its [outcome window](/calibration/horizons#how-outcomes-meet-answers) opens when Brussle makes the answer and closes one `horizon` later. Brussle dates each write when it receives it and each answer when it makes it, and nothing dates a write or an answer in the past (`created_at` dates only a document's creation). So each replayed answer opens its window now, and an outcome from your history falls before it. To measure a horizon judgment, you wait for the horizon to pass. For a backtest, put the next period in the label and keep `horizon` at `0s`.

The engine answers well about what a record shows now, and near chance about what will happen. So ask it a narrow question about the record as it is (such as "Do this machine's recent reports describe a fault that stops it?") and keep the counts of reports as features. The label says whether the machine failed in the next quarter, and the counts and the fit on your labels carry the number for a failure. See [ask about what will happen](/calibration/features#ask-about-what-will-happen).

## Read each record's past, not the time of the replay

A replay writes all of your history now. Send each record's original `created_at`, and relations sort your records in the order they happened. The two bounds of a relation then behave differently:

* **`last_n`** reads by `created_at`, so with `last_n: 10` a machine reads the 10 newest reports that exist at that point of the replay. Use `last_n`.
* **`window`** counts back from the time of a write, which is the later of the newest write to the judged document (as Brussle received it) and the newest `created_at` among its related documents. In a replay the newest write is now, so a `window` of `365d` holds no record created more than 365 days before the replay, and early periods read nothing. Do not use `window` on a replay. See [newest created first](/guides/related-documents#newest-created-first-and-what-an-edit-costs).

To count more records than the engine reads as text, add a second relation on the same records with a larger `last_n` (up to 1,000) and `aggregate`, and name its numbers in `features`. `"render": false` keeps those numbers out of what the engine reads.

A blocking relation reads the newest records of its block. In a replay a whole period arrives at once, so a record early in the period would read the records after it, which it could not have seen. Add `"relative": "before"`, and each record then reads the `last_n` records just before it by `created_at`, oldest first. A relation with `relative` takes no `window` and no `order`.

```json theme={"theme":{"light":"css-variables","dark":"css-variables"}}
{
  "related": {
    "earlier_reports": {
      "match": {"attributes.kind": "report"},
      "join": {"theirs": "attributes.machine_id", "mine": "attributes.machine_id"},
      "last_n": 10,
      "relative": "before",
      "fields": ["created_at", "state.summary"]
    }
  }
}
```

## Run a backtest

Here you backtest a judgment on machines and their service reports, where each report holds its machine's `id` in `attributes.machine_id`. The history covers several quarters, and each machine's label is whether it failed in the next quarter.

**1. Make a namespace for the backtest,** and set a [budget](/guides/import-existing-data#set-a-budget-first) on it, since a backtest is billed as live judging (Brussle judges each touched machine again in each period).

**2. Create the judgment before the first period,** so Brussle judges each period as you write it.

```json POST /v1/namespaces/acme%2Fbacktest/judgments theme={"theme":{"light":"css-variables","dark":"css-variables"}}
{
  "name": "stopping_fault",
  "type": "bool",
  "applies_to": {"attributes.kind": "machine"},
  "question": "Do this machine's recent reports describe a fault that stops it?",
  "context": {
    "fields": ["state.model", "state.site"],
    "related": {
      "reports": {
        "match": {"attributes.kind": "report"},
        "join": {"theirs": "attributes.machine_id", "mine": "id"},
        "last_n": 10,
        "fields": ["created_at", {"path": "state.summary", "max_chars": 400}]
      },
      "report_counts": {
        "match": {"attributes.kind": "report"},
        "join": {"theirs": "attributes.machine_id", "mine": "id"},
        "last_n": 1000,
        "aggregate": {"count": true, "max": ["state.downtime_hours"], "render": false}
      }
    }
  },
  "features": ["report_counts.count", "report_counts.max(state.downtime_hours)"],
  "engine": {"name": "jev", "version": "current"},
  "freshness": {"policy": "on_change", "debounce_ms": 5000}
}
```

* If the create returns a [replay estimate](/guides/related-documents#the-replay-estimate), Brussle has created nothing yet. In a new namespace the estimate is almost zero, so send the create again with `confirm: true`.
* Wait for its `reference_index` jobs before the first period.
* A judgment with features has a debounce of 10 minutes by default, so a short `debounce_ms` makes each period's answers arrive sooner.

**3. Replay each quarter, in order.**

```python Python theme={"theme":{"light":"css-variables","dark":"css-variables"}}
import time
from datetime import datetime, timezone

J = "stopping_fault"

for quarter in quarters[:-1]:  # the last quarter has no next quarter to label it
    written = ns.import_documents(documents_created_in(quarter))  # machines and reports, with their created_at

    # Wait until the judgment has judged through the quarter's last write.
    while ns.judgments.get(J)["judged_through"] < written.revision:
        time.sleep(5)

    # One label for each machine that this quarter touched: what happened in the next quarter.
    now = datetime.now(timezone.utc).isoformat()
    ns.outcomes.append([
        {"document_id": machine, "judgment": J, "value": failed_in(machine, next_quarter(quarter)), "observed_at": now}
        for machine in machines_touched_in(quarter)
    ])
```

* `quarters`, `documents_created_in`, `machines_touched_in`, `failed_in` and `next_quarter` stand for your own code.
* A post of outcomes is at most 2 MB, so split a large period into more posts.

**4. Fit, then read the report.** See [fit the labels](#fit-the-labels) and [read the ranking](#read-the-ranking).

## Wait for each period's answers

Before you write the next period, wait until each answer of this one exists. Otherwise a machine's answer can include records of the next period, no answer shows the machine as it was at the end of this one, and your label joins an older answer.

* **Compare revisions,** as in the example. The import returns the highest `revision` it wrote, and once the judgment's `judged_through` is at or above it, each machine the period touched has its answer (or its last attempt failed). A get of the judgment reads no documents. See [is a judgment caught up?](/concepts/freshness#is-a-judgment-caught-up).
* **A pause stops the wait,** since `judged_through` does not pass a write that waits for judging to start again, or for the [rolling limit](/concepts/freshness#deferral-and-catch-up). If the budget runs out, the loop waits at that period until you raise it, so set a budget high enough for every period.
* **Or poll a query** for answers that are `pending`, to see which machines are not judged yet. When the query returns no row, the period is judged, or its answers wait and read `stale`.
* **`wait_for`** on a write waits for answers too, but it does less. It waits only for the documents in that write (not for a machine whose reports you wrote), it waits for at most 10 seconds, and it judges only the first 16 documents of the write at once. See [limits](/limits).

## Date each label

A label joins the answer that was current at its `observed_at`, which is the answer for the newest write that touched the document at that time (a write to the document itself or to one of its related records).

* `observed_at` must be at or after the `updated_at` of the period's last write that touched the document. If it is earlier, the label joins an older answer or none, and the report counts a label with no answer in `unmatched_outcomes`.
* `observed_at` now, taken after the wait, meets this rule if you use a clock that is in sync.
* Do not use the date from your history, since it falls before the replay and the label joins no answer.

## One label for each answer

An answer keeps one label. The latest `observed_at` counts (of two with the same `observed_at`, the one posted last), and the report counts each replaced label in `unmatched_outcomes.same_window`.

* Post one label for each touched document in each period. A document the period did not touch keeps its answer, so a second label for it replaces the earlier period's label.
* To correct a label, post the new value as of now before you write the next period, because after the next period's writes a label as of now joins the newer answer.

## Fit the labels

* While the current epoch has no fit, Brussle tries each hour, and the first run after the labels meet the [minimums](/calibration/correction#at-least-15-of-each-kind) (50, with at least 15 of each kind) makes the fit. After that, the fit runs each night.
* To fit now, select **Fit now** in the judgment's **Calibration** section or send `POST /namespaces/{ns}/judgments/{name}/calibration/fit`, once an hour for each judgment. See [fit now](/calibration/promotion#fit-now).

Without features, the fit adds a `calibrated` object and `p` stays the engine's. With features, the fit applies from 100 labels with 15 of each kind, and `p` then becomes the [corrected value](/calibration/features#how-the-fit-with-features-works), so expect it to move:

* `p` changes for each answer, with no new judging.
* Documents can cross your thresholds on `p`, so `GET` on the judgment shows a `fit_changed` warning until you set a threshold again. Pick new thresholds with the [threshold recommender](/calibration/thresholds).
* Brussle judges again, and bills, the judgments that read this one, and the event `judgment.fit_applied` reports them.

## Read the ranking

A question about a rare event has a low base rate: few machines fail in a quarter. `accuracy` then says little, because it counts `p` of 0.5 or more as `true`, and "no" for every machine is right for most of them.

Read the ranking numbers in each epoch's `held_out_metrics` instead. They score the engine's `p` (`raw`) and each fit (`scalar`, and with features `features`) on held-out outcomes:

* `auroc` says how well `p` puts the machines that failed above those that did not.
* `precision_at_5` and `precision_at_10` give the share of the top 5% and 10% by `p` that failed.

See [how well answers rank](/calibration/report#how-well-answers-rank).

With features, `features_only` scores a fit on your features alone. If it scores as well as `features`, the counts carry the number and the engine adds nothing.

Compare the precision with `base_rate`. Say 2% of machines failed, and 20% of the top 5% by `p` failed. The top 5% then held failures at 10 times the base rate in your history. That measures your history; it does not say how the next quarter will go.


This documentation is built and hosted on [Mintlify](https://mintlify.com), a developer documentation platform.