> ## Documentation Index
> Fetch the complete documentation index at: https://docs.brussle.com/llms.txt
> Use this file to discover all available pages before exploring further.

# Backtest on history

> Replay the records you already have one period at a time, label each answer with what happened next, and measure the judgment and its correction before you use it live.

A backtest measures a judgment and its correction on your history before you use it live. Replay the records you already have, one period at a time, and label each period's answers with what happened in the next. Your history already holds the outcome, so you do not wait for it.

1. Use a `bool` judgment with the default `horizon` of `0s`. Ask a narrow question about what the record shows. Keep counts as `features`.
2. Read related records with `last_n`, not `window`.
3. Write one period of history, and wait until each of its answers exists.
4. For each judged document the period touched, post one labelled example: what happened in the next period, with `observed_at` now.
5. Do the next period. When the replay is done, let the fit run, and read the report.
6. Compare `features` with `features_only` in the report. The difference is what the engine adds to your counts.

## Labelled examples, not a horizon

A [labelled example](/train/outcomes#post-labelled-examples) joins the answer that was current at its `observed_at`. A label dated after a period's answers exist, and before the next period's writes, joins the answer for that period. That is the backtest.

You cannot backtest a judgment with a `horizon`. Its [outcome window](/train/horizons#how-outcomes-meet-answers) opens when Brussle makes the answer and closes one `horizon` later. Brussle dates each write when it receives it and each answer when it makes it, and nothing dates a write or an answer in the past (`created_at` dates only a document's creation). Each replayed answer opens its window now, and an outcome from your history falls before it. To measure a horizon judgment, wait for the horizon to pass. For a backtest, put the next period in the label and keep `horizon` at `0s`.

The engine answers well about what a record shows now, and near chance about what will happen. Ask it a narrow question about the record as it is, such as "Do this machine's recent reports describe a fault that stops it?", and keep the counts of reports as features. The label says whether the machine failed in the next quarter. The counts and the fit on your labels carry the number for a failure. See [ask about what will happen](/train/features#ask-about-what-will-happen).

## Read each record's past, not the time of the replay

A replay writes all of your history now. Send each record's original `created_at`, so relations sort your records in the order they happened. The two bounds of a relation then behave differently:

* **`last_n` reads by `created_at`.** With `last_n: 10`, a machine reads the 10 newest reports that exist at that point of the replay. Use `last_n`.
* **`window` counts back from the time of a write:** the later of the newest write to the judged document, as Brussle received it, and the newest `created_at` among its related documents. In a replay, the newest write is now, so a `window` of `365d` holds no record created more than 365 days before the replay, and early periods read nothing. Do not use `window` on a replay. See [newest created first](/guides/related-documents#newest-created-first-and-what-an-edit-costs).

To count more records than the engine reads as text, add a second relation on the same records with a larger `last_n` (up to 1,000) and `aggregate`, and name its numbers in `features`. `"render": false` keeps them out of what the engine reads.

**A blocking relation reads the newest records of its block.** In a replay a whole period arrives at once, so a record early in the period would read the records after it, which it could not have seen. Add `"relative": "before"`: each record then reads the `last_n` records just before it by `created_at`, oldest first. A relation with `relative` takes no `window` and no `order`.

```json theme={"theme":{"light":"css-variables","dark":"css-variables"}}
{
  "related": {
    "earlier_reports": {
      "match": {"attributes.kind": "report"},
      "join": {"theirs": "attributes.machine_id", "mine": "attributes.machine_id"},
      "last_n": 10,
      "relative": "before",
      "fields": ["created_at", "state.summary"]
    }
  }
}
```

## Run a backtest

This example backtests a judgment on machines and their service reports. Each report has its machine's `id` in `attributes.machine_id`. The history covers several quarters, and each machine's label is whether it failed in the next quarter.

**1. Make a namespace for the backtest,** and set a [budget](/guides/import-existing-data#set-a-budget-first) on it. A backtest is billed as live judging: each touched machine is judged again in each period.

**2. Create the judgment before the first period,** so each period is judged as you write it.

```json POST /v1/namespaces/acme%2Fbacktest/judgments theme={"theme":{"light":"css-variables","dark":"css-variables"}}
{
  "name": "stopping_fault",
  "type": "bool",
  "applies_to": {"attributes.kind": "machine"},
  "question": "Do this machine's recent reports describe a fault that stops it?",
  "context": {
    "fields": ["state.model", "state.site"],
    "related": {
      "reports": {
        "match": {"attributes.kind": "report"},
        "join": {"theirs": "attributes.machine_id", "mine": "id"},
        "last_n": 10,
        "fields": ["created_at", {"path": "state.summary", "max_chars": 400}]
      },
      "report_counts": {
        "match": {"attributes.kind": "report"},
        "join": {"theirs": "attributes.machine_id", "mine": "id"},
        "last_n": 1000,
        "aggregate": {"count": true, "max": ["state.downtime_hours"], "render": false}
      }
    }
  },
  "features": ["report_counts.count", "report_counts.max(state.downtime_hours)"],
  "engine": {"name": "jev", "version": "current"},
  "freshness": {"policy": "on_change", "debounce_ms": 5000}
}
```

* If the create returns a [replay estimate](/guides/related-documents#the-replay-estimate), nothing is created yet. In a new namespace the estimate is almost zero. Send the create again with `confirm: true`.
* Wait for its `reference_index` jobs before the first period.
* A judgment with features has a debounce of 10 minutes by default. A short `debounce_ms` makes each period's answers arrive sooner.

**3. Replay each quarter, in order.**

```python Python theme={"theme":{"light":"css-variables","dark":"css-variables"}}
import time
from datetime import datetime, timezone

J = "stopping_fault"

for quarter in quarters[:-1]:  # the last quarter has no next quarter to label it
    written = ns.import_documents(documents_created_in(quarter))  # machines and reports, with their created_at

    # Wait until the judgment has judged through the quarter's last write.
    while ns.judgments.get(J)["judged_through"] < written.revision:
        time.sleep(5)

    # One label for each machine that this quarter touched: what happened in the next quarter.
    now = datetime.now(timezone.utc).isoformat()
    ns.outcomes.append([
        {"document_id": machine, "judgment": J, "value": failed_in(machine, next_quarter(quarter)), "observed_at": now}
        for machine in machines_touched_in(quarter)
    ])
```

* `quarters`, `documents_created_in`, `machines_touched_in`, `failed_in` and `next_quarter` stand for your own code.
* A post of outcomes is at most 2 MB. Split a large period into more posts.

**4. Fit, then read the report.** See [fit the labels](#fit-the-labels) and [read the ranking](#read-the-ranking).

## Wait for each period's answers

Before you write the next period, wait until each answer of this one exists. Otherwise a machine's answer can include records of the next period, no answer shows the machine as it was at the end of this one, and your label joins an older answer.

* **Compare revisions,** as in the example. The import returns the highest `revision` it wrote. When the judgment's `judged_through` is at or above it, each machine the period touched has its answer, or its last attempt failed. A get of the judgment reads no documents. See [is a judgment caught up?](/concepts/freshness#is-a-judgment-caught-up).
* **A pause stops the wait.** `judged_through` does not pass a write that waits for judging to start again, or for the [rolling limit](/concepts/freshness#deferral-and-catch-up). If the budget runs out, the loop waits at that period until you raise it, so set a budget high enough for every period.
* **Or poll a query** for answers that are `pending`, to see which machines are not judged yet. When it returns no row, the period is judged, or its answers wait and read `stale`.
* **`wait_for`** on a write waits for answers too, but does less: it waits only for the documents in that write, not for a machine whose reports you wrote; for at most 10 seconds; and it judges only the first 16 documents of the write at once. See [limits](/limits).

## Date each label

A label joins the answer that was current at its `observed_at`: the answer for the newest write that touched the document at that time, whether to the document itself or to one of its related records.

* `observed_at` must be at or after the `updated_at` of the period's last write that touched the document. If it is earlier, the label joins an older answer or none, and the report counts a label with no answer in `unmatched_outcomes`.
* `observed_at` now, taken after the wait, meets this rule. Use a clock that is in sync.
* Do not use the date from your history. It falls before the replay, so the label joins no answer.

## One label for each answer

An answer keeps one label: the latest `observed_at` counts, and of two with the same `observed_at`, the one posted last. The report counts each replaced label in `unmatched_outcomes.same_window`.

* Post one label for each touched document in each period. A document the period did not touch keeps its answer, so a second label for it replaces the earlier period's label.
* To correct a label, post the new value as of now, before you write the next period. After the next period's writes, a label as of now joins the newer answer.

## Fit the labels

* While the current epoch has no fit, Brussle tries each hour. The first run after the labels meet the [minimums](/train/correction#at-least-15-of-each-kind), 50 with at least 15 of each kind, makes the fit. After that it runs each night.
* To fit now, select **Fit now** in the judgment's **Calibration** section, or send `POST /namespaces/{ns}/judgments/{name}/calibration/fit`, once an hour for each judgment. See [fit now](/train/promotion#fit-now).

Without features, the fit adds a `calibrated` object and `p` stays the engine's. With features, the fit applies from 100 labels with 20 of each kind, and then `p` is the [corrected value](/train/features#how-the-fit-with-features-works), so expect it to move:

* `p` changes for each answer, with no new judging.
* Documents can cross your thresholds on `p`. `GET` on the judgment shows a `fit_changed` warning until you set a threshold again. Pick new ones with the [threshold recommender](/train/thresholds).
* The judgments that read this one are judged again and billed. The event `judgment.fit_applied` reports them.

## Read the ranking

A question about a rare event has a low base rate: few machines fail in a quarter. `accuracy` then says little, because it counts `p` of 0.5 or more as `true`, and "no" for every machine is right for most of them.

Read the ranking numbers in each epoch's `held_out_metrics` instead, for the engine's `p` (`raw`) and for each fit (`scalar`, and with features `features`), on held-out outcomes: `auroc`, how well `p` puts the machines that failed above those that did not; and `precision_at_5` and `precision_at_10`, the share of the top 5% and 10% by `p` that failed. See [how well answers rank](/train/report#how-well-answers-rank).

With features, `features_only` scores a fit on your features alone. If it scores as well as `features`, the counts carry the number and the engine adds nothing.

Compare the precision with `base_rate`. If 2% of machines failed and 20% of the top 5% by `p` failed, the top 5% held failures at 10 times the base rate in your history. This measures your history. It does not say how the next quarter will go.


This documentation is built and hosted on [Mintlify](https://mintlify.com), a developer documentation platform.