Skip to main content
A backtest measures a judgment and its correction on your history, before you use it live. You replay the records you already have one period at a time, and label each period’s answers with what happened in the next one. Since your history already holds the outcome, you do not wait for it.
  1. Use a bool judgment with the default horizon of 0s. Ask a narrow question about what the record shows, and keep counts as features.
  2. Read related records with last_n, not window.
  3. Write one period of history, and wait until each of its answers exists.
  4. Post one labelled example for each judged document the period touched, saying what happened in the next period, with observed_at now.
  5. Repeat for the next period. When the replay is done, let the fit run and read the report.
  6. Compare features with features_only in the report, since the difference is what the engine adds to your counts.

​
Labelled examples, not a horizon

A labelled example joins the answer that was current at its observed_at, so a label dated after a period’s answers exist and before the next period’s writes joins the answer for that period. That is the backtest. You cannot backtest a judgment with a horizon, because its outcome window opens when Brussle makes the answer and closes one horizon later. Brussle dates each write when it receives it and each answer when it makes it, and nothing dates a write or an answer in the past (created_at dates only a document’s creation). So each replayed answer opens its window now, and an outcome from your history falls before it. To measure a horizon judgment, you wait for the horizon to pass. For a backtest, put the next period in the label and keep horizon at 0s. The engine answers well about what a record shows now, and near chance about what will happen. So ask it a narrow question about the record as it is (such as “Do this machine’s recent reports describe a fault that stops it?”) and keep the counts of reports as features. The label says whether the machine failed in the next quarter, and the counts and the fit on your labels carry the number for a failure. See ask about what will happen.

​
Read each record’s past, not the time of the replay

A replay writes all of your history now. Send each record’s original created_at, and relations sort your records in the order they happened. The two bounds of a relation then behave differently:
  • last_n reads by created_at, so with last_n: 10 a machine reads the 10 newest reports that exist at that point of the replay. Use last_n.
  • window counts back from the time of a write, which is the later of the newest write to the judged document (as Brussle received it) and the newest created_at among its related documents. In a replay the newest write is now, so a window of 365d holds no record created more than 365 days before the replay, and early periods read nothing. Do not use window on a replay. See newest created first.
To count more records than the engine reads as text, add a second relation on the same records with a larger last_n (up to 1,000) and aggregate, and name its numbers in features. "render": false keeps those numbers out of what the engine reads. A blocking relation reads the newest records of its block. In a replay a whole period arrives at once, so a record early in the period would read the records after it, which it could not have seen. Add "relative": "before", and each record then reads the last_n records just before it by created_at, oldest first. A relation with relative takes no window and no order.

​
Run a backtest

Here you backtest a judgment on machines and their service reports, where each report holds its machine’s id in attributes.machine_id. The history covers several quarters, and each machine’s label is whether it failed in the next quarter. 1. Make a namespace for the backtest, and set a budget on it, since a backtest is billed as live judging (Brussle judges each touched machine again in each period). 2. Create the judgment before the first period, so Brussle judges each period as you write it.
POST /v1/namespaces/acme%2Fbacktest/judgments
  • If the create returns a replay estimate, Brussle has created nothing yet. In a new namespace the estimate is almost zero, so send the create again with confirm: true.
  • Wait for its reference_index jobs before the first period.
  • A judgment with features has a debounce of 10 minutes by default, so a short debounce_ms makes each period’s answers arrive sooner.
3. Replay each quarter, in order.
Python
  • quarters, documents_created_in, machines_touched_in, failed_in and next_quarter stand for your own code.
  • A post of outcomes is at most 2 MB, so split a large period into more posts.
4. Fit, then read the report. See fit the labels and read the ranking.

​
Wait for each period’s answers

Before you write the next period, wait until each answer of this one exists. Otherwise a machine’s answer can include records of the next period, no answer shows the machine as it was at the end of this one, and your label joins an older answer.
  • Compare revisions, as in the example. The import returns the highest revision it wrote, and once the judgment’s judged_through is at or above it, each machine the period touched has its answer (or its last attempt failed). A get of the judgment reads no documents. See is a judgment caught up?.
  • A pause stops the wait, since judged_through does not pass a write that waits for judging to start again, or for the rolling limit. If the budget runs out, the loop waits at that period until you raise it, so set a budget high enough for every period.
  • Or poll a query for answers that are pending, to see which machines are not judged yet. When the query returns no row, the period is judged, or its answers wait and read stale.
  • wait_for on a write waits for answers too, but it does less. It waits only for the documents in that write (not for a machine whose reports you wrote), it waits for at most 10 seconds, and it judges only the first 16 documents of the write at once. See limits.

​
Date each label

A label joins the answer that was current at its observed_at, which is the answer for the newest write that touched the document at that time (a write to the document itself or to one of its related records).
  • observed_at must be at or after the updated_at of the period’s last write that touched the document. If it is earlier, the label joins an older answer or none, and the report counts a label with no answer in unmatched_outcomes.
  • observed_at now, taken after the wait, meets this rule if you use a clock that is in sync.
  • Do not use the date from your history, since it falls before the replay and the label joins no answer.

​
One label for each answer

An answer keeps one label. The latest observed_at counts (of two with the same observed_at, the one posted last), and the report counts each replaced label in unmatched_outcomes.same_window.
  • Post one label for each touched document in each period. A document the period did not touch keeps its answer, so a second label for it replaces the earlier period’s label.
  • To correct a label, post the new value as of now before you write the next period, because after the next period’s writes a label as of now joins the newer answer.

​
Fit the labels

  • While the current epoch has no fit, Brussle tries each hour, and the first run after the labels meet the minimums (50, with at least 15 of each kind) makes the fit. After that, the fit runs each night.
  • To fit now, select Fit now in the judgment’s Calibration section or send POST /namespaces/{ns}/judgments/{name}/calibration/fit, once an hour for each judgment. See fit now.
Without features, the fit adds a calibrated object and p stays the engine’s. With features, the fit applies from 100 labels with 15 of each kind, and p then becomes the corrected value, so expect it to move:
  • p changes for each answer, with no new judging.
  • Documents can cross your thresholds on p, so GET on the judgment shows a fit_changed warning until you set a threshold again. Pick new thresholds with the threshold recommender.
  • Brussle judges again, and bills, the judgments that read this one, and the event judgment.fit_applied reports them.

​
Read the ranking

A question about a rare event has a low base rate: few machines fail in a quarter. accuracy then says little, because it counts p of 0.5 or more as true, and “no” for every machine is right for most of them. Read the ranking numbers in each epoch’s held_out_metrics instead. They score the engine’s p (raw) and each fit (scalar, and with features features) on held-out outcomes:
  • auroc says how well p puts the machines that failed above those that did not.
  • precision_at_5 and precision_at_10 give the share of the top 5% and 10% by p that failed.
See how well answers rank. With features, features_only scores a fit on your features alone. If it scores as well as features, the counts carry the number and the engine adds nothing. Compare the precision with base_rate. Say 2% of machines failed, and 20% of the top 5% by p failed. The top 5% then held failures at 10 times the base rate in your history. That measures your history; it does not say how the next quarter will go.