Backtest on history

1. Use a bool judgment with the default horizon of 0s. Ask a narrow question about what the record shows. Keep counts as features.
2. Read related records with last_n, not window.
3. Write one period of history, and wait until each of its answers exists.
4. For each judged document the period touched, post one labelled example: what happened in the next period, with observed_at now.
5. Do the next period. When the replay is done, let the fit run, and read the report.
6. Compare features with features_only in the report. The difference is what the engine adds to your counts.

​

Labelled examples, not a horizon

​

Read each record’s past, not the time of the replay

• last_n reads by created_at. With last_n: 10, a machine reads the 10 newest reports that exist at that point of the replay. Use last_n.
• window counts back from the time of a write: the later of the newest write to the judged document, as Brussle received it, and the newest created_at among its related documents. In a replay, the newest write is now, so a window of 365d holds no record created more than 365 days before the replay, and early periods read nothing. Do not use window on a replay. See newest created first.

{
  "related": {
    "earlier_reports": {
      "match": {"attributes.kind": "report"},
      "join": {"theirs": "attributes.machine_id", "mine": "attributes.machine_id"},
      "last_n": 10,
      "relative": "before",
      "fields": ["created_at", "state.summary"]
    }
  }
}

​

Run a backtest

{
  "name": "stopping_fault",
  "type": "bool",
  "applies_to": {"attributes.kind": "machine"},
  "question": "Do this machine's recent reports describe a fault that stops it?",
  "context": {
    "fields": ["state.model", "state.site"],
    "related": {
      "reports": {
        "match": {"attributes.kind": "report"},
        "join": {"theirs": "attributes.machine_id", "mine": "id"},
        "last_n": 10,
        "fields": ["created_at", {"path": "state.summary", "max_chars": 400}]
      },
      "report_counts": {
        "match": {"attributes.kind": "report"},
        "join": {"theirs": "attributes.machine_id", "mine": "id"},
        "last_n": 1000,
        "aggregate": {"count": true, "max": ["state.downtime_hours"], "render": false}
      }
    }
  },
  "features": ["report_counts.count", "report_counts.max(state.downtime_hours)"],
  "engine": {"name": "jev", "version": "current"},
  "freshness": {"policy": "on_change", "debounce_ms": 5000}
}

• If the create returns a replay estimate, nothing is created yet. In a new namespace the estimate is almost zero. Send the create again with confirm: true.
• Wait for its reference_index jobs before the first period.
• A judgment with features has a debounce of 10 minutes by default. A short debounce_ms makes each period’s answers arrive sooner.

import time
from datetime import datetime, timezone

J = "stopping_fault"

for quarter in quarters[:-1]:  # the last quarter has no next quarter to label it
    written = ns.import_documents(documents_created_in(quarter))  # machines and reports, with their created_at

    # Wait until the judgment has judged through the quarter's last write.
    while ns.judgments.get(J)["judged_through"] < written.revision:
        time.sleep(5)

    # One label for each machine that this quarter touched: what happened in the next quarter.
    now = datetime.now(timezone.utc).isoformat()
    ns.outcomes.append([
        {"document_id": machine, "judgment": J, "value": failed_in(machine, next_quarter(quarter)), "observed_at": now}
        for machine in machines_touched_in(quarter)
    ])

• quarters, documents_created_in, machines_touched_in, failed_in and next_quarter stand for your own code.
• A post of outcomes is at most 2 MB. Split a large period into more posts.

​

Wait for each period’s answers

• Compare revisions, as in the example. The import returns the highest revision it wrote. When the judgment’s judged_through is at or above it, each machine the period touched has its answer, or its last attempt failed. A get of the judgment reads no documents. See is a judgment caught up?.
• A pause stops the wait. judged_through does not pass a write that waits for judging to start again, or for the rolling limit. If the budget runs out, the loop waits at that period until you raise it, so set a budget high enough for every period.
• Or poll a query for answers that are pending, to see which machines are not judged yet. When it returns no row, the period is judged, or its answers wait and read stale.
• wait_for on a write waits for answers too, but does less: it waits only for the documents in that write, not for a machine whose reports you wrote; for at most 10 seconds; and it judges only the first 16 documents of the write at once. See limits.

​

Date each label

• observed_at must be at or after the updated_at of the period’s last write that touched the document. If it is earlier, the label joins an older answer or none, and the report counts a label with no answer in unmatched_outcomes.
• observed_at now, taken after the wait, meets this rule. Use a clock that is in sync.
• Do not use the date from your history. It falls before the replay, so the label joins no answer.

​

One label for each answer

• Post one label for each touched document in each period. A document the period did not touch keeps its answer, so a second label for it replaces the earlier period’s label.
• To correct a label, post the new value as of now, before you write the next period. After the next period’s writes, a label as of now joins the newer answer.

​

Fit the labels

• While the current epoch has no fit, Brussle tries each hour. The first run after the labels meet the minimums, 50 with at least 15 of each kind, makes the fit. After that it runs each night.
• To fit now, select Fit now in the judgment’s Calibration section, or send POST /namespaces/{ns}/judgments/{name}/calibration/fit, once an hour for each judgment. See fit now.

• p changes for each answer, with no new judging.
• Documents can cross your thresholds on p. GET on the judgment shows a fit_changed warning until you set a threshold again. Pick new ones with the threshold recommender.
• The judgments that read this one are judged again and billed. The event judgment.fit_applied reports them.

​

Read the ranking