horizon, and each result you record counts against the answer it followed, once per window, so frequent judging never makes the numbers look better.
The engine answers well about what a document shows now, and near chance about what will happen. For a question about what will happen, ask the engine a narrow question about the document as it is, keep counts from related documents as features, and let the fit on your outcomes weigh them. The features page shows the full pattern.
Set a horizon
horizon is part of the judgment’s definition: the time after each answer in which an outcome counts against that answer. It is a whole number and a unit (s, m, h or d), such as "30d". The default is 0s, for labelled examples. A change to it makes a new version.
Then post the outcome when the event happens:
attributes.status becoming overdue, an outcome rule posts it for you.
A judgment with a horizon has no labelling queue: its outcome is about the time after the answer, which nobody can label from the document today. It gets positives as they happen, and negatives as you post them or from implicit negatives. It also cannot be backtested.
How outcomes meet answers
Brussle matches outcomes to answers with outcome windows. (A window here is an outcome window, not a relation’swindow.) For example, a judgment has a 30-day horizon, and Brussle judges a document on day 0. That answer’s window runs to day 30. An outcome “it happened”, observed on day 12, labels the answer made on day 0.
- Windows. A document’s first answer opens a window, which runs for the horizon from the time of the answer. Answers made while it is open do not open windows. The first answer after it closes opens the next. Each window is measured on the answer that opened it.
- Labels. An outcome labels the window that its
observed_atfalls in. Two outcomes in one window count once. An outcome you posted wins over one a rule made. Otherwise atruewins. - “It did not happen”, after the window. A
falseobserved after a window closed, and before the next one opens, labels the window that closed. Post “it did not happen” on day 40 and it counts for the answer made on day 0. Atruealready in that window stays, unless a rule made it. - Late outcomes count. Brussle matches outcomes each time the fit or the report reads them, so a result you post weeks after the event labels its window at the next fit.
- Engine epochs. A window counts for the epoch of the answer that opened it.
Outcomes that count nowhere
The report counts these inunmatched_outcomes:
- an outcome observed before the document’s first answer (
before_first_evaluation); - an event (
true) observed after a window closed and before the next answer (after_horizon).
After the event
Once Brussle sees atrue for a document, answers made from then on have nothing left to measure: the event already happened, and often the write that recorded it also caused the answer. Their windows count nowhere, not as a false and not labelled by an outcome. An outcome in such a window counts in unmatched_outcomes.after_positive. For an event that can happen again, such as a second refund, only the first counts.
Deleted and rewritten documents
- A document written again after a delete starts a new window at its first new answer, even after an event. An answer is about one life of its document, so an outcome observed after the first answer of the new life never labels a window from before the delete.
- An outcome observed while its document is deleted labels no window, and counts in
unmatched_outcomes.no_evaluation. A document remembers the last 8 deletions of itsid, so this holds however late you post the outcome, as long as the deletion is still recorded when Brussle stores or reads the outcome, or when you write theidagain.
Post what did not happen
The correction needs the invoices that did not go overdue too. Do one of these:- Post
falseafter the window has passed, at any time. It answers the window that closed, unless an event was already posted in it. - Turn on implicit negatives, if you record every event. Then each window that closes with no outcome counts as
false.
Turn on implicit negatives
On abool judgment with a horizon that is not zero, you can post only what happened, and let every other window count as “no”. This example also records each event from your writes with a rule:
PATCH /namespaces/{ns}/judgments/{name}. It is a setting, like thresholds, so it makes no version. The next fit and every report use it. Each PATCH replaces the whole outcomes object, so send the rules with it.
With implicit negatives on:
- A window that closes with no outcome counts as
false, for the answer that opened it. - A window that is still open counts nothing yet. An answer from yesterday on a 30-day judgment does not count, because its outcome can still come.
- Outcomes you post, or that rules make, always win in their window. Post a positive quickly, or let a rule make it from the write that records the event. Until then, the window counts as a negative when it closes.
- A window whose document was deleted before it closed is left out, unless an outcome labels it: the event may have happened to a deleted document, so its silence is not a “no”. After the deletion is no longer recorded, the document counts as existing until its newest answer.
- A window opened after the event is left out. The event already happened to that document.
- The report shows the counts: implicit outcomes in each epoch’s
implicit_negativesand inoutcomes_by_source, and the windows left out incensored_predictions:window_open(not closed yet),document_deleted(deleted before it closed) andafter_positive(opened after atrue). - A judgment with many judged documents is fitted on a representative sample of them, with the outcomes posted for those documents, so the share of positives stays true.
- you record only some of the events, such as refunds from one payment system but not another;
- outcomes arrive long after the horizon, such as chargebacks that take 90 days to report on a 30-day judgment;
- the judgment has the default
0shorizon. Brussle refuses implicit negatives there.