Skip to main content
Each outcome you record measures the answer it sits beside, and once you have enough of them Brussle fits the judgment’s correction. You can post labels, upload a CSV, or let rules make outcomes from the writes you already send. An outcome records what happened to a judged document. Take the judgment material_change, which asks whether an edit to a contract section changed what the parties must do. Your legal team reviews the edit to section sec_118 and finds that it changed the payment terms, so you post true as the outcome for sec_118. Outcomes are data. Brussle keeps each one beside the exact answer it measures (that answer names its document revision, judgment version and engine version), so each outcome counts for the judgment version and engine epoch of the answer it joins. Outcomes come in two kinds:
  • Labelled examples: a person’s answer to the question for a document, as of a time. You use them to measure a judgment before you rely on it, as post labelled examples shows.
  • Results that come later: what happened after the answer (such as “this invoice went overdue”). These need a horizon on the judgment, which outcomes that arrive later explains.

​
Post labelled examples

A labelled example is an outcome on a judgment with the default horizon of 0s. It records what was true of the document at observed_at. Date each label when the answer was acted on, not when you found out. A label joins the answer to the document as it was at observed_at. If the document changed before you knew the outcome, a label dated now joins the newer answer, and when the change showed the outcome (a status set to closed, a note that the customer gave notice), that answer had already read it. The report then looks perfect and the live answers are wrong. When the document changes before you know the outcome:
  1. Use an outcome rule with implicit negatives, the first choice: a rule outcome joins the answer the write responded to, never the answer the write caused.
  2. Or set a horizon, so each outcome labels the answer whose window it falls in.
  3. Or post the label with observed_at at the time of the decision.
All three give the same correction. Brussle counts the labels that joined an answer made after an earlier one in the report’s coverage.labels_on_later_revisions, and holds a fit made on labels that read like the outcome.
  1. Write the documents.
  2. Post their labels to POST /namespaces/{ns}/outcomes.
value is what was true of the document at observed_at, in the form the judgment type takes: How a label joins its answer:
  • It joins the answer for the revision that was current at observed_at. A label that you post before that answer exists still joins it when the answer arrives. A label dated after the document changed joins the answer to the changed document.
  • A label observed while its document was deleted joins nothing, because no revision was current. This holds however late you post the label, since a document that you write again remembers the last 8 deletions of its id (as long as the deletion is still recorded when you post or read the label, or when you write the id again).
  • An answer keeps one label. A later label for the same answer replaces the earlier one in the correction, the report and the recommender. The label with the latest observed_at counts, and of two with the same observed_at, the one you posted last, so a document that you label each period without a change counts once. Brussle still stores every label, and the report counts the replaced ones in unmatched_outcomes.same_window. A labelling-queue label is counted apart: an answer keeps one queue label as well.
  • Outcomes are append-only, and retries are safe. Brussle stores the same document, judgment, value and observed_at once.
  • Outcomes belong to a namespace. Post them on the namespace’s own path, never on a template prefix.
  • Errors. A value that does not fit the judgment’s type is invalid_request, and an unknown judgment is not_found. A post is at most 2 MB.
To measure a judgment on records you already have, replay them one period at a time and label each period with what happened next, as backtest on history shows.

​
Upload labels as a CSV

  1. In the dashboard, open the judgment.
  2. Go to its Labels tab.
  3. Upload a CSV with the header document_id,value, and optionally an observed_at column in RFC 3339:
Your browser checks each row against the judgment’s type and lists the row errors before it sends anything, then posts the valid rows in batches of 1,000. A row without observed_at uses the time of the upload; when any row lacks it, the upload page says so and asks you to confirm before it sends, because a label dated at the upload joins the document’s current answer, which may be to a document that changed after you reviewed it.

​
Correct an outcome

To correct an outcome, post the correct value as of now. If the document has not changed, the correction joins the same answer and replaces the earlier label.
  • A correction must differ. Its value or its observed_at must differ from each earlier outcome of its document, since the same value at the same time again is a retry, stored once at its first place. To set an outcome back to an earlier value, use a new observed_at.
  • Date it now, not in the past. The judgment’s correction and the report keep the label with the latest observed_at, and so does a relation that reads the outcome. Brussle keeps a correction dated before the outcome it corrects, but a relation never reads it.
  • On a rule’s outcome, a posted outcome always wins in its answer or window. Post one to correct a rule by hand.

​
Record outcomes from your own writes

Most outcomes already reach you as ordinary writes: a ticket closes with the team that handled it, your team records which invoice a payment settles, a moderator reverses a removal. Outcome rules turn those writes into outcomes, so you get labels without posting them. A rule watches one path on the judged document, and a write that changes the path to the value you name is an outcome. You set rules with PATCH /namespaces/{ns}/judgments/{name}, under outcomes. Triage. A choice judgment routes tickets to a team. When a ticket closes, the team that closed it is the right answer. from takes the outcome from the document after the write:
Matching. The settles judgment chooses which open invoice a payment settles. When your team records the match in state.settles, the invoice it names is the right answer. on fires when the path gets a value for the first time:
Moderation. A bool judgment asks “does this post break the rules?”, and its answers cause removals. A reversed removal means the answer was wrong, and an upheld removal means it was right:

​
How a rule fires

  • Each rule has when or on. when.path is attributes.<name> or state.<key>[.<key>...] on the judged document itself (for a judgment with relations, the payment, not a related document). becomes names one value, and in names up to 16. on names one path, and fires when its value changes from missing or null to any other value.
  • It fires on the write that changes the value to that value, or into the list, from any other value. A write that keeps the value fires nothing, so an unrelated edit, a repeated sync or a retried request adds no second outcome.
  • A delete never fires. A write that creates the document fires if the document has the value when it is created.
  • value is the outcome: true or false for a bool, an option for a choice, a level for a score. A bool takes only value.
  • from (choice and score only) takes the value at that path after the write. If that value is not an option or level, or the path is missing, Brussle keeps the outcome but it counts nowhere. The report counts it in unmatched_outcomes.does_not_fit, and that count finds a rule that points at the wrong field.
  • A value that changes back and forth fires each time it arrives. A status that goes overdue, then open, then overdue again gives two outcomes. On a judgment with a horizon, two outcomes in one window still count once.
How rule outcomes count:
  • They join as posted outcomes do, and the report counts them as rule in outcomes_by_source.
  • A posted outcome wins when you post one and a rule makes one for the same answer or window.
  • With the default 0s horizon, a rule outcome joins the answer to the document as it was immediately before the write that fired the rule. That answer is the one the write responded to, not the answer the write itself causes.
  • Its observed_at is the time the write was committed.

​
Rule limits

  • Plans. Outcome rules are on the Team plan and above. Below Team, a PATCH that sets rules returns plan_required, while one that clears them ("rules": []) always works. After a downgrade takes effect, rules keep running, but the correction leaves out what they make from then on; the report counts those outcomes in rule_outcomes_paused, and the dashboard’s Calibration section shows the count. Outcomes collected before stay in use. After an upgrade, new rule outcomes count again (see plans).
  • Up to 4 rules for each judgment, each on one path. Each PATCH replaces the whole outcomes object, so send implicit_negatives and every rule together.
  • Rules apply from when you set them. Brussle does not replay earlier writes, so to label the past, post outcomes for it.
  • Rules read only the judged document’s own writes. A write to a payment is not an outcome for the invoice it settles, so post that outcome yourself.
  • Templates. Neither a namespace that gets a judgment from a template nor the template’s prefix path can set rules. Set rules on a namespace’s own judgment.
  • Errors. A rule that does not fit the judgment’s type (such as from on a bool) and a fifth rule are invalid_request, with the rule and the reason.
  • Features. A feature cannot read a path that a rule reads, because that path holds the outcome.

​
Post what did not happen too

Most teams record what happened (the payment was refunded, the match was reversed), but few write down what did not happen. The correction needs both: at least 15 of each kind, and the recommender 20.
  • For labelled examples, label documents where the answer is “no” (with "value": false) as well as “yes”.
  • For results that come later, post false when an answer’s window passed without the event, or turn on implicit negatives when no outcome really means “no”.

​
Label more than the reviewed documents

Outcomes usually come from documents that someone already looked at, and people look at the documents that crossed a threshold. Almost all outcomes then sit on answers with a high raw probability, so the fit has nothing for the rest of the range, and its corrected numbers for low-probability answers are a guess. On a judgment with features it is worse: inside a slice of the range the engine’s answers barely differ, so the fit leans on the features and gives the engine’s answer almost no weight, and applied to the rest of the range, where the engine carries the signal, it is wrong exactly where a threshold acts. The report warns about this skew for each epoch, in warnings:
  • coverage_truncated: more than 20% of the judgment’s live answers have a raw p outside the range the labels cover. The warning gives the share below the labels and above them, the labelled range, and where its bounds sit among the live answers. A fit on these labels, and a threshold set from them, act on answers the labels say nothing about, so no correction is applied while it warns: answers read the engine’s number (truncated_sample in the report). The threshold recommender carries the same warning, and says what its threshold would act on.
  • above_thresholds: more than 80% of the outcomes are on answers that meet one of the judgment’s thresholds.
  • high_band: on a bool, more than 80% of the outcomes are on answers with p of 0.7 or more.
The report’s coverage gives the lowest and highest raw value the outcomes cover, and its reliability curve shows how many outcomes fall in each band. No correction made on such labels is applied: the report says truncated_sample, and answers read the engine’s number until the labels cover the range. See why an epoch has no correction. To fix the skew, label a random sample of documents as well; the labelling queue draws one for you. Keep labelling the reviewed documents too, since both count.

​
Outcomes that other judgments read

Outcomes also feed relations: another judgment can read the newest outcome of a related document as a fact in its context (a judgment on a customer, for example, can read whether the customer’s past orders were refunded).
  • A relation reads the outcome with the latest observed_at (see correct an outcome).
  • Outcomes posted before relations could read them, including one posted before this feature existed and one posted for a document that did not exist yet: a relation reads such an outcome for the document’s current life only if no deletion of the document is recorded at or after its observed_at. A document remembers the last 8 deletions of its id, so a later life of a document deleted long ago can read such an outcome.

​
Outcomes that join nothing

Brussle keeps an outcome that joins no answer, and the outcome counts if a later answer takes it. The report counts these outcomes in unmatched_outcomes, by reason: