Skip to main content
Each outcome you record measures the answer it sits beside, and enough of them train the judgment’s correction. Post labels, upload a CSV, or let rules make outcomes from the writes you already send. An outcome records what happened to a judged document. For example, the judgment material_change asks whether an edit to a contract section changed what the parties must do. Your legal team reviews the edit to section sec_118 and finds that it changed the payment terms. You post true as the outcome for sec_118. Outcomes are data. Brussle keeps each one beside the exact answer it measures, which names its document revision, judgment version and engine version. Each outcome counts for the judgment version and engine epoch of the answer it joins. There are two kinds:
  • Labelled examples: a person’s answer to the question for a document, as of a time. Use them to measure a judgment before you rely on it. This page shows how.
  • Results that come later: what happened after the answer, such as “this invoice went overdue”. These need a horizon on the judgment. See outcomes that arrive later.

​
Post labelled examples

A labelled example is an outcome on a judgment with the default horizon of 0s. It records what was true of the document at observed_at.
  1. Write the documents.
  2. Post their labels to POST /namespaces/{ns}/outcomes.
value is what was true of the document at observed_at: How a label joins its answer:
  • It joins the answer for the revision that was current at observed_at. A label that you post before that answer exists still joins it when it arrives.
  • A label observed while its document was deleted joins nothing, because no revision was current. A document that you write again remembers the last 8 deletions of its id, so this holds however late you post the label, as long as the deletion is still recorded when you post or read the label, or when you write the id again.
  • An answer keeps one label. A later label for the same answer replaces the earlier one in the correction, the report and the recommender: the label with the latest observed_at counts, and of two with the same observed_at, the one you posted last. A document that you label each period without a change counts once. Brussle still stores every label, and the report counts the replaced ones in unmatched_outcomes.same_window. A labelling-queue label is counted apart: an answer also keeps one queue label.
  • Outcomes are append-only, and retries are safe. The same document, judgment, value and observed_at is stored once.
  • Outcomes belong to a namespace. Post them on the namespace’s own path, never on a template prefix.
  • Errors. A value that does not fit the judgment’s type is invalid_request. An unknown judgment is not_found. A post is at most 2 MB.
To measure a judgment on records you already have, replay them one period at a time and label each period with what happened next. See backtest on history.

​
Upload labels as a CSV

  1. In the dashboard, open the judgment.
  2. Go to its Labels tab.
  3. Upload a CSV with the header document_id,value, and optionally an observed_at column in RFC 3339:
Your browser checks each row against the judgment’s type and lists the row errors before it sends anything. It posts valid rows in batches of 1,000. A row without observed_at uses the time of the upload.

​
Correct an outcome

Post the correct value as of now. If the document has not changed, the correction joins the same answer and replaces the earlier label.
  • A correction must differ. Its value or its observed_at must differ from each earlier outcome of its document. The same value at the same time again is a retry, stored once at its first place. To set an outcome back to an earlier value, use a new observed_at.
  • Date it now, not in the past. The correction and the report keep the label with the latest observed_at, and so does a relation that reads the outcome. Brussle keeps a correction dated before the outcome it corrects, but a relation never reads it.
  • On a rule’s outcome, a posted outcome always wins in its answer or window. Post one to correct a rule by hand.

​
Record outcomes from your own writes

Most outcomes already reach you as ordinary writes: a ticket closes with the team that handled it, your team records which invoice a payment settles, a moderator reverses a removal. Outcome rules turn those writes into outcomes, so you get labels without posting them. A rule watches one path on the judged document. When a write changes the path to the value you name, that write is an outcome. Set rules with PATCH /namespaces/{ns}/judgments/{name}, under outcomes. Triage. A choice judgment routes tickets to a team. When a ticket closes, the team that closed it is the right answer. from takes the outcome from the document after the write:
Matching. The settles judgment chooses which open invoice a payment settles. When your team records the match in state.settles, the invoice it names is the right answer. on fires when the path gets a value for the first time:
Moderation. A bool judgment asks “does this post break the rules?”, and its answers cause removals. A reversed removal means the answer was wrong. An upheld removal means it was right:

​
How a rule fires

  • Each rule has when or on. when.path is attributes.<name> or state.<key>[.<key>...] on the judged document itself: for a judgment with relations, the payment, not a related document. becomes names one value. in names up to 16. on names one path, and fires when its value changes from missing or null to any other value.
  • It fires on the write that changes the value to that value, or into the list, from any other value. A write that keeps the value fires nothing, so an unrelated edit, a repeated sync or a retried request adds no second outcome.
  • A delete never fires. A write that creates the document fires if the document has the value when it is created.
  • value is the outcome: true or false for a bool, an option for a choice, a level for a score. A bool takes only value.
  • from (choice and score only) takes the value at that path after the write. If that value is not an option or level, or the path is missing, Brussle keeps the outcome but it counts nowhere: the report counts it in unmatched_outcomes.does_not_fit. Use that count to find a rule that points at the wrong field.
  • A value that changes back and forth fires each time it arrives. A status that goes overdue, then open, then overdue again gives two outcomes. On a judgment with a horizon, two outcomes in one window still count once.
How rule outcomes count:
  • They join as posted outcomes do, and the report counts them as rule in outcomes_by_source.
  • A posted outcome wins when you post one and a rule makes one for the same answer or window.
  • With the default 0s horizon, a rule outcome joins the answer to the document as it was immediately before the write that fired the rule. That answer is the one the write responded to, not the answer the write itself causes.
  • Its observed_at is the time the write was committed.

​
Rule limits

  • Plans. Outcome rules are on the Team plan and above. Below Team, a PATCH that sets rules returns plan_required, and one that clears them ("rules": []) always works. After a downgrade takes effect, rules keep running, but the correction leaves out what they make from then on: the report counts those outcomes in rule_outcomes_paused, and the dashboard’s Calibration section shows the count. Outcomes collected before stay in use. After an upgrade, new rule outcomes count again. See plans.
  • Up to 4 rules for each judgment, each on one path. Each PATCH replaces the whole outcomes object, so send implicit_negatives and every rule together.
  • Rules apply from when you set them. Brussle does not replay earlier writes. To label the past, post outcomes for it.
  • Rules read only the judged document’s own writes. A write to a payment is not an outcome for the invoice it settles. Post that outcome yourself.
  • Templates. Neither a namespace that gets a judgment from a template nor the template’s prefix path can set rules. Set rules on a namespace’s own judgment.
  • Errors. A rule that does not fit the judgment’s type, such as from on a bool, and a fifth rule are invalid_request, with the rule and the reason.
  • Features. A feature cannot read a path that a rule reads, because that path holds the outcome.

​
Post what did not happen too

Most teams record what happened: the payment was refunded, the match was reversed. Few write down what did not happen. The correction needs both: at least 15 of each kind, and the recommender 20.
  • For labelled examples, label documents where the answer is “no”, with "value": false, as well as “yes”.
  • For results that come later, post false when an answer’s window passed without the event, or turn on implicit negatives when no outcome really means “no”.

​
Label more than the reviewed documents

Outcomes usually come from documents that someone already looked at, and people look at the documents that crossed a threshold. Then almost all outcomes sit on answers with a high raw probability, the fit has nothing for the rest of the range, and its corrected numbers for low-probability answers are a guess. The report warns about this for each epoch, in warnings:
  • above_thresholds: more than 80% of the outcomes are on answers that meet one of the judgment’s thresholds.
  • high_band: on a bool, more than 80% of the outcomes are on answers with p of 0.7 or more.
Its coverage gives the lowest and highest raw value the outcomes cover, and the reliability curve shows how many outcomes fall in each band. What to do: label a random sample of documents as well. The labelling queue draws one for you. Keep labelling the reviewed documents too; both count.

​
Outcomes that other judgments read

Outcomes also feed relations. Another judgment can read the newest outcome of a related document as a fact in its context. For example, a judgment on a customer can read whether the customer’s past orders were refunded.
  • A relation reads the outcome with the latest observed_at. See correct an outcome.
  • Outcomes posted before relations could read them, including one posted before this feature existed and one posted for a document that did not exist yet: a relation reads such an outcome for the document’s current life only if no deletion of the document is recorded at or after its observed_at. A document remembers the last 8 deletions of its id, so a later life of a document deleted long ago can read such an outcome.

​
Outcomes that join nothing

Brussle keeps an outcome that joins no answer, and it counts if a later answer takes it. The report counts these in unmatched_outcomes, by reason: