Skip to main content
Set a target, such as “send edits to legal review, and be right 90% of the time”, and the recommender finds the threshold that meets it on your outcomes, with a 95% interval and the whole precision and recall curve. Label a random sample from the labelling queue, and the recommendation stands for your real traffic rather than for the documents someone already reviewed, with about a third fewer labels. The recommender and drawing from the queue are on the Team plan and above. Below Team they return plan_required (see plans); Developer sees a preview of both.

​
Get a recommendation

  • target is precision:<x> or recall:<x>.
  • A bool without features: the threshold is on the engine’s p, which is what thresholds read. A bool with features: on the corrected p its answers read (see a threshold on one feature for a cut on one feature). A composite: on the combined p.
  • A choice: add the option, such as &option=fraud. The threshold is then on the raw dist[fraud]. A choice without option is invalid_request.
  • A choice among candidates (options.from, see entity matching) is the exception: its threshold is on the raw escape_p, the unmatched queue. option is none_of_the_above or left out; any other is refused.
  • A score gets no recommendation (invalid_request).
The example shows the first of the curve’s 101 points.

​
Read the recommendation

The response is always 200. status says which of three results it is:
  • A precision target gets the lowest threshold that meets it, which keeps the most recall. A recall target gets the highest, which keeps the most precision.
  • On a choice among candidates it is the opposite. A document’s pick is taken when its escape_p is below the threshold. Precision is the share of taken picks that are right; recall is the share of outcomes naming a match whose pick was taken and right. A precision target gets the highest threshold that meets it, a recall target the lowest. Apply it as {"unmatched": {"value": "none_of_the_above", "gte": threshold}}.
  • The curve has 101 points, thresholds 0.00 to 1.00 in steps of 0.01, each with its precision and recall. The recommended threshold is one of them. Precision is null where no answer reaches the threshold.
  • A point counts what a threshold selects. It compares each answer’s number as the answer shows it, as a named threshold and a query filter do, answers equal to the threshold included. A threshold you set from a point selects exactly the answers it counted.
  • The outcomes are from the current epoch: at least 100, with 20 positive and 20 negative. For a choice, 20 whose outcome is the option and 20 whose outcome is another. When the current epoch has too few, the previous epoch’s are used, and from_previous_epoch is true.
  • fitted_on says what the recommendation rests on. It is queue when the epoch’s queue labels alone number 100, with 20 of each kind: it then uses only them, each weighted back to your traffic, so its precision and recall are your traffic’s, not a sample’s in which each band counts the same. Its interval then comes from how many labels each band has and how common the band is in your answers, so labels at your thresholds make it narrower. Otherwise all: each outcome counts once. The Thresholds tab says “on 140 queue labels, weighted to your traffic” or “on 2,300 outcomes, unweighted”. outcomes_by_source counts those outcomes by source; a threshold on a feature and a choice among candidates leave it out.
With outcomes of only one kind, the recommender does not guess:
reason is too_few_outcomes, too_few_positives or too_few_negatives, chosen as for the report’s not_fitted: a missing kind first, then the total, then 20 of a kind.

​
A threshold on one feature

On a bool with features, add feature= to get a cut on one feature instead of on p. A cut is the rule a feature threshold applies:
  • Both sides. Each cut is tried as at least (gte) and as at most (lte), over the values your outcomes have. threshold is ready to apply under a name.
  • Missing values. A document with no value for the feature is never selected, because a feature threshold reads it as false. missing counts them.
  • fit beside it: the recommendation on p for the same outcomes (the example shortens it), so you can compare a rule on one number with the probability that uses all of them.
  • Rules. feature cannot go with option, and must name one of the judgment’s features, or the call is invalid_request.

​
Apply a threshold

Nothing changes until you apply a recommendation. Thresholds are settings: change them with PATCH /namespaces/{ns}/judgments/{name}. The thresholds you send replace the whole set, so include the ones you want to keep:
For a choice, a threshold names its option: {"fraud": {"value": "fraud", "gte": 0.62}}. {} removes all thresholds. The change applies to every answer at its next read, existing ones included. Brussle judges nothing again and makes no version, and your audit log records the change. In the dashboard, the judgment’s Thresholds tab applies a recommendation with one click, after you confirm, and keeps your other thresholds.

​
On Developer

The Thresholds tab shows a preview for a yes-or-no judgment whose report has the same 100 outcomes, with 20 of each kind, from the same epoch: the threshold the recommender would pick, with its precision and recall. It calculates them from the report’s reliability bins, so it looks at thresholds in steps of 0.1, gives no interval, and its pick can be a little above the recommender’s.

​
Label a random sample

Outcomes from reviewed documents are biased: people review what crossed a threshold. The labelling queue gives you documents to label that nobody chose, spread across the full range of probabilities, and weights their labels back to your real traffic. The correction, its held-out test and the recommender’s precision then agree with your traffic as it is.

​
How a draw is made

Most answers sit in one or two parts of the range. A fraud judgment gives p near 0.02 to almost everything, so a plain random sample says nothing about how the engine does at 0.6. On a bool with thresholds, each draw has two halves:
  • Half is spread evenly over the 10 probability bands that have documents. This half is always there and cannot be turned off. It keeps every part of the range labelled, for the correction and for any threshold you set later.
  • Half goes to your thresholds: to the bands at or above each threshold, where the precision at the threshold is least certain. A band gets more when it has more answers, and when its answers are nearer an even yes or no. Nothing goes below the threshold’s band.
  • Each band’s share counts the documents already labelled from the queue or leased there, so a draw catches up where earlier draws took too few, and a second draw in the same week does not take the same documents.
  • On a bool with features, the second half follows the corrected p the thresholds read, while each document still names the band of its raw p, which its label is weighted by.
  • A judgment without thresholds, a choice and a score draw evenly.
  • On a template’s tenant, a draw follows the tenant’s thresholds when it overrides the template’s.
Choosing documents by their number does not bias the correction. The correction corrects the raw number, and the queue chooses only by that number, so more labels near a threshold make the correction more exact there without moving it. Every number that stands for your traffic, such as the held-out test and the recommender’s precision, weighs each label by how common its band is in your answers.

​
Label in the dashboard

  1. Open the judgment.
  2. Go to its Label tab. It shows one document at a time, with the judgment’s question.
  3. Click the answer: yes or no for a bool, an option for a choice, a level for a score. Or skip.
Each answer is posted as an outcome at once. A line above the document counts the two halves of the draw, such as “This draw: 28 near review (p ≥ 0.6), 22 across the range.” Nothing on a document says which half it came from, so it cannot sway the label. The tab shows how many documents in each band have labels, what the next draw would take, and the share of drawn documents that were labelled.

​
Draw from the queue with the API

POST /namespaces/{ns}/judgments/{name}/labelling-queue draws documents. count is 1 to 200, default 50:
The response lists the items, lowest band first:
  • band is one of 10 equal bands of the raw value. probability is the raw value: p for a bool, the probability of the most probable option or level for a choice or score, which value names.
  • The even half gives a band with 2% of the answers as many items as one with 80%. When it does not divide evenly, the rest goes to the bands with the fewest documents already labelled or leased, so repeated small draws reach every band.
  • bands has all 10 bands, lowest first (the example shows two): coverage counts the items from the even half, targeted those from the half at your thresholds. targets names the thresholds this draw followed, with the bands each selects.
  • document is what the labeller needs to see: its attributes, state and images, as references. evaluation_id leads to the exact context the engine read, at GET /namespaces/{ns}/evaluations/{id}.
  • seed (default 0) sets the random order within each band. The same documents, labels and seed give the same items.
Post each label as an ordinary outcome, with its item’s queue_item_id:

​
Leases and queue labels

  • A draw leases its items to you for 7 days, until lease_expires_at, so two people labelling at once do not get the same document. A label is taken while the lease holds.
  • After the lease ends, a label is refused with conflict, because the document is back in the queue for someone else. A queue_item_id from a draw more than 30 days old, or one naming a different document, is invalid_request. Draw new items.
  • A document labelled from the queue leaves it until its answer changes. One not labelled in time goes back.
  • An answer keeps one queue label, the latest recorded, so a retry or a corrected label counts once.
  • Posted outcomes and rule outcomes do not take a document out of the queue. Those are the reviewed documents; leaving them out would make the sample one of only the documents nobody reviewed.
  • Label what the document shows, and label every item. Labels only on the documents a person chose to answer are not a random sample: the easy ones get labelled and the hard ones do not. Skip only a document that cannot be judged; it goes back when its lease ends.

​
How queue labels count

  • Each counts as a queue label, in outcomes_by_source.queue in the report.
  • When an epoch’s queue labels alone meet the fit minimums (50, with 15 of each kind; for a choice or score, two values with 15 each), the correction uses only them, weighted by band, because nobody chose them. The report’s fitted_on becomes queue.
  • Until then, queue labels count as ordinary outcomes beside the others, and widen the range the correction covers.
  • The recommender uses them alone from 100 queue labels with 20 of each kind (see read the recommendation).

​
Preview the queue

GET on the same path previews the queue and gives out nothing. It is ns.judgments.labellingQueue(name) in TypeScript and labelling_queue(name) in Python. For each band it shows the answers, and how many are labelled, leased and available, and also:
  • next_draw: what a draw of the default 50 would take from each band now, and in total.
  • targets: the thresholds a draw follows.
  • drawn and label_rate, for each band and in total: the documents drawn in the last 30 days and, of those labelled or whose lease ended, the share labelled; null until one is. A document still leased and unlabelled is not counted yet.
  • warnings: queue_unlabelled_draws when label_rate is below 0.5.
The preview is on every plan, so on Developer the tab shows what the queue would contain. A judgment with a horizon has no queue (invalid_request): its outcome is what happens after the answer, which nobody can label from the document today. See outcomes that arrive later.