Skip to main content
You set a target (such as “send edits to legal review, and be right 90% of the time”), and the recommender finds the threshold that meets it on your outcomes, with a 95% interval and the whole precision and recall curve. When you label a random sample from the labelling queue, the recommendation stands for your real traffic instead of for the documents someone already reviewed, with about a third fewer labels. The recommender and drawing from the queue are on the Team plan and above, so below Team they return plan_required (see plans). Developer sees a preview of both.

​
Get a recommendation

  • target is precision:<x> or recall:<x>.
  • A bool gets a threshold on the p its answers read. Without features that is the engine’s p, which is what thresholds read. With features it is the corrected p (for a cut on one feature, see a threshold on one feature). A composite gets one on the combined p.
  • A choice needs the option, such as &option=fraud, and without option it is invalid_request. The threshold is then on dist[fraud] as answers read it (the engine’s, or the corrected one on a choice with features).
  • A choice among candidates (options.from, see entity matching) is the exception: its threshold is on escape_p, the unmatched queue. That is the engine’s escape_p, or with features the corrected escape_p and pick that answers read. Leave option out or set it to none_of_the_above, because Brussle refuses any other.
  • A score gets no recommendation (invalid_request).
The example shows the first of the curve’s 101 points.

​
Read the recommendation

The response is always 200, and status says which of three results it is.
  • A precision target gets the lowest threshold that meets it, which keeps the most recall, and a recall target gets the highest, which keeps the most precision.
  • On a choice among candidates it is the opposite, because a document’s pick is taken when its escape_p is below the threshold. A precision target gets the highest threshold that meets it, and a recall target the lowest. Precision is the share of taken picks that are right, and recall is the share of outcomes naming a match whose pick was taken and right. Apply the threshold as {"unmatched": {"value": "none_of_the_above", "gte": threshold}}.
  • The curve has 101 points, each with its precision and recall on every outcome, and the recommended threshold is one of them. Its thresholds run from 0.00 to 1.00 in steps of 0.01, and precision is null where no answer reaches the threshold.
  • precision, recall and interval are read on outcomes the choice never saw. The threshold is chosen on every outcome. Its precision and recall are then read five times over: the outcomes are split five ways, the threshold is chosen again on four parts and read on the fifth, and the five reads are pooled. Picking the best of 101 thresholds on the outcomes you then score it on reads high every time; this read does not. It is usually a little under the curve’s point at the threshold, and it can be under the target. It is what to expect on new answers.
  • A point counts what a threshold selects. It compares each answer’s number as the answer shows it, the way a named threshold and a query filter do, and includes answers equal to the threshold. So a threshold you set from a point selects exactly the answers that point counted.
  • The outcomes come from the current epoch, at least 100 of them, with 20 positive and 20 negative. On a choice that means 20 whose outcome is the option and 20 whose outcome is another. When the current epoch has too few, the recommender uses the previous epoch’s, and from_previous_epoch is true.
  • warnings are the epoch’s calibration warnings, as the report carries them. A recommendation that rests on labels that leave the live answers uncovered (coverage_truncated), or on labels that read like the outcome (reads_like_the_outcome), says so here, before you apply it.
  • acts_on, covered_by and held: on a bool, what the threshold would act on against the labels. See what the threshold would act on.
  • fitted_on says what the recommendation rests on. It is queue when the epoch’s queue labels alone number 100, with 20 of each kind. The recommender then uses only those labels, each weighted back to your traffic, so its precision and recall are your traffic’s (and not those of a sample in which each band counts the same). Its interval then comes from how many labels each band has and how common the band is in your answers, so labels at your thresholds make it narrower. Otherwise fitted_on is all, and each outcome counts once. The Thresholds tab says “on 140 queue labels, weighted to your traffic” or “on 2,300 outcomes, unweighted”. outcomes_by_source counts those outcomes by source. A threshold on a feature and a choice among candidates leave it out.

​
What the threshold would act on

The precision and recall come from the labels. The threshold acts on every live answer. When the labels cover only part of the range, as a reviewed sample does, the two can disagree badly: a fit on the labels gives an answer the labels never covered a corrected p above the threshold, and the precision claimed on the labels says nothing about it. On a bool, each recommendation says:
  • acts_on: the live answers the threshold would select now, as readers read them (the corrected p on a judgment with features).
  • covered_by: the labels whose p lies at or above the threshold, which is what the claimed precision rests on.
  • held: labels_do_not_cover, with a message, when more than 20% of the live answers it would select have a raw p outside the range the labels cover. The threshold is still given, with its curve and interval, and the Thresholds tab shows the hold and its reason before Apply. You can apply it; the claim is what is held. null when the labels cover what it acts on.
What to do: label a random sample from the labelling queue. The labels then cover the range, the hold clears, and the claimed precision is the precision of what the threshold acts on. A choice, a composite and a choice among candidates have no acts_on. The recommender does not guess when your outcomes are all of one kind.
reason is too_few_outcomes, too_few_positives or too_few_negatives, chosen in the same order as for the report’s not_fitted (a missing kind first, then the total, then 20 of a kind).

​
A threshold on one feature

On a bool with features, add feature= to get a cut on one feature instead of on p. A cut is the rule a feature threshold applies.
  • Both sides. The recommender tries each cut as at least (gte) and as at most (lte), over the values your outcomes have, and returns threshold ready to apply under a name.
  • Missing values. A document with no value for the feature is never selected, because a feature threshold reads it as false. missing counts these documents.
  • fit beside it is the recommendation on p for the same outcomes (the example shortens it), so you can compare a rule on one number with the probability that uses all of them.
  • Rules. The call is invalid_request when feature goes with option, or when it does not name one of the judgment’s features.

​
Apply a threshold

Nothing changes until you apply a recommendation. Thresholds are settings, so you change them with PATCH /namespaces/{ns}/judgments/{name}. The thresholds you send replace the whole set, so include the ones you want to keep.
A threshold on a choice names its option, as in {"fraud": {"value": "fraud", "gte": 0.62}}, and {} removes all thresholds. The change applies to every answer at its next read, existing ones included. Brussle judges nothing again and makes no version, and your audit log records the change. The judgment’s Thresholds tab in the dashboard applies a recommendation with one click, after you confirm, and keeps your other thresholds.

​
On Developer

The Thresholds tab shows a preview of the threshold the recommender would pick, with its precision and recall, for a yes-or-no judgment whose report has the same 100 outcomes, with 20 of each kind, from the same epoch. The preview calculates them from the report’s reliability bins, so it looks at thresholds in steps of 0.1, gives no interval, and its pick can be a little above the recommender’s.

​
Label a random sample

Outcomes from reviewed documents are biased, since people review what crossed a threshold. The labelling queue gives you documents to label that nobody chose, spread across the full range of probabilities, and weights their labels back to your real traffic. The correction, its held-out test and the recommender’s precision then agree with your traffic as it is.

​
How a draw is made

Most answers sit in one or two parts of the range. A fraud judgment gives p near 0.02 to almost everything, so a plain random sample says nothing about how the engine does at 0.6. On a bool with thresholds, each draw has two halves:
  • Half is spread evenly over the 10 probability bands that have documents. This half is always there and cannot be turned off, and it keeps every part of the range labelled, for the correction and for any threshold you set later.
  • Half goes to your thresholds, to the bands at or above each threshold, where the precision at the threshold is least certain. A band gets more when it has more answers and when its answers are nearer an even yes or no. Nothing goes below the threshold’s band.
  • Each band’s share counts the documents already labelled from the queue or leased there, so a draw catches up where earlier draws took too few, and a second draw in the same week does not take the same documents.
  • On a bool with features, the second half follows the corrected p the thresholds read, while each document still names the band of its raw p, which its label is weighted by.
  • A judgment without thresholds, a choice and a score draw evenly.
  • A draw on a template’s tenant follows the tenant’s thresholds when the tenant overrides the template’s.
Choosing documents by their number does not bias the correction, because the correction corrects the raw number and the queue chooses only by that number. More labels near a threshold make the correction more exact there without moving it. Every number that stands for your traffic (such as the held-out test and the recommender’s precision) weighs each label by how common its band is in your answers. What the even half costs on skewed traffic. When most of your answers sit in one band, the even half gives that band as many labels as a band with 1% of the answers, and weighting each label back to your traffic makes a few labels carry most of the weight. The labels then count for fewer than there are: the report’s effective_sample_size says how many. 150 labels can count for 60. The correction on the engine’s answer alone needs few, and is applied early. A correction with features needs more: it must beat the correction on the engine’s answer alone on your held-out outcomes, and with the labels counting for 60 that margin is wider than the gain. The report’s features_margin gives the gain, how sure it is, and an estimate of the labels still needed. Until then, answers read the correction on the engine’s answer alone, which the queue’s labels make exact. A judgment with features on skewed traffic needs more queue labels than it needs outcomes.

​
Label in the dashboard

  1. Open the judgment.
  2. Go to its Label tab, which shows one document at a time, with the judgment’s question.
  3. Click the answer (yes or no for a bool, an option for a choice, a level for a score), or skip.
The tab posts each answer as an outcome at once. A line above the document counts the two halves of the draw, such as “This draw: 28 near review (p ≥ 0.6), 22 across the range.” Nothing on a document says which half it came from, so it cannot sway the label. The tab also shows how many documents in each band have labels, what the next draw would take, and the share of drawn documents that were labelled.

​
Draw from the queue with the API

POST /namespaces/{ns}/judgments/{name}/labelling-queue draws documents, and count is 1 to 200, default 50.
The response lists the items, lowest band first.
  • band is one of 10 equal bands of the raw value. probability is the raw value itself, p for a bool and for a choice or score the probability of the most probable option or level, which value names.
  • The even half gives a band with 2% of the answers as many items as one with 80%. When it does not divide evenly, the rest goes to the bands with the fewest documents already labelled or leased, so repeated small draws reach every band.
  • bands has all 10 bands, lowest first (the example shows two). In each, coverage counts the items from the even half and targeted those from the half at your thresholds. targets names the thresholds this draw followed, with the bands each selects.
  • document is what the labeller needs to see (its attributes, state and images, as references), and evaluation_id leads to the exact context the engine read, at GET /namespaces/{ns}/evaluations/{id}.
  • seed (default 0) sets the random order within each band, so the same documents, labels and seed give the same items.
Post each label as an ordinary outcome, with its item’s queue_item_id.

​
Leases and queue labels

  • A draw leases its items to you for 7 days, until lease_expires_at, so two people labelling at once do not get the same document. Brussle takes a label while the lease holds.
  • After the lease ends, Brussle refuses a label with conflict, because the document is back in the queue for someone else. A queue_item_id from a draw more than 30 days old, or one naming a different document, is invalid_request, so draw new items.
  • A document labelled from the queue leaves it until its answer changes, and one not labelled in time goes back.
  • An answer keeps one queue label, the latest recorded, so a retry or a corrected label counts once.
  • Posted outcomes and rule outcomes do not take a document out of the queue. Those are the reviewed documents, and leaving them out would make the sample one of only the documents nobody reviewed.
  • Label what the document shows, and label every item. Labels on only the documents a person chose to answer are not a random sample, since the easy ones get labelled and the hard ones do not. Skip only a document that cannot be judged, and it goes back when its lease ends.

​
How queue labels count

  • Each counts as a queue label, in outcomes_by_source.queue in the report.
  • When an epoch’s queue labels alone meet the fit minimums, the correction uses only them, weighted by band, because nobody chose them, and the report’s fitted_on becomes queue. The minimums are 50 labels, with 15 of each kind. On a choice or score that means two values with 15 each.
  • Until then, queue labels count as ordinary outcomes beside the others, and widen the range the correction covers.
  • The recommender uses them alone from 100 queue labels with 20 of each kind (see read the recommendation).

​
Preview the queue

GET on the same path previews the queue and gives out nothing (ns.judgments.labellingQueue(name) in TypeScript, labelling_queue(name) in Python). For each band it shows the answers and how many are labelled, leased and available, and also:
  • next_draw: what a draw of the default 50 would take from each band now, and in total.
  • targets: the thresholds a draw follows.
  • drawn and label_rate, for each band and in total: the documents drawn in the last 30 days and, of those labelled or whose lease ended, the share labelled. label_rate is null until one is labelled or its lease has ended, and a document still leased and unlabelled is not counted yet.
  • warnings: queue_unlabelled_draws when label_rate is below 0.5.
The preview is on every plan, so on Developer the tab shows what the queue would contain. A judgment with a horizon has no queue (invalid_request), because its outcome is what happens after the answer, which nobody can label from the document today. See outcomes that arrive later.