> ## Documentation Index > Fetch the complete documentation index at: https://docs.brussle.com/llms.txt > Use this file to discover all available pages before exploring further. # Pick thresholds > Get the threshold that meets a precision or recall target on your own outcomes, with its interval, and label a random sample from the queue so that number holds for your real traffic. You set a target (such as "send edits to legal review, and be right 90% of the time"), and the recommender finds the threshold that meets it on your outcomes, with a 95% interval and the whole precision and recall curve. When you label a random sample from the labelling queue, the recommendation stands for your real traffic instead of for the documents someone already reviewed, with about a third fewer labels. The recommender and drawing from the queue are on the Team plan and above, so below Team they return `plan_required` (see [plans](/pricing#plans)). Developer sees a [preview](#on-developer) of both. ## Get a recommendation ```text theme={"theme":{"light":"css-variables","dark":"css-variables"}} GET /namespaces/{ns}/judgments/material_change/thresholds/recommend?target=precision:0.9 ``` ```ts TypeScript theme={"theme":{"light":"css-variables","dark":"css-variables"}} const rec = await ns.judgments.recommendThreshold("material_change", { target: "precision:0.9" }); ``` ```python Python theme={"theme":{"light":"css-variables","dark":"css-variables"}} rec = ns.judgments.recommend_threshold("material_change", target="precision:0.9") ``` * **`target`** is `precision:` or `recall:`. * **A `bool`** gets a threshold on the `p` its answers read. Without features that is the engine's `p`, which is what thresholds read. With features it is the corrected `p` (for a cut on one feature, see [a threshold on one feature](#a-threshold-on-one-feature)). **A composite** gets one on the combined `p`. * **A `choice`** needs the option, such as `&option=fraud`, and without `option` it is `invalid_request`. The threshold is then on `dist[fraud]` as answers read it (the engine's, or the corrected one on a `choice` with features). * **A choice among candidates** (`options.from`, see [entity matching](/guides/entity-matching)) is the exception: its threshold is on `escape_p`, the unmatched queue. That is the engine's `escape_p`, or with features the corrected `escape_p` and pick that answers read. Leave `option` out or set it to `none_of_the_above`, because Brussle refuses any other. * **A `score`** gets no recommendation (`invalid_request`). ```json theme={"theme":{"light":"css-variables","dark":"css-variables"}} { "status": "recommended", "threshold": 0.79, "precision": 0.9, "recall": 0.407, "interval": {"lower": 0.849, "upper": 0.935}, "outcomes": 1432, "fitted_on": "all", "outcomes_by_source": {"posted": 1432, "rule": 0, "queue": 0, "implicit": 0}, "from_previous_epoch": false, "curve": [{"threshold": 0.0, "precision": 0.293, "recall": 1.0}] } ``` The example shows the first of the curve's 101 points. ## Read the recommendation The response is always 200, and `status` says which of three results it is. | `status` | Means | Carries | | - | - | - | | `recommended` | A threshold meets the target | `threshold`, `precision` and `recall` at it, `interval` (the 95% interval of the targeted metric), `outcomes`, `fitted_on`, `outcomes_by_source`, and the `curve` | | `unreachable` | No threshold meets the target | `outcomes`, `fitted_on`, `outcomes_by_source`, `from_previous_epoch` and the `curve`, so you can see what is reachable | | `insufficient_data` | Fewer than 100 outcomes, or fewer than 20 of a kind | `outcomes`, and `reason` and `message` saying what to post | * **A precision target** gets the lowest threshold that meets it, which keeps the most recall, and **a recall target** gets the highest, which keeps the most precision. * **On a choice among candidates it is the opposite,** because a document's pick is taken when its `escape_p` is below the threshold. A precision target gets the highest threshold that meets it, and a recall target the lowest. Precision is the share of taken picks that are right, and recall is the share of outcomes naming a match whose pick was taken and right. Apply the threshold as `{"unmatched": {"value": "none_of_the_above", "gte": threshold}}`. * **The curve** has 101 points, each with its precision and recall on every outcome, and the recommended threshold is one of them. Its thresholds run from 0.00 to 1.00 in steps of 0.01, and precision is `null` where no answer reaches the threshold. * **`precision`, `recall` and `interval` are read on outcomes the choice never saw.** The threshold is chosen on every outcome. Its precision and recall are then read five times over: the outcomes are split five ways, the threshold is chosen again on four parts and read on the fifth, and the five reads are pooled. Picking the best of 101 thresholds on the outcomes you then score it on reads high every time; this read does not. It is usually a little under the curve's point at the threshold, and it can be under the target. It is what to expect on new answers. * **A point counts what a threshold selects.** It compares each answer's number as the answer shows it, the way a named threshold and a query filter do, and includes answers equal to the threshold. So a threshold you set from a point selects exactly the answers that point counted. * **The outcomes** come from the current epoch, at least 100 of them, with 20 positive and 20 negative. On a `choice` that means 20 whose outcome is the option and 20 whose outcome is another. When the current epoch has too few, the recommender uses the previous epoch's, and `from_previous_epoch` is `true`. * **`warnings`** are the epoch's calibration warnings, as the [report](/calibration/report#warnings) carries them. A recommendation that rests on labels that leave the live answers uncovered (`coverage_truncated`), or on labels that read like the outcome (`reads_like_the_outcome`), says so here, before you apply it. * **`acts_on`**, **`covered_by`** and **`held`**: on a `bool`, what the threshold would act on against the labels. See [what the threshold would act on](#what-the-threshold-would-act-on). * **`fitted_on`** says what the recommendation rests on. It is `queue` when the epoch's [queue labels](#label-a-random-sample) alone number 100, with 20 of each kind. The recommender then uses only those labels, each weighted back to your traffic, so its precision and recall are your traffic's (and not those of a sample in which each band counts the same). Its `interval` then comes from how many labels each band has and how common the band is in your answers, so labels at your thresholds make it narrower. Otherwise `fitted_on` is `all`, and each outcome counts once. The **Thresholds** tab says "on 140 queue labels, weighted to your traffic" or "on 2,300 outcomes, unweighted". **`outcomes_by_source`** counts those outcomes by source. A threshold on a feature and a choice among candidates leave it out. ## What the threshold would act on The precision and recall come from the labels. The threshold acts on every live answer. When the labels cover only part of the range, as a reviewed sample does, the two can disagree badly: a fit on the labels gives an answer the labels never covered a corrected `p` above the threshold, and the precision claimed on the labels says nothing about it. On a `bool`, each recommendation says: * **`acts_on`**: the live answers the threshold would select now, as readers read them (the corrected `p` on a judgment with features). * **`covered_by`**: the labels whose `p` lies at or above the threshold, which is what the claimed precision rests on. * **`held`**: `labels_do_not_cover`, with a message, when more than 20% of the live answers it would select have a raw `p` outside the range the labels cover. The threshold is still given, with its curve and interval, and the **Thresholds** tab shows the hold and its reason before **Apply**. You can apply it; the claim is what is held. `null` when the labels cover what it acts on. ```json theme={"theme":{"light":"css-variables","dark":"css-variables"}} { "status": "recommended", "threshold": 0.62, "precision": 0.803, "interval": {"lower": 0.739, "upper": 0.855}, "acts_on": 235, "covered_by": 178, "held": { "reason": "labels_do_not_cover", "message": "The precision claimed here rests on the labels, and 24% of the 235 live answers the threshold selects have a raw p outside the range the labels cover: the labels do not cover what this threshold would act on. Label a random sample of documents, or use the labelling queue, before you rely on it. You can still apply it.", "covered_by": 178, "acts_on": 235, "uncovered": 0.24 }, "warnings": [{"code": "coverage_truncated", "message": "..."}] } ``` **What to do:** label a random sample from the [labelling queue](#label-a-random-sample). The labels then cover the range, the hold clears, and the claimed precision is the precision of what the threshold acts on. A choice, a composite and a choice among candidates have no `acts_on`. The recommender does not guess when your outcomes are all of one kind. ```json theme={"theme":{"light":"css-variables","dark":"css-variables"}} { "status": "insufficient_data", "outcomes": 142, "reason": "too_few_negatives", "message": "Only positive outcomes so far: post outcomes for documents where it did not happen." } ``` `reason` is `too_few_outcomes`, `too_few_positives` or `too_few_negatives`, chosen in the same order as for the report's [`not_fitted`](/calibration/report#why-an-epoch-has-no-correction) (a missing kind first, then the total, then 20 of a kind). ## A threshold on one feature On a `bool` with [features](/calibration/features), add `feature=` to get a cut on one feature instead of on `p`. A cut is the rule a feature threshold applies. ```text theme={"theme":{"light":"css-variables","dark":"css-variables"}} GET /namespaces/{ns}/judgments/goes_overdue/thresholds/recommend?target=precision:0.9&feature=paid_late.count ``` ```ts TypeScript theme={"theme":{"light":"css-variables","dark":"css-variables"}} const cut = await ns.judgments.recommendThreshold("goes_overdue", { target: "precision:0.9", feature: "paid_late.count" }); ``` ```python Python theme={"theme":{"light":"css-variables","dark":"css-variables"}} cut = ns.judgments.recommend_threshold("goes_overdue", target="precision:0.9", feature="paid_late.count") ``` ```json theme={"theme":{"light":"css-variables","dark":"css-variables"}} { "status": "recommended", "feature": "paid_late.count", "threshold": {"feature": "paid_late.count", "gte": 3}, "precision": 0.91, "recall": 0.84, "interval": {"lower": 0.86, "upper": 0.95}, "outcomes": 412, "missing": 6, "from_previous_epoch": false, "fit": {"status": "recommended", "threshold": 0.62, "precision": 0.92, "recall": 0.88}, "curve": [{"side": "gte", "cut": 3, "precision": 0.91, "recall": 0.84}] } ``` * **Both sides.** The recommender tries each cut as at least (`gte`) and as at most (`lte`), over the values your outcomes have, and returns `threshold` ready to apply under a name. * **Missing values.** A document with no value for the feature is never selected, because a feature threshold reads it as `false`. `missing` counts these documents. * **`fit` beside it** is the recommendation on `p` for the same outcomes (the example shortens it), so you can compare a rule on one number with the probability that uses all of them. * **Rules.** The call is `invalid_request` when `feature` goes with `option`, or when it does not name one of the judgment's features. ## Apply a threshold Nothing changes until you apply a recommendation. Thresholds are settings, so you change them with `PATCH /namespaces/{ns}/judgments/{name}`. The `thresholds` you send replace the whole set, so include the ones you want to keep. ```json theme={"theme":{"light":"css-variables","dark":"css-variables"}} {"thresholds": {"review": 0.79, "spot_check": 0.5}} ``` ```ts TypeScript theme={"theme":{"light":"css-variables","dark":"css-variables"}} await ns.judgments.update("material_change", { thresholds: { review: 0.79, spot_check: 0.5 } }); ``` ```python Python theme={"theme":{"light":"css-variables","dark":"css-variables"}} ns.judgments.update("material_change", thresholds={"review": 0.79, "spot_check": 0.5}) ``` A threshold on a `choice` names its option, as in `{"fraud": {"value": "fraud", "gte": 0.62}}`, and `{}` removes all thresholds. The change applies to every answer at its next read, existing ones included. Brussle judges nothing again and makes no version, and your audit log records the change. The judgment's **Thresholds** tab in the dashboard applies a recommendation with one click, after you confirm, and keeps your other thresholds. ### On Developer The **Thresholds** tab shows a preview of the threshold the recommender would pick, with its precision and recall, for a yes-or-no judgment whose report has the same 100 outcomes, with 20 of each kind, from the same epoch. The preview calculates them from the report's reliability bins, so it looks at thresholds in steps of 0.1, gives no interval, and its pick can be a little above the recommender's. ## Label a random sample Outcomes from reviewed documents are biased, since people review what crossed a threshold. The labelling queue gives you documents to label that nobody chose, spread across the full range of probabilities, and weights their labels back to your real traffic. The correction, its held-out test and the recommender's precision then agree with your traffic as it is. ### How a draw is made Most answers sit in one or two parts of the range. A fraud judgment gives `p` near 0.02 to almost everything, so a plain random sample says nothing about how the engine does at 0.6. On a `bool` with thresholds, each draw has two halves: * **Half is spread evenly** over the 10 probability bands that have documents. This half is always there and cannot be turned off, and it keeps every part of the range labelled, for the correction and for any threshold you set later. * **Half goes to your thresholds,** to the bands at or above each threshold, where the precision at the threshold is least certain. A band gets more when it has more answers and when its answers are nearer an even yes or no. Nothing goes below the threshold's band. * **Each band's share counts the documents already labelled from the queue or leased there,** so a draw catches up where earlier draws took too few, and a second draw in the same week does not take the same documents. * On a `bool` with features, the second half follows the corrected `p` the thresholds read, while each document still names the band of its raw `p`, which its label is weighted by. * A judgment without thresholds, a `choice` and a `score` draw evenly. * A draw on a template's tenant follows the tenant's thresholds when the tenant overrides the template's. Choosing documents by their number does not bias the correction, because the correction corrects the raw number and the queue chooses only by that number. More labels near a threshold make the correction more exact there without moving it. Every number that stands for your traffic (such as the held-out test and the recommender's precision) weighs each label by how common its band is in your answers. **What the even half costs on skewed traffic.** When most of your answers sit in one band, the even half gives that band as many labels as a band with 1% of the answers, and weighting each label back to your traffic makes a few labels carry most of the weight. The labels then count for fewer than there are: the report's `effective_sample_size` says how many. 150 labels can count for 60. The correction on the engine's answer alone needs few, and is applied early. A correction with [features](/calibration/features) needs more: it must beat the correction on the engine's answer alone on your held-out outcomes, and with the labels counting for 60 that margin is wider than the gain. The report's `features_margin` gives the gain, how sure it is, and an estimate of the labels still needed. Until then, answers read the correction on the engine's answer alone, which the queue's labels make exact. A judgment with features on skewed traffic needs more queue labels than it needs outcomes. ### Label in the dashboard 1. Open the judgment. 2. Go to its **Label** tab, which shows one document at a time, with the judgment's question. 3. Click the answer (yes or no for a `bool`, an option for a `choice`, a level for a `score`), or skip. The tab posts each answer as an outcome at once. A line above the document counts the two halves of the draw, such as "This draw: 28 near `review` (p ≥ 0.6), 22 across the range." Nothing on a document says which half it came from, so it cannot sway the label. The tab also shows how many documents in each band have labels, what the next draw would take, and the share of drawn documents that were labelled. ## Draw from the queue with the API `POST /namespaces/{ns}/judgments/{name}/labelling-queue` draws documents, and `count` is 1 to 200, default 50. ```json theme={"theme":{"light":"css-variables","dark":"css-variables"}} {"count": 50} ``` The response lists the items, lowest band first. ```json theme={"theme":{"light":"css-variables","dark":"css-variables"}} { "judgment": "material_change", "version": 3, "engine": "jev", "engine_version": "current+2026-10-02.2", "seed": 0, "lease_expires_at": "2026-10-09T14:03:11.204Z", "targets": [{"name": "review", "value": 0.6, "bands": [6, 7, 8, 9]}], "bands": [ {"index": 0, "lower": 0.0, "upper": 0.1, "coverage": 3, "targeted": 0}, {"index": 6, "lower": 0.6, "upper": 0.7, "coverage": 2, "targeted": 17} ], "items": [ { "queue_item_id": "lq_01JAYQ3K5V9W2X4Y6Z8A0B1C2D_0", "document_id": "sec_4812", "band": {"index": 0, "lower": 0.0, "upper": 0.1}, "probability": 0.04, "value": null, "evaluation_id": "ev_01j9a2k5v9w2x4y6z8a0b1c2d3", "document": {"id": "sec_4812", "revision": 88213, "attributes": {"contract": "c_77"}, "state": {"heading": "Payment terms", "text": "..."}, "images": {}} } ] } ``` * **`band`** is one of 10 equal bands of the raw value. **`probability`** is the raw value itself, `p` for a `bool` and for a `choice` or `score` the probability of the most probable option or level, which **`value`** names. * **The even half** gives a band with 2% of the answers as many items as one with 80%. When it does not divide evenly, the rest goes to the bands with the fewest documents already labelled or leased, so repeated small draws reach every band. * **`bands`** has all 10 bands, lowest first (the example shows two). In each, `coverage` counts the items from the even half and `targeted` those from the half at your thresholds. **`targets`** names the thresholds this draw followed, with the bands each selects. * **`document`** is what the labeller needs to see (its `attributes`, `state` and [`images`](/guides/images), as references), and `evaluation_id` leads to the exact context the engine read, at `GET /namespaces/{ns}/evaluations/{id}`. * **`seed`** (default 0) sets the random order within each band, so the same documents, labels and seed give the same items. Post each label as an ordinary outcome, with its item's `queue_item_id`. ```ts TypeScript theme={"theme":{"light":"css-variables","dark":"css-variables"}} const { items } = await ns.judgments.drawLabellingQueue("material_change", { count: 50 }); await ns.outcomes.append( items.map((item) => ({ document_id: item.document_id, judgment: "material_change", value: askTheLabeller(item.document), observed_at: new Date().toISOString(), queue_item_id: item.queue_item_id, })), ); ``` ```python Python theme={"theme":{"light":"css-variables","dark":"css-variables"}} draw = ns.judgments.draw_labelling_queue("material_change", count=50) ns.outcomes.append([ { "document_id": item["document_id"], "judgment": "material_change", "value": ask_the_labeller(item["document"]), "observed_at": datetime.now(timezone.utc).isoformat(), "queue_item_id": item["queue_item_id"], } for item in draw["items"] ]) ``` ## Leases and queue labels * **A draw leases its items to you for 7 days,** until `lease_expires_at`, so two people labelling at once do not get the same document. Brussle takes a label while the lease holds. * **After the lease ends,** Brussle refuses a label with `conflict`, because the document is back in the queue for someone else. A `queue_item_id` from a draw more than 30 days old, or one naming a different document, is `invalid_request`, so draw new items. * **A document labelled from the queue leaves it** until its answer changes, and one not labelled in time goes back. * **An answer keeps one queue label,** the latest recorded, so a retry or a corrected label counts once. * **Posted outcomes and rule outcomes do not take a document out of the queue.** Those are the reviewed documents, and leaving them out would make the sample one of only the documents nobody reviewed. * **Label what the document shows, and label every item.** Labels on only the documents a person chose to answer are not a random sample, since the easy ones get labelled and the hard ones do not. Skip only a document that cannot be judged, and it goes back when its lease ends. ## How queue labels count * **Each counts as a queue label,** in `outcomes_by_source.queue` in the [report](/calibration/report). * **When an epoch's queue labels alone meet the fit minimums,** the correction uses only them, weighted by band, because nobody chose them, and the report's `fitted_on` becomes `queue`. The minimums are 50 labels, with 15 of each kind. On a `choice` or `score` that means two values with 15 each. * **Until then,** queue labels count as ordinary outcomes beside the others, and widen the range the correction covers. * **The recommender** uses them alone from 100 queue labels with 20 of each kind (see [read the recommendation](#read-the-recommendation)). ## Preview the queue `GET` on the same path previews the queue and gives out nothing (`ns.judgments.labellingQueue(name)` in TypeScript, `labelling_queue(name)` in Python). For each band it shows the `answers` and how many are `labelled`, `leased` and `available`, and also: * **`next_draw`**: what a draw of the default 50 would take from each band now, and in total. * **`targets`**: the thresholds a draw follows. * **`drawn`** and **`label_rate`**, for each band and in total: the documents drawn in the last 30 days and, of those labelled or whose lease ended, the share labelled. `label_rate` is `null` until one is labelled or its lease has ended, and a document still leased and unlabelled is not counted yet. * **`warnings`**: `queue_unlabelled_draws` when `label_rate` is below 0.5. The preview is on every plan, so on Developer the tab shows what the queue would contain. A judgment with a horizon has no queue (`invalid_request`), because its outcome is what happens after the answer, which nobody can label from the document today. See [outcomes that arrive later](/calibration/horizons). This documentation is built and hosted on [Mintlify](https://mintlify.com), a developer documentation platform.