> ## Documentation Index > Fetch the complete documentation index at: https://docs.brussle.com/llms.txt > Use this file to discover all available pages before exploring further. # Pick thresholds > Get the threshold that meets a precision or recall target on your own outcomes, with its interval, and label a random sample from the queue so that number holds for your real traffic. Set a target, such as "send edits to legal review, and be right 90% of the time", and the recommender finds the threshold that meets it on your outcomes, with a 95% interval and the whole precision and recall curve. Label a random sample from the labelling queue, and the recommendation stands for your real traffic rather than for the documents someone already reviewed, with about a third fewer labels. The recommender and drawing from the queue are on the Team plan and above. Below Team they return `plan_required` (see [plans](/pricing#plans)); Developer sees a [preview](#on-developer) of both. ## Get a recommendation ```text theme={"theme":{"light":"css-variables","dark":"css-variables"}} GET /namespaces/{ns}/judgments/material_change/thresholds/recommend?target=precision:0.9 ``` ```ts TypeScript theme={"theme":{"light":"css-variables","dark":"css-variables"}} const rec = await ns.judgments.recommendThreshold("material_change", { target: "precision:0.9" }); ``` ```python Python theme={"theme":{"light":"css-variables","dark":"css-variables"}} rec = ns.judgments.recommend_threshold("material_change", target="precision:0.9") ``` * **`target`** is `precision:` or `recall:`. * **A `bool` without features:** the threshold is on the engine's `p`, which is what thresholds read. **A `bool` with features:** on the corrected `p` its answers read (see [a threshold on one feature](#a-threshold-on-one-feature) for a cut on one feature). **A composite:** on the combined `p`. * **A `choice`:** add the option, such as `&option=fraud`. The threshold is then on the raw `dist[fraud]`. A `choice` without `option` is `invalid_request`. * **A choice among candidates** (`options.from`, see [entity matching](/guides/entity-matching)) is the exception: its threshold is on the raw `escape_p`, the unmatched queue. `option` is `none_of_the_above` or left out; any other is refused. * **A `score`** gets no recommendation (`invalid_request`). ```json theme={"theme":{"light":"css-variables","dark":"css-variables"}} { "status": "recommended", "threshold": 0.79, "precision": 0.9, "recall": 0.407, "interval": {"lower": 0.849, "upper": 0.935}, "outcomes": 1432, "fitted_on": "all", "outcomes_by_source": {"posted": 1432, "rule": 0, "queue": 0, "implicit": 0}, "from_previous_epoch": false, "curve": [{"threshold": 0.0, "precision": 0.293, "recall": 1.0}] } ``` The example shows the first of the curve's 101 points. ## Read the recommendation The response is always 200. `status` says which of three results it is: | `status` | Means | Carries | | - | - | - | | `recommended` | A threshold meets the target | `threshold`, `precision` and `recall` at it, `interval` (the 95% interval of the targeted metric), `outcomes`, `fitted_on`, `outcomes_by_source`, and the `curve` | | `unreachable` | No threshold meets the target | `outcomes`, `fitted_on`, `outcomes_by_source`, `from_previous_epoch` and the `curve`, so you can see what is reachable | | `insufficient_data` | Fewer than 100 outcomes, or fewer than 20 of a kind | `outcomes`, and `reason` and `message` saying what to post | * **A precision target** gets the lowest threshold that meets it, which keeps the most recall. **A recall target** gets the highest, which keeps the most precision. * **On a choice among candidates it is the opposite.** A document's pick is taken when its `escape_p` is *below* the threshold. Precision is the share of taken picks that are right; recall is the share of outcomes naming a match whose pick was taken and right. A precision target gets the highest threshold that meets it, a recall target the lowest. Apply it as `{"unmatched": {"value": "none_of_the_above", "gte": threshold}}`. * **The curve** has 101 points, thresholds 0.00 to 1.00 in steps of 0.01, each with its precision and recall. The recommended threshold is one of them. Precision is `null` where no answer reaches the threshold. * **A point counts what a threshold selects.** It compares each answer's number as the answer shows it, as a named threshold and a query filter do, answers equal to the threshold included. A threshold you set from a point selects exactly the answers it counted. * **The outcomes** are from the current epoch: at least 100, with 20 positive and 20 negative. For a `choice`, 20 whose outcome is the option and 20 whose outcome is another. When the current epoch has too few, the previous epoch's are used, and `from_previous_epoch` is `true`. * **`fitted_on`** says what the recommendation rests on. It is `queue` when the epoch's [queue labels](#label-a-random-sample) alone number 100, with 20 of each kind: it then uses only them, each weighted back to your traffic, so its precision and recall are your traffic's, not a sample's in which each band counts the same. Its `interval` then comes from how many labels each band has and how common the band is in your answers, so labels at your thresholds make it narrower. Otherwise `all`: each outcome counts once. The **Thresholds** tab says "on 140 queue labels, weighted to your traffic" or "on 2,300 outcomes, unweighted". **`outcomes_by_source`** counts those outcomes by source; a threshold on a feature and a choice among candidates leave it out. With outcomes of only one kind, the recommender does not guess: ```json theme={"theme":{"light":"css-variables","dark":"css-variables"}} { "status": "insufficient_data", "outcomes": 142, "reason": "too_few_negatives", "message": "Only positive outcomes so far: post outcomes for documents where it did not happen." } ``` `reason` is `too_few_outcomes`, `too_few_positives` or `too_few_negatives`, chosen as for the report's [`not_fitted`](/train/report#why-an-epoch-has-no-correction): a missing kind first, then the total, then 20 of a kind. ## A threshold on one feature On a `bool` with [features](/train/features), add `feature=` to get a cut on one feature instead of on `p`. A cut is the rule a feature threshold applies: ```text theme={"theme":{"light":"css-variables","dark":"css-variables"}} GET /namespaces/{ns}/judgments/goes_overdue/thresholds/recommend?target=precision:0.9&feature=paid_late.count ``` ```ts TypeScript theme={"theme":{"light":"css-variables","dark":"css-variables"}} const cut = await ns.judgments.recommendThreshold("goes_overdue", { target: "precision:0.9", feature: "paid_late.count" }); ``` ```python Python theme={"theme":{"light":"css-variables","dark":"css-variables"}} cut = ns.judgments.recommend_threshold("goes_overdue", target="precision:0.9", feature="paid_late.count") ``` ```json theme={"theme":{"light":"css-variables","dark":"css-variables"}} { "status": "recommended", "feature": "paid_late.count", "threshold": {"feature": "paid_late.count", "gte": 3}, "precision": 0.91, "recall": 0.84, "interval": {"lower": 0.86, "upper": 0.95}, "outcomes": 412, "missing": 6, "from_previous_epoch": false, "fit": {"status": "recommended", "threshold": 0.62, "precision": 0.92, "recall": 0.88}, "curve": [{"side": "gte", "cut": 3, "precision": 0.91, "recall": 0.84}] } ``` * **Both sides.** Each cut is tried as at least (`gte`) and as at most (`lte`), over the values your outcomes have. `threshold` is ready to apply under a name. * **Missing values.** A document with no value for the feature is never selected, because a feature threshold reads it as `false`. `missing` counts them. * **`fit` beside it:** the recommendation on `p` for the same outcomes (the example shortens it), so you can compare a rule on one number with the probability that uses all of them. * **Rules.** `feature` cannot go with `option`, and must name one of the judgment's features, or the call is `invalid_request`. ## Apply a threshold Nothing changes until you apply a recommendation. Thresholds are settings: change them with `PATCH /namespaces/{ns}/judgments/{name}`. The `thresholds` you send **replace the whole set**, so include the ones you want to keep: ```json theme={"theme":{"light":"css-variables","dark":"css-variables"}} {"thresholds": {"review": 0.79, "spot_check": 0.5}} ``` ```ts TypeScript theme={"theme":{"light":"css-variables","dark":"css-variables"}} await ns.judgments.update("material_change", { thresholds: { review: 0.79, spot_check: 0.5 } }); ``` ```python Python theme={"theme":{"light":"css-variables","dark":"css-variables"}} ns.judgments.update("material_change", thresholds={"review": 0.79, "spot_check": 0.5}) ``` For a `choice`, a threshold names its option: `{"fraud": {"value": "fraud", "gte": 0.62}}`. `{}` removes all thresholds. The change applies to every answer at its next read, existing ones included. Brussle judges nothing again and makes no version, and your audit log records the change. In the dashboard, the judgment's **Thresholds** tab applies a recommendation with one click, after you confirm, and keeps your other thresholds. ### On Developer The **Thresholds** tab shows a preview for a yes-or-no judgment whose report has the same 100 outcomes, with 20 of each kind, from the same epoch: the threshold the recommender would pick, with its precision and recall. It calculates them from the report's reliability bins, so it looks at thresholds in steps of 0.1, gives no interval, and its pick can be a little above the recommender's. ## Label a random sample Outcomes from reviewed documents are biased: people review what crossed a threshold. The labelling queue gives you documents to label that nobody chose, spread across the full range of probabilities, and weights their labels back to your real traffic. The correction, its held-out test and the recommender's precision then agree with your traffic as it is. ### How a draw is made Most answers sit in one or two parts of the range. A fraud judgment gives `p` near 0.02 to almost everything, so a plain random sample says nothing about how the engine does at 0.6. On a `bool` with thresholds, each draw has two halves: * **Half is spread evenly** over the 10 probability bands that have documents. This half is always there and cannot be turned off. It keeps every part of the range labelled, for the correction and for any threshold you set later. * **Half goes to your thresholds:** to the bands at or above each threshold, where the precision at the threshold is least certain. A band gets more when it has more answers, and when its answers are nearer an even yes or no. Nothing goes below the threshold's band. * **Each band's share counts the documents already labelled from the queue or leased there,** so a draw catches up where earlier draws took too few, and a second draw in the same week does not take the same documents. * On a `bool` with features, the second half follows the corrected `p` the thresholds read, while each document still names the band of its raw `p`, which its label is weighted by. * A judgment without thresholds, a `choice` and a `score` draw evenly. * On a template's tenant, a draw follows the tenant's thresholds when it overrides the template's. **Choosing documents by their number does not bias the correction.** The correction corrects the raw number, and the queue chooses only by that number, so more labels near a threshold make the correction more exact there without moving it. Every number that stands for your traffic, such as the held-out test and the recommender's precision, weighs each label by how common its band is in your answers. ### Label in the dashboard 1. Open the judgment. 2. Go to its **Label** tab. It shows one document at a time, with the judgment's question. 3. Click the answer: yes or no for a `bool`, an option for a `choice`, a level for a `score`. Or skip. Each answer is posted as an outcome at once. A line above the document counts the two halves of the draw, such as "This draw: 28 near `review` (p ≥ 0.6), 22 across the range." Nothing on a document says which half it came from, so it cannot sway the label. The tab shows how many documents in each band have labels, what the next draw would take, and the share of drawn documents that were labelled. ## Draw from the queue with the API `POST /namespaces/{ns}/judgments/{name}/labelling-queue` draws documents. `count` is 1 to 200, default 50: ```json theme={"theme":{"light":"css-variables","dark":"css-variables"}} {"count": 50} ``` The response lists the items, lowest band first: ```json theme={"theme":{"light":"css-variables","dark":"css-variables"}} { "judgment": "material_change", "version": 3, "engine": "jev", "engine_version": "current+2026-10-02.2", "seed": 0, "lease_expires_at": "2026-10-09T14:03:11.204Z", "targets": [{"name": "review", "value": 0.6, "bands": [6, 7, 8, 9]}], "bands": [ {"index": 0, "lower": 0.0, "upper": 0.1, "coverage": 3, "targeted": 0}, {"index": 6, "lower": 0.6, "upper": 0.7, "coverage": 2, "targeted": 17} ], "items": [ { "queue_item_id": "lq_01JAYQ3K5V9W2X4Y6Z8A0B1C2D_0", "document_id": "sec_4812", "band": {"index": 0, "lower": 0.0, "upper": 0.1}, "probability": 0.04, "value": null, "evaluation_id": "ev_01j9a2k5v9w2x4y6z8a0b1c2d3", "document": {"id": "sec_4812", "revision": 88213, "attributes": {"contract": "c_77"}, "state": {"heading": "Payment terms", "text": "..."}, "images": {}} } ] } ``` * **`band`** is one of 10 equal bands of the raw value. **`probability`** is the raw value: `p` for a `bool`, the probability of the most probable option or level for a `choice` or `score`, which **`value`** names. * **The even half** gives a band with 2% of the answers as many items as one with 80%. When it does not divide evenly, the rest goes to the bands with the fewest documents already labelled or leased, so repeated small draws reach every band. * **`bands`** has all 10 bands, lowest first (the example shows two): `coverage` counts the items from the even half, `targeted` those from the half at your thresholds. **`targets`** names the thresholds this draw followed, with the bands each selects. * **`document`** is what the labeller needs to see: its `attributes`, `state` and [`images`](/guides/images), as references. `evaluation_id` leads to the exact context the engine read, at `GET /namespaces/{ns}/evaluations/{id}`. * **`seed`** (default 0) sets the random order within each band. The same documents, labels and seed give the same items. Post each label as an ordinary outcome, with its item's `queue_item_id`: ```ts TypeScript theme={"theme":{"light":"css-variables","dark":"css-variables"}} const { items } = await ns.judgments.drawLabellingQueue("material_change", { count: 50 }); await ns.outcomes.append( items.map((item) => ({ document_id: item.document_id, judgment: "material_change", value: askTheLabeller(item.document), observed_at: new Date().toISOString(), queue_item_id: item.queue_item_id, })), ); ``` ```python Python theme={"theme":{"light":"css-variables","dark":"css-variables"}} draw = ns.judgments.draw_labelling_queue("material_change", count=50) ns.outcomes.append([ { "document_id": item["document_id"], "judgment": "material_change", "value": ask_the_labeller(item["document"]), "observed_at": datetime.now(timezone.utc).isoformat(), "queue_item_id": item["queue_item_id"], } for item in draw["items"] ]) ``` ## Leases and queue labels * **A draw leases its items to you for 7 days,** until `lease_expires_at`, so two people labelling at once do not get the same document. A label is taken while the lease holds. * **After the lease ends,** a label is refused with `conflict`, because the document is back in the queue for someone else. A `queue_item_id` from a draw more than 30 days old, or one naming a different document, is `invalid_request`. Draw new items. * **A document labelled from the queue leaves it** until its answer changes. One not labelled in time goes back. * **An answer keeps one queue label,** the latest recorded, so a retry or a corrected label counts once. * **Posted outcomes and rule outcomes do not take a document out of the queue.** Those are the reviewed documents; leaving them out would make the sample one of only the documents nobody reviewed. * **Label what the document shows, and label every item.** Labels only on the documents a person chose to answer are not a random sample: the easy ones get labelled and the hard ones do not. Skip only a document that cannot be judged; it goes back when its lease ends. ## How queue labels count * **Each counts as a queue label,** in `outcomes_by_source.queue` in the [report](/train/report). * **When an epoch's queue labels alone meet the fit minimums** (50, with 15 of each kind; for a `choice` or `score`, two values with 15 each), the correction uses only them, weighted by band, because nobody chose them. The report's `fitted_on` becomes `queue`. * **Until then,** queue labels count as ordinary outcomes beside the others, and widen the range the correction covers. * **The recommender** uses them alone from 100 queue labels with 20 of each kind (see [read the recommendation](#read-the-recommendation)). ## Preview the queue `GET` on the same path previews the queue and gives out nothing. It is `ns.judgments.labellingQueue(name)` in TypeScript and `labelling_queue(name)` in Python. For each band it shows the `answers`, and how many are `labelled`, `leased` and `available`, and also: * **`next_draw`**: what a draw of the default 50 would take from each band now, and in total. * **`targets`**: the thresholds a draw follows. * **`drawn`** and **`label_rate`**, for each band and in total: the documents drawn in the last 30 days and, of those labelled or whose lease ended, the share labelled; `null` until one is. A document still leased and unlabelled is not counted yet. * **`warnings`**: `queue_unlabelled_draws` when `label_rate` is below 0.5. The preview is on every plan, so on Developer the tab shows what the queue would contain. **A judgment with a horizon has no queue** (`invalid_request`): its outcome is what happens after the answer, which nobody can label from the document today. See [outcomes that arrive later](/train/horizons). This documentation is built and hosted on [Mintlify](https://mintlify.com), a developer documentation platform.