> ## Documentation Index > Fetch the complete documentation index at: https://docs.brussle.com/llms.txt > Use this file to discover all available pages before exploring further. # Recommend a threshold for a precision or recall target > The threshold on the raw `p` (bool) or raw `dist[option]` (choice) that meets the target on the active version's outcomes. A document counts as predicted positive when the raw value is at least the threshold, and as positive when its outcome is `true` (bool) or equals `option` (choice). - For a precision target, the lowest threshold whose precision meets it, which keeps the most recall. - For a recall target, the highest threshold whose recall meets it, which keeps the most precision. For a `choice` with `options.from` the threshold is on raw `escape_p`, the unmatched queue, and `option` is `none_of_the_above` or left out. A document whose `escape_p` is below the threshold has its pick taken; at or above it, it waits in the queue. Precision is the share of taken picks that equal the outcome, and recall the share of outcomes naming a match whose pick was taken and right, so a precision target takes the highest threshold that meets it, and a recall target the lowest. Apply it as `{"unmatched": {"value": "none_of_the_above", "gte": threshold}}`. The recommendation rests on the outcomes of the current engine epoch: at least 100, with at least 20 positive and 20 negative (for a choice, 20 of `option` and 20 of the other options). When that epoch has too few, it uses the previous epoch's and says so with `from_previous_epoch`, as calibration does. When neither has enough, `insufficient_data` says why and what to post. Nothing changes until the threshold is applied with `PATCH`. Score judgments have no recommendation (`invalid_request`). The recommender is part of the learning loop, on the Team plan and above: below it, `plan_required`. The calibration report and calibrated answers are on every plan. ## OpenAPI ````yaml /api-reference/openapi.yaml get /namespaces/{ns}/judgments/{name}/thresholds/recommend openapi: 3.1.0 info: title: Brussle API version: '1' license: name: Proprietary identifier: LicenseRef-Proprietary description: | The `/v1` contract. It is locked: changes within `/v1` are additive only, and anything breaking is `/v2`. - **Base URL.** `https://api.brussle.com/v1`. - **Evolution.** Response objects may gain fields and enums may gain values. Clients must ignore what they do not know. Request objects reject unknown fields. - **Path segments.** Namespace names (`{ns}`) and document ids (`{id}`) may contain `/`. Send it percent-encoded as `%2F`, so that `acme/prod/tenant_123` is `acme%2Fprod%2Ftenant_123`. - **Templates.** The judgment routes also take a namespace prefix ending in `/*`, such as `acme%2Fprod%2F*`. A judgment created there is a template: every namespace under the prefix inherits it, including ones created later. The most specific matching prefix wins. - **Thresholds are settings.** Named thresholds belong to the judgment, not to a version. They are evaluated at read time against the raw fields, so a change applies at once to every answer, with no recompute and no new version. - **Entity judgments.** A judgment can read the documents that point at the judged one, through `context.related`, and apply only to documents matching `applies_to`. Its answers carry a `watermark`, and creating one that runs `on_change` returns a replay estimate of its monthly cost until you confirm. - **Referenced documents.** A relation can also read the one document the judged document points at, through `join: {theirs: "id", mine: "attributes."}`. A change to what the relation renders of that document re-judges the judged documents that point at it and are inside `freshness.fanout.scope`: a fan-out. - **Relations.** A relation can read plain judgments' answers (banded or cut, never summed), or be a blocking relation over the documents that share a key, which a `choice` can choose among with `options.from`. A recipe can read the document as the judgment last saw it (`context.previous`). A judgment that can fan out confirms its rolling limits once, at create; past them, work is deferred, reads `stale` with an event, and catches up on its own. Nothing waits for a confirm at runtime. Groups keep aggregates per key value for reporting. - **Idempotency.** Every write operation is idempotent by construction: a retried append adds nothing while its values are still in the array, and a retried upsert or patch sets the same content again. Every request that changes something also takes an `Idempotency-Key`, so that creates, activations, backfills, draws and the rest can be retried safely too: a retry with the key gets the first response back, verbatim and marked `Idempotent-Replayed: true`, instead of running again (see the parameter). - **Errors.** Every error is the `Error` envelope; the HTTP status follows the code (see each response). - **Rate limits.** Every response to a request whose key was accepted carries `X-RateLimit-Limit` and `X-RateLimit-Remaining`, which count the key's requests a second; a request refused before its key is checked (`unauthorized`, or a route that does not exist) carries neither. A `rate_limited` response adds `Retry-After`, except a daily allowance that is used up, which says when it renews in `details.resets_at` instead. Every `503` carries `Retry-After` too. Reads of one namespace also have a concurrency limit: past 32 gets and queries in flight at once, one more is `rate_limited` with `Retry-After` and `details.concurrent_reads`, whatever the rate-limit headers show. - **Usage.** Every response that bills carries `usage`. Judging is billed per judgment, engine-neutral, by size class: one judgment answered counts by the tokens of its compiled context and its question together, 1 if standard (up to 2,000 tokens), 4 if large (up to 8,000) and 16 if extra-large (up to 32,000, the engine maximum). No response ever carries an engine's price. servers: - url: https://api.brussle.com/v1 security: - apiKey: [] tags: - name: Namespaces description: Namespaces, their settings and cache warming. - name: Documents description: Writes, point reads and queries. - name: Judgments description: >- Judgment definitions, versions, settings, activation, backfill and template detach. - name: Outcomes and calibration description: | Post what actually happened to judged documents, and read how well answers match it: the calibration report, calibrated answers and threshold recommendations. See [calibration](/concepts/calibration). - name: Groups description: > Groups keep aggregates per value of a key attribute, a group-by you declare for reporting; a template's tenant summary publishes each tenant's groups, banded, as a document the platform can judge. - name: Jobs description: >- Backfill, shadow, periodic, deletion, reference index, evaluation export, resync, group build, discovery and simulation jobs. - name: Engines description: The engine registry. - name: Organization description: | Settings of the whole organization: prefixes marked non-production, which the tenant fee and template calibration pools leave out. - name: Subscriptions description: | Saved queries that send an event when a document starts or stops matching: the query's filter grammar, on a namespace or a template prefix, with the fields each event carries. - name: Webhooks description: | Webhook endpoints, the events feed, the delivery log, redelivery and recovery. Every event type and its body is under `webhooks`. Events are delivered at least once, signed as Standard Webhooks, and kept for 30 days. Webhooks are on every plan, and deliveries are not billed; plans set how many endpoints an organization has. paths: /namespaces/{ns}/judgments/{name}/thresholds/recommend: parameters: - $ref: '#/components/parameters/NamespaceOrTemplate' - $ref: '#/components/parameters/JudgmentName' get: tags: - Outcomes and calibration summary: Recommend a threshold for a precision or recall target description: > The threshold on the raw `p` (bool) or raw `dist[option]` (choice) that meets the target on the active version's outcomes. A document counts as predicted positive when the raw value is at least the threshold, and as positive when its outcome is `true` (bool) or equals `option` (choice). - For a precision target, the lowest threshold whose precision meets it, which keeps the most recall. - For a recall target, the highest threshold whose recall meets it, which keeps the most precision. For a `choice` with `options.from` the threshold is on raw `escape_p`, the unmatched queue, and `option` is `none_of_the_above` or left out. A document whose `escape_p` is below the threshold has its pick taken; at or above it, it waits in the queue. Precision is the share of taken picks that equal the outcome, and recall the share of outcomes naming a match whose pick was taken and right, so a precision target takes the highest threshold that meets it, and a recall target the lowest. Apply it as `{"unmatched": {"value": "none_of_the_above", "gte": threshold}}`. The recommendation rests on the outcomes of the current engine epoch: at least 100, with at least 20 positive and 20 negative (for a choice, 20 of `option` and 20 of the other options). When that epoch has too few, it uses the previous epoch's and says so with `from_previous_epoch`, as calibration does. When neither has enough, `insufficient_data` says why and what to post. Nothing changes until the threshold is applied with `PATCH`. Score judgments have no recommendation (`invalid_request`). The recommender is part of the learning loop, on the Team plan and above: below it, `plan_required`. The calibration report and calibrated answers are on every plan. operationId: recommendThreshold parameters: - name: target in: query required: true description: >- The metric and the value to meet, such as `precision:0.9` or `recall:0.8`. schema: $ref: '#/components/schemas/ThresholdTarget' example: precision:0.9 - name: option in: query description: >- Required for a choice judgment, and refused for a bool one. The option the threshold is for, such as `fraud`. schema: type: string minLength: 1 responses: '200': description: >- A recommendation, an unreachable target, or too few outcomes; `status` says which. headers: X-RateLimit-Limit: $ref: '#/components/headers/RateLimitLimit' X-RateLimit-Remaining: $ref: '#/components/headers/RateLimitRemaining' content: application/json: schema: $ref: '#/components/schemas/ThresholdRecommendation' '400': $ref: '#/components/responses/InvalidRequest' '401': $ref: '#/components/responses/Unauthorized' '402': $ref: '#/components/responses/PlanRequired' '403': $ref: '#/components/responses/Forbidden' '404': $ref: '#/components/responses/NotFound' '409': $ref: '#/components/responses/Conflict' '413': $ref: '#/components/responses/TooLarge' '429': $ref: '#/components/responses/RateLimited' '500': $ref: '#/components/responses/Internal' components: parameters: NamespaceOrTemplate: name: ns in: path required: true description: | A namespace name, or a template prefix ending in `/*`, with any `/` sent as `%2F`: `acme%2Fprod%2Ftenant_123` or `acme%2Fprod%2F*`. schema: $ref: '#/components/schemas/NamespaceOrTemplate' JudgmentName: name: name in: path required: true schema: $ref: '#/components/schemas/Name' schemas: ThresholdTarget: type: string description: A metric and the value to meet, such as `precision:0.9` or `recall:0.8`. pattern: ^(precision|recall):(0?\.[0-9]*[1-9][0-9]*|1|1\.0+)$ ThresholdRecommendation: type: object description: | Discriminated on `status`: `recommended`, `unreachable` when no threshold meets the target on these outcomes, or `insufficient_data` below 100 outcomes or 20 of either class. oneOf: - $ref: '#/components/schemas/ThresholdRecommended' - $ref: '#/components/schemas/ThresholdUnreachable' - $ref: '#/components/schemas/ThresholdInsufficientData' discriminator: propertyName: status mapping: recommended: $ref: '#/components/schemas/ThresholdRecommended' unreachable: $ref: '#/components/schemas/ThresholdUnreachable' insufficient_data: $ref: '#/components/schemas/ThresholdInsufficientData' NamespaceOrTemplate: type: string description: | A namespace name, or a template prefix: a namespace path ending in `/*`, such as `acme/prod/*`, which every namespace under `acme/prod/` inherits judgments from. Up to 256 bytes. Never exactly `.` or `..`, which a URL path can't carry. pattern: ^(?!\.\.?$)[A-Za-z0-9._:/-]+(/\*)?$ minLength: 1 maxLength: 256 Name: type: string description: >- A judgment, attribute or threshold name. Names are path segments in field references, so they never contain `.`. pattern: ^[A-Za-z0-9_-]+$ minLength: 1 maxLength: 128 ThresholdRecommended: type: object description: >- Apply `threshold` with the judgment `PATCH`: `{"escalate": threshold}` for a bool judgment, `{"fraud": {"value": option, "gte": threshold}}` for a choice. required: - status - threshold - precision - recall - interval - outcomes - from_previous_epoch - curve properties: status: const: recommended threshold: $ref: '#/components/schemas/Probability' description: The threshold on the raw value, one of the curve's points. precision: $ref: '#/components/schemas/Probability' description: Precision on these outcomes at the threshold. recall: $ref: '#/components/schemas/Probability' description: Recall on these outcomes at the threshold. interval: $ref: '#/components/schemas/MetricInterval' description: The 95% interval of the targeted metric at the threshold. outcomes: $ref: '#/components/schemas/RecommendationOutcomes' from_previous_epoch: $ref: '#/components/schemas/RecommendationFromPreviousEpoch' curve: $ref: '#/components/schemas/PrecisionRecallCurve' ThresholdUnreachable: type: object description: No threshold meets the target. The curve shows what is reachable. required: - status - outcomes - from_previous_epoch - curve properties: status: const: unreachable outcomes: $ref: '#/components/schemas/RecommendationOutcomes' from_previous_epoch: $ref: '#/components/schemas/RecommendationFromPreviousEpoch' curve: $ref: '#/components/schemas/PrecisionRecallCurve' ThresholdInsufficientData: type: object description: | Too few outcomes, so nothing is recommended: fewer than 100, or fewer than 20 positive or 20 negative ones. `reason` and `message` say which, and what to post. required: - status - outcomes - reason - message properties: status: const: insufficient_data outcomes: type: integer format: int64 minimum: 0 description: >- The outcomes there are, in the current epoch or, when it has more, the previous one. reason: type: string enum: - too_few_outcomes - too_few_positives - too_few_negatives description: See `CalibrationShortfall`. message: type: string description: >- What to post, such as "Only positive outcomes so far: post outcomes for documents where it did not happen." Error: type: object required: - error properties: error: type: object required: - code - message properties: code: $ref: '#/components/schemas/ErrorCode' message: type: string details: type: object description: >- Code-specific detail, such as the scan estimate on a refused query. Every `internal` error has `request_id`. properties: reason: type: string description: > Why, where one code has more than one cause. On `forbidden`, `payment_overdue`: a payment has been unpaid for 14 days, so the organization's keys are read-only until it is paid; reads still work, and paying restores writes within a few minutes. A key that is read-only itself is `forbidden` without a `reason`. On the events feed's `invalid_request`, `cursor_expired`. suggested_context: description: >- On a create without a context recipe, the recipe suggested from the namespace's documents, or null when there are none. oneOf: - $ref: '#/components/schemas/SuggestedContext' - type: 'null' Probability: type: number minimum: 0 maximum: 1 description: > Answers keep probabilities to about seven significant digits and return them as the shortest decimal that stands for the stored value, so an engine's 0.01 reads 0.01. Thresholds, query filters and threshold recommendations compare that decimal. MetricInterval: type: object required: - lower - upper properties: lower: $ref: '#/components/schemas/Probability' upper: $ref: '#/components/schemas/Probability' RecommendationOutcomes: type: integer format: int64 minimum: 100 description: The number of outcomes the recommendation rests on. RecommendationFromPreviousEpoch: type: boolean description: >- True when the current engine epoch has too few outcomes (fewer than 100, or fewer than 20 of either class) and the previous epoch's were used. PrecisionRecallCurve: type: array description: | Precision and recall at each threshold from 0.00 to 1.00 in steps of 0.01, lowest first: 101 points. maxItems: 101 items: $ref: '#/components/schemas/PrecisionRecallPoint' ErrorCode: type: string description: | HTTP status by code: `invalid_request` 400, `unauthorized` 401, `budget_exceeded` 402, `plan_required` 402, `forbidden` 403, `not_found` 404, `conflict` 409, `too_large` 413, `engine_version_unavailable` 422, `insufficient_labels` 422, `idempotency_key_reused` 422, `rate_limited` 429, `internal` 500, `engine_unavailable` 503, `unavailable` 503. enum: - invalid_request - unauthorized - forbidden - not_found - conflict - too_large - rate_limited - budget_exceeded - plan_required - engine_unavailable - engine_version_unavailable - unavailable - insufficient_labels - idempotency_key_reused - internal SuggestedContext: type: object description: | The recipe suggested for a create without `context`, in the refusal's `details.suggested_context`. From a sample of the documents the definition applies to: it keeps short text, numbers, booleans and small attributes, and leaves out ids, timestamps, links, binary-looking and very long values, and rarely present fields; `excluded` says why for each. `max_tokens` is set to keep the context in the standard size class, at most 2,000. The same documents always give the same recipe. required: - recipe - cost_per_answer - whole_state_cost_per_answer - sampled_documents - excluded properties: recipe: $ref: '#/components/schemas/ContextRecipe' cost_per_answer: type: - number - 'null' minimum: 0 description: >- Mean judgments per answer, counted by size class, over the sample with this recipe and the definition's question. The suggested `max_tokens` is at most 2,000, the standard ceiling, less the question's tokens, so a context that fills it stays standard with its question. whole_state_cost_per_answer: type: - number - 'null' minimum: 0 description: The same with the whole `state`. sampled_documents: type: integer minimum: 1 excluded: type: array description: >- Each path left out, and why, so you can put back what the question needs. items: type: object required: - path - reason properties: path: type: string reason: type: string enum: - id - timestamp - url - binary - long - rare - too_many PrecisionRecallPoint: type: object required: - threshold - precision - recall description: | Counts the outcomes whose answer is at or above `threshold`, compared as a named threshold or a query filter compares them, so a threshold set from a point selects what the point counted. properties: threshold: $ref: '#/components/schemas/Probability' precision: description: Null when no outcome's raw value reaches the threshold. oneOf: - $ref: '#/components/schemas/Probability' - type: 'null' recall: $ref: '#/components/schemas/Probability' ContextRecipe: type: object description: | What the engine sees. A create without it is refused with a suggested recipe, unless `use_context` asks for the suggestion or the whole `state`; a version with no recipe sends the whole `state`, subject to the engine limit. additionalProperties: false properties: fields: type: array description: Paths included verbatim, in order. items: type: string pattern: ^(state|attributes)(\.[^.]+)+$ last_n: type: object description: Per array path, keep the last n elements. additionalProperties: type: integer minimum: 1 window: type: object description: >- Per array path whose elements have a timestamp field `at`, keep elements within the window. additionalProperties: $ref: '#/components/schemas/Duration' max_tokens: type: integer minimum: 1 description: Hard cap; the compiler truncates from the end of the last field. related: type: object description: > Up to 4 relations, by name: documents in the same namespace that point at the judged one, or the one document it points at. Each renders as a `related.` entry after `fields`, and a write that changes what it renders makes the judged document's answer `pending` until it is judged again (for a referenced document, only inside the re-judge scope; outside it, the answer is `stale` with `stale_reason` `referenced_changed`). minProperties: 1 maxProperties: 4 propertyNames: $ref: '#/components/schemas/Name' additionalProperties: $ref: '#/components/schemas/Relation' previous: $ref: '#/components/schemas/PreviousRecipe' Duration: type: string description: A whole number and a unit (`s`, `m`, `h` or `d`), such as `7d` or `1h`. pattern: ^[1-9][0-9]*[smhd]$ Relation: type: object description: | Documents that point at the judged document. They are ordered newest `created_at` first: `window` keeps those created within the window, then `last_n` keeps the newest n. At least one of `last_n` and `window` is required, and at least one of `fields` and `aggregate`. A relation reads at most the newest 1,000 documents. With `join: {theirs: "id", mine: "attributes."}` the relation reads the one document the judged document points at, its referenced document, and takes neither `last_n` nor `window`. With `join: {theirs: "attributes.", mine: "attributes."}` it is a blocking relation over the documents that share the judged document's key value. It needs `window`, takes `last_n` up to 254 (up to the `options.from` candidate limit when a `choice` chooses among it with `options.from`), and a change to its block re-judges the block's judged documents. `fields`, `aggregate` and `match` may read plain judgments' answers (`answers..`), banded or cut. additionalProperties: false required: - match - join allOf: - description: A relation renders fields or aggregates. if: not: required: - fields then: required: - aggregate - description: >- A relation over the documents that point at the judged one is bounded by `last_n` or `window`. if: properties: join: type: object properties: mine: const: id then: anyOf: - required: - last_n - required: - window - description: >- A relation that reads the referenced document takes neither `last_n` nor `window`. if: properties: join: type: object properties: theirs: const: id then: not: anyOf: - required: - last_n - required: - window - description: A blocking relation needs `window`, and reads at most 254 documents. if: properties: join: $ref: '#/components/schemas/BlockingJoin' then: required: - window properties: last_n: type: integer maximum: 254 properties: match: $ref: '#/components/schemas/RelationMatch' description: Which documents the relation reads. join: $ref: '#/components/schemas/RelationJoin' last_n: type: integer minimum: 1 maximum: 1000 description: >- Keep the newest n. Not on a relation that reads a referenced document; at most 254 on a blocking relation, and at most the `options.from` candidate limit (see Limits) when `options.from` chooses among it. window: $ref: '#/components/schemas/Duration' description: >- Keep documents created within this long before the later of the judged document's newest write and its newest related creation, at the answer's watermark. An edit to an old related document never moves the window. Not on a relation that reads a referenced document. fields: type: array description: >- Paths to include from each document, rendered under `records`, newest first. An entry may be a `BandedField`, which renders a number as its band. An entry may also be a plain judgment's `answers..value` and `answers..thresholds.` as they are, and its `.p`, `.score` and `.dist.