> ## Documentation Index > Fetch the complete documentation index at: https://docs.brussle.com/llms.txt > Use this file to discover all available pages before exploring further. # See which alternative your outcomes prefer > The active version's alternatives, scored on the labelled outcomes its calibration fit uses: other engines, wordings and context recipes, each answered on the same labelled answers, fitted on its own outcomes the way the version is, and compared with the version on the same outcomes. `corrected_for` is how many alternatives were scored this month: a verdict of `better` or `moves_ranking` holds after correcting for all of them, so trying more makes each claim harder to win. Two groups share the table. The engine and named alternatives (`in_k: true`) are scored only on a confirmed run, billed as judging at each engine's weight. Recipe tuning's narrowings (`in_k: false`) are the free monthly run of `recipe-tuning`, judged only `no_worse` or `worse` and never counted in `corrected_for`. `quote` prices what is unscored before anything runs, and says in `detectable` the smallest gain these labels could call `better`. `status: quoting` means the quote is being priced; read again in a few seconds. `suggestion` names the alternative the outcomes prefer; nothing is applied for you. On every plan. Below Team, the narrowings' numbers are shown and their `definition` and `differs` are null, with `plan_required`; scoring engine and named alternatives needs Team. ## OpenAPI ````yaml /api-reference/openapi.yaml get /namespaces/{ns}/judgments/{name}/alternatives openapi: 3.1.0 info: title: Brussle API version: '1' license: name: Proprietary identifier: LicenseRef-Proprietary description: | The `/v1` contract. It is locked: changes within `/v1` are additive only, and anything breaking is `/v2`. - **Base URL.** `https://api.brussle.com/v1`. - **Evolution.** Response objects may gain fields and enums may gain values. Clients must ignore what they do not know. Request objects reject unknown fields. - **Path segments.** Namespace names (`{ns}`) and document ids (`{id}`) may contain `/`. Send it percent-encoded as `%2F`, so that `acme/prod/tenant_123` is `acme%2Fprod%2Ftenant_123`. - **Templates.** The judgment routes also take a namespace prefix ending in `/*`, such as `acme%2Fprod%2F*`. A judgment created there is a template: every namespace under the prefix inherits it, including ones created later. The most specific matching prefix wins. - **Thresholds are settings.** Named thresholds belong to the judgment, not to a version. They are evaluated at read time against the raw fields, so a change applies at once to every answer, with no recompute and no new version. - **Entity judgments.** A judgment can read the documents that point at the judged one, through `context.related`, and apply only to documents matching `applies_to`. Its answers carry a `watermark`, and creating one that runs `on_change` returns a replay estimate of its monthly cost until you confirm. - **Referenced documents.** A relation can also read the one document the judged document points at, through `join: {theirs: "id", mine: "attributes."}`. A change to what the relation renders of that document re-judges the judged documents that point at it and are inside `freshness.fanout.scope`: a fan-out. - **Relations.** A relation can read other judgments' answers (banded or cut, never summed), up to three judgments deep and never in a cycle, or be a blocking relation over the documents that share a key, which a `choice` can choose among with `options.from`. A recipe can read the document as the judgment last saw it (`context.previous`). A judgment that can fan out confirms its rolling limits once, at create; past them, work is deferred, reads `stale` with an event, and catches up on its own. Nothing waits for a confirm at runtime. Groups keep aggregates per key value for reporting. - **Idempotency.** Every write operation is idempotent by construction: a retried append adds nothing while its values are still in the array, and a retried upsert or patch sets the same content again. Every request that changes something also takes an `Idempotency-Key`, so that creates, activations, backfills, draws and the rest can be retried safely too: a retry with the key gets the first response back, verbatim and marked `Idempotent-Replayed: true`, instead of running again (see the parameter). - **Slow changes become jobs.** Creating a judgment version, a judgment's `PATCH` and a backfill answer as usual when they finish within 20 s. When estimating the cost takes longer, the request answers `202` with a job of type `judgment_create`, `judgment_update` or `backfill_start` instead, and the job finishes the request. Poll `GET /jobs/{id}`: when the job is `done` or `failed`, its `result` is the response the request would have returned. The SDKs wait for it by default. - **Errors.** Every error is the `Error` envelope; the HTTP status follows the code (see each response). - **Rate limits.** A request must be inside two limits: requests a second for its key (1,000 by default), and requests a second for all the organization's API keys together (2,000 by default). A dashboard session counts only toward its user's own per-key limit. Past either, the request is `rate_limited` with `Retry-After`, and `details.rate_limit` (`per_key` or `per_org`) and `details.limit` say which limit refused it. A refused request counts toward neither limit. Each limit is counted over about a second, so a burst can be refused a little before it. Every response to a request whose key was accepted carries `X-RateLimit-Limit` and `X-RateLimit-Remaining`, for the limit with fewer requests left, or for the limit that refused the request. A request refused before its key is known (`unauthorized`, a route that does not exist, or a malformed namespace name) carries neither. A `rate_limited` response adds `Retry-After`, except a daily allowance that is used up, which says when it renews in `details.resets_at` instead. Every `503` carries `Retry-After` too. Reads of one namespace also have a concurrency limit: past 32 gets and queries in flight at once, one more is `rate_limited` with `Retry-After` and `details.concurrent_reads`, whatever the rate-limit headers show. - **Usage.** Every response that bills carries `usage`. Judging is billed per judgment, by size class: one judgment answered counts by the tokens of its compiled context and its question together, 1 if standard (up to 2,000 tokens), 4 if large (up to 8,000) and 16 if extra-large (up to 32,000, the engine maximum). Questions the engine answers about one document in one request share it: the first counts in full, and each question after the first costs a quarter of a judgment, and a quarter more for each further 500 tokens of question text (at least a quarter of its size class). Each request then counts times its engine's weight (`jev` 1, `gpt-6-luna` 2.5; `pplx-decider`, not active yet, 1), rounded up to a quarter of a judgment. No response ever carries an engine's price. servers: - url: https://api.brussle.com/v1 security: - apiKey: [] tags: - name: Namespaces description: Namespaces, their settings and cache warming. - name: Documents description: Writes, point reads and queries. - name: Judgments description: >- Judgment definitions, versions, settings, activation, backfill and template detach. - name: Outcomes and calibration description: | Post what actually happened to judged documents, and read how well answers match it: the calibration report, calibrated answers and threshold recommendations. See [calibration](/concepts/calibration). - name: Groups description: > Groups keep aggregates per value of a key attribute, a group-by you declare for reporting; a template's tenant summary publishes each tenant's groups, banded, as a document the platform can judge. - name: Jobs description: >- Backfill, shadow, periodic, deletion, reference index, evaluation export, resync, group build, discovery and simulation jobs, and the jobs that finish a slow create, `PATCH` or backfill start. - name: Engines description: The engine registry. - name: Organization description: | Settings of the whole organization: prefixes marked non-production, which template calibration pools leave out. The mark changes nothing on the bill: namespaces under it count toward the tenant fee like any other. - name: Subscriptions description: | Saved queries that send an event when a document starts or stops matching: the query's filter grammar, on a namespace or a template prefix, with the fields each event carries. - name: Webhooks description: | Webhook endpoints, the events feed, the delivery log, redelivery and recovery. Every event type and its body is under `webhooks`. Events are delivered at least once, signed as Standard Webhooks, and kept for 30 days. Webhooks are on every plan, and deliveries are not billed; plans set how many endpoints an organization has. paths: /namespaces/{ns}/judgments/{name}/alternatives: parameters: - $ref: '#/components/parameters/NamespaceOrTemplate' - $ref: '#/components/parameters/JudgmentName' get: tags: - Outcomes and calibration summary: See which alternative your outcomes prefer description: | The active version's alternatives, scored on the labelled outcomes its calibration fit uses: other engines, wordings and context recipes, each answered on the same labelled answers, fitted on its own outcomes the way the version is, and compared with the version on the same outcomes. `corrected_for` is how many alternatives were scored this month: a verdict of `better` or `moves_ranking` holds after correcting for all of them, so trying more makes each claim harder to win. Two groups share the table. The engine and named alternatives (`in_k: true`) are scored only on a confirmed run, billed as judging at each engine's weight. Recipe tuning's narrowings (`in_k: false`) are the free monthly run of `recipe-tuning`, judged only `no_worse` or `worse` and never counted in `corrected_for`. `quote` prices what is unscored before anything runs, and says in `detectable` the smallest gain these labels could call `better`. `status: quoting` means the quote is being priced; read again in a few seconds. `suggestion` names the alternative the outcomes prefer; nothing is applied for you. On every plan. Below Team, the narrowings' numbers are shown and their `definition` and `differs` are null, with `plan_required`; scoring engine and named alternatives needs Team. operationId: getAlternatives responses: '200': description: The table, its quote, and its suggestion. headers: X-RateLimit-Limit: $ref: '#/components/headers/RateLimitLimit' X-RateLimit-Remaining: $ref: '#/components/headers/RateLimitRemaining' content: application/json: schema: $ref: '#/components/schemas/Alternatives' '400': $ref: '#/components/responses/InvalidRequest' '401': $ref: '#/components/responses/Unauthorized' '403': $ref: '#/components/responses/Forbidden' '404': $ref: '#/components/responses/NotFound' '429': $ref: '#/components/responses/RateLimited' '500': $ref: '#/components/responses/Internal' components: parameters: NamespaceOrTemplate: name: ns in: path required: true description: | A namespace name, or a template prefix ending in `/*`, with any `/` sent as `%2F`: `acme%2Fprod%2Ftenant_123` or `acme%2Fprod%2F*`. schema: $ref: '#/components/schemas/NamespaceOrTemplate' JudgmentName: name: name in: path required: true schema: $ref: '#/components/schemas/Name' headers: RateLimitLimit: description: >- Requests allowed per second under one of the two limits on an API key, per key or per organization; the one with fewer requests left, or on a `rate_limited` response the one that refused the request. A dashboard session reports its user's limit. Only on responses to a request whose key was accepted; never on `unauthorized`, on a route that does not exist, or on a malformed namespace name. schema: type: integer minimum: 0 RateLimitRemaining: description: >- Requests left in the current second under the limit `X-RateLimit-Limit` gives. It does not count the per-namespace limit on reads in flight. schema: type: integer minimum: 0 RetryAfter: description: Seconds to wait before retrying. schema: type: integer minimum: 0 schemas: Alternatives: type: object description: | A version's alternatives: which engine, recipe or wording its outcomes prefer, what the labels can and cannot tell, and what scoring costs. required: - version - engine_version - status - reason - message - automatic - skip_engines - labels - corrected_for - quote - current - alternatives - suggestion - headline - computed_at - next_run_after - plan_required properties: version: $ref: '#/components/schemas/JudgmentVersionNumber' engine_version: type: - string - 'null' description: >- The engine epoch of the calibration fit whose labelled answers the alternatives are scored on. Null before a fit. status: type: string enum: - not_run - quoting - running - ready - none description: | - `not_run`: nothing scored this month; `reason` says why. - `quoting`: the quote is being priced; read again shortly. - `running`: this month's run is scoring. - `ready`: `suggestion` names what the outcomes prefer. - `none`: the run found nothing better beyond noise, or nothing cheaper; `message` says what was detectable. reason: description: | Not run: `awaiting_confirm` (there is something to score and `automatic` is off: confirm the quote), `awaiting_run` (the next calibration run prices it), `too_few_outcomes` (no calibration fit yet), `composite`, `inherited` (the template has the table), `budget_paused` (the run waits for budget). None: `nothing_better` (nothing beat the version beyond noise; `message` says the smallest gain the labels could show), `accuracy_would_drop`. A narrowing run's own reasons are recipe tuning's. Null when ready, running or quoting. oneOf: - type: string enum: - awaiting_confirm - awaiting_run - too_few_outcomes - composite - inherited - budget_paused - nothing_better - accuracy_would_drop - superseded - type: 'null' message: type: string description: The status in a sentence. automatic: type: boolean description: >- Whether the run starts by itself after a fit, at most monthly (`alternatives.automatic` on the judgment). Off by default. skip_engines: type: array description: Engines the judgment's setting skips as automatic alternatives. items: $ref: '#/components/schemas/EngineRef' labels: type: object description: The labelled answers the alternatives are scored on. required: - count - sampled - outcomes properties: count: type: integer minimum: 0 description: Labelled answers, at most 500, the same for every alternative. sampled: type: boolean description: Whether the fit had more and these are a fixed sample of them. outcomes: oneOf: - $ref: '#/components/schemas/CalibrationOutcomeFilter' - type: 'null' description: >- The outcomes chosen for this month's run, as `runAlternatives` took them; null for the fit's own. corrected_for: type: integer minimum: 0 description: >- Engine and named alternatives scored this month. Every `better` and `moves_ranking` holds after correcting for all of them. quote: oneOf: - $ref: '#/components/schemas/AlternativesQuote' - type: 'null' current: description: >- The active version on the same labels, scored the same way. Null before a run. oneOf: - type: object required: - held_out - cost_per_answer properties: held_out: $ref: '#/components/schemas/AlternativeHeldOut' cost_per_answer: type: number description: Judgments per answer at its engine's weight. - type: 'null' alternatives: type: array description: >- Engine and named alternatives first (`in_k: true`), then recipe tuning's narrowings. items: $ref: '#/components/schemas/Alternative' suggestion: description: >- The alternative the outcomes prefer, by id, and why. Nothing is applied. oneOf: - type: object required: - id - because properties: id: $ref: '#/components/schemas/AlternativeIdValue' because: type: string enum: - better - cheaper description: >- `better`: it beat the version beyond noise after the correction. `cheaper`: a narrowing at least 20% cheaper per answer and no worse. - type: 'null' headline: type: - string - 'null' description: The suggestion in a sentence, with the correction it holds after. computed_at: oneOf: - $ref: '#/components/schemas/Timestamp' - type: 'null' next_run_after: description: >- When the automatic run may next start. Null while `automatic` is off. oneOf: - $ref: '#/components/schemas/Timestamp' - type: 'null' plan_required: description: >- Below Team, why scoring engine and named alternatives, and the narrowings' recipes, are withheld. oneOf: - $ref: '#/components/schemas/AlternativesPlanRequired' - type: 'null' NamespaceOrTemplate: type: string description: | A namespace name, or a template prefix: a namespace path ending in `/*`, such as `acme/prod/*`, which every namespace under `acme/prod/` inherits judgments from. Up to 256 bytes. Never exactly `.` or `..`, which a URL path can't carry. pattern: ^(?!\.\.?$)[A-Za-z0-9._:/-]+(/\*)?$ minLength: 1 maxLength: 256 Name: type: string description: >- A judgment, attribute or threshold name. Names are path segments in field references, so they never contain `.`. pattern: ^[A-Za-z0-9_-]+$ minLength: 1 maxLength: 128 JudgmentVersionNumber: type: integer format: int32 minimum: 1 EngineRef: type: object description: | An engine and its version, which must be active in `GET /engines`; any other is refused with `engine_version_unavailable`. The version is an exact version, which never changes, or `current` for an engine that is not pinned. Today the active versions are `jev` `current` and `gpt-6-luna` `current`; `pplx-decider` has no active version yet. `current` runs whatever model the provider serves now and records the epoch of behaviour behind each answer in its `engine_version`, such as `current+2026-09-24.1`. There are no other aliases. Engines differ in whether they read images (`limits.images` in `GET /engines`) and in their weight on the bill (`gpt-6-luna` counts each judgment 2.5 times). required: - name - version properties: name: type: string enum: - jev - gpt-6-luna - pplx-decider version: type: string minLength: 1 CalibrationOutcomeFilter: type: object description: | The outcomes a named run is fitted on. An outcome left out still counts as an answer's outcome, so it never makes that answer an implicit negative. additionalProperties: false properties: sources: type: array description: >- Where outcomes came from: `posted`, derived by an outcome `rule`, a labelling-`queue` label, or `implicit` negatives. Empty or absent keeps every source. uniqueItems: true items: type: string enum: - posted - rule - queue - implicit since: $ref: '#/components/schemas/Timestamp' description: >- Outcomes observed at or after this time; an implicit negative when its window closed. until: $ref: '#/components/schemas/Timestamp' description: Outcomes observed before this time. AlternativesQuote: type: object description: | What scoring the unscored engine and named alternatives would bill, priced before anything runs, and the smallest gain the labels could show. Each answered call is billed as judging: its size class times its engine's weight, at the graduated prices of the tiers your organization is in now. required: - status - labels - alternatives - judgment_units - tokens - cost_usd - detectable - new_providers - upstream properties: status: type: string enum: - ready - quoting generation: type: integer minimum: 0 description: >- Which quote this is; it moves whenever the alternatives, the outcomes or the engines in use change. Send it as `quote_generation` to confirm exactly this quote. computed_at: oneOf: - $ref: '#/components/schemas/Timestamp' - type: 'null' labels: type: integer minimum: 0 description: Labelled answers each alternative would be scored on. alternatives: type: array items: type: object required: - id - contexts - labels - tokens - judgment_units - cost_usd - new_provider properties: id: $ref: '#/components/schemas/AlternativeIdValue' contexts: type: string enum: - stored - narrowed - live labels: type: integer minimum: 0 tokens: type: integer minimum: 0 judgment_units: type: integer minimum: 0 cost_usd: type: number minimum: 0 new_provider: type: boolean description: >- Its engine is one this namespace has not used; confirming sends the labelled contexts to its provider. judgment_units: type: integer minimum: 0 tokens: type: integer minimum: 0 cost_usd: type: number minimum: 0 detectable: type: object description: >- The smallest gain this run could call `better` on these labels with this many alternatives; smaller gains read `no_worse`. required: - log_loss - auroc - k properties: log_loss: type: number minimum: 0 auroc: type: - number - 'null' minimum: 0 k: type: integer minimum: 1 description: The alternatives it corrects for. new_providers: type: array description: >- Engines that would receive this namespace's labelled contexts for the first time, `name version`. items: type: string upstream: type: array description: >- The backfill a `live` recipe needs first, when it selects by another judgment's answer. Never run by this run. items: type: object required: - alternative - judgment - documents - judgment_units - cost_usd properties: alternative: $ref: '#/components/schemas/AlternativeIdValue' judgment: $ref: '#/components/schemas/Name' documents: type: integer minimum: 0 judgment_units: type: integer minimum: 0 cost_usd: type: number minimum: 0 reason: type: - string - 'null' description: Why nothing could be priced, when nothing was. AlternativeHeldOut: type: object description: >- Numbers on the labelled answers, each read with its own calibration fit made without the answer it is scored on. properties: read_with: type: string enum: - raw - calibrated - features description: What its calibration would read, as the nightly fit would choose. log_loss: type: number ece: type: - number - 'null' brier: type: - number - 'null' auroc: type: - number - 'null' precision_at_5: type: - number - 'null' precision_at_10: type: - number - 'null' accuracy: type: number features_only: type: - number - 'null' description: On a judgment with `features`, the log loss of the features alone. Alternative: type: object description: One alternative and what the outcomes say of it. required: - id - source - drafted - named_by - named_at - differs - definition - in_k - contexts - as_of - labels - labels_skipped - cost_per_answer - unit_reduction - held_out - change - verdict - moves_ranking - scored_at - billed properties: id: $ref: '#/components/schemas/AlternativeIdValue' source: type: string enum: - automatic - named description: >- `automatic`: one per engine this namespace uses, or a recipe-tuning narrowing. `named`: one you posted. drafted: type: boolean description: A wording drafted by a language model and posted by you. named_by: type: - string - 'null' description: Who named it, `key:` or `user:`. named_at: oneOf: - $ref: '#/components/schemas/Timestamp' - type: 'null' differs: type: - array - 'null' description: >- What differs from the active version, in words. Null below Team for a narrowing. items: type: string definition: description: >- The whole definition to post as a version. Null below Team for a narrowing. oneOf: - $ref: '#/components/schemas/JudgmentDefinition' - type: 'null' in_k: type: boolean description: >- Whether it counts in `corrected_for`. Recipe tuning's narrowings do not, and are never `better`. contexts: type: string enum: - stored - narrowed - live description: | How its contexts are made for each labelled answer: `stored`, the context the labelled answer read; `narrowed`, cut from it exactly; `live`, compiled from today's documents and related documents. as_of: type: - string - 'null' enum: - now - null description: >- `now` on a `live` row: its contexts were built from today's related documents and answers, not from what your labelled answers read, so a change in those documents since the label counts against or for it. Null on every other row. labels: type: integer minimum: 0 description: Labelled answers it has an answer for. labels_skipped: type: object description: Labelled answers it was not scored on, by why. properties: deleted: type: integer minimum: 0 image_erased: type: integer minimum: 0 not_derivable: type: integer minimum: 0 failed: type: integer minimum: 0 cost_per_answer: type: - number - 'null' description: Judgments per answer at its engine's weight, from what it sent. unit_reduction: type: - number - 'null' description: A narrowing's saving per answer, a share of the version's cost. held_out: oneOf: - $ref: '#/components/schemas/AlternativeHeldOut' - type: 'null' change: description: Against the active version on the same labelled answers. oneOf: - $ref: '#/components/schemas/AlternativeChange' - type: 'null' verdict: type: string enum: - better - no_worse - worse - untested - unscored description: | - `better`: its calibrated number beat the version's beyond noise, after correcting for every alternative scored this month. - `no_worse`: worse by no more than noise (one standard error or 1%), uncorrected. - `worse`: worse beyond that. - `untested`: too few labelled answers after its skips. - `unscored`: not scored yet. moves_ranking: type: boolean description: >- Whether it ranks your outcomes better (AUROC) beyond noise, after the same correction. scored_at: oneOf: - $ref: '#/components/schemas/Timestamp' - type: 'null' billed: description: >- What scoring it billed. Null for an unscored alternative and for a narrowing, which is never billed. oneOf: - type: object required: - judgment_units - cost_usd properties: judgment_units: type: integer minimum: 0 cost_usd: type: number minimum: 0 - type: 'null' AlternativeIdValue: type: string pattern: ^alt_[0-9a-f]{16}$ description: An alternative's id, the same for the same definition in every run. examples: - alt_3f9a2c7e1b04d6a8 Timestamp: type: string format: date-time description: RFC 3339, UTC. AlternativesPlanRequired: type: object description: >- Why scoring alternatives, and the narrowings' recipes, are withheld, as `plan_required` says it. required: - message - capability - required_plan - plan properties: message: type: string capability: type: string enum: - alternatives required_plan: type: string enum: - team - scale plan: type: string enum: - developer - team - scale Error: type: object required: - error properties: error: type: object required: - code - message properties: code: $ref: '#/components/schemas/ErrorCode' message: type: string details: type: object description: >- Code-specific detail, such as the scan estimate on a refused query. Every `internal` error has `request_id`. properties: reason: type: string description: | Why, where one code has more than one cause. On the events feed's `invalid_request`, `cursor_expired`. suggested_context: description: >- On a create without a context recipe, the recipe suggested from the namespace's documents, or null when there are none. oneOf: - $ref: '#/components/schemas/SuggestedContext' - type: 'null' JudgmentDefinition: type: object description: | A judgment definition, discriminated on `type`. `thresholds` are the named thresholds given with this version. They are not part of the version: the judgment's current thresholds are its `thresholds` setting, evaluated at read time and returned as booleans under `answers.{name}.thresholds`. oneOf: - $ref: '#/components/schemas/BoolJudgmentDefinition' - $ref: '#/components/schemas/ChoiceJudgmentDefinition' - $ref: '#/components/schemas/ScoreJudgmentDefinition' discriminator: propertyName: type mapping: bool: '#/components/schemas/BoolJudgmentDefinition' choice: '#/components/schemas/ChoiceJudgmentDefinition' score: '#/components/schemas/ScoreJudgmentDefinition' AlternativeChange: type: object required: - log_loss - auroc - accuracy properties: log_loss: $ref: '#/components/schemas/AlternativeTest' auroc: oneOf: - $ref: '#/components/schemas/AlternativeTest' - type: 'null' accuracy: type: number description: Accuracy, alternative minus version. ErrorCode: type: string description: | HTTP status by code: `invalid_request` 400, `unauthorized` 401, `invalid_image` 400, `unknown_image` 400, `image_erased` 410, `budget_exceeded` 402, `plan_required` 402, `forbidden` 403, `not_found` 404, `conflict` 409, `too_large` 413, `engine_version_unavailable` 422, `insufficient_labels` 422, `idempotency_key_reused` 422, `rate_limited` 429, `internal` 500, `engine_unavailable` 503, `unavailable` 503. Any route can also answer `unavailable`, with `Retry-After`, when it did not finish within 35 s: it was stopped, and a retry runs it again. Creating a judgment version, a judgment's `PATCH` and a backfill's start answer within 20 s, with a job (`202`) when they need longer, and never begin their change after 25 s. So when one of them is answered `unavailable` this way, nothing changed and the retry makes the change once. enum: - invalid_request - invalid_image - unknown_image - image_erased - unauthorized - forbidden - not_found - conflict - too_large - rate_limited - budget_exceeded - plan_required - engine_unavailable - engine_version_unavailable - unavailable - insufficient_labels - idempotency_key_reused - internal SuggestedContext: type: object description: | The recipe suggested for a create without `context`, in the refusal's `details.suggested_context`. From a sample of the documents the definition applies to: it keeps short text, numbers, booleans and small attributes, and leaves out ids, timestamps, links, binary-looking and very long values, and rarely present fields; `excluded` says why for each. Each name in a document's `images` is one candidate, `images.`; image paths are kept only on an engine that reads images, and only while the context stays in the standard size class with them: images count by their size, so when it has to choose, the suggestion keeps the text and leaves out image fields, the last first, with the reason `image`. `max_tokens` is set to keep the context in the standard size class, at most 2,000. The same documents always give the same recipe. required: - recipe - cost_per_answer - whole_state_cost_per_answer - sampled_documents - excluded properties: recipe: $ref: '#/components/schemas/ContextRecipe' cost_per_answer: type: - number - 'null' minimum: 0 description: >- Mean judgments per answer, counted by size class at the engine's weight as the bill counts them, over the sample with this recipe and the definition's question. The suggested `max_tokens` is at most 2,000, the standard ceiling, less the question's tokens, so a context that fills it stays standard with its question. whole_state_cost_per_answer: type: - number - 'null' minimum: 0 description: The same with the whole `state`. sampled_documents: type: integer minimum: 1 excluded: type: array description: >- Each path left out, and why, so you can put back what the question needs. items: type: object required: - path - reason properties: path: type: string reason: type: string enum: - id - timestamp - url - binary - image - long - rare - too_many BoolJudgmentDefinition: type: object description: | The answer has `p`. With `parts` it is a composite judgment: the engine is asked each part instead of `question`, and `p` combines the parts' answers with weights fitted on the judgment's outcomes. required: - type allOf: - $ref: '#/components/schemas/JudgmentDefinitionCommon' not: description: A single part needs features. required: - parts properties: parts: type: array maxItems: 1 not: required: - features properties: type: const: bool thresholds: type: object description: >- Each threshold is true when `p` is at least the value, or, written `{feature, gte}` or `{feature, lte}`, when one of the judgment's features is at least or at most the number. propertyNames: $ref: '#/components/schemas/Name' additionalProperties: oneOf: - $ref: '#/components/schemas/Probability' - $ref: '#/components/schemas/FeatureThreshold' parts: type: array description: | Composite judgments only: 2 to 8 narrow yes/no questions with names unique within the judgment. They share the judgment's context recipe and engine, so they go in one engine request with its other questions, and each counts toward the 32 questions per request. `question` then documents what the combination means and is not sent to the engine. Parts are part of the version. They cannot reference other judgments. Each part is billed as a judgment. With `features`, one part is enough. minItems: 1 maxItems: 8 items: $ref: '#/components/schemas/JudgmentPart' features: type: array description: | Up to 12 numbers, on any bool judgment: aggregates over related documents the recipe's `related` declares, the judged document's own values, ages, and the engine's previous answer (see `FeatureRef`). Each is recorded as a raw number beside every answer under `features`, and on its evaluation, as of what that evaluation read, and never recomputed later. A threshold can name one, a query or subscription can filter on one, and the fit on your outcomes reads them beside the engine's answer. A composite judgment's combiner also takes them as numeric inputs beside the parts, scaled automatically, a missing value counting as typical. Features add no questions, so they are never billed: a change that moves only a feature records a new answer with no engine call. A judgment with relations created with features debounces for 10 minutes unless its `freshness` sets `debounce_ms`. Changing the list makes a new version. minItems: 1 maxItems: 12 uniqueItems: true items: $ref: '#/components/schemas/FeatureRef' ChoiceJudgmentDefinition: type: object description: | The answer has `value`, `dist` and `escape_p`. The API appends `none_of_the_above` as a final option on every choice judgment. required: - type - options allOf: - $ref: '#/components/schemas/JudgmentDefinitionCommon' properties: type: const: choice options: description: >- The options, whose values must be distinct; or `{from, label}`, one option per document a blocking relation of the recipe selects. oneOf: - type: array minItems: 2 maxItems: 254 items: $ref: '#/components/schemas/ChoiceOption' - $ref: '#/components/schemas/OptionsFrom' thresholds: type: object description: Each threshold is true when `dist[value]` is at least `gte`. propertyNames: $ref: '#/components/schemas/Name' additionalProperties: $ref: '#/components/schemas/ChoiceThreshold' features: type: array description: > Up to 12 numbers the fit on your outcomes weighs beside the engine's distribution, with a bool's sources (see `FeatureRef`), and on a choice among candidates `.each()`, a number per candidate. Each is recorded as a raw number beside every answer and its evaluation, as of what that evaluation read. Once a fit on them beats the engine on outcomes it was not fitted on, every reader reads the fitted numbers: `dist`, `escape_p` and `value`, so the fit can change which option or candidate is picked. Features add no questions and are never billed. Changing the list makes a new version. Needs the features release on the namespace's deployment. minItems: 1 maxItems: 12 uniqueItems: true items: $ref: '#/components/schemas/FeatureRef' ScoreJudgmentDefinition: type: object description: >- The answer has `score`, the probability-weighted mean of level values, and `dist`. required: - type - levels allOf: - $ref: '#/components/schemas/JudgmentDefinitionCommon' properties: type: const: score levels: type: array description: Ordered; level values must be distinct. minItems: 2 maxItems: 10 items: $ref: '#/components/schemas/ScoreLevel' thresholds: type: object description: Each threshold is true when `score` is at least the value. propertyNames: $ref: '#/components/schemas/Name' additionalProperties: type: number features: type: array description: > Up to 12 numbers the fit on your outcomes weighs beside the engine's distribution, with a bool's sources (see `FeatureRef`). Each is recorded as a raw number beside every answer and its evaluation, as of what that evaluation read. Once a fit on them beats the engine on outcomes it was not fitted on, every reader reads the fitted numbers: `dist` and `score`. Features add no questions and are never billed. Changing the list makes a new version. Needs the features release on the namespace's deployment. minItems: 1 maxItems: 12 uniqueItems: true items: $ref: '#/components/schemas/FeatureRef' AlternativeTest: type: object description: >- A paired difference, alternative minus version, on the same labelled answers. required: - mean - interval properties: mean: type: number standard_error: type: - number - 'null' interval: type: object description: The 95% interval of the difference. required: - lower - upper properties: lower: type: number upper: type: number p: type: - number - 'null' description: >- The chance of a difference at least this good if the two were equal, before the correction. ContextRecipe: type: object description: | What the engine sees. A create without it is refused with a suggested recipe, unless `use_context` asks for the suggestion or the whole `state`; a version with no recipe sends the whole `state`, subject to the engine limit, and no images. additionalProperties: false properties: fields: type: array description: > Paths included in order, each verbatim or, as a `ClippedPath`, cut to its first `max_chars` characters. A path is `state.`, `attributes.`, or an image path, `images.`, which reads that name's image or whole list; an image path takes no `max_chars`. An engine that reads images (its `limits.images` in `GET /engines`) reads each, counted by its tiles; an engine that reads none gets each as an `OmittedImage` with `engine_reads_no_images`, and the create warns. Images are written to documents, never sent with the judgment. items: $ref: '#/components/schemas/RecipeField' last_n: type: object description: >- Per array path in `state` or `attributes`, keep the last n elements. Not an image path. additionalProperties: type: integer minimum: 1 window: type: object description: >- Per array path whose elements have a timestamp field `at`, keep elements within the window. additionalProperties: $ref: '#/components/schemas/Duration' max_tokens: type: integer minimum: 1 description: >- Hard cap, in the judgment's engine's tokens. Over it, whole images are left out first (on an engine that reads them; those of relations, then `previous`, then fields, the last first; every image when the text alone is over), then relation records, then `previous` entries, then the end of the last field is cut. related: type: object description: > Up to 4 relations, by name: documents in the same namespace that point at the judged one, or the one document it points at. Each renders as a `related.` entry after `fields`, and a write that changes what it renders makes the judged document's answer `pending` until it is judged again (for a referenced document, only inside the re-judge scope; outside it, the answer is `stale` with `stale_reason` `referenced_changed`). minProperties: 1 maxProperties: 4 propertyNames: $ref: '#/components/schemas/Name' additionalProperties: $ref: '#/components/schemas/Relation' previous: $ref: '#/components/schemas/PreviousRecipe' JudgmentDefinitionCommon: type: object required: - name - question properties: name: $ref: '#/components/schemas/Name' question: type: string minLength: 1 criteria: type: string context: $ref: '#/components/schemas/ContextRecipe' engine: $ref: '#/components/schemas/EngineRef' description: >- Must be `active` in the registry. May be omitted only when the namespace has a `default_engine`. Only an engine with `limits.images` reads the images the recipe selects. horizon: $ref: '#/components/schemas/Horizon' description: > How long after an answer an outcome still labels it, such as `30d` for "the invoice was paid within 30 days". Part of the definition, so changing it creates a version. Defaults to `0s`, which joins labelled examples to current answers. applies_to: $ref: '#/components/schemas/AttributeFilter' description: > The judgment judges, answers and bills only documents whose attributes match. A document that does not match has no answer for it: `answers` omits it. Without it, the judgment judges, answers and bills every document in the namespace, and the create response warns. Part of the definition, so changing it creates a version. Probability: type: number minimum: 0 maximum: 1 description: > Answers keep probabilities to about seven significant digits and return them as the shortest decimal that stands for the stored value, so an engine's 0.01 reads 0.01. Thresholds, query filters and threshold recommendations compare that decimal. FeatureThreshold: type: object description: | A bool judgment's threshold on one of its features: true when the feature is at least `gte`, or at most `lte`, compared exactly with the stored value. No engine and no labels decide it. A missing feature reads `false`, and so does one a later version no longer declares, which the judgment's `threshold_unresolved` warning names. It is as fresh as the answer, so it can trail the related documents by the judgment's debounce. additionalProperties: false required: - feature oneOf: - required: - gte properties: gte: type: number - required: - lte properties: lte: type: number properties: feature: $ref: '#/components/schemas/FeatureRef' gte: type: number lte: type: number JudgmentPart: type: object description: One part of a composite judgment. additionalProperties: false required: - name - question properties: name: $ref: '#/components/schemas/Name' question: type: string minLength: 1 description: A narrow yes/no question about the document. FeatureRef: type: string description: | One number the judgment records beside every answer and its fit can read. It is one of: - An aggregate the recipe declares, as `.count`, `.count_where`, `.labelled` or `.()`, such as `tickets.count` or `invoices.sum(state.amount)`. A `min` or `max` over an answer's `p` or `score` gives the raw extreme, whatever band the context shows; a `latest` over an answer's threshold reads 1 or 0. A `latest` over an answer's `value` is not a number and is `invalid_request`. A `sum` or `avg` over an answer's number is a feature only. A `latest` over a `bool` outcome reads 1 or 0, over a `score` outcome its level; over a `choice` outcome it is `invalid_request`. - The judged document's own value, `state.` or `attributes.`: a number as itself, a boolean as 1 or 0, anything else missing. It needs no relation. - An age in fractional days, `age(created_at)`, `age()` or `age(.())` over an aggregate the recipe declares. It is measured as of the judged document's own `updated_at` at the revision judged, so a write to a related document never moves it, and a new revision of the judged document does. The value must be an RFC 3339 string; a number or any other value reads missing, and a later time reads negative. `age(updated_at)` is `invalid_request`: as of the judged revision it is always 0. - `previous.p`: the engine's own `p` the last time it judged the document with the same engine version, in the same life of the document; on a choice, its `1 - escape_p`. Missing on the first judgment, after the document is created again, after the engine version changes, and on a composite, which has none. On a score it is `invalid_request`. - On a choice among candidates (`options.from`) only, a number per candidate, `.each()` over the `options.from` relation: each candidate's `state.` or `attributes.`, or `age(created_at)`, `age(updated_at)` or `age()` of the candidate, as of the judged document's revision. It is recorded for each candidate the engine read, under the answer's `candidate_features`. A candidate's answers are `invalid_request`: keep the number on the candidate's state. A name the recipe declares as an aggregate always reads that aggregate. A feature that reads a path an outcome rule reads, one inside the other, is `invalid_request`. pattern: >- ^([A-Za-z0-9_-]+\.(count|count_where|labelled|(sum|avg|min|max|latest)\((state|attributes)(\.[^.()]+)+\)|(min|max|latest)\((created_at|updated_at)\)|(sum|avg|min|max)\(answers\.[A-Za-z0-9_-]+\.(p|score|dist\.[^()]+)\)|latest\(answers\.[A-Za-z0-9_-]+\.thresholds\.[A-Za-z0-9_-]+\)|latest\(outcomes\.[A-Za-z0-9_-]+\.value\))|(state|attributes)(\.[^.()]+)+|age\((created_at|(state|attributes)(\.[^.()]+)+|[A-Za-z0-9_-]+\.(latest|min|max)\((created_at|updated_at|(state|attributes)(\.[^.()]+)+)\))\)|previous\.p|[A-Za-z0-9_-]+\.each\(((state|attributes)(\.[^.()]+)+|age\((created_at|updated_at|(state|attributes)(\.[^.()]+)+)\))\))$ ChoiceOption: type: object additionalProperties: false required: - value properties: value: type: string minLength: 1 not: const: none_of_the_above description: type: string OptionsFrom: type: object description: | Choose among a blocking relation's documents in one call: one option per selected candidate, in the relation's order, `value` its `id` and each candidate described by its `label` fields, then `none_of_the_above`. `from` names a blocking relation of the recipe with `last_n` at most the `options.from` candidate limit (see Limits). A judged document with no candidates is answered `none_of_the_above` with `escape_p` 1 and no engine call. `dist` is a shortlist, not a ranking: past the first few candidates the probabilities tie at zero. Its `value` is a bare document id: a relation reads it only through a judged link (`LinkedJoin`), or counts by whether it is `none_of_the_above` (`CountWhere`). Its `horizon` must be `0s`. additionalProperties: false required: - from - label properties: from: $ref: '#/components/schemas/Name' description: A blocking relation in the recipe's `related`. label: type: array description: >- The candidate's paths that describe it to the engine, in order, rendered as text. A label is never an image path (`invalid_request`); to let an engine read a candidate's image, name `images.` in the relation's `fields`. minItems: 1 maxItems: 8 items: $ref: '#/components/schemas/RelationField' ChoiceThreshold: type: object additionalProperties: false required: - value - gte properties: value: type: string minLength: 1 gte: $ref: '#/components/schemas/Probability' ScoreLevel: type: object additionalProperties: false required: - value - label properties: value: type: integer label: type: string minLength: 1 description: type: string RecipeField: description: >- A `state` or `attributes` path of the judged document, or an image path `images.`, rendered verbatim; or a `ClippedPath`. oneOf: - type: string pattern: ^((state|attributes)(\.[^.]+)+|images\.[A-Za-z0-9_-]{1,128})$ - $ref: '#/components/schemas/ClippedPath' Duration: type: string description: A whole number and a unit (`s`, `m`, `h` or `d`), such as `7d` or `1h`. pattern: ^[1-9][0-9]*[smhd]$ Relation: type: object description: > Documents that point at the judged document. They are ordered newest `created_at` first unless `order` says otherwise: `window` keeps those created within the window, the order is applied, then `last_n` keeps the first n. At least one of `last_n` and `window` is required, and at least one of `fields` and `aggregate`. A relation reads at most the newest 1,000 documents, and an `order` sorts within them. With `join: {theirs: "id", mine: "attributes."}` the relation reads the one document the judged document points at, its referenced document, and takes no `window`. With `last_n` (up to 254) it is a named list instead: the attribute holds an array of ids, and the relation reads the documents it names in list order unless `order` says otherwise, each once, at most `last_n` of them. An id that names nothing, or a document outside `match`, renders nothing and is listed in the evaluation's `related_documents` as `unresolved`. An attribute a named list reads holds at most 254 distinct ids. With `join: {theirs: "attributes.", mine: "attributes."}` it is a blocking relation over the documents that share the judged document's key value. It needs `window` unless it sets `relative`, takes `last_n` up to 254 (up to the `options.from` candidate limit when a `choice` chooses among it with `options.from`), and a change to its block re-judges the block's judged documents. With `relative`, each judged document reads the `last_n` documents of its block created just before it (or just after it), in creation order, and a change re-judges only the judged documents whose documents it moved. `fields`, `aggregate` and `match` may read other judgments' answers (`answers..`), banded or cut, and the newest outcome posted for each related document (`outcomes..value`), shown as posted: the one with the latest `observed_at`, of two at the same time the one posted last, about the document's current life and of the judgment's current answer type. Posting one re-judges the judgments that read it, within their `rolling_limit`. A relation reads at most 4 judgments' answers or outcomes. additionalProperties: false required: - match - join allOf: - description: A relation renders fields or aggregates. if: not: required: - fields then: required: - aggregate - description: >- A relation over the documents that point at the judged one is bounded by `last_n` or `window`. if: properties: join: type: object properties: mine: const: id then: anyOf: - required: - last_n - required: - window - description: A relation that reads the referenced document takes no `window`. if: properties: join: type: object properties: theirs: const: id then: not: required: - window - description: >- A relation that reads the referenced document takes `last_n` up to 254, as a named list. if: properties: join: type: object properties: theirs: const: id then: properties: last_n: type: integer maximum: 254 - description: A blocking relation reads at most 254 documents. if: properties: join: $ref: '#/components/schemas/BlockingJoin' then: properties: last_n: type: integer maximum: 254 - description: A blocking relation needs `window` unless it sets `relative`. if: not: required: - relative properties: join: $ref: '#/components/schemas/BlockingJoin' then: required: - window - description: >- Only a blocking relation takes `relative`, and then neither `window` nor `order`. if: required: - relative then: properties: join: $ref: '#/components/schemas/BlockingJoin' not: anyOf: - required: - window - required: - order - description: >- Only a relation over the documents that point at the judged one, or a named list, takes `order`. if: required: - order properties: order: $ref: '#/components/schemas/RelationOrder' then: anyOf: - properties: join: $ref: '#/components/schemas/ReferringJoin' - required: - last_n properties: join: $ref: '#/components/schemas/ReferencedJoin' properties: match: $ref: '#/components/schemas/RelationMatch' description: Which documents the relation reads. join: $ref: '#/components/schemas/RelationJoin' last_n: type: integer minimum: 1 maximum: 1000 description: >- Keep the first n in the relation's order, newest first by default. On a relation that reads a referenced document it makes a named list, keeping the first n documents the list names, in list order unless `order` says otherwise, at most 254; at most 254 on a blocking relation, and at most the `options.from` candidate limit (see Limits) when `options.from` chooses among it. window: $ref: '#/components/schemas/Duration' description: >- Keep documents created within this long before the later of the judged document's newest write and its newest related creation, at the answer's watermark. An edit to an old related document never moves the window. Not on a relation that reads a referenced document. fields: type: array description: >- Paths to include from each document, rendered under `records` in the relation's order. An entry may be a `BandedField`, which renders a number as its band, or a `ClippedPath`, which renders the first `max_chars` characters. An entry may also be a judgment's `answers..value` and `answers..thresholds.` as they are, and its `.p`, `.score` and `.dist.