> ## Documentation Index > Fetch the complete documentation index at: https://docs.brussle.com/llms.txt > Use this file to discover all available pages before exploring further. # Get a calibration fit with its map > One fit, from the newest run that holds it, with the map or model that turns the engine's raw value into the calibrated one (`model`) and the range of raw values it was fitted on (`range`). From these you can reproduce any calibrated number an answer read with this fit shows. `not_found` when no run of the judgment holds the id. ## OpenAPI ````yaml /api-reference/openapi.yaml get /namespaces/{ns}/judgments/{name}/calibration/fits/{fit} openapi: 3.1.0 info: title: Brussle API version: '1' license: name: Proprietary identifier: LicenseRef-Proprietary description: | The `/v1` contract. It is locked: changes within `/v1` are additive only, and anything breaking is `/v2`. - **Base URL.** `https://api.brussle.com/v1`. - **Evolution.** Response objects may gain fields and enums may gain values. Clients must ignore what they do not know. Request objects reject unknown fields. - **Path segments.** Namespace names (`{ns}`) and document ids (`{id}`) may contain `/`. Send it percent-encoded as `%2F`, so that `acme/prod/tenant_123` is `acme%2Fprod%2Ftenant_123`. - **Templates.** The judgment routes also take a namespace prefix ending in `/*`, such as `acme%2Fprod%2F*`. A judgment created there is a template: every namespace under the prefix inherits it, including ones created later. The most specific matching prefix wins. - **Thresholds are settings.** Named thresholds belong to the judgment, not to a version. They are evaluated at read time against the raw fields, so a change applies at once to every answer, with no recompute and no new version. - **Entity judgments.** A judgment can read the documents that point at the judged one, through `context.related`, and apply only to documents matching `applies_to`. Its answers carry a `watermark`, and creating one that runs `on_change` returns a replay estimate of its monthly cost until you confirm. - **Referenced documents.** A relation can also read the one document the judged document points at, through `join: {theirs: "id", mine: "attributes."}`. A change to what the relation renders of that document re-judges the judged documents that point at it and are inside `freshness.fanout.scope`: a fan-out. - **Relations.** A relation can read other judgments' answers (banded or cut, never summed), up to three judgments deep and never in a cycle, or be a blocking relation over the documents that share a key, which a `choice` can choose among with `options.from`. A recipe can read the document as the judgment last saw it (`context.previous`). A judgment that can fan out confirms its rolling limits once, at create; past them, work is deferred, reads `stale` with an event, and catches up on its own. Nothing waits for a confirm at runtime. Groups keep aggregates per key value for reporting. - **Idempotency.** Every write operation is idempotent by construction: a retried append adds nothing while its values are still in the array, and a retried upsert or patch sets the same content again. Every request that changes something also takes an `Idempotency-Key`, so that creates, activations, backfills, draws and the rest can be retried safely too: a retry with the key gets the first response back, verbatim and marked `Idempotent-Replayed: true`, instead of running again (see the parameter). - **Slow changes become jobs.** Creating a judgment version, a judgment's `PATCH` and a backfill answer as usual when they finish within 20 s. When estimating the cost takes longer, the request answers `202` with a job of type `judgment_create`, `judgment_update` or `backfill_start` instead, and the job finishes the request. Poll `GET /jobs/{id}`: when the job is `done` or `failed`, its `result` is the response the request would have returned. The SDKs wait for it by default. - **Errors.** Every error is the `Error` envelope; the HTTP status follows the code (see each response). - **Rate limits.** A request must be inside two limits: requests a second for its key (1,000 by default), and requests a second for all the organization's API keys together (2,000 by default). A dashboard session counts only toward its user's own per-key limit. Past either, the request is `rate_limited` with `Retry-After`, and `details.rate_limit` (`per_key` or `per_org`) and `details.limit` say which limit refused it. A refused request counts toward neither limit. Each limit is counted over about a second, so a burst can be refused a little before it. Every response to a request whose key was accepted carries `X-RateLimit-Limit` and `X-RateLimit-Remaining`, for the limit with fewer requests left, or for the limit that refused the request. A request refused before its key is known (`unauthorized`, a route that does not exist, or a malformed namespace name) carries neither. A `rate_limited` response adds `Retry-After`, except a daily allowance that is used up, which says when it renews in `details.resets_at` instead. Every `503` carries `Retry-After` too. Reads of one namespace also have a concurrency limit: past 32 gets and queries in flight at once, one more is `rate_limited` with `Retry-After` and `details.concurrent_reads`, whatever the rate-limit headers show. - **Usage.** Every response that bills carries `usage`. Judging is billed per judgment, by size class: one judgment answered counts by the tokens of its compiled context and its question together, 1 if standard (up to 2,000 tokens), 4 if large (up to 8,000) and 16 if extra-large (up to 32,000, the engine maximum). Questions the engine answers about one document in one request share it: the first counts in full, and each question after the first costs a quarter of a judgment, and a quarter more for each further 500 tokens of question text (at least a quarter of its size class). Each request then counts times its engine's weight (`jev` 1, `gpt-6-luna` 2.5; `pplx-decider`, not active yet, 1), rounded up to a quarter of a judgment. No response ever carries an engine's price. servers: - url: https://api.brussle.com/v1 security: - apiKey: [] tags: - name: Namespaces description: Namespaces, their settings and cache warming. - name: Documents description: Writes, point reads and queries. - name: Judgments description: >- Judgment definitions, versions, settings, activation, backfill and template detach. - name: Outcomes and calibration description: | Post what actually happened to judged documents, and read how well answers match it: the calibration report, calibrated answers and threshold recommendations. See [calibration](/concepts/calibration). - name: Groups description: > Groups keep aggregates per value of a key attribute, a group-by you declare for reporting; a template's tenant summary publishes each tenant's groups, banded, as a document the platform can judge. - name: Jobs description: >- Backfill, shadow, periodic, deletion, reference index, evaluation export, resync, group build, discovery and simulation jobs, and the jobs that finish a slow create, `PATCH` or backfill start. - name: Engines description: The engine registry. - name: Organization description: | Settings of the whole organization: prefixes marked non-production, which template calibration pools leave out. The mark changes nothing on the bill: namespaces under it count toward the tenant fee like any other. - name: Subscriptions description: | Saved queries that send an event when a document starts or stops matching: the query's filter grammar, on a namespace or a template prefix, with the fields each event carries. - name: Webhooks description: | Webhook endpoints, the events feed, the delivery log, redelivery and recovery. Every event type and its body is under `webhooks`. Events are delivered at least once, signed as Standard Webhooks, and kept for 30 days. Webhooks are on every plan, and deliveries are not billed; plans set how many endpoints an organization has. paths: /namespaces/{ns}/judgments/{name}/calibration/fits/{fit}: parameters: - $ref: '#/components/parameters/NamespaceOrTemplate' - $ref: '#/components/parameters/JudgmentName' - $ref: '#/components/parameters/CalibrationFitId' get: tags: - Outcomes and calibration summary: Get a calibration fit with its map description: | One fit, from the newest run that holds it, with the map or model that turns the engine's raw value into the calibrated one (`model`) and the range of raw values it was fitted on (`range`). From these you can reproduce any calibrated number an answer read with this fit shows. `not_found` when no run of the judgment holds the id. operationId: getCalibrationFit responses: '200': description: The fit. headers: X-RateLimit-Limit: $ref: '#/components/headers/RateLimitLimit' X-RateLimit-Remaining: $ref: '#/components/headers/RateLimitRemaining' content: application/json: schema: $ref: '#/components/schemas/CalibrationFitDetail' '400': $ref: '#/components/responses/InvalidRequest' '401': $ref: '#/components/responses/Unauthorized' '403': $ref: '#/components/responses/Forbidden' '404': $ref: '#/components/responses/NotFound' '429': $ref: '#/components/responses/RateLimited' '500': $ref: '#/components/responses/Internal' components: parameters: NamespaceOrTemplate: name: ns in: path required: true description: | A namespace name, or a template prefix ending in `/*`, with any `/` sent as `%2F`: `acme%2Fprod%2Ftenant_123` or `acme%2Fprod%2F*`. schema: $ref: '#/components/schemas/NamespaceOrTemplate' JudgmentName: name: name in: path required: true schema: $ref: '#/components/schemas/Name' CalibrationFitId: name: fit in: path required: true description: A fit's id, as answers and `GET .../calibration/fits` show it. schema: type: string pattern: ^fit_[0-9a-f]{16}$ headers: RateLimitLimit: description: >- Requests allowed per second under one of the two limits on an API key, per key or per organization; the one with fewer requests left, or on a `rate_limited` response the one that refused the request. A dashboard session reports its user's limit. Only on responses to a request whose key was accepted; never on `unauthorized`, on a route that does not exist, or on a malformed namespace name. schema: type: integer minimum: 0 RateLimitRemaining: description: >- Requests left in the current second under the limit `X-RateLimit-Limit` gives. It does not count the per-namespace limit on reads in flight. schema: type: integer minimum: 0 RetryAfter: description: Seconds to wait before retrying. schema: type: integer minimum: 0 schemas: CalibrationFitDetail: type: object allOf: - $ref: '#/components/schemas/CalibrationFitRow' required: - model - range properties: model: $ref: '#/components/schemas/CalibrationModel' features: description: | On a judgment with `features`, the fit that reads them beside the engine's answer, whether answers read it or not: `engine_weight` and each feature read on its own, as the report's `held_out_metrics.features` gives them, and `penalty`, how strongly that fit pulls every weight toward 0, recorded so a change of it shows per fit. Null on any other fit. oneOf: - type: object required: - engine_weight - per_feature - penalty properties: engine_weight: type: number per_feature: type: array items: $ref: '#/components/schemas/FeatureReading' penalty: type: number - type: 'null' range: description: >- The lowest and highest raw value the fit was made on; outside it, a calibrated number is `extrapolated`. oneOf: - $ref: '#/components/schemas/RawRange' - type: 'null' NamespaceOrTemplate: type: string description: | A namespace name, or a template prefix: a namespace path ending in `/*`, such as `acme/prod/*`, which every namespace under `acme/prod/` inherits judgments from. Up to 256 bytes. Never exactly `.` or `..`, which a URL path can't carry. pattern: ^(?!\.\.?$)[A-Za-z0-9._:/-]+(/\*)?$ minLength: 1 maxLength: 256 Name: type: string description: >- A judgment, attribute or threshold name. Names are path segments in field references, so they never contain `.`. pattern: ^[A-Za-z0-9_-]+$ minLength: 1 maxLength: 128 CalibrationFitRow: type: object description: One fit of one run, for one version and engine epoch. required: - run - fitted_at - cause - version - engine - engine_version - fit - applied - stage - outcomes - pinned - current properties: run: type: string description: The run, newest first; ids sort by when the run was made. fitted_at: $ref: '#/components/schemas/Timestamp' cause: $ref: '#/components/schemas/CalibrationRunCause' label: type: string description: A named run's label. filter: $ref: '#/components/schemas/CalibrationOutcomeFilter' version: $ref: '#/components/schemas/JudgmentVersionNumber' engine: type: string engine_version: type: string fit: description: >- The fit's id. Null when the fit did not beat the raw answers on held-out outcomes, or a named run's outcomes missed the minimums (`not_fitted`). oneOf: - type: string - type: 'null' applied: type: boolean description: >- Whether the fit beat the raw answers on held-out outcomes when it was made, so answers of its run read it. stage: type: string enum: - early - full - features description: >- `early` and `full` as `calibrated.stage` means them; `features` for a judgment whose `p` reads its features and the engine's `p` together. outcomes: type: integer format: int64 minimum: 0 pinned: type: boolean description: True when the run holds this fit because it is pinned. pinned_from: type: string description: For a pinned fit, the run it was pinned from. current: type: boolean description: True on the fits answers of this version and epoch read now. not_fitted: $ref: '#/components/schemas/CalibrationShortfall' outcomes_by_source: $ref: '#/components/schemas/OutcomesBySource' fitted_on: type: string enum: - all - queue source: $ref: '#/components/schemas/FitSource' held_out: description: The fit's held-out test as it was made, with its ranking on a bool. oneOf: - $ref: '#/components/schemas/FitHeldOut' - type: 'null' features_only: description: >- For a judgment with `features`, the same test of a fit on the features alone, without the engine's answer. Null elsewhere. oneOf: - $ref: '#/components/schemas/HeldOut' - type: 'null' role: type: string enum: - current - challenger description: >- `current`: the fit the run kept or made current. `challenger`: the run's new fit, kept beside it, listed directly under it. Absent on runs made before calibration kept the current fit. promotion: description: >- On a `current` row, what the run did with the version and epoch's fit. Null on a `challenger` row. oneOf: - $ref: '#/components/schemas/CalibrationPromotion' - type: 'null' held: description: Why the current fit was kept, when it was. oneOf: - $ref: '#/components/schemas/CalibrationHeld' - type: 'null' versus_current: description: >- The run's new fit against the current one, on the outcomes since the current fit's run. oneOf: - $ref: '#/components/schemas/CalibrationVersus' - type: 'null' CalibrationModel: description: > How the fit turns the engine's raw value into the calibrated one, given as the formula's terms. Write `logit(p) = ln(p / (1 - p))`, with `p` first clamped to [0.005, 0.995], and `σ(z) = 1 / (1 + e^-z)`. - `log_odds_linear`, a bool: calibrated `p` = `σ(a · logit(p) + b)`. - `temperature`: a choice or score's calibrated distribution is `softmax(ln(p_i) / t)` over its options or levels (each `p_i` first clamped as above); on a bool, `σ(logit(p) / t)`. - `points`: linear interpolation of the raw value through the points (`x`, `y`), flat beyond the ends; a choice or score maps each option's or level's probability and divides by their sum. - `features`, a bool with `features`: `σ(intercept + Σ wᵢ · (xᵢ - cᵢ) / sᵢ)` over `inputs`, where the first input, named `log_odds`, is `logit(p)` and the rest are the features' raw values; a missing value contributes 0. - `features_candidates`, a choice among candidates: `existence` is the `features` form on `1 - escape_p`, the chance a match exists; among the candidates with any probability, candidate `j`'s share of it is `softmax(log_probability_weight · ln(p_j) + Σ wᵢ · (x_jᵢ - cᵢ) / sᵢ)` over its `inputs`; `escape_p` is `1 -` the existence chance. - `features_options`, a fixed choice: `softmax` over the options of `log_probability_weight · ln(p_k) + intercept_against_escape + Σ wₖᵢ · (xᵢ - cᵢ) / sᵢ`, `weights` per option in `options`' order, and over `none_of_the_above` of `log_probability_weight · ln(p_escape)`. - `features_levels`, a score: the chance of a level at or below level `j` is `σ(cuts_j + cumulative_log_odds_weight · logit(C_j) - Σ wᵢ · (xᵢ - cᵢ) / sᵢ)`, `C_j` the engine's chance of a level at or below `j`; each level's chance is the difference. Null when the fit did not beat the raw answers. oneOf: - type: object required: - form - a - b properties: form: const: log_odds_linear a: type: number b: type: number - type: object required: - form - t properties: form: const: temperature t: type: number - type: object required: - form - x - 'y' properties: form: const: points x: type: array items: type: number 'y': type: array items: type: number - type: object required: - form - intercept - inputs - weights - centers - scales properties: form: const: features intercept: type: number inputs: type: array items: type: string weights: type: array items: type: number centers: type: array items: type: number scales: type: array items: type: number - type: object required: - form - existence - selection properties: form: const: features_candidates existence: type: object required: - intercept - inputs - weights - centers - scales properties: intercept: type: number inputs: type: array items: type: string weights: type: array items: type: number centers: type: array items: type: number scales: type: array items: type: number selection: type: object required: - log_probability_weight - inputs - weights - centers - scales properties: log_probability_weight: type: number inputs: type: array items: type: string weights: type: array items: type: number centers: type: array items: type: number scales: type: array items: type: number - type: object required: - form - log_probability_weight - intercept_against_escape - options - inputs - weights - centers - scales properties: form: const: features_options log_probability_weight: type: number intercept_against_escape: type: number options: type: array items: type: string inputs: type: array items: type: string weights: type: array items: type: array items: type: number centers: type: array items: type: number scales: type: array items: type: number - type: object required: - form - cumulative_log_odds_weight - levels - cuts - inputs - weights - centers - scales properties: form: const: features_levels cumulative_log_odds_weight: type: number levels: type: array items: type: integer cuts: type: array items: type: number inputs: type: array items: type: string weights: type: array items: type: number centers: type: array items: type: number scales: type: array items: type: number - type: 'null' FeatureReading: type: object description: | One feature of the fit on your outcomes, read on its own, on the same held-out outcomes as the reading it sits in. A `without` no lower than the reading's `log_loss` means the feature adds nothing the others do not; an `alone` near 0 means it predicts the outcome by itself. required: - name - weight - center - scale - missing - without - alone properties: name: $ref: '#/components/schemas/FeatureRef' weight: type: number description: >- How strongly one standard deviation of the feature moves `p`. A positive weight raises `p`, a negative one lowers it. center: type: number description: >- The feature's mean over the outcomes the fit was made on. A missing value reads as this. scale: type: number description: >- The feature's standard deviation over those outcomes, which the weight is per. missing: $ref: '#/components/schemas/Probability' description: >- The share of those outcomes whose answer had no value for the feature. without: type: object description: The held-out log loss of the fit made again without this feature. required: - log_loss properties: log_loss: type: number minimum: 0 alone: type: object description: >- The held-out log loss of a fit on this feature alone, without the engine's answer. required: - log_loss properties: log_loss: type: number minimum: 0 warning: type: string enum: - reads_like_the_outcome description: | `reads_like_the_outcome` when this number alone predicts your outcomes almost perfectly, below a tenth of what always answering the share of `true` outcomes scores. That usually means it is written once the outcome is known, so it will not be there when an answer is needed. It is a statement about your data: it never changes which fit answers read. Check the feature, and drop it in a new version if it is written after the fact. by_option: type: object description: >- A fixed choice only. The feature's weight on each option; `weight` is the largest of them. A positive weight moves answers toward the option. additionalProperties: type: number RawRange: type: object description: >- A range of raw values, `p` or the probability of the most probable option or level. required: - lower - upper properties: lower: $ref: '#/components/schemas/Probability' upper: $ref: '#/components/schemas/Probability' Error: type: object required: - error properties: error: type: object required: - code - message properties: code: $ref: '#/components/schemas/ErrorCode' message: type: string details: type: object description: >- Code-specific detail, such as the scan estimate on a refused query. Every `internal` error has `request_id`. properties: reason: type: string description: | Why, where one code has more than one cause. On the events feed's `invalid_request`, `cursor_expired`. suggested_context: description: >- On a create without a context recipe, the recipe suggested from the namespace's documents, or null when there are none. oneOf: - $ref: '#/components/schemas/SuggestedContext' - type: 'null' Timestamp: type: string format: date-time description: RFC 3339, UTC. CalibrationRunCause: type: string description: | Why a run ran. `nightly`: the scheduled run, or one brought forward (after outcomes first arrive, an engine drift or switch, or a lifted pin). `fit_now`: `POST .../calibration/fit`. `pin`: a pin, which writes a run with the pinned fit. `named`: a named run. `shadow`: a composite's combiner fitted by its confirmed shadow job. enum: - nightly - fit_now - pin - named - shadow CalibrationOutcomeFilter: type: object description: | The outcomes a named run is fitted on. An outcome left out still counts as an answer's outcome, so it never makes that answer an implicit negative. additionalProperties: false properties: sources: type: array description: >- Where outcomes came from: `posted`, derived by an outcome `rule`, a labelling-`queue` label, or `implicit` negatives. Empty or absent keeps every source. uniqueItems: true items: type: string enum: - posted - rule - queue - implicit since: $ref: '#/components/schemas/Timestamp' description: >- Outcomes observed at or after this time; an implicit negative when its window closed. until: $ref: '#/components/schemas/Timestamp' description: Outcomes observed before this time. JudgmentVersionNumber: type: integer format: int32 minimum: 1 CalibrationShortfall: type: object description: | Why there is no calibration or no threshold recommendation. `message` says what to post, such as "Only positive outcomes so far: post outcomes for documents where it did not happen." when every outcome is `true`. required: - reason - message properties: reason: type: string description: | - `too_few_outcomes`: fewer than 50 for calibration, 100 for a threshold recommendation. - `too_few_positives`, `too_few_negatives`: fewer than 15 `true` or `false` outcomes (bool) for calibration, 20 for a recommendation, or for a choice option, fewer than 20 outcomes of it or of the other options. - `too_few_values`: a choice or score has fewer than 2 values with 15 outcomes each. - `awaiting_fit`: the report only. There are enough outcomes, and the next fit, within the hour, calibrates the epoch. - `raw_better`: the report only. The epoch was fitted, but the fit did not beat the raw answers on held-out outcomes, so its answers stay raw. - `within_noise`: the report only. Below 100 outcomes the fit beat the raw answers on held-out outcomes by no more than one standard error (`gain_standard_errors`): calibration is working and not yet sure, and answers stay raw until it is. - `readers_need_full`: the report only. A judgment with `features` applies a fit from 100 outcomes, because thresholds and relations act on its number; the report shows the fit 100 outcomes would apply. When more than one applies, the first of these wins: a kind of outcome missing altogether (all `true` is `too_few_negatives`, all `false` `too_few_positives`, one value only `too_few_values`), then too few outcomes, then too few of a kind. enum: - too_few_outcomes - too_few_positives - too_few_negatives - too_few_values - awaiting_fit - raw_better - within_noise - readers_need_full message: type: string gain_standard_errors: type: number description: >- With `within_noise`, how many standard errors the fit's held-out gain over the raw answers is. OutcomesBySource: type: object description: Joined outcomes by where they came from, summing to `outcomes`. required: - posted - rule - queue - implicit properties: posted: type: integer format: int64 minimum: 0 description: >- Posted to `POST /namespaces/{ns}/outcomes` without a `queue_item_id`. rule: type: integer format: int64 minimum: 0 description: Derived from writes by the judgment's `outcomes.rules`. queue: type: integer format: int64 minimum: 0 description: >- Labels for documents the labelling queue drew, posted with their `queue_item_id`. implicit: type: integer format: int64 minimum: 0 description: Implicit negatives (`outcomes.implicit_negatives`). FitSource: type: string enum: - outcomes - replay description: | `outcomes`: the epoch's own outcomes. `replay`: after an engine drift, until the new epoch's own outcomes meet the minimums, the fit also rests on outcomes of earlier epochs, each with the new engine's answer to the context the outcome labelled; after a version was activated on another engine, on the outcomes of the version it replaced, the same way. FitHeldOut: type: object allOf: - $ref: '#/components/schemas/HeldOut' properties: ranking: type: object description: A bool's ranking on the same outcomes, raw and with the fit. required: - base_rate - raw - calibrated properties: base_rate: type: number minimum: 0 maximum: 1 raw: $ref: '#/components/schemas/FitRanking' calibrated: $ref: '#/components/schemas/FitRanking' HeldOut: type: object description: > Whether a fit beats the raw answers on outcomes it was not fitted on: the raw answers' log loss, and the log loss of fits made without the outcome they score, over the same outcomes. The fit is applied only when `calibrated_log_loss` is lower. Fits made since the lift also give the ECEs and the gain's standard error, and fits made since 2026-10-13 the Brier scores; a tenant's own test gives none of them. Recorded when the fit was made (`fitted_at`), on the answers as they were then: outcomes and answers that change later move `raw` and `calibrated` at once, and this at the next fit. required: - raw_log_loss - calibrated_log_loss properties: raw_log_loss: type: number minimum: 0 calibrated_log_loss: type: number minimum: 0 raw_ece: type: number minimum: 0 calibrated_ece: type: number minimum: 0 standard_error: type: number minimum: 0 description: >- The standard error of the mean per-outcome gain in log loss, raw minus calibrated. raw_brier: type: number minimum: 0 maximum: 1 description: | The raw answers' Brier score: the mean squared gap between the number and what happened, 0 to 1, lower is better, weighted as the log losses are. For a choice or score, the sum over the options of the squared gaps, halved, so a two-option choice reads as a yes-or-no does. Reported beside log loss; a fit is chosen on log loss. Absent on a fit made before it was recorded. calibrated_brier: type: number minimum: 0 maximum: 1 description: >- The Brier score of fits made without the outcome they score, over the same outcomes. holdout: type: string enum: - five_groups - per_document description: | How the outcomes were held out: `five_groups`, five groups of documents, each scored by a fit made without its group; or below 100 outcomes `per_document`, each document scored by a fit made without it. gain_standard_errors: type: number description: >- The gain over the raw answers in log loss, in standard errors. Below 100 outcomes a fit applies only above 1. raw_accuracy: $ref: '#/components/schemas/Probability' description: >- A choice or score only. How often the most probable option or level of the raw answers came true. Absent on a fit made before it was recorded. calibrated_accuracy: $ref: '#/components/schemas/Probability' description: The same for fits made without the outcome they score. raw_level_distance: type: number minimum: 0 description: >- A choice or score only. The mean distance, in positions, between the most probable level of the raw answers and the one that came true; the report shows it on a score. calibrated_level_distance: type: number minimum: 0 description: The same for fits made without the outcome they score. CalibrationPromotion: type: string description: > What a run did with the version and epoch's fit. - `first`: no fit was current (the first fit, a new epoch or version, or outcomes of another kind than the current fit was made on): the run's fit is current. - `promoted`: the run's fit was clearly better on the outcomes since the current fit's run, and moved the number by at least a point; or a promote asked for it. - `withdrawn`: the raw answers (a tenant's fallback) clearly beat the current fit on the outcomes since; the run's fit, or raw, is current. - `held`: the current fit is kept; `held` says why. - `pinned`: a pin holds the fit; the run's fit is scored beside it. enum: - first - promoted - withdrawn - held - pinned CalibrationHeld: type: string description: > Why the current fit was kept. - `too_few_outcomes_since`: fewer than 20 of each kind arrived since its run: no test. - `not_clearly_better`: the new fit did not beat it by more than two standard errors on those outcomes. - `too_small_a_move`: it did, but would move the number by less than 0.01 on average. - `pinned`: a pin holds it. - `too_few_outcomes`: the outcomes fell below the minimums, so nothing was fitted; the current fit is kept. enum: - too_few_outcomes_since - not_clearly_better - too_small_a_move - pinned - too_few_outcomes CalibrationVersus: type: object description: The run's new fit against the current one. required: - outcomes - gain - standard_error - shift properties: outcomes: type: integer format: int64 minimum: 0 description: >- Outcomes that arrived since the current fit's run, on which the two are compared. gain: description: >- The current fit's mean log loss minus the new fit's on them, the new fit read without the outcome it is scored on; positive favours the new fit. Null with fewer than 20 of each kind. oneOf: - type: number - type: 'null' standard_error: oneOf: - type: number minimum: 0 - type: 'null' shift: type: number minimum: 0 description: >- How far the new fit would move the number, on average over every outcome the judgment is fitted on; for a choice or score, the probability that moves between options. moved_picks: type: number minimum: 0 maximum: 1 description: >- On a choice or score with `features`, the share of those outcomes whose most probable option would change. FeatureRef: type: string description: | One number the judgment records beside every answer and its fit can read. It is one of: - An aggregate the recipe declares, as `.count`, `.count_where`, `.labelled` or `.()`, such as `tickets.count` or `invoices.sum(state.amount)`. A `min` or `max` over an answer's `p` or `score` gives the raw extreme, whatever band the context shows; a `latest` over an answer's threshold reads 1 or 0. A `latest` over an answer's `value` is not a number and is `invalid_request`. A `sum` or `avg` over an answer's number is a feature only. A `latest` over a `bool` outcome reads 1 or 0, over a `score` outcome its level; over a `choice` outcome it is `invalid_request`. - The judged document's own value, `state.` or `attributes.`: a number as itself, a boolean as 1 or 0, anything else missing. It needs no relation. - An age in fractional days, `age(created_at)`, `age()` or `age(.())` over an aggregate the recipe declares. It is measured as of the judged document's own `updated_at` at the revision judged, so a write to a related document never moves it, and a new revision of the judged document does. The value must be an RFC 3339 string; a number or any other value reads missing, and a later time reads negative. `age(updated_at)` is `invalid_request`: as of the judged revision it is always 0. - `previous.p`: the engine's own `p` the last time it judged the document with the same engine version, in the same life of the document; on a choice, its `1 - escape_p`. Missing on the first judgment, after the document is created again, after the engine version changes, and on a composite, which has none. On a score it is `invalid_request`. - On a choice among candidates (`options.from`) only, a number per candidate, `.each()` over the `options.from` relation: each candidate's `state.` or `attributes.`, or `age(created_at)`, `age(updated_at)` or `age()` of the candidate, as of the judged document's revision. It is recorded for each candidate the engine read, under the answer's `candidate_features`. A candidate's answers are `invalid_request`: keep the number on the candidate's state. A name the recipe declares as an aggregate always reads that aggregate. A feature that reads a path an outcome rule reads, one inside the other, is `invalid_request`. pattern: >- ^([A-Za-z0-9_-]+\.(count|count_where|labelled|(sum|avg|min|max|latest)\((state|attributes)(\.[^.()]+)+\)|(min|max|latest)\((created_at|updated_at)\)|(sum|avg|min|max)\(answers\.[A-Za-z0-9_-]+\.(p|score|dist\.[^()]+)\)|latest\(answers\.[A-Za-z0-9_-]+\.thresholds\.[A-Za-z0-9_-]+\)|latest\(outcomes\.[A-Za-z0-9_-]+\.value\))|(state|attributes)(\.[^.()]+)+|age\((created_at|(state|attributes)(\.[^.()]+)+|[A-Za-z0-9_-]+\.(latest|min|max)\((created_at|updated_at|(state|attributes)(\.[^.()]+)+)\))\)|previous\.p|[A-Za-z0-9_-]+\.each\(((state|attributes)(\.[^.()]+)+|age\((created_at|updated_at|(state|attributes)(\.[^.()]+)+)\))\))$ Probability: type: number minimum: 0 maximum: 1 description: > Answers keep probabilities to about seven significant digits and return them as the shortest decimal that stands for the stored value, so an engine's 0.01 reads 0.01. Thresholds, query filters and threshold recommendations compare that decimal. ErrorCode: type: string description: | HTTP status by code: `invalid_request` 400, `unauthorized` 401, `invalid_image` 400, `unknown_image` 400, `image_erased` 410, `budget_exceeded` 402, `plan_required` 402, `forbidden` 403, `not_found` 404, `conflict` 409, `too_large` 413, `engine_version_unavailable` 422, `insufficient_labels` 422, `idempotency_key_reused` 422, `rate_limited` 429, `internal` 500, `engine_unavailable` 503, `unavailable` 503. Any route can also answer `unavailable`, with `Retry-After`, when it did not finish within 35 s: it was stopped, and a retry runs it again. Creating a judgment version, a judgment's `PATCH` and a backfill's start answer within 20 s, with a job (`202`) when they need longer, and never begin their change after 25 s. So when one of them is answered `unavailable` this way, nothing changed and the retry makes the change once. enum: - invalid_request - invalid_image - unknown_image - image_erased - unauthorized - forbidden - not_found - conflict - too_large - rate_limited - budget_exceeded - plan_required - engine_unavailable - engine_version_unavailable - unavailable - insufficient_labels - idempotency_key_reused - internal SuggestedContext: type: object description: | The recipe suggested for a create without `context`, in the refusal's `details.suggested_context`. From a sample of the documents the definition applies to: it keeps short text, numbers, booleans and small attributes, and leaves out ids, timestamps, links, binary-looking and very long values, and rarely present fields; `excluded` says why for each. Each name in a document's `images` is one candidate, `images.`; image paths are kept only on an engine that reads images, and only while the context stays in the standard size class with them: images count by their size, so when it has to choose, the suggestion keeps the text and leaves out image fields, the last first, with the reason `image`. `max_tokens` is set to keep the context in the standard size class, at most 2,000. The same documents always give the same recipe. required: - recipe - cost_per_answer - whole_state_cost_per_answer - sampled_documents - excluded properties: recipe: $ref: '#/components/schemas/ContextRecipe' cost_per_answer: type: - number - 'null' minimum: 0 description: >- Mean judgments per answer, counted by size class at the engine's weight as the bill counts them, over the sample with this recipe and the definition's question. The suggested `max_tokens` is at most 2,000, the standard ceiling, less the question's tokens, so a context that fills it stays standard with its question. whole_state_cost_per_answer: type: - number - 'null' minimum: 0 description: The same with the whole `state`. sampled_documents: type: integer minimum: 1 excluded: type: array description: >- Each path left out, and why, so you can put back what the question needs. items: type: object required: - path - reason properties: path: type: string reason: type: string enum: - id - timestamp - url - binary - image - long - rare - too_many FitRanking: type: object required: - auroc properties: auroc: type: number minimum: 0 maximum: 1 precision_at_5: oneOf: - type: number minimum: 0 maximum: 1 - type: 'null' precision_at_10: oneOf: - type: number minimum: 0 maximum: 1 - type: 'null' ContextRecipe: type: object description: | What the engine sees. A create without it is refused with a suggested recipe, unless `use_context` asks for the suggestion or the whole `state`; a version with no recipe sends the whole `state`, subject to the engine limit, and no images. additionalProperties: false properties: fields: type: array description: > Paths included in order, each verbatim or, as a `ClippedPath`, cut to its first `max_chars` characters. A path is `state.`, `attributes.`, or an image path, `images.`, which reads that name's image or whole list; an image path takes no `max_chars`. An engine that reads images (its `limits.images` in `GET /engines`) reads each, counted by its tiles; an engine that reads none gets each as an `OmittedImage` with `engine_reads_no_images`, and the create warns. Images are written to documents, never sent with the judgment. items: $ref: '#/components/schemas/RecipeField' last_n: type: object description: >- Per array path in `state` or `attributes`, keep the last n elements. Not an image path. additionalProperties: type: integer minimum: 1 window: type: object description: >- Per array path whose elements have a timestamp field `at`, keep elements within the window. additionalProperties: $ref: '#/components/schemas/Duration' max_tokens: type: integer minimum: 1 description: >- Hard cap, in the judgment's engine's tokens. Over it, whole images are left out first (on an engine that reads them; those of relations, then `previous`, then fields, the last first; every image when the text alone is over), then relation records, then `previous` entries, then the end of the last field is cut. related: type: object description: > Up to 4 relations, by name: documents in the same namespace that point at the judged one, or the one document it points at. Each renders as a `related.` entry after `fields`, and a write that changes what it renders makes the judged document's answer `pending` until it is judged again (for a referenced document, only inside the re-judge scope; outside it, the answer is `stale` with `stale_reason` `referenced_changed`). minProperties: 1 maxProperties: 4 propertyNames: $ref: '#/components/schemas/Name' additionalProperties: $ref: '#/components/schemas/Relation' previous: $ref: '#/components/schemas/PreviousRecipe' RecipeField: description: >- A `state` or `attributes` path of the judged document, or an image path `images.`, rendered verbatim; or a `ClippedPath`. oneOf: - type: string pattern: ^((state|attributes)(\.[^.]+)+|images\.[A-Za-z0-9_-]{1,128})$ - $ref: '#/components/schemas/ClippedPath' Duration: type: string description: A whole number and a unit (`s`, `m`, `h` or `d`), such as `7d` or `1h`. pattern: ^[1-9][0-9]*[smhd]$ Relation: type: object description: > Documents that point at the judged document. They are ordered newest `created_at` first unless `order` says otherwise: `window` keeps those created within the window, the order is applied, then `last_n` keeps the first n. At least one of `last_n` and `window` is required, and at least one of `fields` and `aggregate`. A relation reads at most the newest 1,000 documents, and an `order` sorts within them. With `join: {theirs: "id", mine: "attributes."}` the relation reads the one document the judged document points at, its referenced document, and takes no `window`. With `last_n` (up to 254) it is a named list instead: the attribute holds an array of ids, and the relation reads the documents it names in list order unless `order` says otherwise, each once, at most `last_n` of them. An id that names nothing, or a document outside `match`, renders nothing and is listed in the evaluation's `related_documents` as `unresolved`. An attribute a named list reads holds at most 254 distinct ids. With `join: {theirs: "attributes.", mine: "attributes."}` it is a blocking relation over the documents that share the judged document's key value. It needs `window` unless it sets `relative`, takes `last_n` up to 254 (up to the `options.from` candidate limit when a `choice` chooses among it with `options.from`), and a change to its block re-judges the block's judged documents. With `relative`, each judged document reads the `last_n` documents of its block created just before it (or just after it), in creation order, and a change re-judges only the judged documents whose documents it moved. `fields`, `aggregate` and `match` may read other judgments' answers (`answers..`), banded or cut, and the newest outcome posted for each related document (`outcomes..value`), shown as posted: the one with the latest `observed_at`, of two at the same time the one posted last, about the document's current life and of the judgment's current answer type. Posting one re-judges the judgments that read it, within their `rolling_limit`. A relation reads at most 4 judgments' answers or outcomes. additionalProperties: false required: - match - join allOf: - description: A relation renders fields or aggregates. if: not: required: - fields then: required: - aggregate - description: >- A relation over the documents that point at the judged one is bounded by `last_n` or `window`. if: properties: join: type: object properties: mine: const: id then: anyOf: - required: - last_n - required: - window - description: A relation that reads the referenced document takes no `window`. if: properties: join: type: object properties: theirs: const: id then: not: required: - window - description: >- A relation that reads the referenced document takes `last_n` up to 254, as a named list. if: properties: join: type: object properties: theirs: const: id then: properties: last_n: type: integer maximum: 254 - description: A blocking relation reads at most 254 documents. if: properties: join: $ref: '#/components/schemas/BlockingJoin' then: properties: last_n: type: integer maximum: 254 - description: A blocking relation needs `window` unless it sets `relative`. if: not: required: - relative properties: join: $ref: '#/components/schemas/BlockingJoin' then: required: - window - description: >- Only a blocking relation takes `relative`, and then neither `window` nor `order`. if: required: - relative then: properties: join: $ref: '#/components/schemas/BlockingJoin' not: anyOf: - required: - window - required: - order - description: >- Only a relation over the documents that point at the judged one, or a named list, takes `order`. if: required: - order properties: order: $ref: '#/components/schemas/RelationOrder' then: anyOf: - properties: join: $ref: '#/components/schemas/ReferringJoin' - required: - last_n properties: join: $ref: '#/components/schemas/ReferencedJoin' properties: match: $ref: '#/components/schemas/RelationMatch' description: Which documents the relation reads. join: $ref: '#/components/schemas/RelationJoin' last_n: type: integer minimum: 1 maximum: 1000 description: >- Keep the first n in the relation's order, newest first by default. On a relation that reads a referenced document it makes a named list, keeping the first n documents the list names, in list order unless `order` says otherwise, at most 254; at most 254 on a blocking relation, and at most the `options.from` candidate limit (see Limits) when `options.from` chooses among it. window: $ref: '#/components/schemas/Duration' description: >- Keep documents created within this long before the later of the judged document's newest write and its newest related creation, at the answer's watermark. An edit to an old related document never moves the window. Not on a relation that reads a referenced document. fields: type: array description: >- Paths to include from each document, rendered under `records` in the relation's order. An entry may be a `BandedField`, which renders a number as its band, or a `ClippedPath`, which renders the first `max_chars` characters. An entry may also be a judgment's `answers..value` and `answers..thresholds.` as they are, and its `.p`, `.score` and `.dist.