> ## Documentation Index > Fetch the complete documentation index at: https://docs.brussle.com/llms.txt > Use this file to discover all available pages before exploring further. # Get the calibration report for the active version > How well the active version's answers match the outcomes joined to its evaluations, per engine epoch, before and after calibration. Calibration is fitted per judgment version and per engine epoch, first within minutes of the judgment's first posted outcome or first outcome rules, and then nightly. From 100 outcomes it is a conservative correction (`stage: early`); with many more, one that follows the engine's own pattern closely (`stage: full`). An epoch also needs 20 outcomes of each class: 20 `true` and 20 `false` for a bool, and two values with 20 each for a choice or score. Until then it has no calibration, its `calibrated` metrics are null, and `not_fitted` says why and what to post. A fit is applied only when it beats the raw answers on outcomes it was not fitted on (`held_out`); otherwise answers stay raw and `not_fitted` is `raw_better`. `unmatched_outcomes` counts the outcomes that joined nothing, by reason. `coverage` and `warnings` say where the outcomes sit, and warn when most come from answers a person reviewed. The raw metrics are computed at request time and include every outcome posted so far. See [calibration](/concepts/calibration). On a prefix path the report pools the outcomes of every namespace under the prefix, which is the fit inherited answers use. ## OpenAPI ````yaml /api-reference/openapi.yaml get /namespaces/{ns}/judgments/{name}/calibration openapi: 3.1.0 info: title: Brussle API version: '1' license: name: Proprietary identifier: LicenseRef-Proprietary description: | The `/v1` contract. It is locked: changes within `/v1` are additive only, and anything breaking is `/v2`. - **Base URL.** `https://api.brussle.com/v1`. - **Evolution.** Response objects may gain fields and enums may gain values. Clients must ignore what they do not know. Request objects reject unknown fields. - **Path segments.** Namespace names (`{ns}`) and document ids (`{id}`) may contain `/`. Send it percent-encoded as `%2F`, so that `acme/prod/tenant_123` is `acme%2Fprod%2Ftenant_123`. - **Templates.** The judgment routes also take a namespace prefix ending in `/*`, such as `acme%2Fprod%2F*`. A judgment created there is a template: every namespace under the prefix inherits it, including ones created later. The most specific matching prefix wins. - **Thresholds are settings.** Named thresholds belong to the judgment, not to a version. They are evaluated at read time against the raw fields, so a change applies at once to every answer, with no recompute and no new version. - **Entity judgments.** A judgment can read the documents that point at the judged one, through `context.related`, and apply only to documents matching `applies_to`. Its answers carry a `watermark`, and creating one that runs `on_change` returns a replay estimate of its monthly cost until you confirm. - **Referenced documents.** A relation can also read the one document the judged document points at, through `join: {theirs: "id", mine: "attributes."}`. A change to what the relation renders of that document re-judges the judged documents that point at it and are inside `freshness.fanout.scope`: a fan-out. - **Relations.** A relation can read plain judgments' answers (banded or cut, never summed), or be a blocking relation over the documents that share a key, which a `choice` can choose among with `options.from`. A recipe can read the document as the judgment last saw it (`context.previous`). A judgment that can fan out confirms its rolling limits once, at create; past them, work is deferred, reads `stale` with an event, and catches up on its own. Nothing waits for a confirm at runtime. Groups keep aggregates per key value for reporting. - **Idempotency.** Every write operation is idempotent by construction: a retried append adds nothing while its values are still in the array, and a retried upsert or patch sets the same content again. Every request that changes something also takes an `Idempotency-Key`, so that creates, activations, backfills, draws and the rest can be retried safely too: a retry with the key gets the first response back, verbatim and marked `Idempotent-Replayed: true`, instead of running again (see the parameter). - **Errors.** Every error is the `Error` envelope; the HTTP status follows the code (see each response). - **Rate limits.** Every response to a request whose key was accepted carries `X-RateLimit-Limit` and `X-RateLimit-Remaining`, which count the key's requests a second; a request refused before its key is checked (`unauthorized`, or a route that does not exist) carries neither. A `rate_limited` response adds `Retry-After`, except a daily allowance that is used up, which says when it renews in `details.resets_at` instead. Every `503` carries `Retry-After` too. Reads of one namespace also have a concurrency limit: past 32 gets and queries in flight at once, one more is `rate_limited` with `Retry-After` and `details.concurrent_reads`, whatever the rate-limit headers show. - **Usage.** Every response that bills carries `usage`. Judging is billed per judgment, engine-neutral, by size class: one judgment answered counts by the tokens of its compiled context and its question together, 1 if standard (up to 2,000 tokens), 4 if large (up to 8,000) and 16 if extra-large (up to 32,000, the engine maximum). No response ever carries an engine's price. servers: - url: https://api.brussle.com/v1 security: - apiKey: [] tags: - name: Namespaces description: Namespaces, their settings and cache warming. - name: Documents description: Writes, point reads and queries. - name: Judgments description: >- Judgment definitions, versions, settings, activation, backfill and template detach. - name: Outcomes and calibration description: | Post what actually happened to judged documents, and read how well answers match it: the calibration report, calibrated answers and threshold recommendations. See [calibration](/concepts/calibration). - name: Groups description: > Groups keep aggregates per value of a key attribute, a group-by you declare for reporting; a template's tenant summary publishes each tenant's groups, banded, as a document the platform can judge. - name: Jobs description: >- Backfill, shadow, periodic, deletion, reference index, evaluation export, resync, group build, discovery and simulation jobs. - name: Engines description: The engine registry. - name: Organization description: | Settings of the whole organization: prefixes marked non-production, which the tenant fee and template calibration pools leave out. - name: Subscriptions description: | Saved queries that send an event when a document starts or stops matching: the query's filter grammar, on a namespace or a template prefix, with the fields each event carries. - name: Webhooks description: | Webhook endpoints, the events feed, the delivery log, redelivery and recovery. Every event type and its body is under `webhooks`. Events are delivered at least once, signed as Standard Webhooks, and kept for 30 days. Webhooks are on every plan, and deliveries are not billed; plans set how many endpoints an organization has. paths: /namespaces/{ns}/judgments/{name}/calibration: parameters: - $ref: '#/components/parameters/NamespaceOrTemplate' - $ref: '#/components/parameters/JudgmentName' get: tags: - Outcomes and calibration summary: Get the calibration report for the active version description: | How well the active version's answers match the outcomes joined to its evaluations, per engine epoch, before and after calibration. Calibration is fitted per judgment version and per engine epoch, first within minutes of the judgment's first posted outcome or first outcome rules, and then nightly. From 100 outcomes it is a conservative correction (`stage: early`); with many more, one that follows the engine's own pattern closely (`stage: full`). An epoch also needs 20 outcomes of each class: 20 `true` and 20 `false` for a bool, and two values with 20 each for a choice or score. Until then it has no calibration, its `calibrated` metrics are null, and `not_fitted` says why and what to post. A fit is applied only when it beats the raw answers on outcomes it was not fitted on (`held_out`); otherwise answers stay raw and `not_fitted` is `raw_better`. `unmatched_outcomes` counts the outcomes that joined nothing, by reason. `coverage` and `warnings` say where the outcomes sit, and warn when most come from answers a person reviewed. The raw metrics are computed at request time and include every outcome posted so far. See [calibration](/concepts/calibration). On a prefix path the report pools the outcomes of every namespace under the prefix, which is the fit inherited answers use. operationId: getCalibrationReport responses: '200': description: The report. headers: X-RateLimit-Limit: $ref: '#/components/headers/RateLimitLimit' X-RateLimit-Remaining: $ref: '#/components/headers/RateLimitRemaining' content: application/json: schema: $ref: '#/components/schemas/CalibrationReport' '400': $ref: '#/components/responses/InvalidRequest' '401': $ref: '#/components/responses/Unauthorized' '403': $ref: '#/components/responses/Forbidden' '404': $ref: '#/components/responses/NotFound' '409': $ref: '#/components/responses/Conflict' '413': $ref: '#/components/responses/TooLarge' '429': $ref: '#/components/responses/RateLimited' '500': $ref: '#/components/responses/Internal' components: parameters: NamespaceOrTemplate: name: ns in: path required: true description: | A namespace name, or a template prefix ending in `/*`, with any `/` sent as `%2F`: `acme%2Fprod%2Ftenant_123` or `acme%2Fprod%2F*`. schema: $ref: '#/components/schemas/NamespaceOrTemplate' JudgmentName: name: name in: path required: true schema: $ref: '#/components/schemas/Name' headers: RateLimitLimit: description: >- Requests allowed per second for this key. Only on responses to a request whose key was accepted. schema: type: integer minimum: 0 RateLimitRemaining: description: >- Requests left in the current one-second window for this key. It does not count the per-namespace limit on reads in flight. schema: type: integer minimum: 0 RetryAfter: description: Seconds to wait before retrying. schema: type: integer minimum: 0 schemas: CalibrationReport: type: object description: | How well the active version's answers match their outcomes. `epochs` holds one entry per engine epoch with outcomes, newest first. An epoch is the `engine_version` its evaluations were computed under: each Jev epoch label, or the exact version for a pinned engine. required: - judgment - type - version - outcomes - outcomes_by_source - rule_outcomes_paused - unmatched_outcomes - censored_predictions - epochs - lift properties: judgment: $ref: '#/components/schemas/Name' type: type: string enum: - bool - choice - score version: $ref: '#/components/schemas/JudgmentVersionNumber' description: The active version the report is for. outcomes: type: integer format: int64 minimum: 0 description: >- Outcomes joined to evaluations of this version, across every epoch, implicit negatives included. outcomes_by_source: $ref: '#/components/schemas/OutcomesBySource' rule_outcomes_paused: type: integer format: int64 minimum: 0 description: >- Rule outcomes left out because the organization's plan did not include outcome rules when they were observed. They are kept, and count again only for outcomes observed while the plan includes rules. 0 on Team and Scale throughout. unmatched_outcomes: $ref: '#/components/schemas/UnmatchedOutcomes' censored_predictions: $ref: '#/components/schemas/CensoredPredictions' selection: $ref: '#/components/schemas/SelectionAccuracy' match_misses: $ref: '#/components/schemas/MatchMisses' epochs: type: array items: $ref: '#/components/schemas/CalibrationEpoch' lift: description: | How much calibration has cut this version's error since the learning loop started, on held-out outcomes. On every plan; the dashboard shows its history on Team and above, and its headline as a preview on Developer. Null for a composite, whose `p` is its combiner's, and on a template's tenant: the template's report has the pool's lift. oneOf: - $ref: '#/components/schemas/CalibrationLift' - type: 'null' NamespaceOrTemplate: type: string description: | A namespace name, or a template prefix: a namespace path ending in `/*`, such as `acme/prod/*`, which every namespace under `acme/prod/` inherits judgments from. Up to 256 bytes. Never exactly `.` or `..`, which a URL path can't carry. pattern: ^(?!\.\.?$)[A-Za-z0-9._:/-]+(/\*)?$ minLength: 1 maxLength: 256 Name: type: string description: >- A judgment, attribute or threshold name. Names are path segments in field references, so they never contain `.`. pattern: ^[A-Za-z0-9_-]+$ minLength: 1 maxLength: 128 JudgmentVersionNumber: type: integer format: int32 minimum: 1 OutcomesBySource: type: object description: Joined outcomes by where they came from, summing to `outcomes`. required: - posted - rule - queue - implicit properties: posted: type: integer format: int64 minimum: 0 description: >- Posted to `POST /namespaces/{ns}/outcomes` without a `queue_item_id`. rule: type: integer format: int64 minimum: 0 description: Derived from writes by the judgment's `outcomes.rules`. queue: type: integer format: int64 minimum: 0 description: >- Labels for documents the labelling queue drew, posted with their `queue_item_id`. implicit: type: integer format: int64 minimum: 0 description: Implicit negatives (`outcomes.implicit_negatives`). UnmatchedOutcomes: type: object description: | Outcomes, posted or derived by rules, that add no sample to the report's version, by reason. They are kept, and count if a later evaluation or window takes them. required: - no_evaluation - before_first_evaluation - after_horizon - same_window - after_positive - does_not_fit properties: no_evaluation: type: integer format: int64 minimum: 0 description: >- The document has no successful evaluation of the version; with a `0s` horizon, of the revision current at `observed_at`. Also an outcome observed while its document was deleted. before_first_evaluation: type: integer format: int64 minimum: 0 description: >- Observed at or before the document's first answer; with `0s`, before its first judged revision was written. after_horizon: type: integer format: int64 minimum: 0 description: >- An event (`true`, or a choice or score value) observed after a prediction window closed and before the next one opened. A `false` there answers the window that closed. same_window: type: integer format: int64 minimum: 0 description: >- Another outcome already labels its prediction window, such as a second event, or a posted outcome beat a derived one there. With `0s`, a derived outcome whose answer another outcome already labels, or a labelling-queue label whose answer keeps a later-recorded one. after_positive: type: integer format: int64 minimum: 0 description: >- Its prediction window opened at or after a `true` for the same document (until the document is deleted and created again). The event had already happened, so that answer predicts nothing. does_not_fit: type: integer format: int64 minimum: 0 description: >- Its value, or the evaluation's output, does not fit the version's type, such as a rule's `from` field holding something that is not an option. CensoredPredictions: type: object description: | Prediction windows with no outcome that are not counted as implicit negatives. All are 0 unless `outcomes.implicit_negatives` is on. required: - window_open - document_deleted - after_positive properties: window_open: type: integer format: int64 minimum: 0 description: The window has not closed yet. It counts once it closes. document_deleted: type: integer format: int64 minimum: 0 description: >- The document was deleted before the window closed, so no outcome does not mean "no". after_positive: type: integer format: int64 minimum: 0 description: >- The window opened at or after a `true` for the same document (until the document is deleted and created again), so the event had already happened. SelectionAccuracy: type: object description: >- For an `options.from` judgment, how often the pick equals the outcome where a match existed. Reported, never fitted. required: - accuracy - outcomes properties: accuracy: type: - number - 'null' minimum: 0 maximum: 1 description: Null with no outcome naming a match. outcomes: type: integer format: int64 minimum: 0 description: Outcomes naming a match. MatchMisses: type: object description: | For an `options.from` judgment, outcomes naming a document its evaluation did not read, by cause: created after the evaluation's watermark, cut by `last_n` (the number that says whether the oldest candidates matter), not in the block (a bad key), or the evaluation failed. required: - candidate_newer_than_evaluation - beyond_last_n - not_in_block - evaluation_failed properties: candidate_newer_than_evaluation: type: integer format: int64 minimum: 0 beyond_last_n: type: integer format: int64 minimum: 0 not_in_block: type: integer format: int64 minimum: 0 evaluation_failed: type: integer format: int64 minimum: 0 CalibrationEpoch: type: object description: | One epoch's metrics. For a composite judgment, `raw` measures the combined `p` and nothing is calibrated: `stage`, `fitted_at` and `calibrated` are null. On a template's tenant, `stage`, `fitted_at`, `not_fitted`, `held_out` and `calibrated` describe the fit the tenant's answers read: its own fit blended with the pool's, or the template's pooled fit (`tenant`). After a Jev drift, an epoch with too few outcomes of its own rests on a replay: the outcomes of earlier epochs, with the new model's answers to their stored contexts. `fit_source` says so, and `replay` has its progress; `raw`, `calibrated` and `coverage` then include the replayed outcomes, and `outcomes` does not. required: - engine - engine_version - outcomes - outcomes_by_source - fitted_on - fit_source - replay - rule_outcomes_paused - implicit_negatives - stage - fitted_at - from_previous_epoch - not_fitted - held_out - coverage - warnings - raw - calibrated - tenant - tenants properties: engine: type: string engine_version: type: string description: The epoch, as evaluations and answers record it. outcomes: type: integer format: int64 minimum: 0 description: >- Outcomes joined to this epoch's evaluations, implicit negatives included. Replayed outcomes are in `replay.outcomes`, not here. outcomes_by_source: $ref: '#/components/schemas/OutcomesBySource' fit_source: description: >- What the epoch's fit rests on, even before the next fit makes it. Null for a composite. oneOf: - $ref: '#/components/schemas/FitSource' - type: 'null' replay: description: >- The replay that re-fitted this epoch after a Jev drift. Null when none ran for it, and on a template's tenant (the template's report has it). oneOf: - $ref: '#/components/schemas/CalibrationReplay' - type: 'null' fitted_on: type: string enum: - all - queue description: | `queue` when this epoch's labelling-queue labels alone meet the minimums: the fit then rests on them only, weighted back to real traffic, and `coverage` and `warnings` describe them. Otherwise `all`: every outcome, each weighing 1. rule_outcomes_paused: type: integer format: int64 minimum: 0 description: >- This epoch's rule outcomes left out because the plan did not include outcome rules when they were observed. Not in `outcomes`. implicit_negatives: type: integer format: int64 minimum: 0 description: >- How many of `outcomes` are implicit negatives (`outcomes.implicit_negatives` on the judgment). 0 when the setting is off. stage: description: Null while nothing is fitted; `not_fitted` says why. oneOf: - $ref: '#/components/schemas/CalibrationStage' - type: 'null' fitted_at: description: >- When the current fit was made (nightly). Null when nothing is fitted. oneOf: - $ref: '#/components/schemas/Timestamp' - type: 'null' from_previous_epoch: type: boolean description: | True when this epoch has no fit of its own yet (after a Jev drift) and its answers read with the previous epoch's calibration, or for a composite judgment the previous epoch's combiner. They are marked `from_previous_epoch` too. not_fitted: description: | Why this epoch's answers have no calibration of their own: no fit yet, or `raw_better` when the fit did not beat the raw answers on held-out outcomes. Null when its fit is applied, and for a composite judgment. oneOf: - $ref: '#/components/schemas/CalibrationShortfall' - type: 'null' held_out: description: >- The test that decides whether the epoch's fit is applied. Null when nothing is fitted. On a tenant with its own fit, on the tenant's outcomes. oneOf: - $ref: '#/components/schemas/HeldOut' - type: 'null' coverage: description: >- The lowest and highest raw value among the outcomes the fit rests on (see `fitted_on`). Null when it has none. oneOf: - $ref: '#/components/schemas/RawRange' - type: 'null' warnings: type: array description: >- Where the outcomes sit that makes calibration less reliable. Empty when there is nothing to say. items: $ref: '#/components/schemas/CoverageWarning' raw: description: The metrics of the raw answers. Null when the epoch has no outcomes. oneOf: - $ref: '#/components/schemas/CalibrationMetrics' - type: 'null' calibrated: description: >- The same metrics after calibration, on the outcomes the fit was made on. Null when nothing is fitted. With `raw_better`, what the fit that was not applied would give. oneOf: - $ref: '#/components/schemas/CalibrationMetrics' - type: 'null' tenant: description: >- On a template's tenant, what its answers read in this epoch and why. Null on a template, on a namespace's own judgment, and for a composite. oneOf: - $ref: '#/components/schemas/TenantCalibration' - type: 'null' tenants: description: >- On a template, what its tenants with outcomes in this epoch read. Null elsewhere, and for a composite. oneOf: - $ref: '#/components/schemas/TenantSpread' - type: 'null' CalibrationLift: type: object description: | The learning loop's lift. Each nightly fit adds a point to the history when its held-out numbers changed, at most one a day per epoch, kept for 400 days. The headline compares, on the same held-out outcomes, the raw answers with what the answers read, in the current epoch only. required: - headline - withheld - model_changed - history properties: headline: description: >- The current epoch's lift. Null while the epoch has no fit; `withheld` says why. oneOf: - $ref: '#/components/schemas/LiftHeadline' - type: 'null' withheld: description: | Why there is no headline: the current epoch is below the calibration minimums (100 outcomes, 20 of each kind), or `awaiting_fit` when the next nightly fit adds its first point. Null when there is a headline. oneOf: - $ref: '#/components/schemas/CalibrationShortfall' - type: 'null' model_changed: description: >- Set when the version has points in an earlier engine epoch than the current one. The engine's model changed, and the headline counts from the current epoch's own first fit. oneOf: - $ref: '#/components/schemas/LiftModelChange' - type: 'null' history: type: array description: The version's points, oldest first, across its epochs. items: $ref: '#/components/schemas/LiftPoint' Error: type: object required: - error properties: error: type: object required: - code - message properties: code: $ref: '#/components/schemas/ErrorCode' message: type: string details: type: object description: >- Code-specific detail, such as the scan estimate on a refused query. Every `internal` error has `request_id`. properties: reason: type: string description: > Why, where one code has more than one cause. On `forbidden`, `payment_overdue`: a payment has been unpaid for 14 days, so the organization's keys are read-only until it is paid; reads still work, and paying restores writes within a few minutes. A key that is read-only itself is `forbidden` without a `reason`. On the events feed's `invalid_request`, `cursor_expired`. suggested_context: description: >- On a create without a context recipe, the recipe suggested from the namespace's documents, or null when there are none. oneOf: - $ref: '#/components/schemas/SuggestedContext' - type: 'null' FitSource: type: string enum: - outcomes - replay description: | `outcomes`: the epoch's own outcomes. `replay`: after a Jev drift, until the new epoch's own outcomes meet the minimums, the fit also rests on outcomes of earlier epochs, each with the new model's answer to the context the outcome labelled. CalibrationReplay: type: object description: | A replay after a Jev drift: the stored compiled contexts of the evaluations the judgment's outcomes labelled, judged again by the same engine in the new epoch. Replays are never answers and never billed. required: - status - reason - from_engine_version - planned - done - outcomes - started_at - finished_at properties: status: type: string enum: - running - done - stopped reason: description: >- Why a stopped replay stopped. `cost_cap`, a monthly limit on replays; `superseded`, the model changed again and the newer epoch has its own replay. Null otherwise. oneOf: - type: string enum: - cost_cap - superseded - type: 'null' from_engine_version: description: The newest earlier epoch whose outcomes are replayed. oneOf: - type: string - type: 'null' planned: type: integer format: int64 minimum: 0 description: The outcomes to replay, newest first. done: type: integer format: int64 minimum: 0 description: Of those, replayed so far. A document deleted since is skipped. outcomes: type: integer format: int64 minimum: 0 description: >- Replayed outcomes the epoch's fit rests on now; 0 once its own outcomes are enough. started_at: $ref: '#/components/schemas/Timestamp' finished_at: oneOf: - $ref: '#/components/schemas/Timestamp' - type: 'null' CalibrationStage: type: string description: > How far along a fit is. - `early`: it rests on the minimum outcomes or somewhat more. The correction is deliberately conservative and its uncertainty is wider, so expect it to move more from one fit to the next. - `full`: it rests on many more outcomes and follows the engine's own pattern closely. enum: - early - full Timestamp: type: string format: date-time description: RFC 3339, UTC. CalibrationShortfall: type: object description: | Why there is no calibration or no threshold recommendation. `message` says what to post, such as "Only positive outcomes so far: post outcomes for documents where it did not happen." when every outcome is `true`. required: - reason - message properties: reason: type: string description: | - `too_few_outcomes`: fewer than 100. - `too_few_positives`, `too_few_negatives`: fewer than 20 `true` or `false` outcomes (bool), or for a choice option, fewer than 20 outcomes of it or of the other options. - `too_few_values`: a choice or score has fewer than 2 values with 20 outcomes each. - `awaiting_fit`: the report only. There are enough outcomes, and the next fit, within a day, calibrates the epoch. - `raw_better`: the report only. The epoch was fitted, but the fit did not beat the raw answers on held-out outcomes, so its answers stay raw. When more than one applies, the first of these wins: a kind of outcome missing altogether (all `true` is `too_few_negatives`, all `false` `too_few_positives`, one value only `too_few_values`), then fewer than 100 outcomes, then fewer than 20 of a kind. enum: - too_few_outcomes - too_few_positives - too_few_negatives - too_few_values - awaiting_fit - raw_better message: type: string HeldOut: type: object description: > Whether a fit beats the raw answers on outcomes it was not fitted on: the raw answers' log loss, and the log loss of fits made without the outcome they score, over the same outcomes. The fit is applied only when `calibrated_log_loss` is lower. Fits made since the lift also give the ECEs and the gain's standard error; a tenant's own test does not. required: - raw_log_loss - calibrated_log_loss properties: raw_log_loss: type: number minimum: 0 calibrated_log_loss: type: number minimum: 0 raw_ece: type: number minimum: 0 calibrated_ece: type: number minimum: 0 standard_error: type: number minimum: 0 description: >- The standard error of the mean per-outcome gain in log loss, raw minus calibrated. RawRange: type: object description: >- A range of raw values, `p` or the probability of the most probable option or level. required: - lower - upper properties: lower: $ref: '#/components/schemas/Probability' upper: $ref: '#/components/schemas/Probability' CoverageWarning: type: object description: | Outcomes bunched where a reviewed sample puts them, so the fit is reliable only there. - `above_thresholds`: more than 80% of the outcomes are on answers that meet one of the judgment's thresholds. - `high_band`: a bool judgment only. More than 80% of the outcomes are on answers with `p` of 0.7 or more. required: - code - message properties: code: type: string enum: - above_thresholds - high_band message: type: string CalibrationMetrics: type: object description: | Metrics over the joined outcomes. For choice and score judgments, confidence is the probability of the most probable option or level, and a hit is an outcome equal to it (the exact level for scores). required: - accuracy - expected_calibration_error - log_loss - reliability properties: accuracy: $ref: '#/components/schemas/Probability' description: >- Share of hits: `p` of at least 0.5 on a `true` outcome or below it on `false` (bool), the exact option or level (choice, score). expected_calibration_error: type: number minimum: 0 description: >- The count-weighted mean gap between `mean_predicted` and `observed` over the reliability bins. log_loss: type: number minimum: 0 description: Mean negative log probability given to the observed outcome. mean_level_distance: type: number minimum: 0 description: >- Score judgments only. Mean absolute distance, in levels, between the most probable level and the observed one. reliability: type: array description: >- The reliability curve, ten equal-width bins of predicted probability from 0 to 1, lowest first. minItems: 10 maxItems: 10 items: $ref: '#/components/schemas/ReliabilityBin' TenantCalibration: type: object description: | What a template's tenant reads. Every tenant starts on the template's pooled calibration. On Scale, a tenant with 100 outcomes in an epoch, 20 of each kind, also gets its own fit, blended with the pool's and weighted more heavily as its own outcomes grow (`weight`). The tenant reads it only when it does best on the tenant's held-out outcomes and clearly beats what the tenant would read otherwise. required: - applied - reason - message - outcomes - weight - held_out properties: applied: type: string description: | - `tenant_shrunk`: the tenant's own fit, blended with the pool's. - `pooled`: the template's pooled fit. - `raw`: nothing; answers carry no `calibrated` object. enum: - tenant_shrunk - pooled - raw reason: type: string description: > - `tenant_better`: its own fit did best on its held-out outcomes, and clearly beat the pool. - `pooled_better`: its own fit did not clearly beat the pooled fit. - `raw_better`: the raw answers did best on its held-out outcomes. - `too_few_outcomes`: fewer than 100 outcomes in the epoch, or 20 of a kind, for a fit of its own; it reads the pool. - `awaiting_fit`: enough outcomes, and the next nightly fit makes its own fit. - `plan_required`: a tenant's own fit is on the Scale plan; every tenant reads the pool. - `non_production`: the tenant is under a non-production prefix, so the template's pool leaves it out: it reads the pool, adds nothing to it and gets no fit of its own. enum: - tenant_better - pooled_better - raw_better - too_few_outcomes - awaiting_fit - plan_required - non_production message: type: string outcomes: type: integer format: int64 minimum: 0 description: >- The tenant's outcomes its own fit was made on, or while it has none, its outcomes in the epoch. weight: type: number minimum: 0 maximum: 1 description: >- How much the tenant's own fit counts in the blend, rising with its outcomes. held_out: description: Its held-out test. Null without a fit of its own. oneOf: - $ref: '#/components/schemas/TenantHeldOut' - type: 'null' TenantSpread: type: object description: >- What a template's tenants with outcomes in an epoch read. The last three sum to `with_outcomes`; tenants with no outcomes read the pool. required: - with_outcomes - tenant_shrunk - pooled - raw properties: with_outcomes: type: integer format: int64 minimum: 0 tenant_shrunk: type: integer format: int64 minimum: 0 description: >- Tenants that read their own fit, blended with the pool's. 0 below Scale. pooled: type: integer format: int64 minimum: 0 raw: type: integer format: int64 minimum: 0 LiftHeadline: type: object description: | "Calibration cut this judgment's error by `error_reduction` since `since`, on `outcomes` outcomes." From the current epoch's newest fit, held-out numbers only. required: - metric - baseline - engine - engine_version - since - as_of - outcomes - applied - raw_log_loss - applied_log_loss - error_reduction - interval properties: metric: type: string enum: - log_loss description: The error measured, the one a fit is judged on. baseline: type: string enum: - raw description: What the error is compared with, the engine's raw answers. engine: type: string engine_version: type: string description: The current epoch. since: $ref: '#/components/schemas/Timestamp' description: The epoch's first fit. as_of: $ref: '#/components/schemas/Timestamp' description: The fit the numbers are from. outcomes: type: integer format: int64 minimum: 100 description: The outcomes that fit rests on, each held out once. applied: type: boolean description: >- Whether answers read the fit. False when the raw answers did better on held-out outcomes, so calibration cuts nothing. raw_log_loss: type: number minimum: 0 applied_log_loss: type: number minimum: 0 description: >- The held-out log loss of what answers read, the fit's when applied, else the raw answers'. error_reduction: type: number maximum: 1 description: >- `(raw_log_loss − applied_log_loss) / raw_log_loss`: 0.38 is "cut its error by 38%". 0 when not applied. interval: description: >- The 95% interval of `error_reduction`. Null when not applied. A lower end at or below 0 means the cut is not yet clear. oneOf: - $ref: '#/components/schemas/LiftInterval' - type: 'null' LiftModelChange: type: object required: - from - to - 'on' properties: from: type: string description: The epoch of the newest earlier point. to: type: string description: The current epoch. 'on': description: >- The day the current epoch began, from its label. Null for a pinned engine's version. oneOf: - type: string format: date - type: 'null' LiftPoint: type: object description: One nightly fit's held-out numbers. required: - fitted_at - version - engine - engine_version - outcomes - outcomes_by_source - fitted_on - source - stage - applied - raw_log_loss - calibrated_log_loss - raw_ece - calibrated_ece - standard_error properties: fitted_at: $ref: '#/components/schemas/Timestamp' version: $ref: '#/components/schemas/JudgmentVersionNumber' engine: type: string engine_version: type: string outcomes: type: integer format: int64 minimum: 0 outcomes_by_source: $ref: '#/components/schemas/OutcomesBySource' description: >- The outcomes the fit rests on; all `queue` when fitted on labelling-queue labels. fitted_on: type: string enum: - all - queue source: $ref: '#/components/schemas/FitSource' stage: $ref: '#/components/schemas/CalibrationStage' applied: type: boolean description: Whether answers read the fit. raw_log_loss: type: number minimum: 0 calibrated_log_loss: type: number minimum: 0 raw_ece: type: number minimum: 0 calibrated_ece: type: number minimum: 0 standard_error: type: number minimum: 0 description: >- The standard error of the mean per-outcome gain in log loss, raw minus calibrated. ErrorCode: type: string description: | HTTP status by code: `invalid_request` 400, `unauthorized` 401, `budget_exceeded` 402, `plan_required` 402, `forbidden` 403, `not_found` 404, `conflict` 409, `too_large` 413, `engine_version_unavailable` 422, `insufficient_labels` 422, `idempotency_key_reused` 422, `rate_limited` 429, `internal` 500, `engine_unavailable` 503, `unavailable` 503. enum: - invalid_request - unauthorized - forbidden - not_found - conflict - too_large - rate_limited - budget_exceeded - plan_required - engine_unavailable - engine_version_unavailable - unavailable - insufficient_labels - idempotency_key_reused - internal SuggestedContext: type: object description: | The recipe suggested for a create without `context`, in the refusal's `details.suggested_context`. From a sample of the documents the definition applies to: it keeps short text, numbers, booleans and small attributes, and leaves out ids, timestamps, links, binary-looking and very long values, and rarely present fields; `excluded` says why for each. `max_tokens` is set to keep the context in the standard size class, at most 2,000. The same documents always give the same recipe. required: - recipe - cost_per_answer - whole_state_cost_per_answer - sampled_documents - excluded properties: recipe: $ref: '#/components/schemas/ContextRecipe' cost_per_answer: type: - number - 'null' minimum: 0 description: >- Mean judgments per answer, counted by size class, over the sample with this recipe and the definition's question. The suggested `max_tokens` is at most 2,000, the standard ceiling, less the question's tokens, so a context that fills it stays standard with its question. whole_state_cost_per_answer: type: - number - 'null' minimum: 0 description: The same with the whole `state`. sampled_documents: type: integer minimum: 1 excluded: type: array description: >- Each path left out, and why, so you can put back what the question needs. items: type: object required: - path - reason properties: path: type: string reason: type: string enum: - id - timestamp - url - binary - long - rare - too_many Probability: type: number minimum: 0 maximum: 1 description: > Answers keep probabilities to about seven significant digits and return them as the shortest decimal that stands for the stored value, so an engine's 0.01 reads 0.01. Thresholds, query filters and threshold recommendations compare that decimal. ReliabilityBin: type: object description: >- Predictions whose probability (`p`, or the confidence for choice and score) falls in `[lower, upper)`; the last bin includes 1. required: - lower - upper - count - mean_predicted - observed properties: lower: $ref: '#/components/schemas/Probability' upper: $ref: '#/components/schemas/Probability' count: type: integer format: int64 minimum: 0 mean_predicted: description: Mean predicted probability in the bin. Null when the bin is empty. oneOf: - $ref: '#/components/schemas/Probability' - type: 'null' observed: description: >- Share of `true` outcomes (bool) or hits (choice, score) in the bin. Null when the bin is empty. oneOf: - $ref: '#/components/schemas/Probability' - type: 'null' TenantHeldOut: type: object description: | Mean log losses over a tenant's outcomes, each read with fits made without it. required: - raw_log_loss - pooled_log_loss - tenant_log_loss properties: raw_log_loss: type: number minimum: 0 pooled_log_loss: type: number minimum: 0 tenant_log_loss: type: number minimum: 0 description: The tenant's own fit blended with the pool's. LiftInterval: type: object required: - lower - upper properties: lower: type: number upper: type: number ContextRecipe: type: object description: | What the engine sees. A create without it is refused with a suggested recipe, unless `use_context` asks for the suggestion or the whole `state`; a version with no recipe sends the whole `state`, subject to the engine limit. additionalProperties: false properties: fields: type: array description: Paths included verbatim, in order. items: type: string pattern: ^(state|attributes)(\.[^.]+)+$ last_n: type: object description: Per array path, keep the last n elements. additionalProperties: type: integer minimum: 1 window: type: object description: >- Per array path whose elements have a timestamp field `at`, keep elements within the window. additionalProperties: $ref: '#/components/schemas/Duration' max_tokens: type: integer minimum: 1 description: Hard cap; the compiler truncates from the end of the last field. related: type: object description: > Up to 4 relations, by name: documents in the same namespace that point at the judged one, or the one document it points at. Each renders as a `related.` entry after `fields`, and a write that changes what it renders makes the judged document's answer `pending` until it is judged again (for a referenced document, only inside the re-judge scope; outside it, the answer is `stale` with `stale_reason` `referenced_changed`). minProperties: 1 maxProperties: 4 propertyNames: $ref: '#/components/schemas/Name' additionalProperties: $ref: '#/components/schemas/Relation' previous: $ref: '#/components/schemas/PreviousRecipe' Duration: type: string description: A whole number and a unit (`s`, `m`, `h` or `d`), such as `7d` or `1h`. pattern: ^[1-9][0-9]*[smhd]$ Relation: type: object description: | Documents that point at the judged document. They are ordered newest `created_at` first: `window` keeps those created within the window, then `last_n` keeps the newest n. At least one of `last_n` and `window` is required, and at least one of `fields` and `aggregate`. A relation reads at most the newest 1,000 documents. With `join: {theirs: "id", mine: "attributes."}` the relation reads the one document the judged document points at, its referenced document, and takes neither `last_n` nor `window`. With `join: {theirs: "attributes.", mine: "attributes."}` it is a blocking relation over the documents that share the judged document's key value. It needs `window`, takes `last_n` up to 254 (up to the `options.from` candidate limit when a `choice` chooses among it with `options.from`), and a change to its block re-judges the block's judged documents. `fields`, `aggregate` and `match` may read plain judgments' answers (`answers..`), banded or cut. additionalProperties: false required: - match - join allOf: - description: A relation renders fields or aggregates. if: not: required: - fields then: required: - aggregate - description: >- A relation over the documents that point at the judged one is bounded by `last_n` or `window`. if: properties: join: type: object properties: mine: const: id then: anyOf: - required: - last_n - required: - window - description: >- A relation that reads the referenced document takes neither `last_n` nor `window`. if: properties: join: type: object properties: theirs: const: id then: not: anyOf: - required: - last_n - required: - window - description: A blocking relation needs `window`, and reads at most 254 documents. if: properties: join: $ref: '#/components/schemas/BlockingJoin' then: required: - window properties: last_n: type: integer maximum: 254 properties: match: $ref: '#/components/schemas/RelationMatch' description: Which documents the relation reads. join: $ref: '#/components/schemas/RelationJoin' last_n: type: integer minimum: 1 maximum: 1000 description: >- Keep the newest n. Not on a relation that reads a referenced document; at most 254 on a blocking relation, and at most the `options.from` candidate limit (see Limits) when `options.from` chooses among it. window: $ref: '#/components/schemas/Duration' description: >- Keep documents created within this long before the later of the judged document's newest write and its newest related creation, at the answer's watermark. An edit to an old related document never moves the window. Not on a relation that reads a referenced document. fields: type: array description: >- Paths to include from each document, rendered under `records`, newest first. An entry may be a `BandedField`, which renders a number as its band. An entry may also be a plain judgment's `answers..value` and `answers..thresholds.` as they are, and its `.p`, `.score` and `.dist.