> ## Documentation Index > Fetch the complete documentation index at: https://docs.brussle.com/llms.txt > Use this file to discover all available pages before exploring further. # Make an alternative a version, and see its shadow report > Creates the alternative's definition as a new version, as `POST /judgments` would, and starts its shadow job, as `activate` without `force` would (`202`). The shadow report carries `alternative`: the outcomes' verdict and how many alternatives it was corrected for when the version was created, beside the backfill's price in `recompute` and, on another engine, `calibration_replay`. Confirming the job is the one step that activates it. The new version's calibration is fitted as any new version's is, and its calibration report's `alternative_selection` shows the numbers it was chosen on beside the same numbers on outcomes observed since. On the Team plan and above. ## OpenAPI ````yaml /api-reference/openapi.yaml post /namespaces/{ns}/judgments/{name}/alternatives/{id}/version openapi: 3.1.0 info: title: Brussle API version: '1' license: name: Proprietary identifier: LicenseRef-Proprietary description: | The `/v1` contract. It is locked: changes within `/v1` are additive only, and anything breaking is `/v2`. - **Base URL.** `https://api.brussle.com/v1`. - **Evolution.** Response objects may gain fields and enums may gain values. Clients must ignore what they do not know. Request objects reject unknown fields. - **Path segments.** Namespace names (`{ns}`) and document ids (`{id}`) may contain `/`. Send it percent-encoded as `%2F`, so that `acme/prod/tenant_123` is `acme%2Fprod%2Ftenant_123`. - **Templates.** The judgment routes also take a namespace prefix ending in `/*`, such as `acme%2Fprod%2F*`. A judgment created there is a template: every namespace under the prefix inherits it, including ones created later. The most specific matching prefix wins. - **Thresholds are settings.** Named thresholds belong to the judgment, not to a version. They are evaluated at read time against the raw fields, so a change applies at once to every answer, with no recompute and no new version. - **Entity judgments.** A judgment can read the documents that point at the judged one, through `context.related`, and apply only to documents matching `applies_to`. Its answers carry a `watermark`, and creating one that runs `on_change` returns a replay estimate of its monthly cost until you confirm. - **Referenced documents.** A relation can also read the one document the judged document points at, through `join: {theirs: "id", mine: "attributes."}`. A change to what the relation renders of that document re-judges the judged documents that point at it and are inside `freshness.fanout.scope`: a fan-out. - **Relations.** A relation can read other judgments' answers (banded or cut, never summed), up to three judgments deep and never in a cycle, or be a blocking relation over the documents that share a key, which a `choice` can choose among with `options.from`. A recipe can read the document as the judgment last saw it (`context.previous`). A judgment that can fan out confirms its rolling limits once, at create; past them, work is deferred, reads `stale` with an event, and catches up on its own. Nothing waits for a confirm at runtime. Groups keep aggregates per key value for reporting. - **Idempotency.** Every write operation is idempotent by construction: a retried append adds nothing while its values are still in the array, and a retried upsert or patch sets the same content again. Every request that changes something also takes an `Idempotency-Key`, so that creates, activations, backfills, draws and the rest can be retried safely too: a retry with the key gets the first response back, verbatim and marked `Idempotent-Replayed: true`, instead of running again (see the parameter). - **Slow changes become jobs.** Creating a judgment version, a judgment's `PATCH` and a backfill answer as usual when they finish within 20 s. When estimating the cost takes longer, the request answers `202` with a job of type `judgment_create`, `judgment_update` or `backfill_start` instead, and the job finishes the request. Poll `GET /jobs/{id}`: when the job is `done` or `failed`, its `result` is the response the request would have returned. The SDKs wait for it by default. - **Errors.** Every error is the `Error` envelope; the HTTP status follows the code (see each response). - **Rate limits.** A request must be inside two limits: requests a second for its key (1,000 by default), and requests a second for all the organization's API keys together (2,000 by default). A dashboard session counts only toward its user's own per-key limit. Past either, the request is `rate_limited` with `Retry-After`, and `details.rate_limit` (`per_key` or `per_org`) and `details.limit` say which limit refused it. A refused request counts toward neither limit. Each limit is counted over about a second, so a burst can be refused a little before it. Every response to a request whose key was accepted carries `X-RateLimit-Limit` and `X-RateLimit-Remaining`, for the limit with fewer requests left, or for the limit that refused the request. A request refused before its key is known (`unauthorized`, a route that does not exist, or a malformed namespace name) carries neither. A `rate_limited` response adds `Retry-After`, except a daily allowance that is used up, which says when it renews in `details.resets_at` instead. Every `503` carries `Retry-After` too. Reads of one namespace also have a concurrency limit: past 32 gets and queries in flight at once, one more is `rate_limited` with `Retry-After` and `details.concurrent_reads`, whatever the rate-limit headers show. - **Usage.** Every response that bills carries `usage`. Judging is billed per judgment, by size class: one judgment answered counts by the tokens of its compiled context and its question together, 1 if standard (up to 2,000 tokens), 4 if large (up to 8,000) and 16 if extra-large (up to 32,000, the engine maximum). Questions the engine answers about one document in one request share it: the first counts in full, and each question after the first costs a quarter of a judgment, and a quarter more for each further 500 tokens of question text (at least a quarter of its size class). Each request then counts times its engine's weight (`jev` 1, `gpt-6-luna` 2.5; `pplx-decider`, not active yet, 1), rounded up to a quarter of a judgment. No response ever carries an engine's price. servers: - url: https://api.brussle.com/v1 security: - apiKey: [] tags: - name: Namespaces description: Namespaces, their settings and cache warming. - name: Documents description: Writes, point reads and queries. - name: Judgments description: >- Judgment definitions, versions, settings, activation, backfill and template detach. - name: Outcomes and calibration description: | Post what actually happened to judged documents, and read how well answers match it: the calibration report, calibrated answers and threshold recommendations. See [calibration](/concepts/calibration). - name: Groups description: > Groups keep aggregates per value of a key attribute, a group-by you declare for reporting; a template's tenant summary publishes each tenant's groups, banded, as a document the platform can judge. - name: Jobs description: >- Backfill, shadow, periodic, deletion, reference index, evaluation export, resync, group build, discovery and simulation jobs, and the jobs that finish a slow create, `PATCH` or backfill start. - name: Engines description: The engine registry. - name: Organization description: | Settings of the whole organization: prefixes marked non-production, which template calibration pools leave out. The mark changes nothing on the bill: namespaces under it count toward the tenant fee like any other. - name: Subscriptions description: | Saved queries that send an event when a document starts or stops matching: the query's filter grammar, on a namespace or a template prefix, with the fields each event carries. - name: Webhooks description: | Webhook endpoints, the events feed, the delivery log, redelivery and recovery. Every event type and its body is under `webhooks`. Events are delivered at least once, signed as Standard Webhooks, and kept for 30 days. Webhooks are on every plan, and deliveries are not billed; plans set how many endpoints an organization has. paths: /namespaces/{ns}/judgments/{name}/alternatives/{id}/version: parameters: - $ref: '#/components/parameters/NamespaceOrTemplate' - $ref: '#/components/parameters/JudgmentName' - $ref: '#/components/parameters/AlternativeId' post: tags: - Outcomes and calibration summary: Make an alternative a version, and see its shadow report description: > Creates the alternative's definition as a new version, as `POST /judgments` would, and starts its shadow job, as `activate` without `force` would (`202`). The shadow report carries `alternative`: the outcomes' verdict and how many alternatives it was corrected for when the version was created, beside the backfill's price in `recompute` and, on another engine, `calibration_replay`. Confirming the job is the one step that activates it. The new version's calibration is fitted as any new version's is, and its calibration report's `alternative_selection` shows the numbers it was chosen on beside the same numbers on outcomes observed since. On the Team plan and above. operationId: createVersionFromAlternative parameters: - $ref: '#/components/parameters/IdempotencyKey' responses: '202': description: The version is created and its shadow job awaits confirmation. content: application/json: schema: $ref: '#/components/schemas/Job' '400': $ref: '#/components/responses/InvalidRequest' '401': $ref: '#/components/responses/Unauthorized' '402': $ref: '#/components/responses/PlanRequired' '403': $ref: '#/components/responses/Forbidden' '404': $ref: '#/components/responses/NotFound' '409': $ref: '#/components/responses/Conflict' '429': $ref: '#/components/responses/RateLimited' '500': $ref: '#/components/responses/Internal' components: parameters: NamespaceOrTemplate: name: ns in: path required: true description: | A namespace name, or a template prefix ending in `/*`, with any `/` sent as `%2F`: `acme%2Fprod%2Ftenant_123` or `acme%2Fprod%2F*`. schema: $ref: '#/components/schemas/NamespaceOrTemplate' JudgmentName: name: name in: path required: true schema: $ref: '#/components/schemas/Name' AlternativeId: name: id in: path required: true description: An alternative's id, as `getAlternatives` lists it. schema: $ref: '#/components/schemas/AlternativeIdValue' IdempotencyKey: name: Idempotency-Key in: header description: | One key per logical request, reused only on its retries. A key belongs to one request: within your organization, the same method, path, query and body. For 24 hours after a successful response, a request with the key and the same body gets that response back verbatim, with `Idempotent-Replayed: true`, and runs nothing. Only a successful response is kept, so the retry of a request that failed runs again. While the first request runs or its response is kept, the key with a different request is `idempotency_key_reused` (422). A request sent while one with its key is still running is `rate_limited` with `Retry-After: 1`, without running: retry it to get the first one's response. In the rare case the key can't be checked, the request runs as if it had none. schema: type: string minLength: 1 maxLength: 255 schemas: Job: type: object description: > A job. A `shadow` job comes from activating a differing version without `force`. It starts in `awaiting_confirm` and runs its sample while it waits; `report` fills in when the sample is done. `confirm`, accepted once `report` is filled in, activates `version`, starts a backfill of the answers from versions that ask differently (`backfill_job_id`), and the job ends `done`; `cancel` leaves the active version unchanged. On a template prefix, `namespace` is the prefix. A `reference_index` job builds the index an entity judgment's relations need; it starts `running`, and `attribute` names the attribute it indexes. An `evaluation_export` job starts `running`, counts evaluations written in `progress.documents_done`, never spends, and has `export`. A `resync` job re-renders the readers of a changed threshold (`resync`) and re-judges the documents whose rendering flipped; a `group_build` job computes a new group's or aggregate's first aggregates with no fan-out (`group`); a `discover` job fills in `proposal`; a `simulation` job fills in `simulation`. Each starts `running`; `discover` and `simulation` never spend. Fan-out is not a job: it runs within the judgment's rolling limit and is deferred past it. A `judgment_create`, `judgment_update` or `backfill_start` job finishes a create, `PATCH` or backfill request that needed more than 20 s (that request's `202`). It starts `running` and ends `done` when the request succeeded or `failed` when it did not; either way `result` is the response the request would have returned, and `error` is the error's code and message when it failed. It never spends, and it cannot be paused or cancelled. required: - id - type - status - namespace - spend_usd - progress - created_at - updated_at properties: id: type: string type: type: string enum: - backfill - shadow - periodic - namespace_delete - reference_index - evaluation_export - resync - group_build - discover - simulation - judgment_create - judgment_update - backfill_start status: type: string enum: - estimating - awaiting_confirm - running - paused - done - failed - cancelled namespace: $ref: '#/components/schemas/NamespaceOrTemplate' judgment: $ref: '#/components/schemas/Name' estimate: oneOf: - $ref: '#/components/schemas/Estimate' - type: 'null' spend_usd: type: number minimum: 0 description: Judging billed by this job so far, in US dollars. progress: type: object required: - documents_done properties: documents_done: type: integer format: int64 minimum: 0 description: | Documents done so far. A `backfill` counts the documents it has judged, and only those in its scope (the judgment's `applies_to` and the backfill's `filters`), so the count climbs to `estimate.documents`. It passes over the rest, and documents the active version has already answered, without counting them. estimated_completion_at: oneOf: - $ref: '#/components/schemas/Timestamp' - type: 'null' error: type: - string - 'null' attribute: type: - string - 'null' pattern: ^attributes\.[A-Za-z0-9_-]+$ description: >- Only on `reference_index` jobs, as `attributes.`. The attribute whose reference index the job builds. Null for other jobs. resync: $ref: '#/components/schemas/ResyncDetail' description: Only on `resync` jobs. group: $ref: '#/components/schemas/Name' description: Only on `group_build` jobs, the group whose aggregates it computes. proposal: description: Only on `discover` jobs; null until done. oneOf: - $ref: '#/components/schemas/DiscoveryProposal' - type: 'null' simulation: description: Only on `simulation` jobs; null until done. oneOf: - $ref: '#/components/schemas/SimulationReport' - type: 'null' version: $ref: '#/components/schemas/JudgmentVersionNumber' description: Shadow jobs only. The version being activated. report: description: | Shadow jobs only. Null while the sample runs; `progress` counts the sampled documents. A composite version's job has a `CompositeShadowReport`, and when it has too few labels it fails with an `error` that starts with `insufficient_labels`. oneOf: - $ref: '#/components/schemas/ShadowReport' - $ref: '#/components/schemas/CompositeShadowReport' - type: 'null' backfill_job_id: type: - string - 'null' description: > Shadow jobs only. The `backfill` job that confirming started once the version was active, which judges again each document whose answer is from a version that asks differently. Its `estimate` is the report's `recompute`, which the confirm agreed to. Null until the job is `done`, and also then when there was nothing to judge again (the judgment has no other version that asks differently, such as a composite's first version, or the `recompute` counts no documents), or when the `recompute` did not fit what is left of the namespace's budget this billing period or the organization's balance. Then the switch still happened: start a backfill yourself, which shows its estimate first. export: $ref: '#/components/schemas/EvaluationExport' description: >- `evaluation_export` jobs only: the range, the evaluations written, and once done the download links. result: description: >- Only on `judgment_create`, `judgment_update` and `backfill_start` jobs. Null until the job is `done` or `failed`; then the response the request would have returned. oneOf: - $ref: '#/components/schemas/RequestResult' - type: 'null' warnings: type: array description: >- Only in the response of the activation that started a shadow job, such as `images_unread`. Absent when there are none, and on `GET /jobs/{id}`. items: $ref: '#/components/schemas/JudgmentWarning' created_at: $ref: '#/components/schemas/Timestamp' updated_at: $ref: '#/components/schemas/Timestamp' NamespaceOrTemplate: type: string description: | A namespace name, or a template prefix: a namespace path ending in `/*`, such as `acme/prod/*`, which every namespace under `acme/prod/` inherits judgments from. Up to 256 bytes. Never exactly `.` or `..`, which a URL path can't carry. pattern: ^(?!\.\.?$)[A-Za-z0-9._:/-]+(/\*)?$ minLength: 1 maxLength: 256 Name: type: string description: >- A judgment, attribute or threshold name. Names are path segments in field references, so they never contain `.`. pattern: ^[A-Za-z0-9_-]+$ minLength: 1 maxLength: 128 AlternativeIdValue: type: string pattern: ^alt_[0-9a-f]{16}$ description: An alternative's id, the same for the same definition in every run. examples: - alt_3f9a2c7e1b04d6a8 Estimate: type: object required: - documents - tokens - judgment_units - cost_usd - duration_s properties: documents: type: integer format: int64 minimum: 0 tokens: type: integer format: int64 minimum: 0 judgment_units: type: integer format: int64 minimum: 0 description: >- Estimated judgments, each counted by its size class times its engine's weight, from a sample of up to 1,000 documents. A backfill judges one judgment at a time, so each document's question counts in full; a composite's parts share one request, the first in full and each other a quarter, and a quarter more for each further 500 tokens of its question text. A whole number, rounded; `cost_usd` prices each quarter. cost_usd: type: number minimum: 0 description: >- What the backfill is expected to cost you, the judgments at the graduated prices of the tiers your organization will be in, counting what it has already used this billing period. duration_s: type: integer format: int64 minimum: 0 description: >- Expected wall-clock seconds if nothing else is judged meanwhile. Other judging comes first (new writes, catch-ups, other backfills) and makes it longer, so read it as the least to expect. downstream: type: array description: >- A backfill of a judgment whose answers other judgments read also re-judges them where a rendered band or cut flips; one line per reader. Absent when it has none. items: $ref: '#/components/schemas/DownstreamLine' Timestamp: type: string format: date-time description: RFC 3339, UTC. ResyncDetail: type: object description: | What a `resync` job re-renders. A confirmed threshold change that readers' relations name starts one (`cause: thresholds`), and so does a new calibration for a bool judgment with `features` (`cause: fit`), with no confirm, since the readers would otherwise read the old values. It goes through the judgment's answers once and re-judges each reader's document whose child now reads differently; those re-judges are billed as any re-judge and reach subscriptions in bulk. Until it reaches a document, that document's answer stays `fresh` under the old values. It spends nothing itself. required: - thresholds_before - thresholds_after - readers properties: thresholds_before: $ref: '#/components/schemas/ThresholdSettings' thresholds_after: $ref: '#/components/schemas/ThresholdSettings' readers: type: array items: $ref: '#/components/schemas/Name' cause: type: string enum: - thresholds - fit description: What it follows. DiscoveryProposal: type: object description: | A new version's `options`: the current ones plus the proposed ones, with a shadow run of that version over the sample. Nothing was created. required: - options - proposed - sampled - escape_documents - unlabelled properties: options: type: array description: Every option of the proposed version, current ones first. items: $ref: '#/components/schemas/ChoiceOption' proposed: type: array items: $ref: '#/components/schemas/ProposedOption' sampled: type: integer minimum: 0 maximum: 200 description: Escape documents in the sample. escape_documents: type: integer format: int64 minimum: 0 description: Escape documents in the window, sampled or not. unlabelled: type: integer minimum: 0 description: >- Sampled documents the proposed version still answered `none_of_the_above`. SimulationReport: type: object description: > What the proposed referenced document would flip, estimated from a weighted sample, and never written. required: - population - sampled - thresholds - noise - without_stored_answer - examples - excluded - fanout - larger_sample properties: population: type: integer format: int64 minimum: 0 description: In-scope judged documents that point at the document. sampled: type: integer minimum: 0 thresholds: type: object description: >- Per named threshold, flips from each sampled document's stored answer to its answer under the proposed document, as a write compares them; from its answer under the current document when it has no stored answer. The interval is the sample's, widened on each side by `noise`'s estimate. additionalProperties: $ref: '#/components/schemas/SimulatedFlips' noise: type: object description: >- Per named threshold, flips from each sampled document's stored answer to a new answer under the current document, with nothing changed. The engine's own variation; a write judges again, so about this many of its flips are noise. Measured on the documents with a stored answer; the interval is the sample's. additionalProperties: $ref: '#/components/schemas/SimulatedFlips' without_stored_answer: type: integer minimum: 0 description: >- Documents judged both ways with no stored answer from the active version. Their flips compare the two new answers, and `noise` leaves them out. examples: type: array description: Sampled documents that flipped, up to 20. maxItems: 20 items: $ref: '#/components/schemas/DocumentId' excluded: type: object description: Sampled documents left out, by reason. properties: failed: type: integer minimum: 0 description: Either evaluation failed. out_of_scope: type: integer minimum: 0 description: >- Gone by the time it was evaluated, such as deleted since it was sampled. fanout: $ref: '#/components/schemas/Estimate' description: >- The real fan-out if the document were written so, priced like a backfill of the in-scope documents. larger_sample: type: - integer - 'null' description: >- A sample size that would narrow the sample's intervals, before noise, enough to decide, when this one cannot; null otherwise. JudgmentVersionNumber: type: integer format: int32 minimum: 1 ShadowReport: type: object description: | What activating `version` would change, from a shadow sample of 1,000 random documents, or every document when there are fewer. Shadow evaluations are in the evaluation log with `shadow: true` and never produce answers. `current` is the answers of the active version for the sampled documents, and `candidate` the shadow evaluations. required: - documents - current - candidate - threshold_flips - recompute properties: downstream: type: array description: >- What activating the version costs the judgments that read its answers, from the sample's band and cut flips; one line per reader. Absent when it has none. items: $ref: '#/components/schemas/DownstreamLine' documents: type: integer format: int64 minimum: 0 description: Documents in the sample. current: $ref: '#/components/schemas/ShadowSide' candidate: $ref: '#/components/schemas/ShadowSide' threshold_flips: type: object description: | Per named threshold, the sampled documents whose threshold boolean would change. The candidate side uses the thresholds that apply once the version is active. propertyNames: $ref: '#/components/schemas/Name' additionalProperties: $ref: '#/components/schemas/ThresholdFlips' recompute: $ref: '#/components/schemas/Estimate' description: >- What a backfill of every document under the new version would cost, as a backfill estimate. calibration_replay: description: >- What confirming would bill to calibrate the new version, when it is on another engine than the active version and the judgment has outcomes. Null otherwise. oneOf: - $ref: '#/components/schemas/CalibrationReplayQuote' - type: 'null' alternative: $ref: '#/components/schemas/ShadowAlternative' description: >- When the candidate was created from an alternative (`createVersionFromAlternative`), the outcomes' verdict on it then. Absent otherwise. CompositeShadowReport: type: object description: > The shadow report of a composite version. The job judged documents with outcomes for this judgment and scored the composite on held-out labels, so every document is scored by a fit that never saw its label. The baseline gets the same treatment: its cut-off is fitted on the same outcomes, never the raw 0.5, so splitting gets no credit for what a fitted threshold alone would give. `verdict` is `better` only when the composite's accuracy beats the baseline's by more than the interval of the difference. Otherwise it is `not_better`, and the threshold recommender on the baseline is the cheaper fix. `confirm` activates the version either way, with the combiner fitted on all the documents. required: - type - documents - labels - composite - baseline - difference - parts - verdict - recompute properties: type: const: composite documents: type: integer format: int64 minimum: 0 description: Labelled documents judged and scored. labels: type: object description: The documents by their outcome. required: - 'true' - 'false' properties: 'true': type: integer format: int64 minimum: 0 'false': type: integer format: int64 minimum: 0 composite: $ref: '#/components/schemas/CrossValidated' baseline: $ref: '#/components/schemas/Baseline' difference: type: object description: >- The composite's held-out accuracy minus the baseline's, with its 95% interval over the documents (paired). required: - accuracy - interval properties: accuracy: type: number minimum: -1 maximum: 1 interval: $ref: '#/components/schemas/DifferenceInterval' parts: type: array description: >- Each part's weight in the combiner fitted on all the documents, on a common scale, so sizes compare. A negative weight means a yes points to `false`. items: $ref: '#/components/schemas/PartWeight' features: type: array description: >- Each feature's weight in the same fit, on the same scale as the parts'. items: $ref: '#/components/schemas/FeatureWeight' verdict: type: string enum: - better - not_better recompute: $ref: '#/components/schemas/Estimate' description: >- What a backfill of every document under the new version would cost, as a backfill estimate, on the new version's engine at its weight. The parts share one request, where the first counts in full and each other a quarter or more. EvaluationExport: type: object description: | An `evaluation_export` job's range and files. `files` is empty until the job is `done`; then `GET /jobs/{id}` links each file until `expires_at`, when the first file is deleted. Other routes that return the job leave `files` empty. When links can't be made right now, the files are listed with `url` null and `links_unavailable` says so: read the job again later. required: - since - until - judgments - evaluations - files - expires_at - links_unavailable properties: since: $ref: '#/components/schemas/Timestamp' until: $ref: '#/components/schemas/Timestamp' judgments: description: The judgments exported; null for every judgment. oneOf: - type: array items: $ref: '#/components/schemas/Name' - type: 'null' evaluations: type: integer format: int64 minimum: 0 description: Evaluations written so far, the same as `progress.documents_done`. files: type: array items: $ref: '#/components/schemas/EvaluationExportFile' expires_at: description: >- When the files are deleted, 7 days after the first was written. Null until the job is done, and when it wrote no file. oneOf: - $ref: '#/components/schemas/Timestamp' - type: 'null' links_unavailable: type: - string - 'null' description: >- Why some or all of `files` have no `url` right now, with a request id for support. Null when every file has its link. The job and its files are unaffected; read the job again later for new links. RequestResult: type: object description: | The response a create, `PATCH` or backfill request would have returned had it finished within 20 s: its HTTP `status` and its `body`. A `judgment_create` job's is `201` with the new version, or `200` with an estimate or a dry run; a `judgment_update` job's is `200` with the settings applied or an estimate; a `backfill_start` job's is `200` with the estimate or the started backfill's `job_id`. A status of 400 or more has an `Error` body, the same as the request would have had. required: - status - body properties: status: type: integer minimum: 200 maximum: 599 description: The HTTP status the request would have had. body: anyOf: - $ref: '#/components/schemas/CreateJudgmentResponse' - $ref: '#/components/schemas/CreateJudgmentDryRun' - $ref: '#/components/schemas/CreateJudgmentEstimate' - $ref: '#/components/schemas/UpdateJudgmentResponse' - $ref: '#/components/schemas/BackfillEstimate' - $ref: '#/components/schemas/JobStarted' - $ref: '#/components/schemas/Error' JudgmentWarning: type: object description: | `taxonomy_drift`: the escape share over the last 7 days (`escape_share`) is above `escape_alert`; it clears when the share falls back. `threshold_unresolved`: the `thresholds` it lists name a feature or an option the active version does not have, so they read `false` until set again; shown on `GET` and in the activation's response. `fit_changed`: on a bool judgment with `features`, what `p` reads changed after its thresholds on `p` were set, so a cut may now select different answers; it clears when a threshold is set again with `PATCH`. `images_unread`: the version names image paths, listed in `paths`, on an engine that reads no images, so the engine reads only each image's digest; only in the activation's response (the `Judgment`, or the shadow `Job`). required: - code - message properties: code: type: string enum: - taxonomy_drift - threshold_unresolved - fit_changed - images_unread message: type: string paths: type: array description: >- For `images_unread`, the image paths the definition names, such as `images.photo` or `images.photo of relation claims`. items: type: string thresholds: type: array description: For `threshold_unresolved`, the thresholds it names. items: $ref: '#/components/schemas/Name' escape_share: type: number minimum: 0 maximum: 1 since: $ref: '#/components/schemas/Timestamp' description: When the share rose above the alert, or when what `p` reads changed. Error: type: object required: - error properties: error: type: object required: - code - message properties: code: $ref: '#/components/schemas/ErrorCode' message: type: string details: type: object description: >- Code-specific detail, such as the scan estimate on a refused query. Every `internal` error has `request_id`. properties: reason: type: string description: | Why, where one code has more than one cause. On the events feed's `invalid_request`, `cursor_expired`. suggested_context: description: >- On a create without a context recipe, the recipe suggested from the namespace's documents, or null when there are none. oneOf: - $ref: '#/components/schemas/SuggestedContext' - type: 'null' DownstreamLine: type: object description: > One reader's cost of a change to a judgment whose answers its relations read: the re-judgments the change would cause a month (for a one-off such as a backfill, in the month it runs), counted from the answer history's band and cut flips over the last 30 days, priced by the reader's size class at your tiers. A reader's re-judgments move what its own readers read, so there is a line for each reader of a reader too, up to three levels down. Below the first level the count is an estimate from the reader's own past flips, not a bound. required: - judgment - judgments_per_month - cost_usd - level properties: judgment: $ref: '#/components/schemas/Name' description: The reader, a judgment whose relation reads this one's answers. judgments_per_month: type: integer format: int64 minimum: 0 description: >- Evaluations of the reader the change would cause a month. It is exact except for a child that moved between parents in the window, which the history does not record. cost_usd: type: number minimum: 0 description: >- Those evaluations at the reader's size class and your organization's tiers, in US dollars. level: type: integer minimum: 1 maximum: 3 description: >- 1 for a judgment that reads the changed one; 2 and 3 for one that reads a reader, through `via`. via: $ref: '#/components/schemas/Name' description: >- Present from level 2. The reader a level up whose re-judgments cause these. basis: type: string enum: - history - replay description: >- What the count is drawn from. `history` is the answer history of the judgment read. `replay` is used where there is none, either a judgment not judged in the last 30 days (each of its re-judgments counted as one of the reader's) or an activation's sampled replay. Absent on a line that is not a reader's. omitted: type: array description: >- Present with `lower_bound`. The judgments whose answer history the estimate left partly or wholly unread, at this level or above. items: $ref: '#/components/schemas/Name' lower_bound: type: boolean const: true description: >- Present when the estimate stopped reading answer history before the end, so the count is at least this many. ThresholdSettings: type: object description: | The judgment's named thresholds, a setting rather than part of a version. Each is a number for a bool judgment (true when `p` is at least it) or a score judgment (true when `score` is at least it), `{value, gte}` for a choice judgment (true when `dist[value]` is at least `gte`), or `{feature, gte}` or `{feature, lte}` for a bool judgment with features. They are evaluated at read time against the raw fields, never the `calibrated` ones. A threshold that does not fit the judgment's type, or names a feature or option its version does not have, is `invalid_request`. propertyNames: $ref: '#/components/schemas/Name' additionalProperties: oneOf: - type: number - $ref: '#/components/schemas/ChoiceThreshold' - $ref: '#/components/schemas/FeatureThreshold' ChoiceOption: type: object additionalProperties: false required: - value properties: value: type: string minLength: 1 not: const: none_of_the_above description: type: string ProposedOption: type: object required: - value - description - absorbs - sample_ids properties: value: type: string description: type: string absorbs: type: integer minimum: 0 description: >- Sampled escape documents the proposed version answered with this option. sample_ids: type: array description: The sampled documents the model said this option covers. items: $ref: '#/components/schemas/DocumentId' SimulatedFlips: type: object required: - sampled_on - sampled_off - estimated - interval properties: sampled_on: type: integer minimum: 0 description: Sampled documents the threshold would newly hold for. sampled_off: type: integer minimum: 0 description: Sampled documents it would stop holding for. estimated: type: integer format: int64 minimum: 0 description: Estimated flips in the population, either way. interval: $ref: '#/components/schemas/LiftInterval' DocumentId: type: string description: >- Up to 128 bytes. Never exactly `.` or `..`, which a URL path can't carry. pattern: ^(?!\.\.?$)[A-Za-z0-9._:/-]+$ minLength: 1 maxLength: 128 ShadowSide: type: object description: One side of a shadow report, discriminated on the judgment `type`. oneOf: - $ref: '#/components/schemas/ShadowValues' - $ref: '#/components/schemas/ShadowOptions' discriminator: propertyName: type mapping: bool: '#/components/schemas/ShadowValues' score: '#/components/schemas/ShadowValues' choice: '#/components/schemas/ShadowOptions' ThresholdFlips: type: object required: - to_true - to_false properties: to_true: type: integer format: int64 minimum: 0 description: Documents false under the active version and true under the new one. to_false: type: integer format: int64 minimum: 0 description: Documents true under the active version and false under the new one. CalibrationReplayQuote: type: object description: > The replay that calibrates a version activated on another engine, priced before you confirm: the outcomes of the active version's answers, newest first and at most 1,000, each answer's stored context judged by the new engine with the new version's question. Each replayed outcome is billed as one judgment of its size class times the new engine's weight, held by your budget like any judging; the monthly replay limit does not stop it. required: - outcomes - tokens - judgment_units - cost_usd properties: outcomes: type: integer format: int64 minimum: 0 description: Outcomes whose answers' contexts would be replayed. tokens: type: integer format: int64 minimum: 0 description: The tokens the new engine would read, each context and the question. judgment_units: type: integer format: int64 minimum: 0 description: >- Judgments, each counted by its size class times the new engine's weight; a whole number, rounded. cost_usd: type: number minimum: 0 description: >- What it would cost you, at the graduated prices of the tiers your organization is in now. ShadowAlternative: type: object description: >- The outcomes' verdict on the alternative this version was created from, as it stood when the version was created. required: - id - verdict - change - k - drafted properties: id: $ref: '#/components/schemas/AlternativeIdValue' verdict: type: string enum: - better - no_worse - worse - untested - unscored change: oneOf: - $ref: '#/components/schemas/AlternativeChange' - type: 'null' k: type: integer minimum: 0 description: How many alternatives the verdict was corrected for. drafted: type: boolean CrossValidated: type: object description: Held-out scores over the report's documents. required: - accuracy - accuracy_interval - auc properties: accuracy: $ref: '#/components/schemas/Probability' description: >- Share of documents whose held-out probability is on the right side of 0.5. accuracy_interval: $ref: '#/components/schemas/MetricInterval' auc: $ref: '#/components/schemas/Probability' description: ROC AUC of the held-out probabilities. Baseline: type: object description: >- The fair baseline, scored like the composite. The active version (`version`), or for a first version the composite's `question` asked on its own (`version` null). required: - version - threshold - accuracy - accuracy_interval - auc allOf: - $ref: '#/components/schemas/CrossValidated' properties: version: oneOf: - $ref: '#/components/schemas/JudgmentVersionNumber' - type: 'null' threshold: description: >- The baseline's fitted cut-off on its `p`, fitted on all the documents. Null when a higher `p` does not mean `true` more often. oneOf: - $ref: '#/components/schemas/Probability' - type: 'null' DifferenceInterval: type: object required: - lower - upper properties: lower: type: number minimum: -1 maximum: 1 upper: type: number minimum: -1 maximum: 1 PartWeight: type: object required: - name - weight properties: name: $ref: '#/components/schemas/Name' weight: type: number FeatureWeight: type: object required: - name - weight properties: name: $ref: '#/components/schemas/FeatureRef' weight: type: number EvaluationExportFile: type: object description: One gzipped JSON Lines file of an evaluation export. required: - url - url_expires_at - evaluations - bytes properties: url: description: >- A signed download link on the API's host. It needs no API key and works until `url_expires_at`. Null when the link can't be made right now (`links_unavailable` says why). oneOf: - type: string format: uri - type: 'null' url_expires_at: description: When `url` stops working; null with it. oneOf: - $ref: '#/components/schemas/Timestamp' - type: 'null' evaluations: type: integer format: int64 minimum: 1 description: Lines in the file. bytes: type: integer format: int64 minimum: 1 description: The file's size, compressed. CreateJudgmentResponse: type: object required: - name - version - active - warnings - definition properties: name: $ref: '#/components/schemas/Name' version: $ref: '#/components/schemas/JudgmentVersionNumber' active: type: boolean warnings: type: array description: >- Non-fatal problems, such as sending the whole `state`, a recipe suggested for you, no `applies_to` (the judgment judges, and bills, every document written to the namespace), a definition that names image paths (`images.`) on an engine that reads no images (decided from the definition alone, so it comes also before any document is written), a relation that renders a referenced document's `updated_at`, which changes on every write to it and so fans out every time, or a relation with an `order`, which reads up to 1,000 documents per judged document to sort them (the warning names the factor), can re-select on a write to the ordered field, may have judged documents with more matching documents than it sorts, or sorts by a long text field. A relation whose `match` selects documents by another judgment's answer is also warned, since it never reads a document whose answer is wrong, or whose change removes what that answer found, so select by topic (a judgment that asks whether a document is about the subject, including changing or removing it) and calibrate on outcomes so its numbers, and thresholds on them, mean what they say. items: type: string definition: $ref: '#/components/schemas/JudgmentDefinition' description: The version exactly as created, with the engine resolved. outcomes: $ref: '#/components/schemas/StarterOutcomes' description: From a starter with a suggested outcome setting only. job_id: type: string description: >- The first of `job_ids`, kept for clients written before `job_ids`. Read `job_ids` to see every job. job_ids: type: array minItems: 1 description: >- The `reference_index` jobs that build the indexes this version's relations need, one per attribute with no index yet, or whose index must be rebuilt to cover this version's documents (another `match`, or the attribute used the other way round), so two when it joins on two new attributes. Present exactly when `job_id` is. Until every job is done, the version's answers are `unavailable`, or `pending` for documents written since that an `on_change` judgment judges once the jobs are done. items: type: string CreateJudgmentDryRun: type: object description: | What a create would make, and what it would cost. Nothing was created, written or billed. required: - dry_run - definition - warnings - cost_per_answer - sampled_documents - estimate - replay - outcomes properties: dry_run: const: true definition: $ref: '#/components/schemas/JudgmentDefinition' warnings: type: array items: type: string cost_per_answer: type: - number - 'null' minimum: 0 description: | Mean judgments per answer over `sampled_documents` of the documents the definition applies to, each counted by its size class as a backfill estimate counts them: 1 when every answer is standard, 4 when every one is large, 2 for a standard composite of five parts with short questions (the first in full, each other a quarter). Null when there is nothing to sample, and on a template's prefix path. sampled_documents: type: integer minimum: 0 estimate: description: | What backfilling every document the definition applies to would cost once it exists. Creating never backfills by itself. Null on a template's prefix path. oneOf: - $ref: '#/components/schemas/Estimate' - type: 'null' replay: description: >- An entity judgment's monthly replay estimate, whatever its policy; null for others. oneOf: - $ref: '#/components/schemas/ReplayEstimate' - type: 'null' outcomes: description: >- For a starter with a suggested outcome setting, what the create would apply; null otherwise. oneOf: - $ref: '#/components/schemas/StarterOutcomes' - type: 'null' CreateJudgmentEstimate: type: object description: >- An unconfirmed create of an entity judgment that runs `on_change`. Nothing was created. required: - replay not: description: A dry run is `CreateJudgmentDryRun`, even with a replay estimate. required: - dry_run properties: dry_run: const: true properties: replay: $ref: '#/components/schemas/ReplayEstimate' fanout: $ref: '#/components/schemas/FanoutLimitEstimate' warnings: type: array description: >- Non-fatal problems the confirmed create would also return, present only when there are some, such as a relation with an `order` whose judged documents already have more than the 1,000 matching documents it sorts. items: type: string graph_ceiling: description: | What this judgment and every judgment that reads its answers, directly or through others, would cost in 30 days if each spent its whole rolling limit, at its size class and your organization's tiers, with every compile billed: an upper bound, unlike the replay. Null when one of them has no rolling limit, which `graph_ceiling_unlimited` names. oneOf: - $ref: '#/components/schemas/GraphCeiling' - type: 'null' graph_ceiling_unlimited: type: array description: >- Present when `graph_ceiling` is null. The judgments among them whose `rolling_limit` is null. items: $ref: '#/components/schemas/Name' UpdateJudgmentResponse: oneOf: - $ref: '#/components/schemas/JudgmentSettingsApplied' - $ref: '#/components/schemas/BackfillEstimate' - $ref: '#/components/schemas/DownstreamEstimate' - $ref: '#/components/schemas/AlternativesConfirmNeeded' BackfillEstimate: type: object required: - estimate properties: estimate: $ref: '#/components/schemas/Estimate' replay: $ref: '#/components/schemas/ReplayEstimate' description: >- When the `PATCH` switches an entity judgment to `on_change`, what it would cost a month from then on, beside the backfill of the documents with no current answer. JobStarted: type: object required: - job_id properties: job_id: type: string ErrorCode: type: string description: | HTTP status by code: `invalid_request` 400, `unauthorized` 401, `invalid_image` 400, `unknown_image` 400, `image_erased` 410, `budget_exceeded` 402, `plan_required` 402, `forbidden` 403, `not_found` 404, `conflict` 409, `too_large` 413, `engine_version_unavailable` 422, `insufficient_labels` 422, `idempotency_key_reused` 422, `rate_limited` 429, `internal` 500, `engine_unavailable` 503, `unavailable` 503. Any route can also answer `unavailable`, with `Retry-After`, when it did not finish within 35 s: it was stopped, and a retry runs it again. Creating a judgment version, a judgment's `PATCH` and a backfill's start answer within 20 s, with a job (`202`) when they need longer, and never begin their change after 25 s. So when one of them is answered `unavailable` this way, nothing changed and the retry makes the change once. enum: - invalid_request - invalid_image - unknown_image - image_erased - unauthorized - forbidden - not_found - conflict - too_large - rate_limited - budget_exceeded - plan_required - engine_unavailable - engine_version_unavailable - unavailable - insufficient_labels - idempotency_key_reused - internal SuggestedContext: type: object description: | The recipe suggested for a create without `context`, in the refusal's `details.suggested_context`. From a sample of the documents the definition applies to: it keeps short text, numbers, booleans and small attributes, and leaves out ids, timestamps, links, binary-looking and very long values, and rarely present fields; `excluded` says why for each. Each name in a document's `images` is one candidate, `images.`; image paths are kept only on an engine that reads images, and only while the context stays in the standard size class with them: images count by their size, so when it has to choose, the suggestion keeps the text and leaves out image fields, the last first, with the reason `image`. `max_tokens` is set to keep the context in the standard size class, at most 2,000. The same documents always give the same recipe. required: - recipe - cost_per_answer - whole_state_cost_per_answer - sampled_documents - excluded properties: recipe: $ref: '#/components/schemas/ContextRecipe' cost_per_answer: type: - number - 'null' minimum: 0 description: >- Mean judgments per answer, counted by size class at the engine's weight as the bill counts them, over the sample with this recipe and the definition's question. The suggested `max_tokens` is at most 2,000, the standard ceiling, less the question's tokens, so a context that fills it stays standard with its question. whole_state_cost_per_answer: type: - number - 'null' minimum: 0 description: The same with the whole `state`. sampled_documents: type: integer minimum: 1 excluded: type: array description: >- Each path left out, and why, so you can put back what the question needs. items: type: object required: - path - reason properties: path: type: string reason: type: string enum: - id - timestamp - url - binary - image - long - rare - too_many ChoiceThreshold: type: object additionalProperties: false required: - value - gte properties: value: type: string minLength: 1 gte: $ref: '#/components/schemas/Probability' FeatureThreshold: type: object description: | A bool judgment's threshold on one of its features: true when the feature is at least `gte`, or at most `lte`, compared exactly with the stored value. No engine and no labels decide it. A missing feature reads `false`, and so does one a later version no longer declares, which the judgment's `threshold_unresolved` warning names. It is as fresh as the answer, so it can trail the related documents by the judgment's debounce. additionalProperties: false required: - feature oneOf: - required: - gte properties: gte: type: number - required: - lte properties: lte: type: number properties: feature: $ref: '#/components/schemas/FeatureRef' gte: type: number lte: type: number LiftInterval: type: object required: - lower - upper properties: lower: type: number upper: type: number ShadowValues: type: object description: The mean and histogram of `p` (bool) or `score` (score). required: - type - mean - histogram properties: type: type: string enum: - bool - score mean: type: number histogram: type: array description: >- Ten equal-width bins, from 0 to 1 for `p` and from the lowest to the highest level for `score`, lowest first. minItems: 10 maxItems: 10 items: $ref: '#/components/schemas/HistogramBin' ShadowOptions: type: object description: >- The mean probability per option, so the per-option shift is `candidate.dist` minus `current.dist`. required: - type - dist properties: type: const: choice dist: $ref: '#/components/schemas/Distribution' AlternativeChange: type: object required: - log_loss - auroc - accuracy properties: log_loss: $ref: '#/components/schemas/AlternativeTest' auroc: oneOf: - $ref: '#/components/schemas/AlternativeTest' - type: 'null' accuracy: type: number description: Accuracy, alternative minus version. Probability: type: number minimum: 0 maximum: 1 description: > Answers keep probabilities to about seven significant digits and return them as the shortest decimal that stands for the stored value, so an engine's 0.01 reads 0.01. Thresholds, query filters and threshold recommendations compare that decimal. MetricInterval: type: object required: - lower - upper properties: lower: $ref: '#/components/schemas/Probability' upper: $ref: '#/components/schemas/Probability' FeatureRef: type: string description: | One number the judgment records beside every answer and its fit can read. It is one of: - An aggregate the recipe declares, as `.count`, `.count_where`, `.labelled` or `.()`, such as `tickets.count` or `invoices.sum(state.amount)`. A `min` or `max` over an answer's `p` or `score` gives the raw extreme, whatever band the context shows; a `latest` over an answer's threshold reads 1 or 0. A `latest` over an answer's `value` is not a number and is `invalid_request`. A `sum` or `avg` over an answer's number is a feature only. A `latest` over a `bool` outcome reads 1 or 0, over a `score` outcome its level; over a `choice` outcome it is `invalid_request`. - The judged document's own value, `state.` or `attributes.`: a number as itself, a boolean as 1 or 0, anything else missing. It needs no relation. - An age in fractional days, `age(created_at)`, `age()` or `age(.())` over an aggregate the recipe declares. It is measured as of the judged document's own `updated_at` at the revision judged, so a write to a related document never moves it, and a new revision of the judged document does. The value must be an RFC 3339 string; a number or any other value reads missing, and a later time reads negative. `age(updated_at)` is `invalid_request`: as of the judged revision it is always 0. - `previous.p`: the engine's own `p` the last time it judged the document with the same engine version, in the same life of the document; on a choice, its `1 - escape_p`. Missing on the first judgment, after the document is created again, after the engine version changes, and on a composite, which has none. On a score it is `invalid_request`. - On a choice among candidates (`options.from`) only, a number per candidate, `.each()` over the `options.from` relation: each candidate's `state.` or `attributes.`, or `age(created_at)`, `age(updated_at)` or `age()` of the candidate, as of the judged document's revision. It is recorded for each candidate the engine read, under the answer's `candidate_features`. A candidate's answers are `invalid_request`: keep the number on the candidate's state. A name the recipe declares as an aggregate always reads that aggregate. A feature that reads a path an outcome rule reads, one inside the other, is `invalid_request`. pattern: >- ^([A-Za-z0-9_-]+\.(count|count_where|labelled|(sum|avg|min|max|latest)\((state|attributes)(\.[^.()]+)+\)|(min|max|latest)\((created_at|updated_at)\)|(sum|avg|min|max)\(answers\.[A-Za-z0-9_-]+\.(p|score|dist\.[^()]+)\)|latest\(answers\.[A-Za-z0-9_-]+\.thresholds\.[A-Za-z0-9_-]+\)|latest\(outcomes\.[A-Za-z0-9_-]+\.value\))|(state|attributes)(\.[^.()]+)+|age\((created_at|(state|attributes)(\.[^.()]+)+|[A-Za-z0-9_-]+\.(latest|min|max)\((created_at|updated_at|(state|attributes)(\.[^.()]+)+)\))\)|previous\.p|[A-Za-z0-9_-]+\.each\(((state|attributes)(\.[^.()]+)+|age\((created_at|updated_at|(state|attributes)(\.[^.()]+)+)\))\))$ JudgmentDefinition: type: object description: | A judgment definition, discriminated on `type`. `thresholds` are the named thresholds given with this version. They are not part of the version: the judgment's current thresholds are its `thresholds` setting, evaluated at read time and returned as booleans under `answers.{name}.thresholds`. oneOf: - $ref: '#/components/schemas/BoolJudgmentDefinition' - $ref: '#/components/schemas/ChoiceJudgmentDefinition' - $ref: '#/components/schemas/ScoreJudgmentDefinition' discriminator: propertyName: type mapping: bool: '#/components/schemas/BoolJudgmentDefinition' choice: '#/components/schemas/ChoiceJudgmentDefinition' score: '#/components/schemas/ScoreJudgmentDefinition' StarterOutcomes: type: object description: | A starter's suggested outcome setting and what the create did with it, or in a dry run what it would do. It is applied with the name's first version in a namespace: implicit negatives on every plan, outcome rules only on a plan with outcome rules. required: - suggested - applied - plan_required - message properties: suggested: $ref: '#/components/schemas/OutcomeSettings' applied: description: >- The setting now on the judgment, or in a dry run the one the create would apply; null when nothing is applied. oneOf: - $ref: '#/components/schemas/OutcomeSettings' - type: 'null' plan_required: description: Why the rules were not applied, when the plan lacks outcome rules. oneOf: - $ref: '#/components/schemas/OutcomeRulesPlanRequired' - type: 'null' message: type: string description: >- What happened (in a dry run, what would happen), and what to do, in a sentence. ReplayEstimate: type: object description: > What an entity judgment would cost a month, from the namespace's last 30 days of writes run through its relations, debounce and ceiling rules. The replay cannot see every edit to a related document, only its creation, its latest write and the deletes still recorded: exact for documents written once, and for often-edited documents the figures are a lower bound (`lower_bound`). It does not count the evaluations an unchanged context saves, which only lower the bill. required: - replayed_days - entities - judgments_per_month - judgment_units_per_month - cost_usd_per_month - bulk_pool_share - lower_bound properties: replayed_days: type: integer minimum: 0 maximum: 30 description: >- Days of writes replayed, fewer than 30 in a younger namespace. The monthly figures are scaled from them to 30 days. entities: type: integer format: int64 minimum: 0 description: Documents the judgment applies to now. judgments_per_month: type: integer format: int64 minimum: 0 description: Evaluations the replay counted, scaled to 30 days. judgment_units_per_month: type: integer format: int64 minimum: 0 description: >- Judgments, each counted by its size class times its engine's weight, from the compiled context and question of a sample of up to 1,000 entities, as if the judgment were judged alone; a composite's parts share one request, the first in full and each other a quarter, and a quarter more for each further 500 tokens of its question text. A whole number, rounded; `cost_usd_per_month` prices each quarter. Judged in one request with other judgments, it bills less. cost_usd_per_month: type: number minimum: 0 description: >- The judgments at the graduated prices of the tiers your organization will be in, counting what it has already used this billing period. A confirmed request is refused with `budget_exceeded` when this is more than the namespace's budget. bulk_pool_share: type: number minimum: 0 description: >- The share of the background judging rate available to this judgment that these evaluations would use, by requests or tokens, whichever is larger. Above 1, the judgment cannot keep up with the writes. Every engine has a rate limit, so it is always a number. lower_bound: type: boolean description: >- True when a related document in the replayed window has a `revision` higher than the writes the replay saw, so it was edited more often than the replay counted, and whenever `excludes` is present. The figures are then a lower bound, shown as "at least". The namespace budget still caps real spend. excludes: type: array description: > What the figures leave out, present only when something is. `fanout`: the judgment has a relation that reads a referenced document, and the replay cannot tell which past writes to it changed what the relation shows, so it cannot count its fan-out; `freshness.fanout.rolling_limit` bounds it. `candidate_edits` and `candidate_deletes`: a blocking relation's candidates edited or deleted in the window, which leave nothing a replay can read. `source_deletes`: a judged link's documents deleted in the window, which leave nothing a replay can read; every write to a document the link's `match` reads is counted as one re-judge of the document it links to. minItems: 1 uniqueItems: true items: type: string enum: - fanout - candidate_edits - candidate_deletes - source_deletes FanoutLimitEstimate: type: object description: | For a judgment with a relation: the rolling limit `confirm: true` sets, from `freshness.fanout` in the request or the default of 300,000, and what it costs if used in full. Work past it is deferred at runtime, never put to you again. Set `freshness.fanout.rolling_limit` to `null` to confirm with no limit: every change then re-judges every in-scope document, with no deferral, and only the namespace's budget caps what fan-out spends. required: - rolling_limit - cost_usd_at_limit properties: rolling_limit: type: - integer - 'null' format: int64 minimum: 0 description: >- Re-judgments by referenced and block changes and by the answers it reads moving, allowed in any 30 days; null for no limit. cost_usd_at_limit: type: - number - 'null' minimum: 0 description: >- What `rolling_limit` re-judgments cost in 30 days at the sample's judgments per answer and your organization's tiers; null with no limit. GraphCeiling: type: object required: - cost_usd - judgments properties: cost_usd: type: number minimum: 0 description: The sum of `judgments`' costs, in US dollars. judgments: type: array description: This judgment first, then each judgment that reads its answers. items: type: object required: - judgment - rolling_limit - cost_usd properties: judgment: $ref: '#/components/schemas/Name' rolling_limit: type: integer format: int64 minimum: 0 description: Its compiles allowed in any 30 days. cost_usd: type: number minimum: 0 description: What that many compiles cost, in US dollars. JudgmentSettingsApplied: type: object required: - judgment properties: judgment: $ref: '#/components/schemas/Judgment' job_id: type: string description: >- The backfill started by a confirmed switch to `on_change`, or the `resync` job a confirmed threshold change or pin started. downstream: type: array description: >- With a confirmed threshold change that readers' relations name, or a confirmed pin on a judgment with `features`, what it costs each reader. items: $ref: '#/components/schemas/DownstreamLine' DownstreamEstimate: type: object description: > A change that moves what other judgments read, sent without `confirm`: a threshold readers' relations name, a pin or a promote on a judgment with `features` that other judgments read, or a tenant summary's groups or bands. Nothing changed. Send it again with `confirm: true`. required: - downstream not: description: >- An applied change is `JudgmentSettingsApplied`, even with `downstream`. required: - judgment properties: downstream: type: array items: $ref: '#/components/schemas/DownstreamLine' AlternativesConfirmNeeded: type: object description: >- Turning `alternatives.automatic` on needs `confirm`; this is the quote of a full run, and nothing changed. required: - alternatives_quote properties: alternatives_quote: $ref: '#/components/schemas/AlternativesQuote' ContextRecipe: type: object description: | What the engine sees. A create without it is refused with a suggested recipe, unless `use_context` asks for the suggestion or the whole `state`; a version with no recipe sends the whole `state`, subject to the engine limit, and no images. additionalProperties: false properties: fields: type: array description: > Paths included in order, each verbatim or, as a `ClippedPath`, cut to its first `max_chars` characters. A path is `state.`, `attributes.`, or an image path, `images.`, which reads that name's image or whole list; an image path takes no `max_chars`. An engine that reads images (its `limits.images` in `GET /engines`) reads each, counted by its tiles; an engine that reads none gets each as an `OmittedImage` with `engine_reads_no_images`, and the create warns. Images are written to documents, never sent with the judgment. items: $ref: '#/components/schemas/RecipeField' last_n: type: object description: >- Per array path in `state` or `attributes`, keep the last n elements. Not an image path. additionalProperties: type: integer minimum: 1 window: type: object description: >- Per array path whose elements have a timestamp field `at`, keep elements within the window. additionalProperties: $ref: '#/components/schemas/Duration' max_tokens: type: integer minimum: 1 description: >- Hard cap, in the judgment's engine's tokens. Over it, whole images are left out first (on an engine that reads them; those of relations, then `previous`, then fields, the last first; every image when the text alone is over), then relation records, then `previous` entries, then the end of the last field is cut. related: type: object description: > Up to 4 relations, by name: documents in the same namespace that point at the judged one, or the one document it points at. Each renders as a `related.` entry after `fields`, and a write that changes what it renders makes the judged document's answer `pending` until it is judged again (for a referenced document, only inside the re-judge scope; outside it, the answer is `stale` with `stale_reason` `referenced_changed`). minProperties: 1 maxProperties: 4 propertyNames: $ref: '#/components/schemas/Name' additionalProperties: $ref: '#/components/schemas/Relation' previous: $ref: '#/components/schemas/PreviousRecipe' HistogramBin: type: object description: Values in `[lower, upper)`; the last bin includes `upper`. required: - lower - upper - count properties: lower: type: number upper: type: number count: type: integer format: int64 minimum: 0 Distribution: type: object description: Probability per option or level value. Sums to 1. additionalProperties: $ref: '#/components/schemas/Probability' AlternativeTest: type: object description: >- A paired difference, alternative minus version, on the same labelled answers. required: - mean - interval properties: mean: type: number standard_error: type: - number - 'null' interval: type: object description: The 95% interval of the difference. required: - lower - upper properties: lower: type: number upper: type: number p: type: - number - 'null' description: >- The chance of a difference at least this good if the two were equal, before the correction. BoolJudgmentDefinition: type: object description: | The answer has `p`. With `parts` it is a composite judgment: the engine is asked each part instead of `question`, and `p` combines the parts' answers with weights fitted on the judgment's outcomes. required: - type allOf: - $ref: '#/components/schemas/JudgmentDefinitionCommon' not: description: A single part needs features. required: - parts properties: parts: type: array maxItems: 1 not: required: - features properties: type: const: bool thresholds: type: object description: >- Each threshold is true when `p` is at least the value, or, written `{feature, gte}` or `{feature, lte}`, when one of the judgment's features is at least or at most the number. propertyNames: $ref: '#/components/schemas/Name' additionalProperties: oneOf: - $ref: '#/components/schemas/Probability' - $ref: '#/components/schemas/FeatureThreshold' parts: type: array description: | Composite judgments only: 2 to 8 narrow yes/no questions with names unique within the judgment. They share the judgment's context recipe and engine, so they go in one engine request with its other questions, and each counts toward the 32 questions per request. `question` then documents what the combination means and is not sent to the engine. Parts are part of the version. They cannot reference other judgments. Each part is billed as a judgment. With `features`, one part is enough. minItems: 1 maxItems: 8 items: $ref: '#/components/schemas/JudgmentPart' features: type: array description: | Up to 12 numbers, on any bool judgment: aggregates over related documents the recipe's `related` declares, the judged document's own values, ages, and the engine's previous answer (see `FeatureRef`). Each is recorded as a raw number beside every answer under `features`, and on its evaluation, as of what that evaluation read, and never recomputed later. A threshold can name one, a query or subscription can filter on one, and the fit on your outcomes reads them beside the engine's answer. A composite judgment's combiner also takes them as numeric inputs beside the parts, scaled automatically, a missing value counting as typical. Features add no questions, so they are never billed: a change that moves only a feature records a new answer with no engine call. A judgment with relations created with features debounces for 10 minutes unless its `freshness` sets `debounce_ms`. Changing the list makes a new version. minItems: 1 maxItems: 12 uniqueItems: true items: $ref: '#/components/schemas/FeatureRef' ChoiceJudgmentDefinition: type: object description: | The answer has `value`, `dist` and `escape_p`. The API appends `none_of_the_above` as a final option on every choice judgment. required: - type - options allOf: - $ref: '#/components/schemas/JudgmentDefinitionCommon' properties: type: const: choice options: description: >- The options, whose values must be distinct; or `{from, label}`, one option per document a blocking relation of the recipe selects. oneOf: - type: array minItems: 2 maxItems: 254 items: $ref: '#/components/schemas/ChoiceOption' - $ref: '#/components/schemas/OptionsFrom' thresholds: type: object description: Each threshold is true when `dist[value]` is at least `gte`. propertyNames: $ref: '#/components/schemas/Name' additionalProperties: $ref: '#/components/schemas/ChoiceThreshold' features: type: array description: > Up to 12 numbers the fit on your outcomes weighs beside the engine's distribution, with a bool's sources (see `FeatureRef`), and on a choice among candidates `.each()`, a number per candidate. Each is recorded as a raw number beside every answer and its evaluation, as of what that evaluation read. Once a fit on them beats the engine on outcomes it was not fitted on, every reader reads the fitted numbers: `dist`, `escape_p` and `value`, so the fit can change which option or candidate is picked. Features add no questions and are never billed. Changing the list makes a new version. Needs the features release on the namespace's deployment. minItems: 1 maxItems: 12 uniqueItems: true items: $ref: '#/components/schemas/FeatureRef' ScoreJudgmentDefinition: type: object description: >- The answer has `score`, the probability-weighted mean of level values, and `dist`. required: - type - levels allOf: - $ref: '#/components/schemas/JudgmentDefinitionCommon' properties: type: const: score levels: type: array description: Ordered; level values must be distinct. minItems: 2 maxItems: 10 items: $ref: '#/components/schemas/ScoreLevel' thresholds: type: object description: Each threshold is true when `score` is at least the value. propertyNames: $ref: '#/components/schemas/Name' additionalProperties: type: number features: type: array description: > Up to 12 numbers the fit on your outcomes weighs beside the engine's distribution, with a bool's sources (see `FeatureRef`). Each is recorded as a raw number beside every answer and its evaluation, as of what that evaluation read. Once a fit on them beats the engine on outcomes it was not fitted on, every reader reads the fitted numbers: `dist` and `score`. Features add no questions and are never billed. Changing the list makes a new version. Needs the features release on the namespace's deployment. minItems: 1 maxItems: 12 uniqueItems: true items: $ref: '#/components/schemas/FeatureRef' OutcomeSettings: type: object description: | How the judgment's outcomes are read and derived, a setting rather than part of a version. See [calibration](/concepts/calibration). additionalProperties: false properties: rules: type: array maxItems: 4 items: $ref: '#/components/schemas/OutcomeRule' description: | Derive outcomes from the judged document's own writes, so labels arrive without posting them. Each rule is checked on every write to a document from when it is set; earlier writes are never replayed. Derived outcomes are stored with the write, join like posted ones, and a posted outcome beats a derived one in the same prediction window. With a `0s` horizon, a derived outcome labels the answer to the document as it was just before the write that revealed it. Omitted when there are none. Not accepted on a template's prefix path. Setting rules needs the Team plan or above (`plan_required`); clearing them never does. Outcomes derived while the plan does not include rules are kept but left out of calibration (`rule_outcomes_paused` in the report). implicit_negatives: type: boolean default: false description: | For a `bool` judgment with a non-zero `horizon` only. When true, a prediction window that has closed with no outcome counts as a `false` outcome, so you only need to post what happened. A window still open counts nothing yet, one whose document was deleted before it closed is left out unless an outcome labels it, and one opened at or after a `true` for the document is left out; the report counts them in `censored_predictions`. Leave it off when a missing outcome does not mean "no", for example when you only record some of what happens. On a namespace that inherits the judgment, set it on the template's prefix path: its calibration is shared. OutcomeRulesPlanRequired: type: object description: >- Why a starter's outcome rules were not applied, as `plan_required` says it. required: - message - capability - required_plan - plan properties: message: type: string capability: type: string enum: - outcome_rules required_plan: type: string enum: - team - scale plan: type: string enum: - developer - team - scale Judgment: type: object required: - name - active_version - freshness - thresholds - outcomes - judged_through - versions properties: name: $ref: '#/components/schemas/Name' active_version: description: Null once the judgment is deactivated. oneOf: - $ref: '#/components/schemas/JudgmentVersionNumber' - type: 'null' judged_through: $ref: '#/components/schemas/JudgedThrough' freshness: $ref: '#/components/schemas/FreshnessSettings' thresholds: $ref: '#/components/schemas/ThresholdSettings' calibration: $ref: '#/components/schemas/CalibrationSetting' description: >- The calibration setting. Always present on a judgment read from a current deployment; `mode` is `auto` until a pin is set. outcomes: $ref: '#/components/schemas/OutcomeSettings' alternatives: $ref: '#/components/schemas/AlternativesSetting' description: >- The alternatives setting; absent while it is the default (`automatic` off, no engine skipped). template: $ref: '#/components/schemas/NamespaceOrTemplate' description: >- Present when the judgment is inherited, the template prefix it comes from, such as `acme/prod/*`. overrides: type: array description: >- Present when the judgment is inherited. The settings this namespace overrides; the others follow the template. uniqueItems: true items: type: string enum: - thresholds - freshness fanout_sources: $ref: '#/components/schemas/FanoutSources' fanout_window: $ref: '#/components/schemas/FanoutWindow' escape_alert: $ref: '#/components/schemas/EscapeAlert' warnings: type: array description: >- Conditions to act on, such as `taxonomy_drift`. Absent when there are none. items: $ref: '#/components/schemas/JudgmentWarning' readers: type: array description: >- The judgments whose relations read this one's answers. While there are any, a change that would break them is `conflict` naming them in `details.readers`. That is deleting or deactivating it, a policy other than `on_change`, a fan-out scope on a judgment that reads a referenced document or a block, removing a threshold a reader names, or a version that is a composite with parts or chooses among candidates (unless the reader only counts whether the value is `none_of_the_above`), whose answers lack a part a reader reads, that narrows `applies_to` below a reader's `match`, or that would make a reader read more than three deep (`details.chain` is the chain). For a judgment read through judged links, it is also a version that no longer chooses among candidates, or whose candidates share another key attribute. Absent when there are none. items: $ref: '#/components/schemas/Name' upstream: type: array description: >- Each chain of judgments this one reads the answers of, from the one it reads first down to one that reads none, with their debounces summed. That is roughly how long a change at the end of the chain takes to reach this judgment, before its own debounce. Absent when it reads no answers. items: type: object required: - path - debounce_ms properties: path: type: array minItems: 1 maxItems: 3 items: $ref: '#/components/schemas/Name' debounce_ms: type: integer format: int64 minimum: 0 versions: type: array description: >- Every version, oldest first. Versions are kept until the namespace is deleted. items: $ref: '#/components/schemas/JudgmentVersion' stats: $ref: '#/components/schemas/JudgmentStats' AlternativesQuote: type: object description: | What scoring the unscored engine and named alternatives would bill, priced before anything runs, and the smallest gain the labels could show. Each answered call is billed as judging: its size class times its engine's weight, at the graduated prices of the tiers your organization is in now. required: - status - labels - alternatives - judgment_units - tokens - cost_usd - detectable - new_providers - upstream properties: status: type: string enum: - ready - quoting generation: type: integer minimum: 0 description: >- Which quote this is; it moves whenever the alternatives, the outcomes or the engines in use change. Send it as `quote_generation` to confirm exactly this quote. computed_at: oneOf: - $ref: '#/components/schemas/Timestamp' - type: 'null' labels: type: integer minimum: 0 description: Labelled answers each alternative would be scored on. alternatives: type: array items: type: object required: - id - contexts - labels - tokens - judgment_units - cost_usd - new_provider properties: id: $ref: '#/components/schemas/AlternativeIdValue' contexts: type: string enum: - stored - narrowed - live labels: type: integer minimum: 0 tokens: type: integer minimum: 0 judgment_units: type: integer minimum: 0 cost_usd: type: number minimum: 0 new_provider: type: boolean description: >- Its engine is one this namespace has not used; confirming sends the labelled contexts to its provider. judgment_units: type: integer minimum: 0 tokens: type: integer minimum: 0 cost_usd: type: number minimum: 0 detectable: type: object description: >- The smallest gain this run could call `better` on these labels with this many alternatives; smaller gains read `no_worse`. required: - log_loss - auroc - k properties: log_loss: type: number minimum: 0 auroc: type: - number - 'null' minimum: 0 k: type: integer minimum: 1 description: The alternatives it corrects for. new_providers: type: array description: >- Engines that would receive this namespace's labelled contexts for the first time, `name version`. items: type: string upstream: type: array description: >- The backfill a `live` recipe needs first, when it selects by another judgment's answer. Never run by this run. items: type: object required: - alternative - judgment - documents - judgment_units - cost_usd properties: alternative: $ref: '#/components/schemas/AlternativeIdValue' judgment: $ref: '#/components/schemas/Name' documents: type: integer minimum: 0 judgment_units: type: integer minimum: 0 cost_usd: type: number minimum: 0 reason: type: - string - 'null' description: Why nothing could be priced, when nothing was. RecipeField: description: >- A `state` or `attributes` path of the judged document, or an image path `images.`, rendered verbatim; or a `ClippedPath`. oneOf: - type: string pattern: ^((state|attributes)(\.[^.]+)+|images\.[A-Za-z0-9_-]{1,128})$ - $ref: '#/components/schemas/ClippedPath' Duration: type: string description: A whole number and a unit (`s`, `m`, `h` or `d`), such as `7d` or `1h`. pattern: ^[1-9][0-9]*[smhd]$ Relation: type: object description: > Documents that point at the judged document. They are ordered newest `created_at` first unless `order` says otherwise: `window` keeps those created within the window, the order is applied, then `last_n` keeps the first n. At least one of `last_n` and `window` is required, and at least one of `fields` and `aggregate`. A relation reads at most the newest 1,000 documents, and an `order` sorts within them. With `join: {theirs: "id", mine: "attributes."}` the relation reads the one document the judged document points at, its referenced document, and takes no `window`. With `last_n` (up to 254) it is a named list instead: the attribute holds an array of ids, and the relation reads the documents it names in list order unless `order` says otherwise, each once, at most `last_n` of them. An id that names nothing, or a document outside `match`, renders nothing and is listed in the evaluation's `related_documents` as `unresolved`. An attribute a named list reads holds at most 254 distinct ids. With `join: {theirs: "attributes.", mine: "attributes."}` it is a blocking relation over the documents that share the judged document's key value. It needs `window` unless it sets `relative`, takes `last_n` up to 254 (up to the `options.from` candidate limit when a `choice` chooses among it with `options.from`), and a change to its block re-judges the block's judged documents. With `relative`, each judged document reads the `last_n` documents of its block created just before it (or just after it), in creation order, and a change re-judges only the judged documents whose documents it moved. `fields`, `aggregate` and `match` may read other judgments' answers (`answers..`), banded or cut, and the newest outcome posted for each related document (`outcomes..value`), shown as posted: the one with the latest `observed_at`, of two at the same time the one posted last, about the document's current life and of the judgment's current answer type. Posting one re-judges the judgments that read it, within their `rolling_limit`. A relation reads at most 4 judgments' answers or outcomes. additionalProperties: false required: - match - join allOf: - description: A relation renders fields or aggregates. if: not: required: - fields then: required: - aggregate - description: >- A relation over the documents that point at the judged one is bounded by `last_n` or `window`. if: properties: join: type: object properties: mine: const: id then: anyOf: - required: - last_n - required: - window - description: A relation that reads the referenced document takes no `window`. if: properties: join: type: object properties: theirs: const: id then: not: required: - window - description: >- A relation that reads the referenced document takes `last_n` up to 254, as a named list. if: properties: join: type: object properties: theirs: const: id then: properties: last_n: type: integer maximum: 254 - description: A blocking relation reads at most 254 documents. if: properties: join: $ref: '#/components/schemas/BlockingJoin' then: properties: last_n: type: integer maximum: 254 - description: A blocking relation needs `window` unless it sets `relative`. if: not: required: - relative properties: join: $ref: '#/components/schemas/BlockingJoin' then: required: - window - description: >- Only a blocking relation takes `relative`, and then neither `window` nor `order`. if: required: - relative then: properties: join: $ref: '#/components/schemas/BlockingJoin' not: anyOf: - required: - window - required: - order - description: >- Only a relation over the documents that point at the judged one, or a named list, takes `order`. if: required: - order properties: order: $ref: '#/components/schemas/RelationOrder' then: anyOf: - properties: join: $ref: '#/components/schemas/ReferringJoin' - required: - last_n properties: join: $ref: '#/components/schemas/ReferencedJoin' properties: match: $ref: '#/components/schemas/RelationMatch' description: Which documents the relation reads. join: $ref: '#/components/schemas/RelationJoin' last_n: type: integer minimum: 1 maximum: 1000 description: >- Keep the first n in the relation's order, newest first by default. On a relation that reads a referenced document it makes a named list, keeping the first n documents the list names, in list order unless `order` says otherwise, at most 254; at most 254 on a blocking relation, and at most the `options.from` candidate limit (see Limits) when `options.from` chooses among it. window: $ref: '#/components/schemas/Duration' description: >- Keep documents created within this long before the later of the judged document's newest write and its newest related creation, at the answer's watermark. An edit to an old related document never moves the window. Not on a relation that reads a referenced document. fields: type: array description: >- Paths to include from each document, rendered under `records` in the relation's order. An entry may be a `BandedField`, which renders a number as its band, or a `ClippedPath`, which renders the first `max_chars` characters. An entry may also be a judgment's `answers..value` and `answers..thresholds.` as they are, and its `.p`, `.score` and `.dist.