Skip to main content
A choice judgment answers with one of its options, or with none_of_the_above when none of them fits. That escape option is where a new kind of complaint, a new abuse pattern or a new product question shows up first. Discovery reads what landed there and proposes the options your judgment is missing. It never changes the judgment: you review the proposal, edit it, and create the new version yourself. Discovery is for choice judgments with fixed options. A bool or score judgment has no escape option; watch its calibration report instead. A choice that chooses among candidates has one, but it means “no candidate matches”, not a missing option, so discovery does not apply to it either.

​
Know when to look: the escape alert

escape_alert is a setting of a choice judgment: the share of its answers over the last 7 days that were none_of_the_above, above which you want to know. It is off until you set it, and changing it creates no version.
When the share rises above it, three things happen, and nothing else:
  • The judgment gets a warning. GET on the judgment lists taxonomy_drift in warnings, with the share and when it rose. The dashboard shows it on the judgment’s page. It clears when the share falls back.
  • One event. The events feed, and every webhook endpoint that asks for it, gets one judgment.taxonomy_drift per rise, not one per answer. Staying above sends nothing more.
  • Nothing changes on its own. Your answers, options and bill stay as they are. You run discovery when you want to.
The share needs at least 20 answers in the 7 days before it can raise the warning, so a quiet judgment’s first escape isn’t a drift. The alert takes a number above 0 and at most 1; null turns it off. A namespace that inherits the judgment from a template follows the template’s alert, and gets its own warning and event from its own answers.

​
Turn on suggestions

Discovery sends a sample of your documents to a third-party LLM provider, the same one that suggests parts. That provider is one of the subprocessors listed in the data processing agreement. So it is off by default: an org admin turns on Suggestions in the organization’s settings in the dashboard. Until then, discover is refused with forbidden.

​
Run discovery

POST /namespaces/{ns}/judgments/{name}/discover with how far back to look (window, 7 days by default) and the most new options you want (count, 1 to 10, 5 by default):
It answers 202 with a discover job, which:
  1. Samples the escape. It takes a sample of the documents whose answer in the window is none_of_the_above, or whose escape_p is at or above the judgment’s threshold on none_of_the_above if you set one.
  2. Asks for options. It sends the sample, compiled with the judgment’s context recipe exactly as it is judged, to a third-party LLM provider with your question and your current options. The provider proposes up to count new options, each with a description and the sampled documents it covers.
  3. Checks them. It checks the proposed options against the sample in shadow, beside the current ones, as a shadow report checks a new version. This is free and changes no answer.
Discovery is free, and limited to 10 calls per organization a day, across all its namespaces and judgments. These are its own: discovery takes none of suggested parts’ daily calls, and they take none of its. Past them it is rate_limited, with details.limit (10) and when the allowance renews in details.resets_at, the next midnight UTC. The call is taken when the job starts, and a busy model is waited out rather than failing the job. A job that fails before the model has run for it gives the call back.

​
Read the proposal

When the job is done, its proposal is a new version’s options, the current ones first:
  • escape_documents is how many documents in the window escaped; sampled is how many of them the job read.
  • absorbs is how many sampled documents the proposed version answered with that option; sample_ids are the ones the LLM said the option covers. Open a few of them to see what the option really means.
  • unlabelled is how many sampled documents the proposed version still answered none_of_the_above. A large number means the escape holds more than one new thing, or things no option should cover.
The provider may propose fewer options than count, or none when the escaped documents share nothing new. That job is still done: proposed is empty, options are your current ones, and unlabelled counts the sampled documents that escaped.

​
When a job fails

A job fails, with proposal null, only when it got no options to check. Its error says why:

​
Review before you create the version

The proposal is a draft. The review is where most of its value is, so don’t accept it unedited:
  • Some proposals are a narrower case of an option you have. Added as they are, they take documents from their parent option. Fold a proposal like that into the parent option’s description instead of adding it, or delete it.
  • A new option next to an old one won’t take all of its documents. The neighbour keeps some of them. Sharpen both descriptions so the line between them is clear, and read the shadow report when you activate.
  • A small new kind may not come back as its own option. When it is a small part of what escaped, look for it in unlabelled and in the sampled documents.
Then create the version with the options you settled on, as you create any version. Activating it runs the ordinary shadow report over up to 1,000 of your documents, which shows what would change before anything does. In the dashboard, the judgment’s Discover tab shows the proposal as an editable list of options, with the change to the definition beside it, and creates the version from what you edited.

​
Limits