When the engine changes

A correction belongs to one judgment version and one engine epoch. When the engine changes, you do not start again from zero: Brussle replays your recent outcomes through the new behaviour and fits it at once, usually within the hour. You do not have to do anything, and after a change the engine made by itself the replay is free.

​

Two ways the engine changes

• The engine changes by itself. An engine version that is not pinned, such as Jev current, follows its provider. Each change in its behaviour that Brussle detects starts a new epoch, and you get an event. Each answer records its epoch in engine_version. A new epoch never makes an answer stale (see answers from an older version). An exact engine version never changes, so it is one epoch for its whole life.
• You move the judgment to another engine. You activate a version whose engine has a different name or exact version, for example from Jev current to gpt-6-luna current. The new version’s answers come from the new engine, and the earlier version’s answers read stale with older_version.

Either way, the new behaviour starts with no outcomes of its own, but the outcomes you already posted still say what came true. A new version on the same engine is different: it starts with no fit, and needs its own outcomes before its answers have calibrated. Only a version on another engine gets a replay.

​

The replay

Brussle starts the replay within minutes after the engine changes by itself, and within minutes after you activate a version on another engine. You cannot start one yourself.

• What is replayed. After the engine changes by itself: the outcomes that joined an answer of the active version in an earlier epoch. After you move the judgment: the outcomes that joined an answer of the version that was active before. Newest first, up to a fixed number. Each outcome replays the exact answer it measured, so its window is kept. If the new version has a different horizon, each outcome is measured with the new one.
• What is sent. The stored context of each of those answers, the text and the images the engine read then, with the question of the active version. After a change by itself, it goes to the same engine through the same provider your answers already use. After a move, it goes to the new engine, which you chose for all new answers. No provider gets data that your answers do not already send it (see where your data goes).
  • The context is sent as it was stored, also when the new version’s recipe differs.
  • A new engine that reads no images gets only the images’ references, as for its own answers. A new engine that reads images gets no image the earlier engine did not read.
  • Brussle does not replay a document you deleted after the answer, or a context that sent an image you erased after it. The fit uses the other outcomes.
• What is recorded. Each call is a replay evaluation of the active version in the evaluation log, whose replay_of names the evaluation it replayed. A replay is never an answer: nothing in your answers, queries or thresholds changes, and its log does not count toward your storage.
• The fit. Each replayed answer and its outcome is a sample of the new behaviour. Brussle fits them under the same rules as any fit: the minimums, the held-out test, queue weights and, on templates, each tenant’s own fit. The fit is made when the replay ends, usually within the hour. An outcome that also joined an answer of the new behaviour counts once. Once the new behaviour has 50 outcomes of its own, with 15 of each kind, Brussle fits it on those alone.

Replay is part of calibration, so it runs on every plan, and it never slows your answers.

​

What a replay costs

• After the engine changes by itself, the replay is free. It bills no judgments.
• After you move the judgment, the replay is billed as judging: each replayed outcome bills one judgment of its size class, times the weight of the new engine. For example, 812 standard contexts on gpt-6-luna bill 812 × 2.5 = 2,030 judgments. A call that fails bills nothing. The shadow report gives this cost in calibration_replay before you confirm; with force: true you skip the shadow report and its quote. The namespace’s budget and your organization’s balance hold the replay as they hold any judging. The report shows what it billed.
• A judgment with no outcomes has nothing to replay. It bills nothing, and its shadow report has no calibration_replay.

A monthly allowance for each organization limits the free replays: the replay after a change by itself, and recipe tuning. It does not limit the billed replay after a move, which the budget holds instead. A free replay that reaches the allowance stops with cost_cap. Brussle then fits on what was replayed, if that meets the minimums. If there is still no fit the next month, the replay runs again on the outcomes it did not replay.

​

While the replay runs

After the engine changes by itself, answers from the new epoch read the previous epoch’s correction, marked "from_previous_epoch": true, so you can see they were corrected for the old behaviour.

• For a bool without features, a choice and a score, this lasts until the new epoch has a fit, and at most 30 days from the day the behaviour changed. A fit for other behaviour is a guess, so after 30 days these answers have no calibrated until the epoch has a fit.
• A bool with features keeps the previous epoch’s fit for its p until the new epoch has its own, however long that takes, and a composite keeps its combiner the same way. Thresholds, filters, ranking, subscriptions and relations read that p, so it changes only when Brussle selects a new fit.
• The threshold recommender uses the previous epoch’s outcomes in the same way, and says so with from_previous_epoch.
• When the previous epoch is pinned, its answers read the pinned fit. Lift a pin only after the replay is done.

After you move the judgment, the new version’s answers have no calibrated, and a bool with features reads its raw p. The earlier engine’s fit corrects that engine’s numbers, not the new one’s, so Brussle never applies it to the new version. The new version has calibrated from the end of the replay, usually within the hour. Answers it made before then get calibrated when you read them again.

​

Read the replay in the report

In each epoch of the report:

• fit_source is replay while the epoch’s fit uses replayed outcomes, and outcomes once its own are enough. null for a composite.
• replay describes the replay that fitted the epoch again, or is null when none ran:
  • status: running, done or stopped, with a reason of cost_cap or superseded when stopped.
  • cause: drift when the engine changed by itself, engine_switch when you activated this version on another engine.
  • from_engine and from_engine_version: the engine and epoch its outcomes came from. from_version: after an engine_switch, the version whose outcomes it replayed; null otherwise.
  • planned and done: how many outcomes it plans to replay, and has replayed.
  • outcomes: the replayed outcomes the fit uses now.
  • judgment_units: the judgments it billed, at the engine’s weight. 0 after a drift.
  • started_at and finished_at.
• The epoch’s own outcomes count only outcomes that joined answers of the new behaviour, so the report’s totals count each outcome once.
• lift says the engine changed (model_changed) and starts the new behaviour’s headline from its first fit.

See also after the engine changes and change the engine yourself.