Skip to main content
Judge the records you imported after you see the cost: a backfill estimate gives the documents, tokens, judgments, cost and time before anything runs, and nothing runs until you confirm it. This page compares creating judgments before and after the import, and shows how to estimate, confirm and follow a backfill. Set a budget on the namespace before judging starts.

​
Option A: judgments first

  1. Create the judgments.
  2. Import the documents.
Brussle judges each imported document in the background as it arrives, as for any other change under on_change. wait_for makes only the first 16 documents of each write sooner. You get answers during the import, but no estimate before it starts, and nothing to confirm. Only the namespace budget, or moving the judgment’s policy away from on_change, stops the judging. Option A also judges some documents more than once:
  • A document that you write again during the import is judged again when its compiled context changed, for example on a second pass that fills in a field.
  • A judgment with related documents judges each judged document when its related documents arrive: once for each burst, and at least once for each ceiling while they keep arriving. With option A, import the related documents first, such as invoices before payments.
  • A judgment that reads the document that each judged document points at fans out each time you write that document after the documents that point at it, and judges again the documents inside its re-judge scope. With option A, import the referenced documents first, such as products before order lines, and send created_at, so that only recently created judged documents are in the scope.
Option A suits small imports, and namespaces that you fill slowly as new data arrives.

​
Option B: import first, then backfill

  1. Import all the documents, with no judgments.
  2. Create each judgment.
  3. For each judgment, ask for a backfill estimate.
  4. Read the estimate.
  5. Send the backfill again with confirm: true.
An on_change judgment created on a namespace that already has documents judges changes from then on. It leaves the existing documents for a backfill that you confirm, and Brussle never starts a backfill without your confirmation. Until the backfill reaches them, the existing documents’ answers read unavailable. The estimate looks like this:
It shows the documents it includes, their tokens, the judgments they would bill, counted by size class, the cost, and how long the backfill will take. Nothing runs until you send the request again with confirm: true, which starts a backfill job. A filters expression limits a backfill to the documents that match, so you can judge the documents you need first, such as open ones, and the rest later. Each backfill has its own estimate. For a judgment with relations, Brussle sometimes builds a reference index first. The create response then lists reference_index jobs in job_ids, and the judgment answers when they are done. Start its backfill after them.

​
The replay estimate

When you create an on_change judgment with relations, Brussle first returns a replay estimate: the monthly cost, worked out from your recent writes.
  • Records that you imported with their original created_at count as created at that time, so the replay estimate leaves out the older ones.
  • Without created_at, most of the writes it replays are the import itself. If the namespace is younger than 30 days, Brussle scales them up to a month.
The backfill estimate is the one that prices judging what you imported. Straight after you import old records into a new namespace, the replay estimate can be almost zero until your own traffic arrives. Ask for it again after some days of live writes, and until then let the namespace budget limit the spend.

​
Why B for large imports

  • You see the cost before you pay it. Each backfill shows documents, tokens, judgments, cost and duration, and runs only when you confirm it. Option A spends while the import runs.
  • The budget is checked first. If a backfill’s estimate does not fit in what is left of the namespace’s budget, or is more than your organization’s balance, Brussle refuses the confirm with budget_exceeded, before it spends anything. With option A, you find out only when the budget pauses judging during the import.
  • Each document is judged once. A backfill judges what you imported in its final form, with each related document in place. A second write of a document, or a related document that arrives during the import, costs nothing more.
  • You control the speed. A backfill runs in the background, after the namespace’s ordinary judging, and you can pause, resume or cancel it. duration_s is the time it takes if Brussle judges nothing else at the same time; for a large import, this can be days. Ordinary judging comes first, so a busy namespace makes the backfill take longer.

​
A namespace that already has judgments

To import into a namespace whose judgments are on_change:
  1. Switch each judgment to manual with a PATCH.
  2. Import the documents.
  3. Switch each judgment back to on_change. This request changes nothing yet: it returns a backfill estimate for the documents with no current answer.
  4. Read the estimate.
  5. Send the request again with confirm: true. Brussle switches the policy and starts the backfill.
See switching to on_change. While a judgment is manual, the answers of documents that change read stale.

​
Watch a backfill

  • The job. GET /jobs/{id} (db.jobs.get(id)) returns its status, progress.documents_done, spend_usd so far, the estimate, estimated_completion_at, and error if it failed.
  • Pause, resume or cancel. POST /jobs/{id}/pause, /resume and /cancel control it. A pause or a cancel takes effect in about one second. Requests already sent to the engine finish and are billed, and nothing more is sent. The job’s progress.documents_done and spend_usd count each document that it judged. The dashboard shows the same jobs.
  • Answers. Each answer has its freshness. For an on_change judgment, query for answers that are pending or unavailable, as in is a judgment caught up?. If the query returns a row, some documents are not judged yet. When the judgment existed before the import, compare the import’s revision with the judgment’s judged_through instead, which reads no documents.
  • Spend and the budget. stats.spend_month_usd in GET /namespaces/{ns} is the namespace’s judging cost in the current billing period. budget_paused says whether the budget paused judging.

​
Judge many tenants

  • Import every existing tenant first, then create the template judgments. A namespace that existed before the template judgment leaves its documents for a backfill.
  • One backfill on the prefix path includes each tenant. Its estimate counts up to 100 of the namespaces and scales the count to all of them. Once confirmed, each namespace stays inside its own budget, and Brussle skips a namespace whose budget is paused.
  • A tenant created after the template is judged in full from its first write, as in option A. Set its budget after its first write, before the rest of its import.