Images in documents

A document’s state can hold images: a photo of a claim, a scan of a contract page, the picture on a product listing. Brussle stores each image one time and puts a reference to it in the document.

Only some engines read images. Every engine reads the text, numbers and attributes of a document the same way; an engine that does not read images gets each image’s reference in its place, and the evaluation says so. The engines table shows which engine versions read images.

An image is useful in the same cases as text. A judgment that reads one photo and nothing else is the single-row case, and a column in your own database can hold that answer. Brussle helps when the answer depends on other rows: a claim photo checked against its policy, a product photo against the listing that it belongs to, a page scan against its previous version.

Send an image

Anywhere in state, at any depth and in arrays, write an image as an object with one key, $image, and the image as a base64 data URL:

POST /v1/namespaces/acme%2Fclaims

{"upsert": [{
  "id": "claim_1042",
  "attributes": {"kind": "claim", "policy_id": "pol_88"},
  "state": {
    "description": "Rear door dented in a car park.",
    "photos": [{"$image": "data:image/jpeg;base64,/9j/4AAQSkZJRg..."}]
  }
}]}

The image is a PNG, a JPEG or a WebP: data:image/png;base64,, data:image/jpeg;base64, or data:image/webp;base64,. Write image/jpeg, not image/jpg.
Send the base64 in the standard alphabet, on one line. Brussle refuses line breaks with invalid_image.
Brussle does not fetch links. An $image that is a URL is refused with invalid_image.
Put each image under a key, such as state.photo. state itself is never an image: Brussle stores an $image key at the top level of state as plain text.
patch and append take images in the same way as upsert. A patch replaces each top-level key that it sends. To add an image to an array, use append. An append skips an image that the array already holds.
The write’s response does not hold the references. Get the document to read them.

What the document holds

When you read the document, each image is a reference in place of the data URL:

{"$image": "sha256:e0db4765554182d845b920d6d678bda68fe41fa74ffb41d36f715e5e6e5f3ccc", "type": "image/jpeg", "width": 1440, "height": 960}

$image is the image’s digest. The same image always has the same digest.
type, width and height describe the image as Brussle stored it.

You can rely on these things:

An image that is too large is made smaller. If an image covers more than 2,048 tiles of 32 × 32 pixels, Brussle scales it down until it fits. The image keeps its aspect ratio and its format. Brussle turns it the right way up, and removes its metadata.
An image that fits is stored exactly as you sent it, with its metadata, such as the EXIF location of a photo. An engine that reads images gets those bytes. Remove metadata that you do not want stored or sent before you write the image.
What is stored is what the model reads. The stored image is the only copy. An evaluation names the image by its digest, so you can always see the image that an answer read. Keep your originals if you need them at full size.
Each image is stored one time. Many documents can name the same image. A write that sends an image that the namespace already holds stores nothing new.
Sending the same image again changes nothing. The same image gives the same reference. Thus, a patch that sends a document’s image again makes no new revision, and no judgment runs again.

Name a stored image

A write can also send the reference itself, as a get of the document returns it. Use this to put an image that the namespace holds into another document without sending the image again:

{"patch": [{"id": "claim_1043", "state": {"photos": [{"$image": "sha256:e0db47...", "type": "image/jpeg", "width": 1440, "height": 960}]}}]}

Brussle refuses a reference to an image that the namespace does not hold with unknown_image. Each namespace has its own images, so a reference from one namespace does not work in another. Send the image itself instead. A policy simulation stores nothing, so its document takes images only as references. It refuses an image sent as a data URL with invalid_image. Write the image to a document first, then use the reference that the document returns.

Read an image

GET /namespaces/{ns}/images/{digest} returns the image’s bytes, where {digest} is the reference’s $image, such as sha256:e0db47.... You can send the colon as it is or as %3A. In both SDKs, ns.image(digest) returns the image’s bytes and its type.

Any key that can read the namespace can read its images. Read a tenant’s images on the tenant’s namespace, not on a template prefix.
The bytes under a digest never change. Thus, the response has Cache-Control: private, max-age=31536000, immutable, and you can keep it.
An unknown digest gets not_found. A digest that is not sha256: and 64 lowercase hex digits gets invalid_request.

Which engines read images

Each engine’s page has an Images row. It says how many images the engine reads in one request, or that the engine reads no images. A judgment reads the images that its context recipe selects, in the document’s own fields, in its previous version and in the related documents that it reads. An image is one value: max_chars never cuts it, and a field that holds an image is never cut to a string. max_chars does not apply to a field that holds an image: Brussle sends the whole field, its text included. Keep long text and images in separate fields. An evaluation’s context names each image by its reference, never by its bytes. Thus, you can always see which images an answer read, and get each one with GET /namespaces/{ns}/images/{digest}.

Judge the images in a document

You do not send images to a judgment. You write them to documents, and the judgment’s context recipe names the fields that hold them, as it names any other field. Choose an engine that reads images. This judgment reads each claim’s description and its photos:

POST /v1/namespaces/acme%2Fclaims/judgments

{
  "name": "damage_matches",
  "type": "bool",
  "applies_to": {"attributes.kind": "claim"},
  "question": "Do the photos show damage that matches the description?",
  "criteria": "Yes only if a photo shows damage in the place and of the kind that the description gives. Blurred or unrelated photos are no.",
  "context": {"fields": ["state.description", "state.photos"]},
  "engine": {"name": "gpt-6-luna", "version": "current"},
  "freshness": {"policy": "on_change"}
}

state.photos is the array of images in the claim. The engine reads each image in it, and the text of state.description.
A related document’s images are read the same way: name the field in the relation’s fields, such as the policy’s photo of the car when it was insured.
A field can hold text and images together, such as an array of notes where some notes are images. The engine reads both.
On an engine that reads no images, the same recipe works, but the engine gets each image’s digest in its place. See the warning.

When the engine does not read an image

The engine reads the image’s digest instead of the image, and the evaluation’s context shows this form:

{"$image": "sha256:e0db47...", "omitted": "engine_reads_no_images"}

engine_reads_no_images. The engine reads no images. This is not a cut, so it does not make context_truncated true.
over_the_limit. The engine reads images, but this one did not fit. A context holds at most 16 images. When a context is over its token limit, Brussle leaves out whole images before it cuts any text. It leaves out the images of related documents first, then those of the previous version, then those of the document’s own fields, from the last field. If the text alone is over the limit, Brussle leaves out every image, and then cuts the text. A field that holds an image is never cut: Brussle keeps it whole or drops it whole. context_truncated is then true.

The warning. When you create a judgment on an engine that reads no images, the response has a warning if its recipe reads fields that hold images. Brussle checks the namespace’s first 100 documents for this. Thus, a create on a template prefix, a create before you write documents, or a create whose documents hold images only further on, gets no warning. Setting the namespace’s default_engine gives no warning either. Check the engine’s limits.images in GET /engines yourself. The answer is the engine’s answer to exactly what the context shows.

What images cost

An image counts toward the judgment’s size class by its size in pixels, the same as text counts by its tokens. An engine reads an image in tiles of 32 × 32 pixels, and each tile is about one token. For example, a photo of 1440 × 960 pixels is 45 × 30 tiles, or 1,350 tokens, plus the short text of its reference. With a short record beside it, that is still a standard judgment. Two such photos are more than 2,000 tokens, so the judgment is large. To bound the cost, keep fewer photos in the fields that the recipe reads.

A full-size photo makes a large judgment. Brussle scales a large photo, such as one from a phone camera, to about 2,048 tiles, which is about 2,048 tokens. That alone is more than a standard judgment, so with any question the judgment is large and counts as 4. To keep a judgment standard, send photos of at most about 1,500 × 1,000 pixels, with a short record beside them.
The engine’s weight applies. On gpt-6-luna, whose weight is 2.5, a standard judgment counts as 2.5 judgments, and a large one as 10.
An image that the engine does not read costs only its digest, about 100 characters.
A document whose images and other read fields did not change is not judged again. The reference names the image by its digest, so sending the same image again changes nothing. When a field that the recipe reads changes, the engine reads the images again, and they count again.
An estimate counts images in the same way as the bill.

A suggested context recipe keeps image fields only on an engine that reads images, and only while the context stays a standard judgment. When it has to choose, it keeps the text and leaves out image fields, the last one first, with the reason image. Thus, it leaves out full-size photos. On an engine that reads no images, it leaves out image fields with the reason image. To judge full-size photos, name their fields in your recipe yourself, and do not copy the suggested max_tokens: a context over max_tokens leaves images out as over_the_limit.

How long images stay

An image stays as long as the namespace. When you delete a document, its images stay, because the evaluations of the document still name them. You cannot delete one image. When you delete the namespace, Brussle deletes its images.

Billing

A write’s bytes_written counts each image as its reference, plus the bytes of each image that the write stores for the first time. An image that the namespace already holds adds nothing.
Each stored image counts in the namespace’s stored bytes one time, however many documents name it.
Reading an image is billed as bytes_scanned: the image’s size, and at least what a document get costs.

Limits and errors

Rule	If you break it
An image is a PNG, JPEG or WebP, sent as a base64 data URL, and its bytes are the format that its data URL names	`invalid_image`
An image value is `{"$image": "data:..."}` or a reference, with no other keys	`invalid_image`
A reference names an image that the namespace holds, with that image’s `type`, `width` and `height`	`unknown_image`, or `invalid_image` if the type or size is different
An image is at most 20 MB, decoded, and at most 64 megapixels	`too_large`
A document holds at most 16 images. Each place counts, also when two places hold the same image. A `patch` or `append` counts the images that the document holds after it.	`too_large`

When an image breaks a rule, Brussle refuses the whole write, and stores none of its images. The error’s details.id names the document, and details.path names the image. A document with more than 16 images can have no details.path, and after a patch or append, no details. If Brussle refuses a write for another reason, for example the 1 MB state limit, the images that the write sent can stay stored. You do not pay for them. A later write that sends them uses them. A document’s 1 MB state limit counts each image as its reference, not its bytes. A write is still at most 64 MB, and this limit counts the data URLs. Base64 makes an image about a third larger, so one write holds at most three images of 20 MB. See limits.

Get started

Before you build

Documents and namespaces

Judgments

Relations: answers that read other rows

Keeping answers current

Versions and history

Outcomes

Subscriptions and events

Many tenants

Reference

Images in documents

Send an image

What the document holds

Name a stored image

Read an image

Which engines read images

Judge the images in a document

When the engine does not read an image

What images cost

How long images stay

Billing

Limits and errors

​Send an image

​What the document holds

​Name a stored image

​Read an image

​Which engines read images

​Judge the images in a document

​When the engine does not read an image

​What images cost

​How long images stay

​Billing

​Limits and errors

Send an image

What the document holds

Name a stored image

Read an image

Which engines read images

Judge the images in a document

When the engine does not read an image

What images cost

How long images stay

Billing

Limits and errors