Eyes (eyes)

Category: intelligence | Form: | Symbol: Ey

A document reader (OCR / image-to-text) within an Intelligence Lab

An Eyes is a specific OCR model (e.g. Mistral OCR, Baidu Unlimited-OCR) configured with the page and confidence settings it should use. Agents with eyes can read scanned documents, photographed forms, and PDFs — including handwriting. Select eyes by ID; the runtime finds the parent lab and handles the document request.

Guide

A document reader (OCR / image-to-text) within an Intelligence Lab

What It Does

An Eyes represents a single OCR model within a Lab (provider) — for example Mistral OCR or a self-hosted Unlimited-OCR deployment. It defines the reader’s identity (provider + model_id), how it should render what it finds (table format, page cap), an optional structured-extraction contract (annotation_schema + annotation_prompt), and the confidence reporting used to route uncertain readings to a human. When something needs to turn an image or PDF into text, it references an eyes — the runtime resolves which lab the eyes belongs to and uses that lab’s connection details.

Eyes are atoms with nested residence: they have no children and live inside a Lab element, which provides the API endpoint and credentials. An eyes can also carry a per-element base_url that wins over the parent lab’s base_url — useful for routing one eyes to a self-hosted deployment while siblings keep the lab’s hosted-API URL.

Within the Lab, eyes is the reading sense: eyes turns documents into text, ears turns speech into text, mouth turns text into speech, and brain does the thinking over all of it.

Why eyes and not a vision brain

Before this element existed, OCR models were configured as brain elements with vision: true. That works for a vision LLM answering questions about an image, but it forces every document request through chat-completions — and the providers that are best at documents expose a dedicated endpoint instead. Mistral’s POST /v1/ocr returns page markdown, tables, per-page confidence scores, and a schema-filled annotation in a single call; none of that fits a chat-completions request. Eyes exists so the document path can speak the document protocol.

The practical distinction:

• eyes — “what does this page say?” Reading text off a document, including handwriting, with structure and confidence preserved.
• brain (vision) — “what is happening in this picture?” Reasoning about image content.

Element Definition

Property	Value
Type	eyes
Category	intelligence
Form	atom
Residence	nested
Symbol	Ey / #8B5CF6 (icon visibility)
Activity type	resource
Streaming	false
Handler	EyesHandler
Allowed visibility	collaborator
States	draft (initial) → active → error

Properties

Field	Type	Default	Description
provider	string (enum)	mistral-ocr	OCR provider dispatch key. mistral-ocr posts to the dedicated document endpoint; vllm-openai drives a self-hosted vision model over chat-completions; custom behaves as vllm-openai against an arbitrary endpoint
model_id	string	—	Model identifier sent to the provider API (e.g. mistral-ocr-4). Max 256 chars
display_name	string	—	Human label for this reader. Max 128 chars
base_url	string (url)	—	Optional per-element endpoint override. Wins over the parent lab’s base_url. Leave empty to inherit. Max 512 chars
credential_ref	string (secret picker)	—	Reference to a secret element with the provider API key. Optional — falls back to the parent lab’s credentials
table_format	string (enum)	html	How tables are rendered. html preserves cell boundaries in ruled forms; markdown reads better as prose
include_images	boolean	false	Return cropped images of figures alongside the text. Increases response size substantially
max_pages	integer	—	Cap on pages read from a multi-page document. Empty reads all pages
prompt_mode	string (enum)	—	Trained prompt mode for self-hosted DeepSeek-OCR-lineage models: docparse / multipage / freeocr / grounding. Ignored by mistral-ocr
window_size	integer	1024	Repetition-suppression window for self-hosted models (64–4096). Ignored by mistral-ocr
annotation_schema	object (json)	—	JSON Schema describing the fields to extract. When set, the reader fills this shape directly instead of returning only free text
annotation_prompt	string (textarea)	—	Guidance sent alongside the schema — what the document is, what language, whether to expect handwriting, what each field means. Max 4096 chars
confidence_granularity	string (enum)	page	Whether the provider reports page or word confidence, or none
min_confidence	number	—	Review threshold (0–1). Pages scoring below are flagged needs_review. Advisory — text is still returned
pricing	object	—	Cost reference (USD): per_page, and/or input_per_mtok / output_per_mtok for token-priced models

Ports

Direction	Port	Schema	Required	Description
Input	request	ReadRequest	no	Document to read
Output	info	EyesInfo	yes	Eyes metadata for the document-reader picker UI
Output	result	ReadResponse (event)	no	Document reading result

Capabilities

Capability	Description
optical-character-recognition	Turn images and PDFs into text
handwriting	Read handwritten as well as printed text
table-extraction	Preserve table structure from ruled forms
structured-extraction	Fill a caller-supplied JSON Schema directly from the document
confidence-scoring	Report per-page or per-word confidence for review routing

Operations

Op	Method	Path	Auth	Description
info	GET	info	read	Provider, model, and reading configuration
read	POST	read	execute	Read a document (image or PDF) into text
test	POST	test	execute	Verify the OCR connection with a built-in test document

Reading a document

POST /api/{circle}/{eyes-slug}/ops/read
{
  "document_b64": "iVBORw0KGgo…",
  "content_type": "image/png"
}

The element’s saved annotation_schema and annotation_prompt apply automatically, so a configured eyes needs only the document. Both can be overridden per request when a caller wants a different shape from the same reader.

Structured extraction

Set annotation_schema on the element to declare what the document contains, and annotation_prompt to say what it is. Because the schema lives on the element, each document type gets its own eyes and callers never restate the contract:

annotation_prompt: >-
  A Swedish crop-damage inspection form, filled in by hand in cursive.
  Read handwritten values into the typed fields. Decimal commas are Swedish
  (1,5 means 1.5). Leave a field null when the cell is genuinely empty —
  do not infer a value from neighbouring rows.
annotation_schema:
  type: object
  properties:
    farm_name: { type: string }
    inspection_date: { type: string }
    rows:
      type: array
      items:
        type: object
        properties:
          crop: { type: string }
          hectares: { type: number }
          yield_range: { type: string }
          damage_pct: { type: number }

The response then carries both the free text and the filled shape:

{
  "text": "…markdown with an HTML table…",
  "annotation": { "farm_name": "…", "rows": [ … ] },
  "pages": [ { "index": 0, "average_confidence": 0.97, "minimum_confidence": 0.11, "needs_review": true } ],
  "needs_review": true
}

Not every reader can fill a schema

annotation_schema is only as good as the reader behind it. Measured on the same Swedish crop-damage form (2026-07-30):

provider	schema filled?	result
mistral-ocr (dedicated /ocr endpoint)	yes	four rows with crops and hectares, including handwritten 1,5 read as 1.5
vllm-openai (self-hosted vision model)	no	{"rows": []} — valid JSON, finish_reason: stop, and empty

The self-hosted reader accepts the schema and satisfies its grammar without following its instructions, so it returns a well-formed empty shell rather than an error. The same call reads the document into a 25-cell table when asked for text. So: use vllm-openai for text, and a dedicated document endpoint when you need fields — and treat an empty annotation as a signal to check which reader produced it, not as evidence the document was blank.

Confidence is the review signal

On handwritten forms the average page confidence stays high while the minimum collapses — the low minimum lands on exactly the fields a human should check. That asymmetry is why min_confidence gates on the minimum rather than the average, and why needs_review is surfaced per page as well as overall.

A reader that returns a confidently wrong value with no hedge is more dangerous than one that returns nothing, because nothing announces itself and a wrong number does not. Set min_confidence on any document whose values are acted on automatically.

Composition

• Lives inside a lab, which supplies the endpoint and credentials.
• Attaches rate-limit.
• Pairs with brain for reasoning over what was read, and with document / files for storing the source and the result.
• Siblings: ears (speech → text), mouth (text → speech), brain (the LLM).

Relationships

• Attaches to: rate-limit

Capabilities

• optical-character-recognition: Turn images and PDFs into text
• handwriting: Read handwritten as well as printed text
• table-extraction: Preserve table structure from ruled forms
• structured-extraction: Fill a caller-supplied JSON Schema directly from the document
• confidence-scoring: Report per-page or per-word confidence for review routing

Properties

Property	Type	Default	Description
provider	string	"mistral-ocr"	OCR provider dispatch key. mistral-ocr posts to the provider’s dedicated document endpoint (POST {base_url}/ocr), which returns page markdown plus optional tables, confidence scores, and a schema-filled annotation in one call. vllm-openai drives a self-hosted vision model over chat-completions (POST {base_url}/chat/completions) with an image content part. custom behaves as vllm-openai against an arbitrary endpoint.
model_id	string	—	Model identifier sent to the provider API (e.g. mistral-ocr-4)
display_name	string	—	Human label for this reader
base_url	string	—	Optional per-element endpoint override. Wins over the parent lab’s base_url. Use for routing one eyes to an internal deployment (e.g. https://gpu.triform.cloud/ocr/v1) while keeping the lab’s hosted-API URL for siblings. Leave empty to inherit from the parent lab.
credential_ref	string	—	Reference to secret element with provider API key. Optional — falls back to the parent lab’s credentials when unset.
table_format	string	"html"	How tables are rendered in the extracted text. html preserves cell boundaries in ruled forms, which matters when a value’s meaning comes from which column it sits in; markdown reads better as prose.
include_images	boolean	false	Return cropped images of figures alongside the text. Increases response size substantially.
max_pages	integer	—	Cap on pages read from a multi-page document. Leave empty to read all pages.
prompt_mode	string	—	Trained prompt mode for self-hosted DeepSeek-OCR-lineage models (vllm-openai). These models do not follow arbitrary instructions but do have distinct trained modes, and no single mode is best for every document — docparse wins on dense ruled tables while multipage and grounding recover handwriting the others miss. Ignored by mistral-ocr. Leave empty for the provider default.
window_size	integer	1024	Repetition-suppression window for self-hosted models (vllm-openai). Larger windows more than halve runaway repeated rows on dense ruled forms. Independent of prompt_mode, which governs recall rather than repetition. Ignored by mistral-ocr.
annotation_schema	object	—	Optional JSON Schema describing the fields to extract from the document. When set, the reader fills this shape directly instead of returning only free text — one call does both OCR and extraction. This is the customer-configurable contract: it is where a form’s field names, types, and expectations are declared.
annotation_prompt	string	—	Guidance sent alongside annotation_schema — what the document is, what language it is in, whether to expect handwriting, and what each field means. Carries the context a schema alone cannot express (e.g. "a Swedish crop-damage inspection form, handwritten in cursive").
confidence_granularity	string	"page"	Whether the provider reports per-page or per-word confidence scores. Confidence is the review-routing signal: on handwritten forms the average stays high while the minimum collapses on exactly the hard cursive, so a low minimum is the cue to route a field to a human. Not all providers report it.
min_confidence	number	—	Optional review threshold. Pages or words scoring below this are flagged in the response as needing human review. Requires confidence_granularity other than none. Advisory — the reader still returns the text.
pricing	object	—	Cost reference (USD)

Operations

activity

Get /ops/activity | Auth: Read

Get activity events for this element

Scope depends on element capabilities: individual elements query by element_id, project-form elements with activity-scope-members include member activities, circle-level elements with activity-scope-all query the entire circle. Gracefully returns empty list if activities table is missing (old circles).

attachments

Get /ops/attachments | Auth: Read

List all modifiers and resources attached to this element

Returns both modifiers (policy enforcement) and resources (data injection) with is_modifier flag to distinguish. Items in the generated MODIFIER_TYPES list are modifiers; everything else is a resource. Includes cascade_policy and version pin info.

batch_stats

Get /ops/batch_stats | Auth: Read

Get per-element statistics for all children of this element

Returns per-child stats plus an aggregate. Most meaningful on compound or manifest form elements (repositories, circles, projects); atoms have no children so the result is an empty children array with a zeroed aggregate. Uses efficient GROUP BY SQL. Weighted averages for eval scores.

compose

Post /ops/compose | Auth: Execute

Batch add and remove modifiers on this element in a single call

Declarative composition: add modifiers by ref path (slug or path@version) and remove by attachment ID, all in one atomic call on the target element. Each ‘add’ entry resolves the source element, validates topology, attaches with optional priority and cascade policy. Each ‘remove’ entry deletes the attachment row. Returns a summary of what was added and removed. Example: compose({ add: [{ref: “my-prompt”}, {ref: “rate-limit/api@v2”, priority: 50}], remove: [{attachment_id: “uuid”}] })

context

Get /ops/context | Auth: Read

Get connected elements (graph traversal)

Graph traversal showing all connected elements with their relationship type (contains, contained_by, references, referenced_by, attaches, etc.). Use ?depth=N to control traversal depth (default 1) and ?types=actor,data to filter by element types.

cost

Get /ops/cost | Auth: Read

Get direct and recursively rolled-up wallet cost for this element

Ledger rows stay owned by the element that incurred them (for inference, the resolved brain). Containers, labs and apps return a deduplicated rollup across containment and explicit element references. Default period is day.

create

Post /ops/create | Auth: Write

Create child element

POST to the parent path — element_type goes in the request body, NOT the URL. Both element_type and slug are required and must be non-empty. Name is derived from slug if omitted. Writes to both Git and PostgreSQL. All elements are stored flat under the circle — no intermediate library wrapper rows.

delete

Delete /ops/delete | Auth: Admin

Delete element (soft delete)

Soft delete — sets state to ‘deleted’ but retains the record. Cannot delete elements that have children (has_no_bond precondition) or active runs. Requires admin auth and confirmation.

disable

Post /ops/disable | Auth: Admin

Disable element (hides and prevents use)

Idempotent — safe to call on already-disabled elements. Optionally pass a reason string. Disabled elements cannot be invoked or executed. Inverse of enable.

enable

Post /ops/enable | Auth: Admin

Enable element (makes usable and visible)

Idempotent — safe to call on already-enabled elements. Transitions element to ready/enabled state. Cannot enable deleted elements. Inverse of disable.

export_bundle

Get /ops/export/bundle | Auth: Read

Export element as downloadable git bundle

On non-root-namespace elements, returns a binary git bundle. On root-namespace (circle) elements, dispatch hands off to the circle’s own export_bundle op, which returns a multi-element JSON envelope with one base64 bundle per child element — this is intentional, not an error.

get

Get /ops/get | Auth: Read

Get element details

Element is already resolved by the routing layer — this returns the cached element, not a fresh DB query. Use the path /api/{circle}/{slug} to address elements.

import_bundle

Post /ops/import/bundle | Auth: Write

Import git bundle into element

Accepts a base64-encoded git bundle in the JSON bundle_base64 field. Use overwrite=true to replace existing elements with same slug (default skips duplicates). Imported elements get new UUIDs. Returns counts of imported/skipped elements and any errors.

info

Get /ops/info | Auth: Read

Get eyes metadata

Returns provider, model, and reading configuration — used by the document-reader picker UI.

intention

Get /ops/intention | Auth: Read

Get element intention with full inheritance chain

Returns three levels: direct (this element’s intention), inherited (from category and root), and resolved (final merged intention). Useful for understanding an element’s purpose in context of its hierarchy.

promote

Post /ops/promote | Auth: Admin

Promote element configuration to a target environment

Only for manifest-form elements (projects). Environments advance: dev → demo → live. dev→demo requires member+ role, demo→live requires admin. Freezes member versions at promotion time (creates snapshot). Persists environment config to spec.environments.

read

Post /ops/read | Auth: Execute

Read a document (image or PDF) into text

One call does OCR and — when an annotation schema is configured or supplied — structured extraction. Pass the document as base64 or a URL. The element’s saved annotation_schema / annotation_prompt are used unless overridden here, so a configured eyes needs only the document.

readme

Get /ops/readme | Auth: Read

Get element README.md content

Reads README.md from the element’s git repository. Returns empty content (not an error) if no README exists. Always returns markdown format.

readme_update

Post /ops/readme_update | Auth: Write

Update element README.md content

Creates or overwrites README.md in the element’s git repo. Commits to the draft branch. Content must be provided as a markdown string.

remove-modifier

Post /ops/remove-modifier | Auth: Execute

Remove an attached modifier from this element by attachment ID

Removes a modifier/resource attachment by its row ID. The ID comes from the attachments or context API. This is the reverse of attach — called on the target element, not the source.

restore

Post /ops/restore | Auth: Admin

Restore element to a specific version

Automatically snapshots the current state before restoring (creates a ‘Before restore to vN’ version entry). Writes restored spec to git as .triform/spec.yaml. Git failures warn but don’t fail the operation — DB state is authoritative. Cannot restore deleted elements.

schema

Get /ops/schema | Auth: Read

Get element input/output schema (MCP tools/list compatible)

Returns type-level port schemas from the TypeRegistry — not instance-specific overrides. Includes direction (input/output), required flag, and JSON schema per port. Useful for understanding what data an element accepts and produces.

source

Get /ops/source | Auth: Read

Get any file’s content from the element’s git repository

Reads an arbitrary file from the element’s CAS-backed git tree by its relative path. Same store as readme, just generalized. Path safety: rejects .. traversal, leading /, and null bytes. Use this to view main.py for action elements, asset files for SPAs, etc. Returns empty content (not an error) if the file doesn’t exist.

source_branches

Get /ops/source/branches | Auth: Read

List Source branches for this element

Returns the standard draft/demo/live Source branches, their current commits, and promotion relationships. Use GET /api/{element_path}/ops/source/branches.

source_diff3

Get /ops/source/diff3 | Auth: Read

Preview a three-way merge of two commits without writing

Read-only. Computes what a merge of theirs (default: your base_commit) into ours (default: the draft tip) would produce: per-file conflicts with ours/theirs/base content, plus clean/fast_forward flags. Use after a 409 to see exactly what moved before re-splicing your change. GET /api/{element_path}/ops/source/diff3?theirs={sha}.

source_merge

Post /ops/source/merge | Auth: Write

Three-way merge a commit into the draft branch

Merges theirs (default: your base_commit) into the draft branch. Clean (non-overlapping edits) → commits and advances draft, returns merged: true + the new commit. Conflicts → writes nothing and returns the conflict set to resolve and retry. Refused inside an active version-set scope (use version_sets_merge). POST /api/{element_path}/ops/source/merge {“theirs”: “{sha}”}.

source_promote

Post /ops/source/promote | Auth: Write

Promote Source branch forward

Promotes draft to demo or demo to live through the generated element op path. Direct Git pushes to demo/live are blocked by Source policy.

source_repair

Post /ops/source/repair | Auth: Write

Inspect or repair the element Source index

Runs Source repair through the element operation path. Defaults to dry_run=true; set dry_run=false only after reviewing a dry-run report.

source_status

Get /ops/source/status | Auth: Read

Get Source control status for this element

Returns the branch-aware clone URL, checkout commands, current draft commit, child source-link count, portable export summary, Source health, warnings, and auth hints for the addressed element. Use the element-first path: GET /api/{element_path}/ops/source/status.

source_validate

Post /ops/source/validate | Auth: Read

Validate Source branch contents

Validates a Source branch before accepting local Git workflow changes or promotion. Defaults to branch=draft and rejects runtime data, generated output, secret material, and unreadable CAS refs.

stats

Get /ops/stats | Auth: Read

Get aggregate statistics for this element

Health status is computed: error if errors_per_day > 5 or success_rate < 0.8, warning if errors_per_day > 0 or success_rate < 0.95. Firing alerts escalate health to error/warning. Default period is ‘day’. Returns runs_per_day, success_rate, avg_duration_ms, and more.

test

Post /ops/test | Auth: Execute

Verify the OCR connection

Reads a short built-in test document to verify the connection works.

tree

Get /ops/tree | Auth: Read

Get the element’s position in the graph — ancestors, children, references, and subtree statistics

Uses per-circle ElementGraph cache for O(1) lookups. Returns ancestors (containment chain), children (direct), members (references), referenced_by (reverse refs), attachments, and subtree stats. Default depth is 3, max is 10. Pass ?include_metadata=true for name/state on each node.

update

Patch /ops/update | Auth: Write

Update element

Partial update — send only the fields you want to change. spec, name, and intention are all independently optional. spec MUST be a JSON object when present; deep-merged into the existing spec by default. Empty {"spec":{}} preserves existing spec content but still records a new version (no-op for content, not for version state). To clear/replace the entire spec wholesale send {"spec":{...},"deep":false}. List-typed spec fields use replace semantics (the patch list replaces the existing list, no array merging). Coordinates Git + DB writes. Slug cannot be changed after creation.

update_meta

Patch /ops/update_meta | Auth: Write

Update element metadata (lightweight merge — does NOT bump version or snapshot spec)

Shallow JSONB merge into element.meta. Top-level keys in the provided value replace existing meta values; other keys are preserved. Used for UI metadata like canvas positions, panel state, viewer preferences. Wire-shape op_name is update_meta (distinct from update) so SSE subscribers + the cache auto-invalidator can distinguish lightweight metadata changes from spec edits without inspecting the payload. The MutatingElementStore wrapper stamps this op_name on the lifecycle event emitted by update_element_meta storage calls.

version

Get /ops/version | Auth: Read

Get current version or full history

Returns current version by default. Pass ?history=true for full version history (up to ?limit=N, default 50). Versions are backed by the element_versions table. Every spec update creates a new version entry.

Error Codes

Code	Class	Retryable	Description
EYES_UNAVAILABLE	internal	yes	Eyes’ parent lab is unreachable or credentials missing
EYES_CREDENTIAL_MISSING	auth	no	Provider API key not set
EYES_DOCUMENT_UNSUPPORTED	validation	no	Input document format not supported by the provider
EYES_DOCUMENT_MISSING	validation	no	Neither document_b64 nor document_url was supplied
EYES_READ_FAILED	internal	yes	Provider returned an error while reading the document