Reference

Every name a .dreamrc can use, with its config keys. The architecture is why the system has this shape, the spec is the grammar, and the contract says what your data must look like; this page is the lookup table — the names here are exactly the registries the validator checks, so anything not listed fails validation with a did-you-mean. For the pipeline and the registries in code, see library internals.

Kinds — the catalog's, and the view's

Both say "kind" about the same field, and they mean different things. Keeping the two apart is the whole design.

Inventory kinds — what the catalog states

Field.kind says how the BYTES ARE ADDRESSED and nothing about what they mean. Six values, closed, and nothing turns a name or a shape into one of them:

For a loose file the extension picks the addressing and only that: .mp4 / .webm → video, .jpg / .png → image, .csv / .parquet → tensor. .vtt, .glb and .json are all file — "labelled spans", "geometry" and "keypoints" are meanings, and meanings come from the .dreamrc.

meta is raw facts, recorded because they are free: dtype, shape, names, ext, schema, codec. A decoder reads them to do its job — the schema name selects an MCAP decoder, the extension selects a parser — and nothing reads them to decide what a field is.

Payload kinds — what a view asks for

read(ref, { as }) decodes into one of these. The as comes from the slot the field was bound to, so the interpretation is written in the .dreamrc:

Ask for one the bytes cannot become and the read throws naming both sides (observation.state is float32 [6]; keypoints needs [J,2] or [J,3]) instead of drawing something wrong.

Which payloads a field can serve — and when you write as:

The bridge between the two tables. It reads the addressing kind and nothing else — no name, no shape, no extension:

Intersect a field's row with the payloads its slot reads (per component):

• exactly one survives — nothing to write; the container already settled it (a video bound to videoStack.cameras, a tensor bound to pointCloud);
• more than one — the binding says which with as:, and until it does the panel refuses and prints the choice (a .json bound to overlays is a skeleton file or a caption track, and nothing about the bytes says which);
• none — that slot cannot show that field, and it says so by name.

A declared as: short-circuits the table entirely: it is the author speaking, and it is how a LeRobot subtask_index is bound as: segments and joins its label table.

Where a payload comes from — a feature in the dataset's own container, or a standard file beside it (WebVTT/SRT, COCO, glTF, parquet) — is in the contract. No payload requires a format of our own — where a DreamLake structure exists (the motion-track Parquet profile v1), an existing standard carries the same payload.

Data the library does not know

The payload set versions with the library, but shipping your own end to end needs no library change:

1. Declare where it is — dataset.annotations: { gaze: "annotations/gaze_{episode_index:06d}.json" }. The field lands in every episode's catalog as a file with its ext in meta. Nothing is claimed about the contents.
2. Render it — the host registers a component (registerComponent({ name: 'gazePanel', component, reads: { fields: ['file'] } })) whose slot reads file, fetches the URL and parses its own wire format. The .dreamrc writes view: gazePanel and binds the field — no schema change, no library change.
3. Promote it — once a payload proves general (the way segments covers tasks/subtasks/actions/phases), it graduates into the library: a Payload member, a decoder that accepts the wire shapes found in the wild (see normalizeSegments), and a built-in component with a slot that reads it.

The same registries extend the other axes: registerFormat for a new dataset layout, registerStorage for a new backend — including composite backends (an overlay bucket layered over a read-only base is just another two-method Storage whose list merges and whose resolveUrl checks the overlay first).

Payload contracts in full

The exact structures a read(ref, { as }) returns — what a component is handed, and therefore what a host-registered component should expect. These are the in-memory contract: our one self-defined schema layer, versioned with the library (why it exists). Every type below is exported from @dreamlake/viz/dataset-viz, and the source (dataset-viz/types.ts) is the authority — this listing mirrors it.

type Payload =
  // A camera stream. `window` marks the sub-range of `url` this episode
  // occupies (LeRobot v3 concatenates episodes into one file per camera).
  | { kind: 'video'; url: string; window?: { from: number; to: number } }
  | { kind: 'series'; timestamps: number[]; columns: Record<string, number[]> }
  | { kind: 'image'; url: string }
  | { kind: 'frames'; count: number; fps?: number; frameAt: (i: number) => Promise<string> }
  | { kind: 'file'; url: string; ext?: string }
  | { kind: 'keypoints'; keypoints: KeypointTrack }
  | { kind: 'segments'; segments: Segment[] }
  | { kind: 'pose3d'; timestamps: number[]; joints: number[][]; shape: number[]; layout?: string }
  // Static 3D geometry — node names inside it are the join key for
  // per-frame transform3d / vertices3d tracks.
  | { kind: 'mesh3d'; url: string; format: 'gltf' | 'glb' | 'obj' }
  | { kind: 'transform3d'; timestamps: number[]; values: Float32Array; layout: TransformLayout }
  // Per-frame vertex positions for a deforming mesh, fetched lazily;
  // topology comes from the paired mesh3d node.
  | { kind: 'vertices3d'; count: number; fps?: number; vertexCount: number; at: (i: number) => Promise<Float32Array> }
  // Raw values, never pre-colorized — the component owns the colormap.
  | { kind: 'depth'; count: number; fps?: number; at: (i: number) => Promise<DepthFrame> }
  | { kind: 'pointcloud'; count: number; fps?: number; at: (i: number) => Promise<CloudFrame> }

/** One labelled time span — the canonical shape for every segment-class
 *  track (tasks, subtasks, actions, phases, subtitle files). */
interface Segment {
  start: number
  end: number
  label: string
}

/** Per-frame 2D keypoints, normalized out of whatever the source was (COCO
 *  JSON, a per-frame tensor). Coordinates are pixels in width × height;
 *  frames are sparse — a missing key means "no detection". */
interface KeypointTrack {
  width: number
  height: number
  /** Frame index → detections. Frame k is shown at k / fps seconds. */
  frames: Map<number, KeypointDetection[]>
  fps: number
  /** Index pairs drawn as bones. COCO-17 and Hand-21 presets are built in. */
  skeleton?: [number, number][]
  jointNames?: string[]
}

interface KeypointDetection {
  /** [x, y] per joint, pixels. */
  points: number[][]
  score?: number
  /** Free label — "left" / "right" / a class name. */
  group?: string
  /** [x1, y1, x2, y2] pixels, when the source carries one. */
  box?: [number, number, number, number]
}

/** One depth map. `data` is row-major width×height; `scale` converts a raw
 *  value to metres (0.001 for millimetre uint16; omit when unknown — the
 *  component then normalizes to the frame's own range). 0 = no reading. */
interface DepthFrame {
  width: number
  height: number
  data: Float32Array | Uint16Array | Uint8Array
  scale?: number
}

/** One cloud frame. `xyz` is N×3 metres; `rgb` optional N×3, either 0-255
 *  bytes or 0-1 floats. */
interface CloudFrame {
  xyz: Float32Array
  rgb?: Uint8Array | Float32Array
}

/** Component order of a transform3d row. Both orders occur in the wild
 *  (glTF/three use xyzw; robotics and MANO exports use wxyz), so the
 *  adapter states which it read instead of guessing downstream. */
type TransformLayout = 'txyz_qwxyz' | 'txyz_qxyzw'

Storage drivers

Declared in a standalone file's storage: block, or injected by the host for a file at its dataset root (who writes it). Configs carry identifiers only — never credentials.

Hosts add drivers with registerStorage(name, factory); the whole contract is list(path) + resolveUrl(path). (The legacy schema-viz pages use basePath/id for their own http/hf drivers — a different subsystem; a .dreamrc always uses the keys above.)

The http index.json manifest

A static host can't enumerate itself, so the http driver lists a directory by fetching <dir>/index.json. Every directory a glob walks needs one — for episodes: "episodes/*/" that is episodes/index.json plus one inside each episode folder (formats with episodes: auto fetch known paths and need none). The shape mirrors what list() returns:

{
  "entries": [
    { "name": "run_a", "path": "episodes/run_a", "type": "dir" },
    { "name": "cam_ego.mp4", "path": "episodes/run_a/cam_ego.mp4", "type": "file" }
  ]
}

name + type ("file" | "dir") are required; path is the storage-relative (or absolute) location — entries may point anywhere, which is how a manifest can reference files hosted elsewhere. Don't list index.json itself; .dreamrc need not be listed either (the app fetches it directly).

Formats

The dataset.format adapters. Common to all: the catalog is an inventory — each field's address plus the facts the container reported — and read(ref, { as }) is where anything is decoded. dataset.annotations declares extra tracks (spec), merged in by the core — never a per-format concern.

lerobot

LeRobot v2.0 / v2.1 / v3.0. The entry file is meta/info.json.

The inventory is meta/info.json's features, verbatim — one entry per feature, carrying its dtype, shape and names as found. dtype is the only thing read, and only to decide addressing: video → video, image → frames, language / string → text, any numeric dtype → tensor. Anything else is omitted with a warning naming it, including a depth VIDEO (the feature's own video.is_depth_map flag) — no browser decodes 12-bit log-quantized H.265.

Exactly four bookkeeping columns are dropped — timestamp, frame_index, episode_index, index — and nothing else. task_index and subtask_index stay: an integer pointing into a label table is data, and a view binding one as: segments is what fetches meta/tasks* and joins it. Timeline: per-episode length at the dataset's fps.

folder

Folder-per-episode, no manifest — the directory layout is the contract.

The inventory is the directory listing and nothing more: one field per file, addressed by its basename, with path, ext and size in meta. The extension picks the ADDRESSING only — mp4/webm → video, jpg/png → image, csv/parquet → tensor, and everything else — .vtt, .glb, .json — is file. Consecutive numbered stills collapse into one frames field. annotations/ is listed too, and a file there replaces a same-named one at the root.

No file is opened and no name is read for meaning; a view binding subtasks.vtt as segments is what parses it, and the extension's second say — at read time — picks only which parser. Series tables take their x-axis from a timestamp-like column, row index otherwise. Timeline is null — media components probe durations themselves.

umi

Zarr stores, two modes detected from the store itself.

The inventory reads each array's own metadata. An image codec is the store declaring that one chunk is one encoded image, so that array is frames (byte-ranged a frame at a time — the store is never downloaded whole); numeric arrays are tensor, carrying dtype, shape and codec. Nothing is synthesized: a names list whose length does not match the array's width names nothing, and a [T,J,3] keypoint array is a tensor like any other until a view binds it as: pose3d. Arrays this reader cannot window (three dims or more) are omitted with a warning. Timeline from frame counts at fps.

mcap

MCAP v1 indexed containers — one episode per *.mcap file, read in place over HTTP range requests (a 512MB file costs ~130KB before any field is read).

The inventory is the channel list, each channel carrying its schema name and encoding in meta. Addressing follows what the schema says a message IS: json channels with numeric leaves → tensor (dot paths are the columns, e.g. linear_accel.x; dims come from the jsonschema or the first message), string-only leaves → text; Foxglove image schemas → frames (one message is one image, byte-ranged per frame); FrameTransform → one tensor per child frame; a SceneUpdate carrying a ModelPrimitive → file.

The schema name is then a DECODER selector at read time — a foxglove.PointCloud channel bound as: pointcloud, a tf channel bound as: transform3d — never a conclusion at catalog time. Channels outside this v1 scope — ros1msg/cdr/ros2idl decoding, Grid, attachments, bz2 chunks — are omitted from the catalog with a console warning naming topic + encoding, never mislisted. Unindexed files fail with a clear error (repack with mcap recover). Timeline: message start→end from the summary statistics.

View components

Live example of each — a minimal .dreamrc and its render — on the view components catalog. Here: the config surface.

A slot is where meaning is stated. Each component declares, per binding slot, the payload kinds it reads — and that one declaration is what read(ref, { as }) is called with, what a * glob is filtered by, and what an author's as: is checked against. Bind a field to series and it is read as traces; bind the same field to tracks and it is read as spans. A slot a component does not declare is rejected rather than ignored: overlays: on a lineChart is an author expecting to see something, and silence there is the failure mode this design removes.

Whether a binding needs an as: follows from the field, not from the view — the rule. The media/geometry slots (cameras, cloud, geometry) take bare refs with nowhere to write one, so they are always settled by the field's own addressing kind; series, tracks and overlays take { field, as } entries. A * glob selects only the fields whose addressing kind can serve the slot — cameras: ["*"] on videoStack takes the cameras and leaves the tensors — while an explicit ref always passes and fails loudly if it cannot be read.

Bindings are the keys the view interprets — its slot names (cameras / cloud / geometry / series / tracks / overlays); every other key passes through as a prop to the underlying Episode* component, so its documented props are all available. A split: row is a fixed-height strip (height on the split, default 280) — its sizing keys (width fixed box · flex stretch share · minWidth squish floor · child height override) are consumed by the layout, not the component; media sizes its width from its aspect, charts fill their slot, and overflow scrolls horizontally, synced across episodes under a SyncScrollProvider. "row" marks components that render in the compact layout="row" strips (dataset lists); the rest appear in grid layout only.

Hosts add views with registerComponent({ name, component, reads, slotNames?, rows? }) — reads is the slot → payload declaration above (canonical keys: fields, series, tracks, overlays), slotNames renames a slot's config-surface key the way videoStack surfaces fields as cameras, and a component that omits reads opts out of binding checks. A component receives { fields, info?, read, timeline, cursor, config, layout }.