Reference

Every name a .dreamrc can use, with its config keys. The architecture is why the system has this shape, the spec is the grammar, and the contract says what your data must look like; this page is the lookup table — the names here are exactly the registries the validator checks, so anything not listed fails validation with a did-you-mean. For the pipeline and the registries in code, see the architecture's under-the-hood section.

Kinds — the catalog's, and the view's

Both say "kind" about the same field, and they mean different things. Keeping the two apart is the whole design.

Inventory kinds — what the catalog states

Field.kind says how the BYTES ARE ADDRESSED and nothing about what they mean. Six values, closed, and nothing turns a name or a shape into one of them:

For a loose file the extension picks the addressing and only that: .mp4 / .webm → video, .jpg / .png → image, .csv / .parquet → tensor. .vtt, .glb and .json are all file — "labelled spans", "geometry" and "keypoints" are meanings, and meanings come from the .dreamrc.

meta is raw facts, recorded because they are free: dtype, shape, names, ext, schema, codec. A decoder reads them to do its job — the schema name selects an MCAP decoder, the extension selects a parser — and nothing reads them to decide what a field is.

Payload kinds — what a view asks for

read(ref, { as }) decodes into one of these. The as comes from the slot the field was bound to, so the interpretation is written in the .dreamrc:

Ask for one the bytes cannot become and the read throws naming both sides (observation.state is float32 [6]; keypoints needs [J,2] or [J,3]) instead of drawing something wrong.

Which payloads a field can serve — and when you write as:

The bridge between the two tables. It reads the addressing kind and nothing else — no name, no shape, no extension:

Intersect a field's row with the payloads its slot reads (per component):

• exactly one survives — nothing to write; the container already settled it (a video bound to videoStack.cameras, a tensor bound to pointCloud);
• more than one — the binding says which with as:, and until it does the panel refuses and prints the choice (a .json bound to overlays is a skeleton file or a caption track, and nothing about the bytes says which);
• none — that slot cannot show that field, and it says so by name.

A declared as: short-circuits the table entirely: it is the author speaking, and it is how a LeRobot subtask_index is bound as: segments and joins its label table.

Where a payload comes from — a feature in the dataset's own container, or a standard file beside it (WebVTT/SRT, COCO, glTF, parquet) — is in the contract. No payload requires a format of our own — where a DreamLake structure exists (the motion-track Parquet profile v1), an existing standard carries the same payload.

Data the library does not know

The payload set versions with the library, but shipping your own end to end needs no library change:

1. Declare where it is — dataset.annotations: { gaze: "annotations/gaze_{episode_index:06d}.json" }. The field lands in every episode's catalog as a file with its ext in meta. Nothing is claimed about the contents.
2. Render it — the host registers a component (registerComponent({ name: 'gazePanel', component, reads: { fields: ['file'] } })) whose slot reads file, fetches the URL and parses its own wire format. The .dreamrc writes view: gazePanel and binds the field — no schema change, no library change.
3. Promote it — once a payload proves general (the way segments covers tasks/subtasks/actions/phases), it graduates into the library: a Payload member, a decoder that accepts the wire shapes found in the wild (see normalizeSegments), and a built-in component with a slot that reads it.

The same registries extend the other axes: registerFormat for a new dataset layout, registerStorage for a new backend — including composite backends (an overlay bucket layered over a read-only base is just another two-method Storage whose list merges and whose resolveUrl checks the overlay first).

Payload contracts in full

The exact structures a read(ref, { as }) returns — what a component is handed, and therefore what a host-registered component should expect. These are the in-memory contract: our one self-defined schema layer, versioned with the library (why it exists). Every type below is exported from @dreamlake/viz/dataset-viz, and the source (dataset-viz/types.ts) is the authority — this listing mirrors it.

type Payload =
  // A camera stream. `window` marks the sub-range of `url` this episode
  // occupies (LeRobot v3 concatenates episodes into one file per camera).
  | { kind: 'video'; url: string; window?: { from: number; to: number } }
  | { kind: 'series'; timestamps: number[]; columns: Record<string, number[]> }
  | { kind: 'image'; url: string }
  | { kind: 'frames'; count: number; fps?: number; frameAt: (i: number) => Promise<string> }
  | { kind: 'file'; url: string; ext?: string }
  | { kind: 'keypoints'; keypoints: KeypointTrack }
  | { kind: 'segments'; segments: Segment[] }
  | { kind: 'pose3d'; timestamps: number[]; joints: number[][]; shape: number[]; layout?: string }
  // Static 3D geometry — node names inside it are the join key for
  // per-frame transform3d / vertices3d tracks.
  | { kind: 'mesh3d'; url: string; format: 'gltf' | 'glb' | 'obj' }
  | { kind: 'transform3d'; timestamps: number[]; values: Float32Array; layout: TransformLayout }
  // Per-frame vertex positions for a deforming mesh, fetched lazily;
  // topology comes from the paired mesh3d node.
  | {
      kind: 'vertices3d'
      count: number
      fps?: number
      vertexCount: number
      at: (i: number) => Promise<Float32Array>
    }
  // Raw values, never pre-colorized — the component owns the colormap.
  | { kind: 'depth'; count: number; fps?: number; at: (i: number) => Promise<DepthFrame> }
  | { kind: 'pointcloud'; count: number; fps?: number; at: (i: number) => Promise<CloudFrame> }

/** One labelled time span — the canonical shape for every segment-class
 *  track (tasks, subtasks, actions, phases, subtitle files). */
interface Segment {
  start: number
  end: number
  label: string
}

/** Per-frame 2D keypoints, normalized out of whatever the source was (COCO
 *  JSON, a per-frame tensor). Coordinates are pixels in width × height;
 *  frames are sparse — a missing key means "no detection". */
interface KeypointTrack {
  width: number
  height: number
  /** Frame index → detections. Frame k is shown at k / fps seconds. */
  frames: Map<number, KeypointDetection[]>
  fps: number
  /** Index pairs drawn as bones. COCO-17 and Hand-21 presets are built in. */
  skeleton?: [number, number][]
  jointNames?: string[]
}

interface KeypointDetection {
  /** [x, y] per joint, pixels. */
  points: number[][]
  score?: number
  /** Free label — "left" / "right" / a class name. */
  group?: string
  /** [x1, y1, x2, y2] pixels, when the source carries one. */
  box?: [number, number, number, number]
}

/** One depth map. `data` is row-major width×height; `scale` converts a raw
 *  value to metres (0.001 for millimetre uint16; omit when unknown — the
 *  component then normalizes to the frame's own range). 0 = no reading. */
interface DepthFrame {
  width: number
  height: number
  data: Float32Array | Uint16Array | Uint8Array
  scale?: number
}

/** One cloud frame. `xyz` is N×3 metres; `rgb` optional N×3, either 0-255
 *  bytes or 0-1 floats. */
interface CloudFrame {
  xyz: Float32Array
  rgb?: Uint8Array | Float32Array
}

/** Component order of a transform3d row. Both orders occur in the wild
 *  (glTF/three use xyzw; robotics and MANO exports use wxyz), so the
 *  adapter states which it read instead of guessing downstream. */
type TransformLayout = 'txyz_qwxyz' | 'txyz_qxyzw'

Storage drivers

Declared in a standalone file's storage: block, or injected by the host for a file at its dataset root (who writes it). Configs carry identifiers only — never credentials.

Hosts add drivers with registerStorage(name, factory); the whole contract is list(path) + resolveUrl(path). (The legacy schema-viz pages use basePath/id for their own http/hf drivers — a different subsystem; a .dreamrc always uses the keys above.)

The http index.json manifest

A static host can't enumerate itself, so the http driver lists a directory by fetching <dir>/index.json. Every directory a glob walks needs one — for episodes: "episodes/*/" that is episodes/index.json plus one inside each episode folder (formats with episodes: auto fetch known paths and need none). The shape mirrors what list() returns:

{
  "entries": [
    { "name": "run_a", "path": "episodes/run_a", "type": "dir" },
    { "name": "cam_ego.mp4", "path": "episodes/run_a/cam_ego.mp4", "type": "file" }
  ]
}

type ("file" | "dir") defaults to "file", name falls back to the last segment of path, and a bare JSON array of filename strings lists plain files; path is the storage-relative (or absolute) location — entries may point anywhere, which is how a manifest can reference files hosted elsewhere. Don't list index.json itself; .dreamrc need not be listed either (the app fetches it directly).

Formats

The dataset.format adapters. Common to all: the catalog is an inventory — each field's address plus the facts the container reported — and read(ref, { as }) is where anything is decoded. dataset.annotations declares extra tracks (spec), merged in by the core — never a per-format concern.

lerobot

LeRobot v2.0 / v2.1 / v3.0. The entry file is meta/info.json.

The inventory is meta/info.json's features, verbatim — one entry per feature, carrying its dtype, shape and names as found. dtype is the only thing read, and only to decide addressing: video → video, image → frames, language / string → text, any numeric dtype → tensor. Anything else is omitted with a warning naming it, including a depth VIDEO (the feature's own video.is_depth_map flag) — no browser decodes 12-bit log-quantized H.265.

Exactly four bookkeeping columns are dropped — timestamp, frame_index, episode_index, index — and nothing else. task_index and subtask_index stay: an integer pointing into a label table is data, and a view binding one as: segments is what fetches meta/tasks* and joins it. Timeline: per-episode length at the dataset's fps.

folder

Folder-per-episode, no manifest — the directory layout is the contract.

The inventory is the directory listing and nothing more: one field per file, addressed by its basename, with path, ext and size in meta. The extension picks the ADDRESSING only — mp4/webm → video, jpg/png → image, csv/parquet → tensor, and everything else — .vtt, .glb, .json — is file. Consecutive numbered stills collapse into one frames field. annotations/ is listed too, and a file there replaces a same-named one at the root.

No file is opened and no name is read for meaning; a view binding subtasks.vtt as segments is what parses it, and the extension's second say — at read time — picks only which parser. Series tables take their x-axis from a timestamp-like column, row index otherwise. Timeline is null — media components probe durations themselves.

umi

Zarr stores, two modes detected from the store itself.

The inventory reads each array's own metadata. An image codec is the store declaring that one chunk is one encoded image, so that array is frames (byte-ranged a frame at a time — the store is never downloaded whole); numeric arrays are tensor, carrying dtype, shape and codec. Nothing is synthesized: a names list whose length does not match the array's width names nothing, and a [T,J,3] keypoint array is a tensor like any other until a view binds it as: pose3d. Arrays this reader cannot window (three dims or more) are omitted with a warning. Timeline from frame counts at fps.

mcap

MCAP v1 indexed containers — one episode per *.mcap file, read in place over HTTP range requests (a 512MB file costs ~130KB before any field is read).

The multi-file arrangement has a live worked example — a folder of nuScenes scenes, one episode per file — as template 3 and in the gallery.

The inventory is the channel list, each channel carrying its schema name and encoding in meta. Addressing follows what the schema says a message IS: Foxglove image schemas → frames (one message is one image, byte-ranged per frame); FrameTransform → one tensor per child frame (/tf::box7); a SceneUpdate → one file field per model-bearing entity (/scene::box7), which a 3D view joins to its /tf::box7 track by the :: segment. Schema-name dispatch applies across wire formats: a json channel bearing a well-known name decodes exactly like its protobuf twin (byte fields arrive base64 and are converted by name — a stub jsonschema costs nothing), so json writers get images and 3D too. Other json channels list by their leaves: numeric → tensor (dot paths are the columns, e.g. linear_accel.x; dims come from the jsonschema or the first message), string-only → text.

ROS bags decode through the same paths — ROS 2 (cdr messages with ros2msg schema text, rosbag2's own MCAP output) and ROS 1 (ros1 + ros1msg, rosbag conversions). WELL-KNOWN ROS types are normalized to their foxglove TWIN's shape and dispatch on the twin name (recorded in meta.twin beside the honest meta.schema): tf2_msgs/TFMessage → FrameTransforms, sensor_msgs CompressedImage/Image/PointCloud2 → CompressedImage/RawImage/PointCloud, geometry_msgs/PoseStamped and nav_msgs/Odometry → PoseInFrame. Every other ros channel takes first-message leaf discovery like json (JointState, IMU, …). A tf child read as: transform3d composes its LOCAL parent→child transforms down the declared parent_frame_id chain (across /tf + /tf_static) into fixed-frame poses — so a tf-only humanoid recording renders as a moving skeleton (axis triads + bones between tracked frames), and a models: URDF turns that skeleton into the robot itself.

The schema name is then a DECODER selector at read time — a foxglove.PointCloud channel bound as: pointcloud, a tf channel bound as: transform3d — never a conclusion at catalog time. Channels outside this v1 scope — ros1msg/ros2idl decoding, Grid, attachments, bz2 chunks — are omitted from the catalog with a console warning naming topic + encoding, never mislisted. Unindexed files fail with a clear error (repack with mcap recover). Timeline: message start→end from the summary statistics.

The default mcap view

defaultMcapViews(fields) (exported from @dreamlake/viz/dataset-viz) turns one episode's catalog into a ready layout: image schemas → frameStack, each foxglove.PointCloud → a pointCloud pane, tf children and PoseInFrame → one recon3d (with as: transform3d written explicitly, up: z — the mcap ecosystem's own convention, ROS REP-103 / the Foxglove 3D panel's fixed +Z-up display — and originMarker: axes, since a world-frame recording's origin is the floor, not a recording camera), remaining numeric channels → lineCharts, string channels → a timeline, closed by the fields catalog (no metaPanel — the preview chrome already carries the file's name and duration). Every 3D pane gets a scrubbable timeline ruler directly under it: the 3D views follow the cursor but cannot set it, and on a page tall enough to scroll, the playhead must never be out of reach — span-shaped tracks ride those rulers instead of a separate tail timeline. Cloud panes are bounded like charts: the busiest four (they are live WebGL contexts, and AV recordings carry eight lidar topics), folded into a two-column grid when there are more than two. Charts are bounded by three mechanical rules (addressing and shape facts, never a guess at meaning): at most six panes ranked by recorded volume (messages × dims), topics wider than 48 dims never chart by default, and a topic that path-prefixes another (/plan over /plan/oracle) is the writer's own grouping — the pair shares one chart while the combined width stays under 12 traces. Three or more charts fold into a two-column grid; everything that didn't chart is still in the fields catalog. A sensor_msgs/JointState topic also binds to the 3D pane's joints: slot, and its joint names — like a tf tree's frame ids — can select a known robot description from robot-presets.ts, which then turns those angles into a pose per link. It returns [] when nothing yields a data view, so callers fall back honestly instead of rendering empty chrome. mcapDreamrcYaml(views) serializes the result as a complete .dreamrc.

This is the one sanctioned default layout, and the precedent is precise: a declarative standard schema may drive a default view; a name or a shape never may. A Foxglove schema name is the writer declaring meaning by published spec — reading it is reading, the same trust lerobot places in dtype: video. The generator consumes only such declarations plus each kind's meaning-free reading (numbers chart as numbers), so a plain [512,6] tensor is only ever a chart here — nothing short of the file declaring foxglove.PointCloud, or a human writing as: pointcloud, makes it a point cloud. It is a caller-side helper: views: stays required in a .dreamrc, an authored config always wins, and no other format gains a fallback by its existence.

View components

Live example of each — a minimal .dreamrc and its render — on the view components catalog. Here: the config surface.

A slot is where meaning is stated. Each component declares, per binding slot, the payload kinds it reads — and that one declaration is what read(ref, { as }) is called with, what a * glob is filtered by, and what an author's as: is checked against. Bind a field to series and it is read as traces; bind the same field to tracks and it is read as spans. A slot a component does not declare is rejected rather than ignored: overlays: on a lineChart is an author expecting to see something, and silence there is the failure mode this design removes.

Whether a binding needs an as: follows from the field, not from the view — the rule. The media/geometry slots (cameras, cloud, geometry) take bare field names with nowhere to write one, so they are always settled by the field's own addressing kind; series, tracks and overlays take { field, as } entries. A * glob selects only the fields whose addressing kind can serve the slot — cameras: ["*"] on videoStack takes the cameras and leaves the tensors — while an explicit ref always passes and fails loudly if it cannot be read.

Bindings are the keys the view interprets — its slot names (cameras / cloud / geometry / series / tracks / overlays); every other key passes through as a prop to the underlying Episode* component, so its documented props are all available. Every view takes width / height / aspectRatio — a sized view fills its box with row-strip fill semantics (how); the per-view height defaults below apply when nothing sizes the view. A split: row is a fixed-height strip (height on the split, default 280) — its per-child keys (width fixed box · aspectRatio width from strip height · flex stretch share · minWidth squish floor · child height override) are consumed by the layout, not the component; media sizes its width from its aspect, charts fill their slot, and overflow scrolls horizontally, synced across episodes under a SyncScrollProvider.

Hosts add views with registerComponent({ name, component, reads, slotNames? }) — reads is the slot → payload declaration above (canonical keys: fields, series, tracks, overlays), slotNames renames a slot's config-surface key the way videoStack surfaces fields as cameras, and a component that omits reads opts out of binding checks. A component receives { fields, info?, read, timeline, cursor, config }.