What your data must look like

This page is the data side of the contract: what your container must declare, the shape each payload demands, and where annotation tracks live. The program never decides what your data means — your .dreamrc does (the architecture) — so the requirements are much smaller than a list of naming rules. There are only two:

1. The container must say which bytes are encoded media, so they can be addressed a frame at a time instead of being read as numbers. Every format already does this — LeRobot's dtype, a zarr array's codec, an MCAP channel's schema, a file's extension.
2. A column bound to a view must have the shape that view decodes. Ask for a skeleton and the numbers must be [J,2] or [J,3]. That is not a rule we impose; it is what a skeleton is.

Everything else — which camera a hand belongs to, whether a [7] column is a pose or a gripper command, what a .json beside your video contains — is stated in the .dreamrc. You never rename a feature to make it render.

Which datasets can be read

Four formats, named in dataset.format. Each is somebody else's specification, read as published:

folder is the escape hatch and asks nothing of the layout beyond one directory per episode. HDF5 (RoboMimic, ManiSkill, AgiBotWorld) and RLDS/TFRecord (Open X-Embodiment) have no adapter yet — most publishers also ship a LeRobot export, which does. A format we do not read gets an adapter, not a conversion demand (the extension path).

What the viewer sees before you configure anything

The catalog is an inventory, not a classification. Point the checker at any dataset and it lists what exists, with the facts the container reported and no conclusions drawn from them:

episode_000000  (7 fields)
  video    observation.images.ego        h264 1080×1920
  tensor   observation.keypoints_2d.left.ego   float32 [21,3]
  tensor   observation.state             float32 [6]  names: shoulder_pan.pos, …
  tensor   subtask_index                 int64 [1]
  text     language_instruction
  file     recon/scene.glb               ext: glb

The kinds describe how the bytes are addressed, never what they mean:

That listing is what you write the config against — it is the raw truth, and an agent authoring a .dreamrc reads exactly this. Binding a field to a view slot is what states its meaning, and asking for something the bytes cannot become fails by name (observation.state is float32 [6]; keypoints needs [J,2] or [J,3]) instead of rendering a guess.

What each payload needs

The shape requirements, in full. This is the whole contract on the data side:

Annotations belong inside the container

A payload's preferred home is inside the dataset, expressed in the container's own idiom: a keypoint feature sits in the same parquet as your state and action columns, gets read by episode row range, and travels with the dataset.

What that looks like per container, and the one thing each must get right:

LeRobot — meta/info.json features gives every column its dtype, shape and names, and those appear verbatim in the listing. The one thing to get right: cameras must be dtype: video or dtype: image, so they are addressed as media. Episode boundaries come from meta/episodes — v3 packs many episodes into one parquet and one mp4 per camera, and each episode's row range and video window are read from there. A labelled span is an index column plus a label table (meta/<name>s.jsonl — LeRobot's own task_index + meta/tasks.jsonl pattern, one level down); hand keypoints are a [J,2] or [J,3] float feature like any other.

MCAP — every channel is listed with its schema name and encoding. Foxglove schemas (PointCloud, FrameTransform, CompressedImage, SceneUpdate, …) have decoders; cdr / ros2msg / ros1msg do not, so a plain ROS 2 bag currently yields nothing.

Zarr — an array's codec says whether it holds encoded images or numbers. Episodes come from meta/episode_ends or the store's attributes.

A folder of files — one directory per episode; a file's basename is its track name; annotations/ is searched. Three placement conventions, and none of them says what a file contains.

Files beside the data

Two cases genuinely cannot go inside, and only two:

1. The container cannot model it. Static geometry is the real example — there is no LeRobot feature shape for a mesh.
2. The dataset is not yours. A public mirror you cannot write into, so any extra tracks (and the config itself) have to live elsewhere.

For those, put a file beside the data in an established format and name it in the config. The original dataset is never modified — delete the extra files and it is untouched.

Declaring a track

dataset.annotations maps a track name to a path. The merge happens after the format adapter runs, so a declared track lands in the same catalog as the dataset's own fields and binds the same way:

dataset:
  format: lerobot
  episodes: auto
  annotations:
    subtasks: "annotations/subtasks/episode_{episode_index:06d}.vtt"
    hands:    "annotations/hands/episode_{episode_index:06d}.json"
    scene:    "recon/scene.glb"

• Paths are relative to the dataset root; templates use the same {var} / {var:06d} style LeRobot itself uses. Variables: episode_index, episode_name, episode_path (glob mode).
• A declaration is an address, not a claim. The file lands in the catalog as file with its extension recorded, and the view that binds it says what to make of it. The extension gets its say at read time, where all it picks is a PARSER: asked for segments, a .vtt goes through the WebVTT reader and a .json through the COCO one.
• { path, kind } is you saying it once, in the declaration instead of at every binding — legitimate because it is the author speaking, not the library deducing. A binding's own as: still wins, and either way the decoder checks the bytes and fails loudly when they cannot produce what was asked.
• A declaration overrides a native track of the same name. That is user intent: replacing a dataset's coarse task segments with a refined re-annotation is exactly what this is for. Undeclared native tracks stay.
• The folder format auto-discovers anything in each episode's annotations/ folder, so declaring is never required when files live there.

Which format to write

Use the standard that already exists for the job — none of these is ours:

Time spans — WebVTT, the web's own "time range → text" format:

WEBVTT

1
00:00:00.000 --> 00:00:02.500
pick up shelf board

2D keypoints — COCO, stock, plus a top-level fps because COCO indexes images and a player needs seconds (our one documented extension to COCO):

{
  "fps": 30,
  "images": [{ "id": 0, "file_name": "frame_000000.jpg", "width": 1920, "height": 1080 }],
  "annotations": [
    { "id": 1, "image_id": 0, "category_id": 1,
      "keypoints": [473.4, 714.2, 2, 532.2, 710.5, 2],
      "bbox": [418, 524, 277, 249], "score": 0.79 }
  ],
  "categories": [{ "id": 1, "name": "hand", "keypoints": ["wrist", "thumb_cmc"],
                   "skeleton": [[1, 2]] }]
}

Every key is COCO's, including the 1-based skeleton pairs. A frame with no detection simply has no annotation — absence, not a zero coordinate.

Geometry and motion — glTF. Export one .glb with your objects as named nodes: the node name is the join key, so a node called ruler binds the per-frame pose track named ruler, and no side file describes the relationship. If the objects move, animate the glTF — per-node translation/rotation/scale channels with their own keyframe times are already standardized, so one animated .glb carries geometry AND motion in a file Blender, three.js and every other glTF tool reads. (The first clip's TRS channels are read; skinning, morph targets and multiple clips are not yet.)

Per-frame numbers — Parquet, when the motion is not in a glTF:

Quaternion order is read from the COLUMN ORDER: qw first means w,x,y,z, qw last means x,y,z,w. These column names are a decoder's requirement, consulted only once a binding has asked for transform3d or vertices3d — never evidence about what a file holds.

Cameras: the encoding that actually matters

The single most common cause of a sluggish dataset, and it applies to every container. Scrubbing means seeking, and a seek decodes forward from the previous keyframe — keyframes 10 seconds apart feel stuck no matter how fast the machine is:

ffmpeg -i raw.mp4 \
  -c:v libx264 -pix_fmt yuv420p \
  -g 30 -keyint_min 30 -sc_threshold 0 \   # a keyframe every ~1 s at 30fps
  -movflags +faststart \                   # index up front — playback starts immediately
  out.mp4

H.264 + yuv420p decodes everywhere; HEVC, AV1 and MPEG-4 Part 2 do not. Independent frames scrub better than any video — one fetch per frame, no decode chain, exact at any file size.

Verify

npx tsx scripts/check-dreamrc.mts hf:your-name/your-dataset   # a Hub dataset
npx tsx scripts/check-dreamrc.mts https://bucket/…/my-dataset # any object storage
npx tsx scripts/check-dreamrc.mts ./draft.dreamrc             # before you upload

It resolves the dataset exactly as the app does: reads the .dreamrc, decodes every bound field, and prints what came back. With no config it prints the inventory instead — which is how you find out what to write.

Checklist

• Cameras are declared as media by the container (dtype: video / dtype: image / an image codec / a media extension)
• Videos are H.264 yuv420p, +faststart, keyframes ≈ 1 s apart
• Every column you intend to render has the shape its payload needs
• Not-measured is NaN, not zero
• Span index columns have their label table
• Annotations live inside the container where it can hold them; sidecar files are established formats, declared in dataset.annotations
• A .dreamrc at the dataset root, no storage: block — the app injects it (spec)
• Resolved once from a shell, with every binding reporting the payload you expected