Media overlays

EpisodeVideoStack and EpisodeFrameStack accept per-source overlays — annotation layers drawn over the media: bounding boxes for detections, keypoint skeletons for poses and hands. Attach them to the source; the tile keeps them aligned with the pixels at any player size.

import { EpisodeVideoStack, type MediaOverlay } from '@dreamlake/viz'

const overlays: MediaOverlay[] = [handSkeletons, objectBoxes]

<EpisodeVideoStack
  videos={[{ id: 'ego', src, overlays }]}
  duration={67.5}
/>

A stack-level showOverlays prop (default true) is the master switch; each overlay also has its own visible flag and opacity.

Hand skeletons + captions on a video

Two layers on one video, each converted with one call: real 21-joint hand detections (handOverlayFromDetections — reads pixel space, fps, and skeleton from the file, applies the per-finger HAND21_STYLE) and labelled subtask segments rendered as subtitle-style captions (tracksOverlayFromSubtasks). Both stay glued to the frame during playback and while scrubbing:

Bounding boxes on a frame stack

Hand-authored synthetic detections in normalized coordinates, keyed by frame index. The cup only exists on frames 30–75 — sparse keys draw nothing, so boxes appear and disappear with the data instead of going stale:

PreviewSource

FRAME 1 / 106

frontsynthetic detections

FRAME 1 / 106

topno overlay

Declaring coordinates

Overlay coordinates live in the media's space — never the player's. Each overlay says which flavor via space:

type OverlaySpace =
  | { units: 'pixel'; width: number; height: number } // annotation-time frame size
  | { units: 'normalized' }                           // x, y already in 0..1

• Pixel coordinates carry the frame size they were produced against (not the runtime videoWidth, which can differ after transcode/rotation).
• Out-of-frame points are legal — partially visible detections draw and get clipped, never clamped.
• Stroke widths and dot radii are CSS px at render size, so a 2px bone reads as 2px on any player.

What draws at time t

Each overlay's data field says how to look up "the items for right now". Missing data draws nothing — stale items are never carried over.

type OverlayFrameData<T> =
  | { by: 'frame'; frames: Record<number, T[]>; fps?: number; frameOffset?: number }
  | { by: 'time'; entries: { time: number; items: T[] }[]; tolerance?: number }

On a video, the lookup follows the frame the <video> actually presents (overlaySync: 'media', the default), so drawings never lead or lag the pixels; 'cursor' follows the time prop instead (deterministic — SSR, tests).

Reference

bbox

interface BBoxItem {
  box: [number, number, number, number]  // [x1, y1, x2, y2]
  label?: string
  score?: number                         // 0..1
  trackId?: string | number              // same id ⇒ same color across frames
  color?: string                         // explicit override
}

keypoints

interface KeypointsItem {
  points: ([number, number] | [number, number, number])[] // z reserved for 3D
  pointScores?: number[]
  score?: number
  box?: [number, number, number, number]  // drawn when style.showBox
  group?: string                          // coloring key, e.g. 'left' / 'right'
  color?: string
}

interface KeypointsOverlay {
  kind: 'keypoints'
  skeleton?: [number, number][]  // bones as index pairs into points
  jointNames?: string[]
  data: OverlayFrameData<KeypointsItem>
  style?: KeypointsStyle
  // …common fields: id, space, visible, opacity, minScore
}

tracks — subtitle-style captions

Labelled time ranges rendered like video subtitles: whichever block spans the current time draws as an outlined caption at the bottom of the tile. The data shape is the same TimelineTrack / TrackBlock that EpisodeTimeline takes, so one dataset feeds both the caption overlay and a timeline track row:

interface TracksOverlay {
  kind: 'tracks'
  tracks: { id: string; name?: string; blocks: TrackBlock[] }[]
  style?: { fontSize?: number; color?: string } // default 12px / white
  // …common fields: id, visible, opacity
}

type TrackBlock = { id?: string; start: number; end: number; label: string } // seconds

Time-range based — no coordinate space, no frame keys.

Data formats

Each converter — and the matching schema format: value — expects exactly the JSON shape below. Data in any other shape needs format: raw (a file that already contains MediaOverlay JSON) or a custom conversion in code.

handJoints → keypoints overlay

HandJointsFile — the hand-detection pipeline output (this is the format of the live example's claru_ego__Ceramics.json). Frames are keyed by 0-based index of the original video; coordinates are full-resolution pixels in upright orientation. Field-by-field, the conversion is:

{
  "width": 1920, "height": 1080,       // → space { units: "pixel", width, height }
  "src_fps": 29.987,                   // → data.fps (playback time → frame key)
  "joint_order": ["wrist", "…21"],     // → jointNames
  "bones": [[0, 1], "…20 pairs"],      // → skeleton (4 per finger, thumb → pinky)
  "frames": {                          // sparse → data.frames (missing key = no hands)
    "0": [{
      "is_right": 1,                   // → group "right" | "left"
      "det_conf": 0.98,                // → score
      "bbox": [x1, y1, x2, y2],        // → box
      "keypoints_2d": [[x, y], "…21"]  // → points
    }]
  },
  "total_frames": 2024
}

import { handOverlayFromDetections, type HandJointsFile } from '@dreamlake/viz'

const overlay = handOverlayFromDetections(file) // + per-finger HAND21_STYLE

subtasks → caption overlay + timeline track

SubtasksFile — subtask segment annotations: one labelled range per segment, in seconds from video start:

{
  "video": "713488",
  "task": "mount a wall shelf",        // → track name
  "labeled_subtasks": [
    { "start_sec": 0.0, "end_sec": 2.5, "subtask": "pick up shelf board" },
    { "start_sec": 2.5, "end_sec": 10.5, "subtask": "mount shelf board on wall tracks" }
  ]
}

Two converters read it — the same blocks drive the subtitle overlay and an EpisodeTimeline track:

import { tracksOverlayFromSubtasks, trackBlocksFromSubtasks } from '@dreamlake/viz'

const overlay = tracksOverlayFromSubtasks(file) // kind: 'tracks' captions
const timelineTracks = [
  { id: 'subtasks', blocks: trackBlocksFromSubtasks(file) }, // <EpisodeTimeline tracks>
]

See both live in the schema example: a videoStack panel names the files by field ref with format: handJoints / format: subtasks, and a timeline panel shows the same subtasks as a track row.