DreamLake
Browse docs

Media overlays

Draw boxes, keypoints, and captions on video or image sequences through each source's overlays. Source coordinates keep detections aligned as tiles resize.

Hand skeletons + captions on a video

This specimen combines COCO hand keypoints with WebVTT task captions:

REC · 00:00.000
Ceramicsclaru_ego · hand skeletons + subtask captions
Code HandJointsSpec.tsx
tsx
import {
  EpisodeVideoStack,
  keypointsOverlayFromTrack,
  tracksOverlayFromSegments,
} from '@dreamlake/viz'
import { parseCocoKeypoints, parseWebVtt } from '@dreamlake/viz/dataset-viz'

const overlays = [
  keypointsOverlayFromTrack(parseCocoKeypoints(cocoJson)),
  tracksOverlayFromSegments(parseWebVtt(vttText)),
]

<EpisodeVideoStack
  videos={[{ id: 'ego', src, overlays }]}
  duration={duration}
/>

Parse files first, then convert their data with the overlay helpers. showOverlays toggles the stack's layers; each layer also has visible and opacity.

Bounding boxes on a frame stack

Scrub through the cup detection on frames 30–75. Frame keys with no detection draw nothing, so an earlier box does not persist.

FRAME 1 / 106
frontsynthetic detections
FRAME 1 / 106
topno overlay
Code BBoxOverlaySpec.tsx

Declaring coordinates

Declare the coordinate space used to produce the annotations:

ts
type OverlaySpace =
  | { units: "pixel"; width: number; height: number }
  | { units: "normalized" };

Pixel width and height describe the annotation image, even if video was later transcoded. Normalized values are fractions of the frame. Out-of-frame points clip at the media edge. Stroke widths and joint radii stay in display CSS pixels.

What draws at time t

Choose frame keys or ascending timestamp entries. Missing frame keys and timestamps outside tolerance produce no items.

ts
type OverlayFrameData<T> =
  | {
      by: "frame";
      frames: Record<number, T[]>;
      fps?: number;
      frameOffset?: number;
    }
  | { by: "time"; entries: { time: number; items: T[] }[]; tolerance?: number };
SourceFrame lookupTime lookup
Videoround(time * fps) + frameOffset; supply the source frame rate.Nearest entry within tolerance, default 0.1 seconds.
Image sequenceDisplayed image's frameIndex, or its array index, plus frameOffset.Match against that image's timestamp.

Use frameOffset to shift indices. Preserve original frame indices in subsampled image sequences, and keep time annotations on the lookup clock.

Video sync modes

The default overlaySync="media" follows the presented frame through requestVideoFrameCallback, or less precise timeupdate events. This avoids showing requested-time annotations on an image still waiting to seek. Use overlaySync="cursor" to follow the supplied time directly.

With a video window, media time belongs to the source file while cursor time is episode-relative. Prepare annotation timestamps and frame offsets for the selected mode.

Reference

Boxes and keypoints share id, space, visible, opacity, and minScore. The defaults are visible, full opacity, and no minimum score. Captions omit coordinate space and score thresholds.

bbox

ts
interface BBoxItem {
  box: [number, number, number, number]; // x1, y1, x2, y2
  label?: string;
  score?: number;
  trackId?: string | number;
  color?: string;
}

const boxes: MediaOverlay = {
  id: "objects",
  kind: "bbox",
  space: { units: "normalized" },
  data: {
    by: "frame",
    fps: 30,
    frames: {
      0: [{ box: [0.2, 0.3, 0.5, 0.7], label: "Cup", score: 0.92 }],
    },
  },
};
BBoxStyle fieldDefaultPurpose
strokeWidth1.5Border width in CSS pixels.
stroke / fillpalette / 'transparent'Box colors.
showLabel / showScoretrue / falseShow the label and optionally append a score such as Cup 0.92.
colorBy'label'Assign colors by 'label', 'trackId', or 'fixed'.
palettebuilt-inColors cycled for the selected key.

Use colorBy="trackId" with stable trackId values to color tracked objects, or override a detection's color with color.

keypoints

ts
interface KeypointsItem {
  points: ([number, number] | [number, number, number])[];
  pointScores?: number[];
  score?: number;
  box?: [number, number, number, number];
  group?: string;
  color?: string;
}

interface KeypointsOverlay {
  id: string;
  kind: "keypoints";
  space: OverlaySpace;
  skeleton?: [number, number][];
  jointNames?: string[];
  data: OverlayFrameData<KeypointsItem>;
  style?: KeypointsStyle;
  visible?: boolean;
  opacity?: number;
  minScore?: number;
}

skeleton connects pairs of indices into points; omit it for dots only. The 2D renderer ignores an optional third coordinate. Use 3D scenes for spatial tracks.

KeypointsStyle fieldDefaultPurpose
jointRadius / boneWidth2.5 / 1.5Joint and bone sizes in CSS pixels.
boneColors—Colors in skeleton-edge order.
jointColorsderivedJoint colors; by default, each inherits the bone ending at it.
rootColor'#ffffff'Color for joints without an incoming bone.
groupColors—Color a whole instance by its group.
palettebuilt-inFallback colors for groups.
showBoxfalseAlso draw the item's detection box.
minPointScore0Omit joints whose point confidence is below this threshold.

tracks — subtitle-style captions

Each caption track displays the interval containing the current media time; its end is exclusive. It shares the timeline block shape.

ts
interface TracksOverlay {
  id: string;
  kind: "tracks";
  tracks: { id: string; name?: string; blocks: TrackBlock[] }[];
  visible?: boolean;
  opacity?: number;
  style?: { fontSize?: number; color?: string };
}
type TrackBlock = { id?: string; start: number; end: number; label: string };

Captions default to 12px white text and require neither frame keys nor a coordinate space.

Data formats

Reuse parsed WebVTT segments for both captions and a timeline row:

tsx
import {
  tracksOverlayFromSegments,
  trackBlocksFromSegments,
} from "@dreamlake/viz";

const overlay = tracksOverlayFromSegments(segments);
const tracks = [{ id: "subtasks", blocks: trackBlocksFromSegments(segments) }];

Segment helpers accept { start, end, label }[] in seconds. keypointsOverlayFromTrack accepts { width, height, fps, frames, skeleton?, jointNames? } with frames: Map<number, KeypointFrameData[]>. Its default HAND21_STYLE can be replaced for another skeleton.

handJoints → keypoints overlay

The schema adapter's format: handJoints reads hand-pipeline JSON with image dimensions, frame rate, joint names, bones, and frame-keyed detections:

json
{
  "width": 1920,
  "height": 1080,
  "src_fps": 29.987,
  "joint_order": ["wrist", "thumb_base"],
  "bones": [[0, 1]],
  "frames": {
    "0": [
      {
        "is_right": 1,
        "det_conf": 0.98,
        "bbox": [100, 120, 300, 400],
        "keypoints_2d": [
          [150, 350],
          [180, 280]
        ]
      }
    ]
  },
  "total_frames": 2024
}

Coordinates use full-resolution, upright-image pixels. This abbreviated example has two joints; supply the full joint order and skeleton for a hand.

subtasks → caption overlay + timeline track

format: subtasks reads intervals in seconds from video start:

json
{
  "video": "713488",
  "task": "mount a wall shelf",
  "labeled_subtasks": [
    { "start_sec": 0, "end_sec": 2.5, "subtask": "pick up shelf board" },
    { "start_sec": 2.5, "end_sec": 10.5, "subtask": "mount shelf board" }
  ]
}

Use the same file for videoStack captions and timeline blocks, as in the schema example. Choose format: raw for existing MediaOverlay data.