Browse docs
Media overlays
Draw boxes, keypoints, and captions on video or image sequences through each
source's overlays. Source coordinates keep detections aligned as tiles resize.
Hand skeletons + captions on a video
This specimen combines COCO hand keypoints with WebVTT task captions:
Code HandJointsSpec.tsx
Parse files first, then convert their data with the overlay helpers.
showOverlays toggles the stack's layers; each layer also has visible and
opacity.
Bounding boxes on a frame stack
Scrub through the cup detection on frames 30–75. Frame keys with no detection draw nothing, so an earlier box does not persist.


Code BBoxOverlaySpec.tsx
Declaring coordinates
Declare the coordinate space used to produce the annotations:
Pixel width and height describe the annotation image, even if video was
later transcoded. Normalized values are fractions of the frame. Out-of-frame
points clip at the media edge. Stroke widths and joint radii stay in display
CSS pixels.
What draws at time t
Choose frame keys or ascending timestamp entries. Missing frame keys and timestamps outside tolerance produce no items.
| Source | Frame lookup | Time lookup |
|---|---|---|
| Video | round(time * fps) + frameOffset; supply the source frame rate. | Nearest entry within tolerance, default 0.1 seconds. |
| Image sequence | Displayed image's frameIndex, or its array index, plus frameOffset. | Match against that image's timestamp. |
Use frameOffset to shift indices. Preserve original frame indices in
subsampled image sequences, and keep time annotations on the lookup clock.
Video sync modes
The default overlaySync="media" follows the presented frame through
requestVideoFrameCallback, or less precise timeupdate events. This avoids
showing requested-time annotations on an image still waiting to seek.
Use overlaySync="cursor" to follow the supplied time directly.
With a video window, media time belongs to the source file while cursor time
is episode-relative. Prepare annotation timestamps and frame offsets for the
selected mode.
Reference
Boxes and keypoints share id, space, visible, opacity, and minScore.
The defaults are visible, full opacity, and no minimum score. Captions omit
coordinate space and score thresholds.
bbox
BBoxStyle field | Default | Purpose |
|---|---|---|
strokeWidth | 1.5 | Border width in CSS pixels. |
stroke / fill | palette / 'transparent' | Box colors. |
showLabel / showScore | true / false | Show the label and optionally append a score such as Cup 0.92. |
colorBy | 'label' | Assign colors by 'label', 'trackId', or 'fixed'. |
palette | built-in | Colors cycled for the selected key. |
Use colorBy="trackId" with stable trackId values to color tracked objects,
or override a detection's color with color.
keypoints
skeleton connects pairs of indices into points; omit it for dots only.
The 2D renderer ignores an optional third coordinate. Use
3D scenes for spatial tracks.
KeypointsStyle field | Default | Purpose |
|---|---|---|
jointRadius / boneWidth | 2.5 / 1.5 | Joint and bone sizes in CSS pixels. |
boneColors | — | Colors in skeleton-edge order. |
jointColors | derived | Joint colors; by default, each inherits the bone ending at it. |
rootColor | '#ffffff' | Color for joints without an incoming bone. |
groupColors | — | Color a whole instance by its group. |
palette | built-in | Fallback colors for groups. |
showBox | false | Also draw the item's detection box. |
minPointScore | 0 | Omit joints whose point confidence is below this threshold. |
tracks — subtitle-style captions
Each caption track displays the interval containing the current media time; its end is exclusive. It shares the timeline block shape.
Captions default to 12px white text and require neither frame keys nor a coordinate space.
Data formats
Reuse parsed WebVTT segments for both captions and a timeline row:
Segment helpers accept { start, end, label }[] in seconds.
keypointsOverlayFromTrack accepts { width, height, fps, frames, skeleton?, jointNames? } with frames: Map<number, KeypointFrameData[]>. Its default
HAND21_STYLE can be replaced for another skeleton.
handJoints → keypoints overlay
The schema adapter's format: handJoints reads hand-pipeline JSON with image
dimensions, frame rate, joint names, bones, and frame-keyed detections:
Coordinates use full-resolution, upright-image pixels. This abbreviated example has two joints; supply the full joint order and skeleton for a hand.
subtasks → caption overlay + timeline track
format: subtasks reads intervals in seconds from video start:
Use the same file for videoStack captions and timeline blocks, as in the
schema example.
Choose format: raw for existing MediaOverlay data.