DreamLake
Browse docs

Storage

Implement storage when your source needs custom access. It resolves file URLs and lists directories; the adapter interprets their data.

Choosing a driver

LocationDriverConfiguration
Public HTTP fileshttpA URL prefix in basePath.
Public HuggingFace datasethfRepository id, with optional revision and path.
Private application dataAn application-provided driverIdentifiers needed by your access layer.

Keep credentials in the application access layer. A schema supplies location identifiers; its driver obtains authorized URLs and listings.

The http driver (public)

resolveUrl joins relative paths to basePath and accepts absolute URLs. list reads an index.json in the requested directory.

yaml
storage:
  driver: http
  basePath: /viz-samples/episode_365/

For directory discovery, the manifest can be a list of filenames:

index.jsonjson
["camera-a.mp4", "camera-b.mp4", "measurements.csv"]

It can also contain objects with name, path, type (file or dir) and optional size. These may be wrapped in an { "entries": [...] } object. basePath defaults to an empty string.

HuggingFace storage

The hf driver uses public resolve URLs and the paginated Hub tree API. It does not need index.json.

yaml
storage:
  driver: hf
  id: lerobot/aloha_static_coffee
  revision: main
KeyDefaultMeaning
idrequiredRepository identifier, such as organization/dataset.
basePath''Directory prefix inside the repository.
revision'main'Revision used for resolve and tree requests.
repoType'datasets'HuggingFace repository type used in the URL.

Private data — injected by the host

Wrap your authorized listing and download-URL operations in a Storage factory:

ts
import type { Listing, StorageFactory } from "@dreamlake/viz/schema-viz";

type Access = {
  list(
    root: string,
    path: string,
    options?: { token?: string; limit?: number },
  ): Promise<Listing>;
  downloadUrl(
    root: string,
    path: string,
    options?: { ttlSeconds?: number },
  ): Promise<string>;
};

export function makeProjectStorage(access: Access): StorageFactory {
  return (config) => {
    const root = String(config.root);
    return {
      list: (path, options) => access.list(root, path, options),
      resolveUrl: (path, options) => access.downloadUrl(root, path, options),
    };
  };
}

Return URLs that browser media elements can use, such as signed URLs or authorized same-origin endpoints. The application handles their expiry.

The schema names the registered driver:

yaml
version: 1
sources:
  clips:
    adapter: filesystem
    storage: { driver: project, root: recording-folder }
panels:
  - { view: videoStack, source: clips, fields: ["*"], columns: 2 }

Register once, or per preview

Register the factory at application startup:

ts
import { registerStorage } from "@dreamlake/viz/schema-viz";

registerStorage("project", projectStorageFactory);

Or provide it when mounting a preview:

tsx
<DatasetPreview
  schema={schema}
  extensions={{ storages: { project: projectStorageFactory } }}
/>

Both paths use the module's shared registry. An extensions entry is added only if its name is absent; it is not scoped to that preview and persists for later previews. registerStorage explicitly replaces an existing entry. registerAdapter and registerView likewise support named registration, while extensions.adapters and extensions.views skip names already registered. Use distinct names when configurations need different implementations.

Writing your own driver

The public interface is:

ts
interface Storage {
  resolveUrl(path: string, opts?: { ttlSeconds?: number }): Promise<string>;
  list(
    path: string,
    opts?: { token?: string; limit?: number },
  ): Promise<Listing>;
}

interface Listing {
  entries: Entry[];
  nextToken?: string;
}

interface Entry {
  name: string;
  path: string;
  type: "file" | "dir";
  size?: number;
}

Return paths accepted by resolveUrl. Opaque identifiers can serve as paths while name supplies the extension. Listing is not recursive; check your adapter's pagination behavior before relying on a paginated backend.

What storage does not do

Storage handles locations; the adapter handles format and field kind. A file entry's type is not its data kind.