DreamLake
Browse docs

Read file data

Loaders turn a URL into data for your own components. Choose by format and read size: CSV and JSONL sample an initial range; JSON and Parquet read the whole file within a reported-size cap; NumPy and MCAP return metadata.

Range requests save bandwidth only when the server honors them. Use FilePreview to load and render in one component.

Overview

Shared shape

Call a loader with a URL and optional LoaderOpts; its result is specific to the format:

LoaderOpts.tsts
interface LoaderOpts {
  size?: number;
  signal?: AbortSignal;
  maxBytes?: number;
  maxRows?: number;
}

Supply known size to avoid a size probe and signal to cancel fetching. Byte and row limits apply only to the formats documented below.

useLoader

useLoader supplies data, loading and error state. Put request inputs in its dependency array: a change clears the old result and aborts its request. Unmounting also aborts.

tsx
import { loadCsv, useLoader, StatusView, TableView } from "@dreamlake/viz";

export function CsvTable({ url }: { url: string }) {
  const { data, error, loading } = useLoader(
    (signal) => loadCsv(url, { signal, maxRows: 100 }),
    [url],
  );
  if (loading) return <StatusView kind="loading" />;
  if (error) return <StatusView kind="error" message={error.message} />;
  if (!data) return null;
  return (
    <TableView
      cols={data.header.map((name) => ({ name }))}
      rows={data.rows}
      totalRows={data.totalRows}
      truncated={data.truncated}
    />
  );
}
ts
interface LoaderState<T> {
  data: T | null;
  error: Error | null;
  loading: boolean;
}

function useLoader<T>(
  load: (signal: AbortSignal) => Promise<T>,
  deps: unknown[],
): LoaderState<T>;

HTTP helpers

These helpers are also exported from @dreamlake/viz:

FunctionResultRequest
fetchHead(url, signal?){ size, contentType?, lastModified? }A one-byte Range GET to discover size.
fetchRange(url, start, end, signal?)Uint8ArrayA byte range with an inclusive end.
fetchText(url, signal?)stringComplete response as text.
fetchBytes(url, signal?)Uint8ArrayComplete response as bytes.

fetchHead uses a one-byte GET, compatible with GET-signed URLs. Size comes from Content-Range or Content-Length; expose these headers for cross-origin reads.

A successful full response is accepted when Range is ignored. For partial NumPy/MCAP transfers, use a server that supports Range.

loadText

Fetches the entire string. Only signal applies; check file size yourself before loading large text.

ts
import { loadText } from "@dreamlake/viz";

const text = await loadText(url, { signal });

Tabular formats

loadCsv

Samples up to 5 MiB and returns at most 1,000 string-valued rows. Detects comma, semicolon, tab or pipe delimiters, including quoted fields and a UTF-8 BOM.

csv.tsts
import { loadCsv } from "@dreamlake/viz";

const data = await loadCsv(url, { signal, maxRows: 100 });
// data.header: string[]
// data.rows: string[][]
// data.shown: number
// data.totalRows: number | null
// data.bytesRead: number
// data.truncated: boolean
// data.delimiter: ',' | ';' | '\t' | '|'

Convert numeric columns yourself. totalRows is null for a partial byte sample; otherwise it is the returned count, still limited by maxRows. truncated also covers the row cap. bytesRead counts decoded characters, not transferred bytes.

loadParquet

Reads and caches the whole file with hyparquet. Defaults: 5 MiB size cap and 1,000 returned rows. A reported size above the cap throws ParquetTooLargeError.

parquet.tsts
import {
  loadParquet,
  ParquetTooLargeError,
  MAX_PARQUET_BYTES,
} from "@dreamlake/viz";

try {
  const data = await loadParquet(url, { signal });
  // data.cols: { name: string; type: string }[]
  // data.rows: unknown[][]
  // data.totalRows: number
} catch (error) {
  if (error instanceof ParquetTooLargeError) {
    console.log(error.sizeBytes, error.capBytes);
  } else {
    throw error;
  }
}

maxBytes limits reported file size; maxRows limits output, not downloads. Top-level columns contain display values: nested objects are serialized, big integers become strings, dates use ISO strings, and binary values show byte length.

Structured formats

loadJson

Fetches a complete document within maxBytes (default MAX_JSON_BYTES, 10 MiB). An excessive reported size throws JsonTooLargeError with sizeBytes and capBytes.

json.tsts
import { loadJson, JsonTooLargeError, MAX_JSON_BYTES } from "@dreamlake/viz";

const data = await loadJson(url, { signal });
// data.raw: string
// data.parsed: unknown
// data.parseError: string | null
// data.bytes: number

Invalid JSON returns parseError, parsed: null and the original text. Fetch errors throw. bytes counts characters; the size cap uses the supplied or server-reported length.

loadJsonl

Samples up to 1 MiB and returns at most 1,000 JSON records by default. Invalid lines are skipped.

jsonl.tsts
import { loadJsonl } from "@dreamlake/viz";

const data = await loadJsonl(url, { signal });
// data.records: unknown[]
// data.totalLines: number
// data.truncated: boolean

totalLines counts nonempty lines, or estimates them for a partial sample. It may exceed records.length. truncated means the byte range was cut short; reaching maxRows alone does not set it.

Binary formats

loadNpy

Reads dtype, shape and storage order without decoding values. A 256-byte initial request is followed by a larger header request when necessary.

npy.tsts
import { loadNpy } from "@dreamlake/viz";

const meta = await loadNpy(url, { signal });
// meta.magic: string
// meta.version: string
// meta.headerBytes: number
// meta.dtype: string
// meta.shape: number[]
// meta.elements: number
// meta.itemSize: number
// meta.dataBytes: number
// meta.fortranOrder: boolean
// meta.rangeUsed: string

Supports NPY v1–v3 scalar dtype descriptors, not structured dtypes. dataBytes is calculated from shape and item size. maxBytes and maxRows do not apply.

loadMcap

Reads the header, footer and available summary, without decoding messages. Expect two range requests, one more for a summary, and a size probe unless size is supplied. Summary size determines the amount read.

mcap.tsts
import { loadMcap } from "@dreamlake/viz";

const meta = await loadMcap(url, { signal });
// meta.library: string
// meta.profile: string
// meta.compression: string
// meta.durationMs: number
// meta.messageCount: number
// meta.chunkCount: number
// meta.attachmentCount: number
// meta.channels: McapChannelStat[]
// meta.timestamps: { start: string; end: string }
// meta.rangeUsed: string

Channels report topic, schema, count and rounded messageRate. Missing summary statistics produce zero counts and unavailable timestamps; the loader does not scan messages. Byte/row caps do not apply. Use McapPreview to inspect message data.