DreamLake

DreamLake Python

A Python SDK for ML data management. Upload any file, organize it into a project/episode/folder tree, and pull it back — by path, by folder, or as a curated bundle.

What it is

DreamLake stores ML artifacts (video, audio, images, label tracks, transcripts, parquet, csv — anything) as nodes in a hierarchical tree. The Python SDK is a thin client that handles the upload/download protocol and exposes a small surface:

FunctionPurpose
dl.upload(file, path=...)Upload any file. Type auto-detected from extension.
dl.download(path_or_id, to=...)Download a file, folder, or episode (recursive).
dl.bindr(name, members=[...])Create a curated collection across the tree.
dl.download_bindr(name, to=...)Download every member of a bindr into one tree.
dl.Prefix(project=..., prefix=...)Context manager for scoping subsequent calls.

Hello, DreamLake

hello_dreamlake.pypython
import dreamlake as dl

with dl.Prefix(project="robotics@alice", prefix="/session-042"):
    dl.upload("./video.mp4", path="camera/front")
    dl.upload("./labels.jsonl", path="detections/yolo")

    dl.download("camera/front/video.mp4", to="./out/")

Three things happen in this snippet:

  1. The Prefix context scopes everything to the robotics project (owned by alice) and the session-042 episode.
  2. Each upload is auto-routed: extension → kind (video, label-track), MIME type, BSS staging path. The file is hashed (SHA-256, 16-char prefix), streamed directly to S3 in multipart chunks, then registered as a node in the tree. Folders along the path (camera/front, detections/yolo) are created on demand.
  3. download resolves the relative path against the current Prefix and streams the file back.

How files are organized

Everything is a node in one tree, distinguished by kind:

robotics                                ← kind="project"
└─ session-042                          ← kind="episode"
   ├─ camera/front/video.mp4            ← kind="video"
   └─ detections/yolo/labels.jsonl      ← kind="label-track"

The kind field is free-form — parquet, csv, npy, pickle, or anything else works the same way as the built-in media kinds. There's no separate registry to extend.

Bindrs — curated collections by path

A bindr groups any set of nodes (files or whole folders) into a named collection. Useful for assembling a training set out of scattered data, sharing a slice of a project, or downloading "everything I need for this experiment."

python
with dl.Prefix(project="robotics@alice", prefix="/session-042"):
    dl.bindr("training-set", members=[
        "camera/front",
        "detections/yolo",
    ])

# Later — pull every file referenced by the bindr
dl.download_bindr("training-set", project="robotics@alice", to="./train/")

Bindrs don't copy data. They store path references; downloads stream from the original location.

What it isn't

  • Not a processing engine. Uploads don't trigger HLS splitting, transcript chunking, or feature extraction. Those run as separate pipelines you opt into.
  • Not a versioning system. Files are content-addressed by hash for dedup, but there's no commit graph. If you need versioning, encode it in your paths.
  • Not local-only. Everything is cloud-backed (S3 + a metadata service). Local caching is on the roadmap.

Where to next