DreamLake Python
A Python SDK for ML data management. Upload any file, organize it into a project/episode/folder tree, and pull it back — by path, by folder, or as a curated bundle.
What it is
DreamLake stores ML artifacts (video, audio, images, label tracks, transcripts, parquet, csv — anything) as nodes in a hierarchical tree. The Python SDK is a thin client that handles the upload/download protocol and exposes a small surface:
| Function | Purpose |
|---|---|
dl.upload(file, path=...) | Upload any file. Type auto-detected from extension. |
dl.download(path_or_id, to=...) | Download a file, folder, or episode (recursive). |
dl.bindr(name, members=[...]) | Create a curated collection across the tree. |
dl.download_bindr(name, to=...) | Download every member of a bindr into one tree. |
dl.Prefix(project=..., prefix=...) | Context manager for scoping subsequent calls. |
Hello, DreamLake
Three things happen in this snippet:
- The
Prefixcontext scopes everything to theroboticsproject (owned byalice) and thesession-042episode. - Each
uploadis auto-routed: extension → kind (video,label-track), MIME type, BSS staging path. The file is hashed (SHA-256, 16-char prefix), streamed directly to S3 in multipart chunks, then registered as a node in the tree. Folders along the path (camera/front,detections/yolo) are created on demand. downloadresolves the relative path against the currentPrefixand streams the file back.
How files are organized
Everything is a node in one tree, distinguished by kind:
The kind field is free-form — parquet, csv, npy, pickle, or anything else works the same way as the built-in media kinds. There's no separate registry to extend.
Bindrs — curated collections by path
A bindr groups any set of nodes (files or whole folders) into a named collection. Useful for assembling a training set out of scattered data, sharing a slice of a project, or downloading "everything I need for this experiment."
Bindrs don't copy data. They store path references; downloads stream from the original location.
What it isn't
- Not a processing engine. Uploads don't trigger HLS splitting, transcript chunking, or feature extraction. Those run as separate pipelines you opt into.
- Not a versioning system. Files are content-addressed by hash for dedup, but there's no commit graph. If you need versioning, encode it in your paths.
- Not local-only. Everything is cloud-backed (S3 + a metadata service). Local caching is on the roadmap.
Where to next
- Upload Files — the basic upload flow with auto-detection
- Upload an Episode — bulk upload a directory of mixed files
- Bindrs — curated cross-tree collections
- Download Files — files, folders, episodes, bindrs