# DreamLake Python

A Python SDK for ML data management. Upload any file, organize it into a project/episode/folder tree, and pull it back — by path, by folder, or as a curated bundle.

## What it is

DreamLake stores ML artifacts (video, audio, images, label tracks, transcripts, parquet, csv — anything) as nodes in a hierarchical tree. The Python SDK is a thin client that handles the upload/download protocol and exposes a small surface:

| Function | Purpose |
| --- | --- |
| `dl.upload(file, path=...)` | Upload any file. Type auto-detected from extension. |
| `dl.download(path_or_id, to=...)` | Download a file, folder, or episode (recursive). |
| `dl.bindr(name, members=[...])` | Create a curated collection across the tree. |
| `dl.download_bindr(name, to=...)` | Download every member of a bindr into one tree. |
| `dl.Prefix(project=..., prefix=...)` | Context manager for scoping subsequent calls. |

## Hello, DreamLake

```python file="hello_dreamlake.py"
import dreamlake as dl

with dl.Prefix(project="robotics@alice", prefix="/session-042"):
    dl.upload("./video.mp4", path="camera/front")
    dl.upload("./labels.jsonl", path="detections/yolo")

    dl.download("camera/front/video.mp4", to="./out/")
```

Three things happen in this snippet:

1. The `Prefix` context scopes everything to the `robotics` project (owned by `alice`) and the `session-042` episode.
2. Each `upload` is auto-routed: extension → kind (`video`, `label-track`), MIME type, BSS staging path. The file is hashed (SHA-256, 16-char prefix), streamed directly to S3 in multipart chunks, then registered as a node in the tree. Folders along the path (`camera/front`, `detections/yolo`) are created on demand.
3. `download` resolves the relative path against the current `Prefix` and streams the file back.

## How files are organized

Everything is a node in one tree, distinguished by `kind`:

```
robotics                                ← kind="project"
└─ session-042                          ← kind="episode"
   ├─ camera/front/video.mp4            ← kind="video"
   └─ detections/yolo/labels.jsonl      ← kind="label-track"
```

The `kind` field is free-form — `parquet`, `csv`, `npy`, `pickle`, or anything else works the same way as the built-in media kinds. There's no separate registry to extend.

## Bindrs — curated collections by path

A bindr groups any set of nodes (files or whole folders) into a named collection. Useful for assembling a training set out of scattered data, sharing a slice of a project, or downloading "everything I need for this experiment."

```python
with dl.Prefix(project="robotics@alice", prefix="/session-042"):
    dl.bindr("training-set", members=[
        "camera/front",
        "detections/yolo",
    ])

# Later — pull every file referenced by the bindr
dl.download_bindr("training-set", project="robotics@alice", to="./train/")
```

Bindrs don't copy data. They store path references; downloads stream from the original location.

## What it isn't

- **Not a processing engine.** Uploads don't trigger HLS splitting, transcript chunking, or feature extraction. Those run as separate pipelines you opt into.
- **Not a versioning system.** Files are content-addressed by hash for dedup, but there's no commit graph. If you need versioning, encode it in your paths.
- **Not local-only.** Everything is cloud-backed (S3 + a metadata service). Local caching is on the roadmap.

## Where to next

- [Upload Files](/examples-upload-files.md) — the basic upload flow with auto-detection
- [Upload an Episode](/examples-upload-episode.md) — bulk upload a directory of mixed files
- [Bindrs](/examples-bindrs.md) — curated cross-tree collections
- [Download Files](/examples-download.md) — files, folders, episodes, bindrs
