DreamLake

Upload an Episode

A typical robot experiment run produces a mix of files in nested folders. Mirror the local layout into a DreamLake episode in one pass.

An episode is the unit of an experiment run — a contiguous capture of related data. Everything inside it shares one logical scope.

Local layout

~/captures/run-100/
├── camera/
│   ├── front/clip.mp4
│   └── rear/clip.mp4
├── audio/
│   ├── mic-1/sample.wav
│   └── mic-2/sample.wav
├── labels/yolo/detections.jsonl
├── data/train.parquet
└── meta.json

One call — dl.upload_folder

upload_episode.pypython
import dreamlake as dl

with dl.Prefix(project="robotics@alice", prefix="/run-100"):
    dl.upload_folder("~/captures/run-100")

That single line:

  • Walks the local directory tree recursively
  • Uploads each file in parallel (default 4 workers)
  • Mirrors the local folder structure on the server — a file at ~/captures/run-100/camera/front/clip.mp4 lands at /run-100/camera/front/clip.mp4 in DreamLake
  • Auto-detects kind from extension for each file
  • Skips hidden / temp files (.DS_Store, *.tmp, *.swp, .*) by default
  • Returns a summary: {"total", "uploaded", "failed", "errors"}

Tuning

python
dl.upload_folder(
    "~/captures/run-100",
    project="robotics@alice",       # no Prefix context needed
    prefix="/run-100",
    parallel=8,                     # bump for fast networks
    ignore=["*.bak", "checkpoints/*"],  # custom skip patterns
    recursive=False,                # flat — only direct children
    on_file=lambda f, i, n: print(f"  [{i+1}/{n}] {f.name}"),
)

Resulting tree

robotics                                              ← project
└─ run-100                                             ← episode
   ├─ camera/front/clip.mp4
   ├─ camera/rear/clip.mp4
   ├─ audio/mic-1/sample.wav
   ├─ audio/mic-2/sample.wav
   ├─ labels/yolo/detections.jsonl
   ├─ data/train.parquet
   └─ meta.json

Dedup is automatic

Re-running the same script is safe — uploads are content-addressed by hash. A file that's already in staging at {owner}/{project}/staging/{hash} won't be re-uploaded. Folders along the path are upsert-only (no duplicates).

When to drop down to the loop

For special cases — selective filters, custom kinds per file, attaching tags — write the loop yourself:

upload_loop.pypython
from pathlib import Path
import dreamlake as dl

ROOT = Path("~/captures/run-100").expanduser()

with dl.Prefix(project="robotics@alice", prefix=f"/{ROOT.name}"):
    for file in ROOT.rglob("*"):
        if not file.is_file(): continue
        if file.suffix == ".npy" and file.stat().st_size > 1e9:
            continue  # skip huge tensors
        rel = file.parent.relative_to(ROOT)
        sub = "" if str(rel) == "." else str(rel)   # top-level files
        dl.upload(str(file), path=sub)

Best practices

  • One episode per experiment run. Use a stable name (run-100, 2026-04-15-trial-01) so re-uploads land in the same place.
  • Treat the path as the only schema you need. Folder structure encodes meaning (camera/front/, labels/yolo/) without needing tags or metadata.
  • Don't fight kind detection. If the SDK calls your file text because it's .txt, that's fine — kind doesn't gate storage, only display.
  • Use ignore= for local clutter. dl.upload_folder filters .DS_Store, *.tmp, *.lock, *.swp, and hidden files by default. Add your own patterns for project-specific clutter.

What you can do next

  • Bindrs — bundle subsets of this episode into named collections
  • Download Files — pull the whole episode back to a different machine