DreamLake

Upload Files

The simplest case: one file, one project, one folder.

upload.pypython
import dreamlake as dl

with dl.Prefix(project="robotics@alice", prefix="/session-042"):
    dl.upload("./video.mp4", path="camera/front")

The SDK does five things behind the scenes:

  1. Hashes the file (SHA-256, 16 hex chars) — used as the S3 staging key.
  2. Detects kind="video" and Content-Type: video/mp4 from the .mp4 extension.
  3. Streams the bytes to S3 in 10 MB multipart chunks via presigned URLs. Bytes never traverse the BSS server.
  4. Registers a File record in BSS.
  5. Creates a kind="video" Node in the tree at /session-042/camera/front/video.mp4, auto-creating the camera/front folders along the way.

Multiple uploads share the context

The Prefix context manager scopes every call inside the block — project, episode (= last segment of prefix), and base path. Three uploads, three different sub-folders, one episode:

python
with dl.Prefix(project="robotics@alice", prefix="/session-042"):
    dl.upload("./video.mp4",   path="camera/front")
    dl.upload("./mic.wav",     path="audio/mic-1")
    dl.upload("./labels.jsonl", path="detections/yolo")

Resulting tree:

robotics                                       ← kind="project"
└─ session-042                                  ← kind="episode"
   ├─ camera/front/video.mp4                    ← kind="video"
   ├─ audio/mic-1/mic.wav                       ← kind="audio"
   └─ detections/yolo/labels.jsonl              ← kind="label-track"

Auto-detected kinds

ExtensionDetected kind
.mp4 .mkv .mov .webmvideo
.wav .mp3 .aac .opus .flacaudio
.jpg .png .webp .gif .bmp .tiffimage
.vtt .srttext-track
.jsonllabel-track
.parquetparquet
.csvcsv
.npy .npznpy
.jsonjson
.txt .md .logtext
anything elsefile

The kind string is free-form. Pass type="my-kind" to override detection or label something the SDK doesn't know about:

python
with dl.Prefix(project="robotics@alice", prefix="/session-042"):
    dl.upload("./checkpoint.bin", path="models/v3", type="model-weights")

Without a Prefix context

Pass project and prefix directly. Useful for one-off scripts.

python
dl.upload(
    "./video.mp4",
    project="robotics@alice",
    prefix="/session-042",
)

The episode is taken from the Prefix context (or, equivalently, the last segment of the prefix= arg above). Without a Prefix context the file lands directly under the project — prefix= only sets the folder path, not an episode wrapper. For the episode + sub-folder pattern, use Prefix:

python
with dl.Prefix(project="robotics@alice", prefix="/session-042"):
    dl.upload("./video.mp4", path="camera/front")

What you can do next

  • Upload an Episode — bulk-upload a directory of mixed files
  • Bindrs — curate files across folders into named collections
  • Download Files — pull files, folders, or whole episodes back