README.md

# LVDU — Long Video Data for Understanding

Working storage for the video-understanding datasets used across this PhD.
One top-level directory per dataset; every dataset follows the same shape so
tooling written against one works against all of them.

## Datasets

| Directory | Records | Media | Size | Source |
|---|---:|---:|---:|---|
| `LLaVA-Video-83K/` | 83,000 | 71,224 | 429 GiB | stratified 5% subset of `lmms-lab/LLaVA-Video-178K` (+ ShareGPTVideo frames) |

## Convention

```
<dataset>/
├── README.md            # dataset card: provenance, how it was built, usage
├── annotations/         # json/jsonl, upstream schema kept unchanged
├── metadata/
│   ├── manifest.parquet     # record -> media + stratification columns
│   └── shard_index.parquet  # member -> (shard, offset, nbytes)
├── shards/<kind>/*.tar  # uncompressed tar, ~2 GiB each
└── <extras>/            # embeddings, oversized loose media, ...
```

Rules that make this work:

- **Shards are uncompressed.** A member is then a contiguous byte range, so any
  file is one `pread` away and can be decoded from memory. Compression would
  force sequential scans.
- **Member names are the upstream relative paths.** Annotations never need
  rewriting, and a shard can be unpacked with plain `tar` if something goes wrong.
- **`shard_index.parquet` is the contract.** Anything that reads media goes
  through it; nothing hardcodes shard filenames.
- **Oversized files stay loose.** Anything big enough that buffering it in RAM
  would hurt (>64 MiB for video) sits outside the shards, with an index row
  whose `shard` is empty.

Buckets are not versioned — files here can be overwritten or deleted in place.
The citable, version-pinned copy of the annotations lives in a separate dataset
repo per dataset; this bucket holds the bulk media.