processing/README.md

# Camera-Dataset build

Builds reference/target video pairs that **share a camera trajectory but show different
scenes**, from the Kwai
[CameraClone-Dataset](https://huggingface.co/datasets/KwaiVGI/CameraClone-Dataset)
(`CamCloneMaster`, SIGGRAPH Asia 2025).

| split | path | pairs |
|---|---|---|
| `train` | `/mnt/dataset/xinyuy/datasets/LTX-2/Camera-Dataset` | 2,000 |
| `val` | `/mnt/dataset/xinyuy/datasets/LTX-2/Camera-Dataset-Val` | 50 |

The two are **disjoint**: `val` selection pre-seeds its "already used" sets from the train
split, so they share no trajectory group, no 3D scene location (target *or* reference), and
no video file. `verify` asserts all three in both directions.

Output matches the layout of the sibling `LTX-2/Depth-Dataset`, `Pose-Dataset` and
`Inpainting-Dataset`, so it drops into the same training pipeline:

```
Camera-Dataset/
├── 0401_g207_c02_s4860.mp4          target   (the video to generate)
├── 0401_g207_c02_s4860_camera.mp4   reference (same camera motion, different scene)
├── ...
└── dataset.json                     [{caption, media_path, reference_path}, ...]
```

Name = `<date>_g<traj_group>_c<cam_index>_s<target_scene_id>`. The `_camera` suffix follows
the existing `_depth` / `_pose` / `_painting` convention. `caption` is the source caption
verbatim — it describes the **target**, so the prompt describes the scene and the reference
supplies only the motion.

## Where the pairing comes from

Inside one `data/<date>/traj_<group>_<cam>/` directory, every video was rendered with the
**same camera trajectory** at a **different 3D location**. So any `(video_path,
ref_video_path)` row of `CamCloneDataset.csv` is a valid pair by construction — verified
across all 1,154,819 rows: the reference is in the same `traj_*` dir 100% of the time, and
is never the same scene file as the target.

The four locations in a group really are different scenes. Group 1 of batch 0316 covers
scenes {1, 550, 935, 1224}, captioned "ancient temple complex", "grand ornate room",
"modern office at twilight", "cozy bar".

## Source data (measured, not assumed)

| | |
|---|---|
| CSV rows | 1,154,819 |
| Trajectory dirs / groups / locations | 101,896 / 10,192 / 40,767 |
| Locations per group | 4 (one group has 3) |
| Video | HEVC, 15 fps, 77 frames |
| Resolution | 1344x768 (batches 0316, 0317) · 1008x576 (0401, 0402, 0404, 0407, 0410) — both exactly 1.75 |

## Two findings that shaped the design

**1. The camera index is not a motion type.** `cam07` measured across four different groups
gives `(pan −9, tilt +16)`, `(−20, −16)`, `(+14, −1)`, `(−48, −8)`. Trajectories are
generated per group, so nothing can be inferred from the `_NN` suffix — motion has to be
measured from pixels.

**2. Motion is already well balanced, so the build does not rebalance it.** 1,234 randomly
sampled trajectories:

| component | share | | component | share |
|---|---|---|---|---|
| pan_left | 34.7% | | roll_cw | 14.9% |
| tilt_up | 31.0% | | zoom_in | 14.6% |
| tilt_down | 30.6% | | roll_ccw | 13.9% |
| pan_right | 28.5% | | zoom_out | 12.1% |

Directions are near-symmetric, active-axis counts spread 1/2/3/4 at 36/24/21/9%, path shape
is 56% linear / 25% arc / 19% complex, and of 179 distinct motion classes the largest holds
7.4%. That is a *better* spread than the hand-curated Cameraman v2 LoRA set (27% → 7%).
Imposing quotas would have cost ~27 min of measurement to correct a skew that isn't there.

The one real skew — **11% near-static trajectories**, dead weight for a camera-motion LoRA —
is handled with a cap, not a quota system.

## Scene diversity

Scene ids are not contiguous per 3D environment, so there is no structural label to sample
on. Diversity therefore comes from the captions: TF-IDF (1–2 grams, English stopwords) +
`MiniBatchKMeans(k=60)` over 40,767 candidate captions, then selection round-robins across
clusters, always drawing from the least-used one.

Hard constraints on the final set: one pair per trajectory group (2,000 of 10,192 available,
so every pair has a distinct camera trajectory), each target location used once, each
reference location used once, and a per-batch quota proportional to batch size.

## Camera motion estimator

`measure_trajectory()` — every 4th frame downscaled to 320px, ORB(1500) + BFMatcher
(crossCheck) + `estimateAffinePartial2D(RANSAC, thr=2.0)` between consecutive frames,
accumulated into `[pan, tilt, roll°, 100·log scale, straightness]`.

RANSAC treats the animated character as outliers, so the fit follows the background — the
camera. `straightness = |net displacement| / path length` separates linear moves (≈1.0) from
arcs and splines (<0.85).

Validated on a known-shared trajectory: `0401/group500/cam01` measures pan
**−451.4 / −456.2 / −442.8 / −450.4** across its four different scenes. Farneback optical
flow was tried first and is too contaminated by character motion and scene parallax — the
same trajectory gave tilt −7.9 / +2.6 / +9.6 / −0.4. Do not go back to it.

The 2D signature is scene-dependent for complex 3D moves (parallax differs by scene depth),
so it is used only for labelling and the static cap. It never gates pair correctness, which
comes from the dataset's own directory structure.

## Video format — native (`NATIVE = True`)

The build ships the source video **untouched**: a container remux, no re-encode.

```
ffmpeg -i <src> -c:v copy -an <dst>
```

* **HEVC / yuvj420p / 15 fps / 77 frames**, at each batch's own resolution —
  1344x768 for 0316 and 0317, 1008x576 for the rest. Both videos of a pair always come
  from the same trajectory dir, hence the same batch, so a pair is never mixed-resolution.
* **Lossless**: verified bit-identical to the source on 24 sampled clips by comparing the
  video-stream MD5 (`ffmpeg -map 0:v -c copy -f md5 -`). ffmpeg still demuxes the input,
  so a corrupt source would fail loudly rather than pass through.
* Takes ~11 s for all 4,000 clips, and the whole dataset is 3.0 GB.

**Note on frame count:** 77 is not 8n+1, so an LTX trainer with an 8x temporal VAE will
have to bucket or truncate these itself (73 is the nearest valid count below). Setting
`NATIVE = False` switches the encode stage to the normalised path instead — 1280x720,
h264 yuv420p, trimmed to `OUT_FRAMES = 73` with `scale=1280:-2,crop=1280:720`, target and
reference trimmed identically so motion stays synced. That path takes ~10 min on 96 cores
(the A100s in this box have no NVENC) and yields ~9.2 GB. `verify` adapts to whichever
mode is set.

## Running it

```bash
python3 build_camera_dataset.py --stage all                 # train, ~2 min in native mode
python3 build_camera_dataset.py --split val --stage all     # val, ~1 min (run train first)
python3 build_camera_dataset.py --split val --stage verify  # re-check an existing build
```

Stages are individually runnable and resumable — `index`, `select`, `measure`, `trim`,
`encode`, `emit`, `verify`. `measure` caches to `motion_labels*.csv` and `encode` skips clips
that already exist, so an interrupted run picks up where it stopped. `index.parquet` and
`candidates.parquet` (captions + clusters) are split-independent and reused, so only the
first build pays the ~35 s caption fetch and clustering; `--split val` skips `index`
entirely.

Build **train before val** — the val split reads `final_pairs.csv` to know what to hold out,
and exits with a clear message if it is missing.

Tunables live at the top of the script: `N_FINAL`, `N_CANDIDATES`, `STATIC_CAP`,
`N_CLUSTERS`, `OUT_W/OUT_H/OUT_FRAMES`, and the motion thresholds
`T_PAN/T_TILT/T_ROLL/T_ZOOM` (units are pixels/degrees on the 320px analysis frame).

## Artifacts

Val artifacts carry a `_val` suffix; train artifacts are unsuffixed.

| file | contents |
|---|---|
| `camera_dataset_mapping.csv` / `_val` | **the deliverable mapping** — every output file to its original path, plus motion labels, caption cluster and caption |
| `index.parquet` | parsed path index of all 1,154,819 CSV rows (shared) |
| `candidates.parquet` | 40,767 candidates with captions and cluster ids (shared) |
| `selected_candidates.csv` / `_val` | the shortlisted pairs (2,400 train / 70 val) |
| `motion_labels.csv` / `_val` | measured signature + motion class per shortlisted trajectory |
| `final_pairs.csv` / `_val` | the pairs that were shipped (2,000 train / 50 val) |

`camera_dataset_mapping.csv` columns: `media_path, reference_path, target_src_path,
reference_src_path, date, traj_group, cam_index, target_scene_id, ref_scene_id, src_width,
src_height, motion_components, n_axes, path_shape, pan, tilt, roll, zoom, straightness,
caption_cluster, caption`.

The source `data/` tree (277 GB, 407,550 files) is read-only to this build.