Instructions to use xiaomoguhzz/VisionEncoder with libraries, inference providers, notebooks, and local apps. Follow these links to get started.
- Libraries
- Transformers
How to use xiaomoguhzz/VisionEncoder with Transformers:
# Load model directly from transformers import AutoModel model = AutoModel.from_pretrained("xiaomoguhzz/VisionEncoder", device_map="auto") - Notebooks
- Google Colab
- Kaggle
Add V12.4.6 video decode manifest
Browse files- data/12.4.6/README.md +27 -0
- data/12.4.6/decode_manifest.jsonl.gz +3 -0
- data/12.4.6/failures.jsonl +2 -0
- data/12.4.6/metadata.json +34 -0
data/12.4.6/README.md
ADDED
|
@@ -0,0 +1,27 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
# V12.4.6 S0 视频解码清单
|
| 2 |
+
|
| 3 |
+
本目录只保存复现 V12.4.6 S0 视频帧 cache 所需的数据清单和解码合同,不包含解码后的 233 GB shard。
|
| 4 |
+
|
| 5 |
+
## 文件
|
| 6 |
+
|
| 7 |
+
- `decode_manifest.jsonl.gz`:178,490 个成功视频的 canonical 顺序、相对路径、精确采样帧下标、FPS、总帧数、JPEG 大小与 CRC32。
|
| 8 |
+
- `failures.jsonl`:原始构建中跳过的 2 个视频。
|
| 9 |
+
- `metadata.json`:原始 cache 的数据合同和统计。
|
| 10 |
+
|
| 11 |
+
`video_path` 均相对于本地 `LLaVA-Video-178K/` 数据根目录。解码后必须保持 manifest 的行顺序,训练时的 `DistributedSampler(seed=42)` 依赖该顺序。
|
| 12 |
+
|
| 13 |
+
## 解码合同
|
| 14 |
+
|
| 15 |
+
1. 使用 PyAV 读取视频,video stream 设为单线程。
|
| 16 |
+
2. 直接使用 manifest 中的 32 个 `frame_indices`。这些下标对应全时长均匀采样:`np.linspace(0, total_frames - 1, 32, dtype=int)`。
|
| 17 |
+
3. 将目标帧转为 RGB24,并以 Pillow `BICUBIC` 直接缩放到 `384 x 384`。
|
| 18 |
+
4. 按帧序排列为 `2 x 16` contact sheet,保存为 JPEG,`quality=95`。
|
| 19 |
+
5. 跳过 `failures.jsonl` 中的两个视频。
|
| 20 |
+
|
| 21 |
+
预期结果为 178,490 条成功记录和 2 条失败记录。`metadata.json` 中的 `index_sha256` 对应原始未脱敏 cache index,不对应本目录的 portable manifest。
|
| 22 |
+
|
| 23 |
+
读取示例:
|
| 24 |
+
|
| 25 |
+
```bash
|
| 26 |
+
gzip -dc data/12.4.6/decode_manifest.jsonl.gz | head -n 1
|
| 27 |
+
```
|
data/12.4.6/decode_manifest.jsonl.gz
ADDED
|
@@ -0,0 +1,3 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
version https://git-lfs.github.com/spec/v1
|
| 2 |
+
oid sha256:8086184aa3316570b8fb17013f99950a3aecabb7693a74d7a6767a58f85aacc8
|
| 3 |
+
size 16427343
|
data/12.4.6/failures.jsonl
ADDED
|
@@ -0,0 +1,2 @@
|
|
|
|
|
|
|
|
|
|
| 1 |
+
{"error":"ValueError('missing sampled frames from 30_60_s_academic_v0_1/academic_source/activitynet/v_MkKUQ4MMHd8.mp4: [934]')","source_row":113315,"video_path":"30_60_s_academic_v0_1/academic_source/activitynet/v_MkKUQ4MMHd8.mp4"}
|
| 2 |
+
{"error":"ValueError('missing sampled frames from 1_2_m_academic_v0_1/academic_source/activitynet/v_qEU4vKowVo4.mp4: [693]')","source_row":136437,"video_path":"1_2_m_academic_v0_1/academic_source/activitynet/v_qEU4vKowVo4.mp4"}
|
data/12.4.6/metadata.json
ADDED
|
@@ -0,0 +1,34 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
{
|
| 2 |
+
"failures": 2,
|
| 3 |
+
"index_sha256": "b612ec89bd36768a906bc3333fc570e9392db1df0bcb2d3175da5448835b88a0",
|
| 4 |
+
"num_frames": 32,
|
| 5 |
+
"num_workers": 32,
|
| 6 |
+
"prefetch_factor": 2,
|
| 7 |
+
"records": 178490,
|
| 8 |
+
"resolution": 384,
|
| 9 |
+
"sampling": "uniform_full_duration",
|
| 10 |
+
"schema": "visionencoder.v12_1_s0_video_frame_cache",
|
| 11 |
+
"sheet_grid": [
|
| 12 |
+
2,
|
| 13 |
+
16
|
| 14 |
+
],
|
| 15 |
+
"source": "v11_2_1_current",
|
| 16 |
+
"student_frame_positions": [
|
| 17 |
+
0,
|
| 18 |
+
2,
|
| 19 |
+
4,
|
| 20 |
+
6,
|
| 21 |
+
8,
|
| 22 |
+
10,
|
| 23 |
+
12,
|
| 24 |
+
14,
|
| 25 |
+
16,
|
| 26 |
+
18,
|
| 27 |
+
20,
|
| 28 |
+
22,
|
| 29 |
+
24,
|
| 30 |
+
26,
|
| 31 |
+
28,
|
| 32 |
+
30
|
| 33 |
+
]
|
| 34 |
+
}
|