Zhan-Lun commited on
Commit
b4f97a7
·
verified ·
1 Parent(s): 4332508

Update 4B model card

Browse files
Files changed (1) hide show
  1. README.md +25 -28
README.md CHANGED
@@ -41,19 +41,15 @@ been removed from the released checkpoint.
41
 
42
  ### Training data
43
 
44
- We created filtered, balanced, and multilingual curated subsets from the six
45
- public datasets below. **This 4B model was trained on a 500,000-sample curated
46
- subset.** The companion 9B-backbone model was trained on a separate
47
- 750,000-sample curated subset.
48
-
49
- | Dataset source |
50
- | :--- |
51
- | [ColPali Train Set](https://huggingface.co/datasets/vidore/colpali_train_set) |
52
- | [Docmatix-IR](https://huggingface.co/datasets/Tevatron/docmatix-ir) |
53
- | [VisRAG In-Domain](https://huggingface.co/datasets/openbmb/VisRAG-Ret-Train-In-domain-data) |
54
- | [VisRAG Synthetic](https://huggingface.co/datasets/openbmb/VisRAG-Ret-Train-Synthetic-data) |
55
- | [VDR Multilingual Train](https://huggingface.co/datasets/llamaindex/vdr-multilingual-train) |
56
- | [Wiki-SS-NQ](https://huggingface.co/datasets/Tevatron/wiki-ss-nq) |
57
 
58
  ## 🛠️ Model specifications
59
 
@@ -81,10 +77,10 @@ subset.** The companion 9B-backbone model was trained on a separate
81
 
82
  ## 📊 Evaluation results
83
 
84
- The table reports NDCG@10 × 100 on the eight **public** ViDoRe V3 tasks. Private
85
- Nuclear and Telecom scores are intentionally omitted and will be added after
86
- official evaluation. Each task value is the mean of its six language subsets;
87
- the public average is the unweighted mean of the eight public task values.
88
 
89
  The result artifacts record MTEB 2.18.5, Transformers 5.14.1, PyTorch 2.9.0
90
  with CUDA 12.8, `bfloat16`, FlashAttention 2.8.3, and batch size 32. The release
@@ -97,8 +93,8 @@ Model encoding runs in `bfloat16`. Before MaxSim scoring, query and document
97
  embeddings are moved to CPU and converted to `float32`. All reported ViDoRe
98
  results use this FP32 scoring path.
99
 
100
- | Model | CompSci | Energy | FinanceEn | FinanceFr | HR | Industrial | Pharma | Physics | **Avg. public** |
101
- | :--- | ---: | ---: | ---: | ---: | ---: | ---: | ---: | ---: | ---: |
102
  | **[webAI-ColVec1.1-8b](https://huggingface.co/webAI-Official/webAI-ColVec1.1-8b)** | 80.14 | 70.09 | **71.89** | **55.10** | 68.49 | **57.41** | 67.88 | 51.29 | **65.29** |
103
  | [VultronRetriever Prime](https://huggingface.co/vultr/VultronRetrieverPrime-Qwen3.5-8B) | 79.81 | **70.26** | 69.01 | 54.51 | 66.82 | **57.41** | **68.19** | **51.73** | 64.72 |
104
  | [webAI-ColVec1-9b](https://huggingface.co/webAI-Official/webAI-ColVec1-9b) | **80.92** | 69.77 | 68.28 | 53.72 | **70.04** | 57.18 | 67.32 | 48.38 | 64.45 |
@@ -235,18 +231,19 @@ change retrieval scores.
235
 
236
  ### Strengths
237
 
238
- - Strong public-task performance on ViDoRe V3.
239
- - OCR-free retrieval over visually complex document pages.
240
- - Fine-grained MaxSim matching with compact 640-dimensional token vectors.
241
- - Multilingual training data and evaluation.
242
- - FlashAttention 2 support across compatible CUDA GPUs and optimized
243
- FlashAttention 3 inference on Hopper GPUs.
 
 
244
 
245
  ### Limitations
246
 
247
- - Multi-vector indexes require more storage than single-vector embeddings.
248
- - Scores can vary slightly between FlashAttention 2, FlashAttention 3, and
249
- SDPA.
250
 
251
  ## License
252
 
 
41
 
42
  ### Training data
43
 
44
+ We created filtered, balanced, and multilingual curated subsets from six public
45
+ datasets: [ColPali Train Set](https://huggingface.co/datasets/vidore/colpali_train_set),
46
+ [Docmatix-IR](https://huggingface.co/datasets/Tevatron/docmatix-ir),
47
+ [VisRAG In-Domain](https://huggingface.co/datasets/openbmb/VisRAG-Ret-Train-In-domain-data),
48
+ [VisRAG Synthetic](https://huggingface.co/datasets/openbmb/VisRAG-Ret-Train-Synthetic-data),
49
+ [VDR Multilingual Train](https://huggingface.co/datasets/llamaindex/vdr-multilingual-train),
50
+ and [Wiki-SS-NQ](https://huggingface.co/datasets/Tevatron/wiki-ss-nq). This
51
+ Qwen3.5-4B-backbone model was trained on a 500,000-sample curated subset as well
52
+ as synthetically generated data.
 
 
 
 
53
 
54
  ## 🛠️ Model specifications
55
 
 
77
 
78
  ## 📊 Evaluation results
79
 
80
+ The table reports NDCG@10 scores on the eight **public** ViDoRe V3 tasks as
81
+ percentages rather than values between 0 and 1 (for example, 0.80 is shown as
82
+ 80.00). Each task value is the mean of its six language subsets; the public
83
+ average is the unweighted mean of the eight public task values.
84
 
85
  The result artifacts record MTEB 2.18.5, Transformers 5.14.1, PyTorch 2.9.0
86
  with CUDA 12.8, `bfloat16`, FlashAttention 2.8.3, and batch size 32. The release
 
93
  embeddings are moved to CPU and converted to `float32`. All reported ViDoRe
94
  results use this FP32 scoring path.
95
 
96
+ | Model | Computer Science | Energy | FinanceEn | FinanceFr | HR | Industrial | Pharmaceuticals | Physics | **Avg. public** |
97
+ | :--- | :---: | :---: | :---: | :---: | :---: | :---: | :---: | :---: | :---: |
98
  | **[webAI-ColVec1.1-8b](https://huggingface.co/webAI-Official/webAI-ColVec1.1-8b)** | 80.14 | 70.09 | **71.89** | **55.10** | 68.49 | **57.41** | 67.88 | 51.29 | **65.29** |
99
  | [VultronRetriever Prime](https://huggingface.co/vultr/VultronRetrieverPrime-Qwen3.5-8B) | 79.81 | **70.26** | 69.01 | 54.51 | 66.82 | **57.41** | **68.19** | **51.73** | 64.72 |
100
  | [webAI-ColVec1-9b](https://huggingface.co/webAI-Official/webAI-ColVec1-9b) | **80.92** | 69.77 | 68.28 | 53.72 | **70.04** | 57.18 | 67.32 | 48.38 | 64.45 |
 
231
 
232
  ### Strengths
233
 
234
+ - **Performance:** State-of-the-art retrieval performance among 4B models on
235
+ the public ViDoRe V3 tasks, with excellent multimodal document retrieval
236
+ results.
237
+ - **Complex layouts:** Excellent handling of chart-rich PDFs and
238
+ domain-specific documents.
239
+ - **End-to-end retrieval:** OCR-free retrieval on unseen multimodal documents
240
+ without using an intermediate vision-language model to generate summaries.
241
+ - **Multilingualism:** Strong performance on non-English document inputs.
242
 
243
  ### Limitations
244
 
245
+ - **Storage Cost:** Still larger than single-vector baselines despite the
246
+ smaller token dimension.
 
247
 
248
  ## License
249