Zhan-Lun commited on
Commit
5a6eef0
·
verified ·
1 Parent(s): 73a9131

Update 8B model card

Browse files
Files changed (1) hide show
  1. README.md +24 -29
README.md CHANGED
@@ -38,25 +38,21 @@ L2-normalized multi-vector embeddings for late-interaction retrieval.
38
  The model uses bidirectional attention in Qwen3.5's full-attention layers and
39
  a learned 640-dimensional projection head. The unused language-model head has
40
  been removed from the released checkpoint. Consequently, the released
41
- embedding model contains 8,395,317,104 parameters, reflected by the
42
  `webAI-ColVec1.1-8b` repository name, while its original backbone is
43
  Qwen3.5-9B.
44
 
45
  ### Training data
46
 
47
- We created filtered, balanced, and multilingual curated subsets from the six
48
- public datasets below. **This Qwen3.5-9B-backbone model was trained on a
49
- 750,000-sample curated subset.** The companion 4B model was trained on a
50
- separate 500,000-sample curated subset.
51
-
52
- | Dataset source |
53
- | :--- |
54
- | [ColPali Train Set](https://huggingface.co/datasets/vidore/colpali_train_set) |
55
- | [Docmatix-IR](https://huggingface.co/datasets/Tevatron/docmatix-ir) |
56
- | [VisRAG In-Domain](https://huggingface.co/datasets/openbmb/VisRAG-Ret-Train-In-domain-data) |
57
- | [VisRAG Synthetic](https://huggingface.co/datasets/openbmb/VisRAG-Ret-Train-Synthetic-data) |
58
- | [VDR Multilingual Train](https://huggingface.co/datasets/llamaindex/vdr-multilingual-train) |
59
- | [Wiki-SS-NQ](https://huggingface.co/datasets/Tevatron/wiki-ss-nq) |
60
 
61
  ## 🛠️ Model specifications
62
 
@@ -84,10 +80,9 @@ separate 500,000-sample curated subset.
84
 
85
  ## 📊 Evaluation results
86
 
87
- The table reports NDCG@10 × 100 on the eight **public** ViDoRe V3 tasks. Private
88
- Nuclear and Telecom scores are intentionally omitted and will be added after
89
- official evaluation. Each task value is the mean of its six language subsets;
90
- the public average is the unweighted mean of the eight public task values.
91
 
92
  The result artifacts record MTEB 2.18.5, Transformers 5.14.1, PyTorch 2.9.0
93
  with CUDA 12.8, `bfloat16`, FlashAttention 2.8.3, and batch size 32. The release
@@ -100,8 +95,8 @@ Model encoding runs in `bfloat16`. Before MaxSim scoring, query and document
100
  embeddings are moved to CPU and converted to `float32`. All reported ViDoRe
101
  results use this FP32 scoring path.
102
 
103
- | Model | CompSci | Energy | FinanceEn | FinanceFr | HR | Industrial | Pharma | Physics | **Avg. public** |
104
- | :--- | ---: | ---: | ---: | ---: | ---: | ---: | ---: | ---: | ---: |
105
  | **webAI-ColVec1.1-8b (this model)** | 80.14 | 70.09 | **71.89** | **55.10** | 68.49 | **57.41** | 67.88 | 51.29 | **65.29** |
106
  | [VultronRetriever Prime](https://huggingface.co/vultr/VultronRetrieverPrime-Qwen3.5-8B) | 79.81 | **70.26** | 69.01 | 54.51 | 66.82 | **57.41** | **68.19** | **51.73** | 64.72 |
107
  | [webAI-ColVec1-9b](https://huggingface.co/webAI-Official/webAI-ColVec1-9b) | **80.92** | 69.77 | 68.28 | 53.72 | **70.04** | 57.18 | 67.32 | 48.38 | 64.45 |
@@ -238,18 +233,18 @@ change retrieval scores.
238
 
239
  ### Strengths
240
 
241
- - Strong public-task performance on ViDoRe V3.
242
- - OCR-free retrieval over visually complex document pages.
243
- - Fine-grained MaxSim matching with compact 640-dimensional token vectors.
244
- - Multilingual training data and evaluation.
245
- - FlashAttention 2 support across compatible CUDA GPUs and optimized
246
- FlashAttention 3 inference on Hopper GPUs.
 
247
 
248
  ### Limitations
249
 
250
- - Multi-vector indexes require more storage than single-vector embeddings.
251
- - Scores can vary slightly between FlashAttention 2, FlashAttention 3, and
252
- SDPA.
253
 
254
  ## License
255
 
 
38
  The model uses bidirectional attention in Qwen3.5's full-attention layers and
39
  a learned 640-dimensional projection head. The unused language-model head has
40
  been removed from the released checkpoint. Consequently, the released
41
+ embedding model contains ~8.4B parameters, reflected by the
42
  `webAI-ColVec1.1-8b` repository name, while its original backbone is
43
  Qwen3.5-9B.
44
 
45
  ### Training data
46
 
47
+ We created filtered, balanced, and multilingual curated subsets from six public
48
+ datasets: [ColPali Train Set](https://huggingface.co/datasets/vidore/colpali_train_set),
49
+ [Docmatix-IR](https://huggingface.co/datasets/Tevatron/docmatix-ir),
50
+ [VisRAG In-Domain](https://huggingface.co/datasets/openbmb/VisRAG-Ret-Train-In-domain-data),
51
+ [VisRAG Synthetic](https://huggingface.co/datasets/openbmb/VisRAG-Ret-Train-Synthetic-data),
52
+ [VDR Multilingual Train](https://huggingface.co/datasets/llamaindex/vdr-multilingual-train),
53
+ and [Wiki-SS-NQ](https://huggingface.co/datasets/Tevatron/wiki-ss-nq). This
54
+ Qwen3.5-9B-backbone model was trained on a 750,000-sample curated subset as well
55
+ as synthetically generated data.
 
 
 
 
56
 
57
  ## 🛠️ Model specifications
58
 
 
80
 
81
  ## 📊 Evaluation results
82
 
83
+ The table reports NDCG@10 scores on the eight **public** ViDoRe V3 tasks as
84
+ percentages rather than values between 0 and 1 (for example, 0.80 is shown as
85
+ 80.00). Each task value is the mean of its six language subsets; the public average is the unweighted mean of the eight public task values.
 
86
 
87
  The result artifacts record MTEB 2.18.5, Transformers 5.14.1, PyTorch 2.9.0
88
  with CUDA 12.8, `bfloat16`, FlashAttention 2.8.3, and batch size 32. The release
 
95
  embeddings are moved to CPU and converted to `float32`. All reported ViDoRe
96
  results use this FP32 scoring path.
97
 
98
+ | Model | Computer Science | Energy | FinanceEn | FinanceFr | HR | Industrial | Pharmaceuticals | Physics | **Avg. public** |
99
+ | :--- | :---: | :---: | :---: | :---: | :---: | :---: | :---: | :---: | :---: |
100
  | **webAI-ColVec1.1-8b (this model)** | 80.14 | 70.09 | **71.89** | **55.10** | 68.49 | **57.41** | 67.88 | 51.29 | **65.29** |
101
  | [VultronRetriever Prime](https://huggingface.co/vultr/VultronRetrieverPrime-Qwen3.5-8B) | 79.81 | **70.26** | 69.01 | 54.51 | 66.82 | **57.41** | **68.19** | **51.73** | 64.72 |
102
  | [webAI-ColVec1-9b](https://huggingface.co/webAI-Official/webAI-ColVec1-9b) | **80.92** | 69.77 | 68.28 | 53.72 | **70.04** | 57.18 | 67.32 | 48.38 | 64.45 |
 
233
 
234
  ### Strengths
235
 
236
+ - **Performance:** State-of-the-art retrieval performance on the public ViDoRe
237
+ V3 tasks, with excellent multimodal document retrieval results.
238
+ - **Complex layouts:** Excellent handling of chart-rich PDFs and
239
+ domain-specific documents.
240
+ - **End-to-end retrieval:** OCR-free retrieval on unseen multimodal documents
241
+ without using an intermediate vision-language model to generate summaries.
242
+ - **Multilingualism:** Strong performance on non-English document inputs.
243
 
244
  ### Limitations
245
 
246
+ - **Storage Cost:** Still larger than single-vector baselines despite the
247
+ smaller token dimension.
 
248
 
249
  ## License
250