multi-vector-encoder/mLateOn-medical

Multi-Vector Encoder

This is a Multi-Vector Encoder model finetuned from lightonai/mLateOn-unsupervised on the miriad-4.4_m-split dataset using the sentence-transformers library. It maps inputs to sequences of 128-dimensional token-level vectors and scores them with late interaction (MaxSim), useful for semantic search with late interaction.

mLateOn-medical is a domain specialist. It was trained on 1 million medical question and passage pairs from MIRIAD in 14 hours on a single RTX 3090. On a held-out benchmark of 1,000 medical questions searching 200,000 passages, it beats every general-purpose retrieval model I evaluated, dense, sparse, lexical, and multi-vector alike, including dense models with roughly 30 times as many active parameters.

The full recipe, the evaluation protocol, and the experiments behind each choice are described in Training and Finetuning Multi-Vector Embedding Models with Sentence Transformers.

Model Details

Model Description

• Model Type: Multi-Vector Encoder
• Base model: lightonai/mLateOn-unsupervised
• Maximum Sequence Length: 8192 tokens
• Output Dimensionality: 128 dimensions
• Similarity Function: MaxSim
• Supported Modality: Text
• Training Dataset:
  • miriad-4.4_m-split

Model Sources

• Documentation: Sentence Transformers Documentation
• Documentation: Multi-Vector Encoder Documentation
• Repository: Sentence Transformers on GitHub
• Hugging Face: Multi-Vector Encoders on Hugging Face

Full Model Architecture

MultiVectorEncoder(
  (0): Transformer({'transformer_task': 'feature-extraction', 'modality_config': {'text': {'method': 'forward', 'method_output_name': 'last_hidden_state'}}, 'module_output_name': 'token_embeddings', 'architecture': 'ModernBertModel'})
  (1): Dense({'in_features': 768, 'out_features': 1536, 'bias': False, 'activation_function': 'torch.nn.modules.linear.Identity', 'module_input_name': 'token_embeddings', 'module_output_name': 'token_embeddings', 'use_residual': True})
  (2): Dense({'in_features': 1536, 'out_features': 768, 'bias': False, 'activation_function': 'torch.nn.modules.linear.Identity', 'module_input_name': 'token_embeddings', 'module_output_name': 'token_embeddings', 'use_residual': True})
  (3): Dense({'in_features': 768, 'out_features': 128, 'bias': False, 'activation_function': 'torch.nn.modules.linear.Identity', 'module_input_name': 'token_embeddings', 'module_output_name': 'token_embeddings'})
  (4): MultiVectorMask({'skiplist_words': ['!', '"', '#', '$', '%', '&', "'", '(', ')', '*', '+', ',', '-', '.', '/', ':', ';', '<', '=', '>', '?', '@', '[', '\\', ']', '^', '_', '`', '{', '|', '}', '~'], 'skiplist_tasks': ['document'], 'keep_only_token_ids': None})
  (5): Normalize({'module_input_name': 'token_embeddings', 'module_output_name': 'token_embeddings'})
)

Usage

Direct Usage (Sentence Transformers)

First install the Sentence Transformers library:

pip install -U sentence-transformers

Then you can load this model and run inference.

from sentence_transformers import MultiVectorEncoder

# Download from the 🤗 Hub
model = MultiVectorEncoder("multi-vector-encoder/mLateOn-medical")
# Run inference: each input becomes a sequence of per-token vectors (variable length).
queries = [
    'How does the polymorphism in the IL28B gene region 8099917 affect the response to treatment in patients with AHC?\n',
]
documents = [
    'Several studies suggest that symptomatic and jaundiced patients have a higher rate of SVC and SVR (8, 9, 18, 23, 25, (34) (35) (36) (37) , possibly associated with a better response of the immune system (29, 36) .\n\n The monitoring of viral kinetics can predict the SVC. RNA-HCV undetectable at week 4 in our series was related to a cure rate of 98%; in these patients treatment with PEG-IFN is unnecessary (35) . In contrast, only 69% of patients with detectable HCV-RNA at week 4 were cured. Those treated for persistent viremia at week 4 did not benefit over those untreated, implying that if there is not spontaneous viral clearance at week 4 a bad outcome is not improbable, and reinforces the value of the early determination of HCV-RNA (positive predictive value 95%).\n\n In patients with undetectable HCV-RNA at week 12 our results showed that 90% were cured versus only 9% with persistent viremia. Yet these patients did not benefit from treatment because the cure rate was identical for both those treated and untreated (91 and 92%). Treatment, therefore, would be unnecessary for patients with undetectable HCV-RNA at week 12.\n\n A new factor that can help to decide whether to treat or not to treat is the polymorphism in the IL28B gene region 8099917. Grebely et al. have found that TT homozygosity, but not GG/GT, was the only factor associated to SVC in patients with recent HCV. Furthermore, TT patients were more frequently icteric. These results could not be confirmed in treated patients. Based in these data, they suggested that this polymorphism must be investigated in patients with AHC and early treatment could be started in individuals with non-TT genotypes (38) .\n\n In summary, in Spain the most likely cause of AHC is nosocomial transmission, followed at a distance by IDUs and sexual transmission. Nearly 60% of AHC patients received antiviral therapy with a significantly higher SVR when compared to the untreated group. Given the high rate of spontaneous recovery, from our results, especially for jaundiced patients, treatment should not be initiated too early and it can be delayed until 12 weeks from symptomonset. At week 4, undetectable HCV-RNA was associated with a satisfactory outcome; thus, treatment could be avoided in these patients. HCV-RNA negativity at week 12 is also associated with a very high rate of viral clearance; on the contrary, the persistence of detectable viremia suggests a bad outcome.',
    "The ratio is therefore reliable for use in normalizing the excreted urinary cotinine concentration in urine spot samples, when 24-hour samples are not available or not practical. There has been much debate in the scientific community over whether the ratio is useful (38) , and this affirmative finding helps to settle that debate. In addition, the results indicate that cotinine is best used to distinguish recent smokers from nonsmokers. It is not possible to distinguish nonrecent smokers-those that reported being current smokers that did not smoke on the day of urine collection-from nonsmokers. Given the leveling off of the cotinine at higher smoking intensities, it is also difficult to use urinary cotinine to assess a person's smoking intensity at higher levels (greater CPD) of smoking.\n\n In the radiochemical analysis of the urine samples, the median 210 Pb activity concentration in the 24-hour urine samples of smokers (12 mBq) was statistically greater than that in nonsmokers (8 mBq). The median urinary 210 Po activity concentration in smokers (19 mBq) was also close to being statistically greater than that in nonsmokers (15 mBq Po is absorbed (39) , whereas only 15-20% of ingested 210 Pb is absorbed (40) ] to a larger extent than it is influenced by inhalation intakes or internal body stores. Most recently, a group in central Italy published their study of the 210 Po activity concentration in the 24-hour urine samples of 132 residents and also failed to find a statistically significant difference in the urine of smokers versus nonsmokers (41) .\n\n Significant correlations were observed, however, between the total urinary 210 Pb activity and the smoking indices (cotinine, CPD, years). Regression models indicated that CPD is statistically predicted by total urinary 210 Pb activity, although with less variability being accounted for than in the model using total urinary cotinine as a predictor. 210 Pb, however, with its longer apparent half-life, indicates past smoking (months), whereas cotinine, with its short half-life, indicates recent smoking (days). A logistic regression model indicated that a 1-unit increase in total urinary 210 Pb activity is associated with an estimated 25% increase in the odds of being a smoker, indicating that 210 Pb activity can be used to predict the probability of having smoked within the past months. Following validation, practical applications might include both epidemiologic studies and smoking cessation studies. For the latter, repeated intrasubject measurements could be made over time to monitor a person's progress in quitting smoking. In closing, the results of this study indicate a promising new use of 210 Pb as a biomarker in predicting a person's retrospective smoking status.\n\n No potential conflicts of interest were disclosed.",
    "Patients with acyanotic CHD (with or without increased pulmonary blood flow) were reported to be more susceptible to developing pulmonary TB. No case of PTB was reported in patients with TOF, which is a cyanotic CHD with decreased pulmonary blood flow. Van der Merwe et al. explained that patients with cyanotic CHD have decreased pulmonary blood flow and cyanosis, which can inhibit the growth of M. tuberculosis whereas in acyanotic CHD, normal or increased pulmonary blood flow and normal pulmonary arterial saturation provides an encouraging environment for bacterial growth [4] . Ifere et al. have also reported two cases of PTB in presence of CHD other than TOF [5] . Olguntürk et al. reported PTB in a pediatric patient with tricuspid valve pathology, ASD, and pulmonary hypertension. The clinical suspicion was only confirmed on autopsy as all other investigations were negative [6] .\n\n To our knowledge, there are only two reported cases of pulmonary TB in the presence of TOF [7] [8] . Radovic et al. [7] reported a middle-aged male patient with sputum positive TB who was investigated for CHD because of deteriorating cyanosis and respiratory insufficiency, even with anti-TB drugs. Echo was consistent with TOF. Gunay et al. [8] presented a young female, a diagnosed case of TOF, with respiratory complains, who was diagnosed with PTB and managed with first-line anti-TB drugs for two months. However, the literature doesn't report any case of TOF with secondary PTB and secondary non-invasive pulmonary aspergilloma as in our reported case.\n\n There are various diagnostic tools for pulmonary aspergilloma, including serum GM level, BAL GM, culture of Aspergillus species in sputum or BAL fluid, and serum Aspergillus precipitating antibodies. BAL GM assays were more sensitive than serum GM assays [9] .\n\n Undiagnosed congenital heart defects rarely present in adults. Physicians dealing with PTB should keep a suspicion of underlying TOF in mind. Similarly, physicians dealing with known cases of TOF presenting with respiratory distress should also keep the differential diagnosis of pulmonary tuberculosis and aspergilloma in mind.\n\n Human subjects: Consent was obtained by all participants in this study.\n\n In compliance with the ICMJE uniform disclosure form, all authors declare the following: Payment/services info: All authors have declared that no financial support was received from any organization for the submitted work. Financial relationships: All authors have declared that they have no financial relationships at present or within the previous three years with any organizations that might have an interest in the submitted work. Other relationships: All authors have declared that there are no other relationships or activities that could appear to",
]
query_embeddings = model.encode_query(queries)
document_embeddings = model.encode_document(documents)
print(query_embeddings[0].shape, document_embeddings[0].shape)
# (35, 128) (516, 128)

# Get the MaxSim similarity scores
similarities = model.similarity(query_embeddings, document_embeddings)
print(similarities)
# tensor([[29.7060, 10.7889,  9.2684]])

Evaluation

Metrics

Multi Vector Information Retrieval

• Datasets: monitor and final
• Evaluated with MultiVectorInformationRetrievalEvaluator

Metric	monitor	final
maxsim_accuracy@1	0.966	0.929
maxsim_accuracy@10	0.996	0.989
maxsim_precision@10	0.0996	0.0989
maxsim_recall@10	0.996	0.989
maxsim_ndcg@10	0.9819	0.9608
maxsim_mrr@10	0.9773	0.9515
maxsim_map@10	0.9773	0.9515

Training Details

Training Dataset

miriad-4.4_m-split

• Dataset: miriad-4.4_m-split at 596b9ab
• Size: 1,000,000 training samples
• Columns: question and passage_text
• Approximate statistics based on the first 100 samples:

  	question	passage_text
  type	string	string
  modality	text	text
  details	min: 48 characters mean: 106.6 characters max: 317 characters	min: 2837 characters mean: 4576.82 characters max: 7387 characters

• Samples:

  question	passage_text
  What factors may contribute to increased pulmonary conduit durability in patients who undergo the Ross operation compared to those with right ventricular outflow tract obstruction?	I n 1966, Ross and Somerville 1 reported the first use of an aortic homograft to establish right ventricle-to-pulmonary artery continuity in a patient with tetralogy of Fallot and pulmonary atresia. Since that time, pulmonary position homografts have been used in a variety of right-sided congenital heart lesions. Actuarial 5-year homograft survivals for cryopreserved homografts are reported to range between 55% and 94%, with the shortest durability noted in patients less than 2 years of age. 4 Pulmonary position homografts also are used to replace pulmonary autografts explanted to repair left-sided outflow disease (the Ross operation). Several factors may be likely to favor increased pulmonary conduit durability in Ross patients compared with those with right ventricular outflow tract obstruction, including later age at operation (allowing for larger homografts), more normal pulmonary artery architecture, absence of severe right ventricular hypertrophy, and more natural positioning of ...
  How does MCAM expression in hMSC affect the growth and maintenance of hematopoietic progenitors?	After culture in a 3-dimensional hydrogel-based matrix, which constitutes hypoxic conditions, MCAM expression is lost. Concordantly, Tormin et al. demonstrated that MCAM is down-regulated under hypoxic conditions. 10 Furthermore, it was shown by others and our group that oxygen tension causes selective modification of hematopoietic cell and mesenchymal stromal cell interactions in co-culture systems as well as influence HSPC metabolism. [44] [45] [46] Thus, the observed differences between Sharma et al. and our data in HSPC supporting capacity of hMSC are likely due to the different culture conditions used. Further studies are required to clarify the influence of hypoxia in our model system. Altogether these findings provide further evidence for the importance of MCAM in supporting HSPC. Furthermore, previous reports have shown that MCAM is down-regulated in MSC after several passages as well as during aging and differentiation. 19, 47 Interestingly, MCAM overexpression in hMSC enhance...
  What is the relationship between Fanconi anemia and breast and ovarian cancer susceptibility genes?	( 31 ) , of which 5% -10 % may be caused by genetic factors ( 32 ) , up to half a million of these patients may be at risk of secondary hereditary neoplasms. The historic observation of twofold to fi vefold increased risks of cancers of the ovary, thyroid, and connective tissue after breast cancer ( 33 ) presaged the later syndromic association of these tumors with inherited mutations of BRCA1, BRCA2, PTEN, and p53 ( 16 ) . By far the largest cumulative risk of a secondary cancer in BRCA mutation carriers is associated with cancer in the contralateral breast, which may reach a risk of 29.5% at 10 years ( 34 ) . The Breast Cancer Linkage Consortium ( 35 , 36 ) also documented threefold to fi vefold increased risks of subsequent cancers of prostate, pancreas, gallbladder, stomach, skin (melanoma), and uterus in BRCA2 mutation carriers and twofold increased risks of prostate and pancreas cancer in BRCA1 mutation carriers; these results are based largely on self-reported family history inf...

• Loss: CachedMultiVectorMultipleNegativesRankingLoss with these parameters:

  {
      "scale": 1.0,
      "similarity_fct": "colbert_scores",
      "mini_batch_size": 16,
      "mini_batch_num_tokens": null,
      "score_mini_batch_size": 16,
      "gather_across_devices": false
  }
  

Training Hyperparameters

Non-Default Hyperparameters

• per_device_train_batch_size: 128
• num_train_epochs: 1
• learning_rate: 0.0001
• warmup_steps: 0.05
• bf16: True
• per_device_eval_batch_size: 16
• prompts: {'question': '[Q] ', 'passage_text': '[D] '}
• batch_sampler: no_duplicates
• max_length: 8192

All Hyperparameters

Click to expand

• per_device_train_batch_size: 128
• num_train_epochs: 1
• max_steps: -1
• learning_rate: 0.0001
• lr_scheduler_type: linear
• lr_scheduler_kwargs: None
• warmup_steps: 0.05
• optim: adamw_torch_fused
• optim_args: None
• weight_decay: 0.0
• adam_beta1: 0.9
• adam_beta2: 0.999
• adam_epsilon: 1e-08
• optim_target_modules: None
• gradient_accumulation_steps: 1
• average_tokens_across_devices: True
• max_grad_norm: 1.0
• label_smoothing_factor: 0.0
• bf16: True
• fp16: False
• bf16_full_eval: False
• fp16_full_eval: False
• tf32: None
• gradient_checkpointing: False
• gradient_checkpointing_kwargs: None
• torch_compile: False
• torch_compile_backend: None
• torch_compile_mode: None
• use_liger_kernel: False
• liger_kernel_config: None
• use_cache: False
• neftune_noise_alpha: None
• torch_empty_cache_steps: None
• auto_find_batch_size: False
• log_on_each_node: True
• logging_nan_inf_filter: True
• include_num_input_tokens_seen: no
• log_level: passive
• log_level_replica: warning
• disable_tqdm: False
• project: huggingface
• trackio_space_id: None
• trackio_bucket_id: None
• trackio_static_space_id: None
• per_device_eval_batch_size: 16
• prediction_loss_only: True
• eval_on_start: False
• eval_do_concat_batches: True
• eval_use_gather_object: False
• eval_accumulation_steps: None
• include_for_metrics: []
• batch_eval_metrics: False
• save_only_model: False
• save_on_each_node: False
• enable_jit_checkpoint: False
• push_to_hub: False
• hub_private_repo: None
• hub_model_id: None
• hub_strategy: every_save
• hub_always_push: False
• hub_revision: None
• load_best_model_at_end: False
• ignore_data_skip: False
• restore_callback_states_from_checkpoint: False
• full_determinism: False
• seed: 42
• data_seed: None
• use_cpu: False
• accelerator_config: {'split_batches': False, 'dispatch_batches': None, 'even_batches': True, 'use_seedable_sampler': True, 'non_blocking': False, 'gradient_accumulation_kwargs': None}
• parallelism_config: None
• dataloader_drop_last: False
• dataloader_num_workers: 0
• dataloader_pin_memory: True
• dataloader_persistent_workers: False
• dataloader_prefetch_factor: None
• remove_unused_columns: True
• label_names: None
• train_sampling_strategy: random
• length_column_name: length
• ddp_find_unused_parameters: None
• ddp_bucket_cap_mb: None
• ddp_broadcast_buffers: False
• ddp_static_graph: None
• ddp_backend: None
• ddp_timeout: 1800
• fsdp: None
• fsdp_config: None
• deepspeed: None
• debug: []
• skip_memory_metrics: True
• do_predict: False
• resume_from_checkpoint: None
• warmup_ratio: None
• local_rank: -1
• prompts: {'question': '[Q] ', 'passage_text': '[D] '}
• batch_sampler: no_duplicates
• multi_dataset_batch_sampler: proportional
• router_mapping: {}
• learning_rate_mapping: {}
• max_length: 8192

Training Logs

Epoch	Step	Training Loss	monitor_maxsim_ndcg@10	final_maxsim_ndcg@10
0.0500	391	0.3596	-	-
0.1001	782	0.0226	0.9731	-
0.1501	1173	0.0196	-	-
0.2002	1564	0.0187	0.9733	-
0.2502	1955	0.0150	-	-
0.3003	2346	0.0158	0.9752	-
0.3503	2737	0.0141	-	-
0.4004	3128	0.0135	0.9806	-
0.4504	3519	0.0123	-	-
0.5004	3910	0.0121	0.9809	-
0.5505	4301	0.0118	-	-
0.6005	4692	0.0119	0.9776	-
0.6506	5083	0.0106	-	-
0.7006	5474	0.0103	0.9837	-
0.7507	5865	0.0085	-	-
0.8007	6256	0.0083	0.9809	-
0.8508	6647	0.0077	-	-
0.9008	7038	0.0076	0.9835	-
0.9509	7429	0.0069	-	-
1.0	7813	-	0.9819	-
-1	-1	-	-	0.9608

Training Time

• Training: 14.1 hours
• Evaluation: 25.3 minutes
• Total: 14.5 hours

Framework Versions

• Python: 3.11.13
• Sentence Transformers: 6.0.0
• Transformers: 5.14.1
• PyTorch: 2.11.0+cu128
• Accelerate: 1.5.2
• Datasets: 3.5.0
• Tokenizers: 0.22.2

Citation

BibTeX

Sentence Transformers

@inproceedings{reimers-2019-sentence-bert,
    title = "Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks",
    author = "Reimers, Nils and Gurevych, Iryna",
    booktitle = "Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing",
    month = "11",
    year = "2019",
    publisher = "Association for Computational Linguistics",
    url = "https://arxiv.org/abs/1908.10084",
}

CachedMultiVectorMultipleNegativesRankingLoss

@misc{gao2021scaling,
    title={Scaling Deep Contrastive Learning Batch Size under Memory Limited Setup},
    author={Luyu Gao and Yunyi Zhang and Jiawei Han and Jamie Callan},
    year={2021},
    eprint={2101.06983},
    archivePrefix={arXiv},
    primaryClass={cs.LG}
}