| | --- |
| | pipeline_tag: sentence-similarity |
| | tags: |
| | - feature-extraction |
| | - sentence-similarity |
| | language: |
| | - de |
| | - en |
| | - es |
| | - fr |
| | - it |
| | - nl |
| | - ja |
| | - pt |
| | - zh |
| | --- |
| | |
| | # Model Card for `vectorizer.raspberry` |
| |
|
| | This model is a vectorizer developed by Sinequa. It produces an embedding vector given a passage or a query. The passage vectors are stored in our vector index and the query vector is used at query time to look up relevant passages in the index. |
| |
|
| | Model name: `vectorizer.raspberry` |
| |
|
| | ## Supported Languages |
| |
|
| | The model was trained and tested in the following languages: |
| |
|
| | - English |
| | - French |
| | - German |
| | - Spanish |
| | - Italian |
| | - Dutch |
| | - Japanese |
| | - Portuguese |
| | - Chinese (simplified) |
| |
|
| | Besides these languages, basic support can be expected for additional 91 languages that were used during the pretraining of the base model (see Appendix A of XLM-R paper). |
| |
|
| | ## Scores |
| |
|
| | | Metric | Value | |
| | |:-----------------------|------:| |
| | | Relevance (Recall@100) | 0.613 | |
| |
|
| | Note that the relevance score is computed as an average over 14 retrieval datasets (see |
| | [details below](#evaluation-metrics)). |
| |
|
| | ## Inference Times |
| |
|
| | | GPU | Quantization type | Batch size 1 | Batch size 32 | |
| | |:------------------------------------------|:------------------|---------------:|---------------:| |
| | | NVIDIA A10 | FP16 | 1 ms | 5 ms | |
| | | NVIDIA A10 | FP32 | 2 ms | 18 ms | |
| | | NVIDIA T4 | FP16 | 1 ms | 12 ms | |
| | | NVIDIA T4 | FP32 | 3 ms | 52 ms | |
| | | NVIDIA L4 | FP16 | 2 ms | 5 ms | |
| | | NVIDIA L4 | FP32 | 4 ms | 24 ms | |
| |
|
| | ## GPU Memory usage |
| |
|
| | | Quantization type | Memory | |
| | |:-------------------------------------------------|-----------:| |
| | | FP16 | 550 MiB | |
| | | FP32 | 1050 MiB | |
| |
|
| | Note that GPU memory usage only includes how much GPU memory the actual model consumes on an NVIDIA T4 GPU with a batch |
| | size of 32. It does not include the fix amount of memory that is consumed by the ONNX Runtime upon initialization which |
| | can be around 0.5 to 1 GiB depending on the used GPU. |
| |
|
| | ## Requirements |
| |
|
| | - Minimal Sinequa version: 11.10.0 |
| | - [CUDA compute capability](https://developer.nvidia.com/cuda-gpus): above 7.5 |
| |
|
| | ## Model Details |
| |
|
| | ### Overview |
| |
|
| | - Number of parameters: 107 million |
| | - Base language |
| | model: [mMiniLMv2-L6-H384-distilled-from-XLMR-Large](https://huggingface.co/nreimers/mMiniLMv2-L6-H384-distilled-from-XLMR-Large) ([Paper](https://arxiv.org/abs/2012.15828), [GitHub](https://github.com/microsoft/unilm/tree/master/minilm)) |
| | - Insensitive to casing and accents |
| | - Output dimensions: 256 (reduced with an additional dense layer) |
| | - Training procedure: Query-passage-negative triplets for datasets that have mined hard negative data, Query-passage |
| | pairs for the rest. Number of negatives is augmented with in-batch negative strategy |
| |
|
| | ### Training Data |
| |
|
| | The model have been trained using all datasets that are cited in the [all-MiniLM-L6-v2](https://huggingface.co/sentence-transformers/all-MiniLM-L6-v2) model. In addition to that, this model has been trained on the datasets cited in [this paper](https://arxiv.org/pdf/2108.13897.pdf) on the 9 aforementioned languages. |
| |
|
| | ### Evaluation Metrics |
| |
|
| | To determine the relevance score, we averaged the results that we obtained when evaluating on the datasets of the |
| | [BEIR benchmark](https://github.com/beir-cellar/beir). Note that all these datasets are in English. |
| |
|
| | | Dataset | Recall@100 | |
| | |:------------------|-----------:| |
| | | Average | 0.613 | |
| | | | | |
| | | Arguana | 0.957 | |
| | | CLIMATE-FEVER | 0.468 | |
| | | DBPedia Entity | 0.377 | |
| | | FEVER | 0.820 | |
| | | FiQA-2018 | 0.639 | |
| | | HotpotQA | 0.560 | |
| | | MS MARCO | 0.845 | |
| | | NFCorpus | 0.287 | |
| | | NQ | 0.756 | |
| | | Quora | 0.992 | |
| | | SCIDOCS | 0.456 | |
| | | SciFact | 0.906 | |
| | | TREC-COVID | 0.100 | |
| | | Webis-Touche-2020 | 0.413 | |
| |
|
| | We evaluated the model on the datasets of the [MIRACL benchmark](https://github.com/project-miracl/miracl) to test its multilingual capacities. Note that not all training languages are part of the benchmark, so we only report the metrics for the existing languages. |
| |
|
| | | Language | Recall@100 | |
| | |:----------------------|-----------:| |
| | | French | 0.650 | |
| | | German | 0.528 | |
| | | Spanish | 0.602 | |
| | | Japanese | 0.614 | |
| | | Chinese (simplified) | 0.680 | |
| |
|
