Jsevisal/CrossEncoder-ModernBERT-base-qnli

Model Card for Model ID

ModernBert version of CrossEncoders QNLI models. Used to determine if a passage contains the answer to a question. In this case the model has been train on GLUE.

Model Details

Model Description

This model is a fine-tuned version of answerdotai/ModernBERT-base on GLUE QNLI dataset.

It achieves the following results on the evaluation set:

• Accuracy Score: 0.9319
• F1 Score: 0.9322

Usage

Pre-trained models can be used like this:

from sentence_transformers import CrossEncoder
model = CrossEncoder('Jsevisal/CrossEncoder-ModernBERT-base-qnli')
scores = model.predict([('Query1', 'Paragraph1'), ('Query2', 'Paragraph2')])

#e.g.
scores = model.predict([('How many people live in Berlin?', 'Berlin had a population of 3,520,031 registered inhabitants in an area of 891.82 square kilometers.'), ('What is the size of New York?', 'New York City is famous for the Metropolitan Museum of Art.')])

Usage with Transformers AutoModel

You can use the model also directly with Transformers library (without SentenceTransformers library):

from transformers import AutoTokenizer, AutoModelForSequenceClassification
import torch

model = AutoModelForSequenceClassification.from_pretrained('Jsevisal/CrossEncoder-ModernBERT-base-qnli')
tokenizer = AutoTokenizer.from_pretrained('Jsevisal/CrossEncoder-ModernBERT-base-qnli')

features = tokenizer(['How many people live in Berlin?', 'What is the size of New York?'], ['Berlin had a population of 3,520,031 registered inhabitants in an area of 891.82 square kilometers.', 'New York City is famous for the Metropolitan Museum of Art.'],  padding=True, truncation=True, return_tensors="pt")

model.eval()
with torch.no_grad():
    scores = torch.nn.functional.sigmoid(model(**features).logits)
    print(scores)

Training procedure

Training hyperparameters

The following hyperparameters were used during training:

• learning_rate: 8e-05
• train_batch_size: 32
• eval_batch_size: 32
• seed: 42
• optimizer: Use OptimizerNames.ADAMW_TORCH with betas=(0.9,0.98) and epsilon=1e-06 and optimizer_args=No additional optimizer arguments
• lr_scheduler_type: linear
• num_epochs: 2

Framework versions

• Transformers 4.49.0.dev0
• Pytorch 2.5.1+cu124
• Datasets 3.2.0
• Tokenizers 0.21.0