hunter-lab/sentence-level-ignorance-classifier

Model Card for Model ID

This model is a binary classifier that determines whether a sentence is a statement of ignorance or not.

Model Details

Model Description

• This model consists of a linear classification head on top of BioMedBERT, which is a BERT-based encoder (110 million parameters, 12 layers, 768 hidden).

• All layers besides 10, 11, the pooler, and classificaton head were frozen to reduce overfitting on the small training set.

• Developed by: Nathan Gelfand, Darya Shlyk

• Funded by: Larry Hunter

• Model type: Supervised fine-tune of a transformer-encoder for text classification

• Language(s) (NLP): English

• License: MIT

• Finetuned from model: microsoft/BiomedNLP-BiomedBERT-base-uncased-abstract

Model Sources [optional]

• Repository: [More Information Needed]

Uses

Direct Use

A statement of ignorance, or knowledge gap statement, is a piece of text indicating a lack of knoweldge on some subject. In the field of biomedical research, knowledge gap statements often indicate unanswered questions, which can be useful for researchers looking for new topics of study. We built this sentence-level binary classifier so that researchers could parse through biomedical papers to identify research directives.

[More Information Needed]

Out-of-Scope Use

[More Information Needed] -->

Bias, Risks, and Limitations

This model was trained on a dataset of 937 annotated statements from articles about monogenic epilepsies. Due to the small dataset size, it is possible that the classifier will perform better on statements related to monogenic epilepsies.

The annotated statements were based on the guidelines below. Therefore, the model's classifications will also be restricted to these guidelines. annotation_guidelines.html

Recommendations

Users (both direct and downstream) should be made aware of the risks, biases and limitations of the model. More information needed for further recommendations.

How to Get Started with the Model

Use the code below to get started with the model.

!pip install transformers 
from transformers import pipeline

classifier = pipeline("text-classification",model="hunter-lab/sentence-level-ignorance-classifier")

sentence = "Consequently, there is a pressing need to enhance basic mechanistic and clinical studies, with a focus on individual differences, to advance the exploration of BCAAs in treating neurological disorders and to provide more robust evidence supporting their clinical applications."

result = classifier(sentence)

print(result)

Training Details

Training Data

Our training data consists of 937 statements from articles extracted from PubMed that are related to monogenic epilepsies (fill out later). Each statement was annotated as "ignorance" or "not ignorance" based on the guidelines in annotation_guidelines.html.
There are 745 training statements, 104 validation statements, and 88 testing statements.

Training Procedure

We split the data roughly 80-10-10 into train, test, and validation sets, grouping statements by article PMID using StratifiedGroupKFold to avoid leakage.

Due to the dominance of negative statements in the dataset we made a wrapper of the Trainer class called WeightedTrainer. This wrapper includes a custom compute_loss function that performs cross-entropy loss while taking into account the modified weight of each class, which is calculated as (total labels) / (number of classes x count of this class label).

Otherwise, the procedure is fairly standard: tokenize the datasets, feed them into the trainer, train, and evaluate.

Training Hyperparameters

training_args = TrainingArguments(
    output_dir=str(MODEL_DIR / "checkpoints"),
    num_train_epochs=20,
    per_device_train_batch_size=8,
    per_device_eval_batch_size=8,
    learning_rate=2e-5,
    weight_decay=0.01,
    warmup_ratio=0.1,
    fp16=True,
    seed=42,
    eval_strategy="epoch",
    save_strategy="epoch",
    save_total_limit=2,
    load_best_model_at_end=True,
    metric_for_best_model="f1",
)

[More Information Needed] -->

Evaluation

Testing Data, Factors & Metrics

Testing Data

https://huggingface.co/datasets/hunter-lab/ignorance-classifier-testing-data

[More Information Needed] -->

Metrics

Precision, recall, and macro f1 were used in evaluation, as they are standard for testing the performance of a binary classifier.

[More Information Needed]

Results

{'eval_loss': 0.26312151551246643, 'eval_f1': 0.9351319475158484, 'eval_precision_class_1': 0.9629629629629629, 'eval_recall_class_1': 0.8666666666666667}