NbAiLab/nb-embed-edu-scorer

Borealis EDU scorer

An ordinal document-quality scorer for educational usefulness in Nordic-language web documents. It fine-tunes the encoder from NbAiLab/borealis-embed-212m and predicts an EDU score from 0 to 5, where higher is more educationally useful.

This repository contains the selected model from a three-run full-training confirmation. It is a custom model: the encoder is stored in encoder/ and the ordinal classification head is stored in ordinal_head.pt.

Training data

The model was trained on the eight language configurations of NbAiLab/nb-fineweb2-edu-sample-v2-rated, pinned at revision af20ece2464419b70249ae4d0e934a9924473f81:

• dan, fao, fin, isl, nno, nob, sami, and swe
• The dataset's mixed-sample configuration was deliberately excluded.
• Documents are truncated to at most 4,096 Borealis tokenizer tokens.

The base encoder revision is 2ae20a7ca72bbfaf526d9b1b371c6b326ccfc7f2. The model mean-pools encoder token states and applies five cumulative ordinal thresholds to score the six ordered labels 0–5.

Validation

Validation uses each language configuration's validation split (146,638 documents total). nob and nno receive weight 3 each; every other language receives weight 1. The selection metric is the mean of weighted normalized MAE and weighted quadratic-weighted kappa (QWK).

Metric	Value
Weighted MAE	0.62663
Weighted normalized MAE	0.87467
Weighted QWK	0.74604
Joint selection score	0.81036

Language	MAE	QWK
Danish (dan)	0.60926	0.79525
Faroese (fao)	0.64115	0.70458
Finnish (fin)	0.65290	0.72897
Icelandic (isl)	0.64745	0.71382
Norwegian Nynorsk (nno)	0.62981	0.76401
Norwegian Bokmal (nob)	0.57526	0.76587
Sami (sami)	0.74969	0.64146
Swedish (swe)	0.60389	0.77880

Use

Download the repository and use the included standalone JSONL scorer. Each input line must contain a text field.

git clone https://huggingface.co/NbAiLab/nb-embed-edu-scorer
cd nb-embed-edu-scorer
pip install torch transformers
python inference.py < input.jsonl > scored.jsonl

Each output row includes a continuous edu_score in the 0–5 range and a heuristic confidence measure. Use the score as a ranking or filtering signal, not as a definitive judgment of educational quality.

Limitations and responsible use

The labels and model reflect automatic EDU-quality ratings and may encode language, source, and annotation biases. Performance is weakest for the smaller Sami validation set. Evaluate on the intended domain before deployment, and do not use the score as the sole basis for high-impact decisions.

License and attribution

This checkpoint is a fine-tune of Borealis Embed. Users must comply with the base model's license and terms; see NbAiLab/borealis-embed-212m for the authoritative license information. The training dataset is subject to its own terms.