scienceverse/bibr-paper-classifier

bibr paper classifier

A multitask classifier that predicts, from a scientific paper's title + abstract:

• OECD field of science, level 1 (oecd_domain) — 6 domains
• OECD field of science, level 2 (oecd_subdomain) — 36 subdomains
• paper type — empirical, review, meta-analysis, case-study, commentary, corrigendum, erratum, retraction

It is the field/type classification component of bibr, a scientific-paper metadata extraction pipeline. In bibr it replaces a per-paper LLM classification call: a shared SPECTER2 encoder with three linear heads runs locally at zero marginal cost and is substantially more accurate than the zero-shot LLM it supersedes.

Architecture

allenai/specter2_base encoder → mean-pooled last hidden state → three linear heads (l1_head, l2_head, paper_type_head). Input text is "{title} [SEP] {abstract}".

Results (held-out test)

Measured on the deployment condition — title + a clean abstract, as provided at inference:

task	macro-F1	micro-F1
OECD L1 (oecd_domain)	0.742	0.768
OECD L2 (oecd_subdomain)	0.461	0.599
paper type	0.936	0.934

paper_type per class: retraction 0.99, erratum 0.98, case-study 0.94, meta-analysis 0.93, commentary 0.92, empirical 0.90, review 0.89. corrigendum is not learnable from the available data (n=2) and is handled by a deterministic title guard in bibr rather than this model.

For comparison, the zero-shot LLM this model replaces scores L1 macro 0.56 (title-only) / 0.35 (title+abstract) against the same labels.

Confidence gating

The heads are softmax-scored; paper_type logits are temperature-scaled (paper_type_temperature in inference_config.json, Guo et al. 2017). Recommended emission policy: emit oecd_domain and paper_type always, and emit oecd_subdomain only above a confidence threshold (else null). Illustrative L2 precision/coverage: thr 0.5 → 66% coverage @ 0.60 precision; thr 0.8 → 36% coverage @ 0.74 precision.

Training data & labels

• OECD labels are derived from the OpenAlex primary_topic taxonomy (CC0), mapped to the OECD Fields of Science hierarchy. They are not LLM-generated. Title/abstract pairs whose abstract was crossed from an unrelated work (a known OpenAlex data artifact) were detected with an embedding-cosine consistency filter and trained title-only to avoid poisoning the input.
• paper_type labels for the rarer classes come from MEDLINE/PubMed PublicationType metadata (public domain), with a smaller set of LLM-adjudicated labels for the ambiguous majority.

Intended use & limitations

Designed for English scientific papers with a title and abstract. OECD field assignment is inherently ambiguous for interdisciplinary work; L2 in particular should be treated as a confidence-gated hint, not ground truth. Not suitable as a sole basis for high-stakes categorization.

License

Released under AGPL-3.0, matching the bibr project. Training-data sources (OpenAlex, PubMed) carry their own open licenses noted above.