chronologic-authenticity-deberta

A DeBERTa-v3-large classifier that separates authentic English prose from 1831–1930 from language-model imitation of it. It is one of the two style instruments in the Chronologic-EN benchmark. The companion model is chronologic-date-deberta.

What the output means. The model has a single logit. sigmoid(logit) is P(synthetic), the probability that the passage was written by a language model. A low score means the passage reads as genuine period prose. A high score means it reads as machine imitation. The inference widget shows this as the synthetic label.

Intended use and out-of-scope use

This model is a model-level benchmark instrument. It measures how far a language model's answers, taken together, sit from matched authentic prose. It is not a passage-level AI-text detector. Don't use it to decide whether a single document, essay or submission was machine-written. Single-passage scores are noisy. It flags about 9% of genuine period prose as synthetic in some collections. And single-passage scoring is exactly where fine-tuned models slip through (see Limitations).

Raw outputs are not Chronologic scores. The benchmark never reads this probability as a verdict. It turns the probability into a percentile against a reference corpus of authentic passages, and that step needs the code in the GitHub repository and the scored reference corpus (the CHRONOLOGIC_DATA directory). A number you compute from this model alone can't be compared with published Authenticity Fidelity scores.

How the benchmark uses it

Each answer's detector output is converted to a percentile among authentic passages from within ±10 years of the question's target date, in the same length bin, scored by the same frozen model. A model writing indistinguishably from period prose would produce uniform percentiles. Authenticity Fidelity (0–100) is the mean percentile, rescaled against the value that resampled authentic passages themselves produce (a pseudo-model baseline). So 100 means indistinguishable from genuine prose of the target period.

Training data

• Positives: passages of English prose published 1831–1930, drawn from the same volume roster as chronologic-date-deberta, whose card describes the sources (IDI, COHA, the Chicago Novel Corpus and others).
• Negatives are elicited from the positives themselves, so topic and content stay roughly constant and style carries the signal. There are five elicitation modalities:
  • 40% infill of a gap between real bookending sentences
  • 20% continuation of a real passage
  • 10% paraphrase
  • 20% generation constrained by a title, a date and a one-sentence summary
  • 10% few-shot imitation
• The pool holds 29,681 newly elicited generations. It also reuses earlier infill generations and adds generations from Talkie, a model trained only on pre-1931 text, run locally. Generators come from a deliberately diverse set of model families, including base models.
• Normalization. Both classes go through the same typography folding (quotes, dashes, whitespace), after side-specific cleaning: OCR repair on the historical side, and Markdown/JSON stripping on the generated side. Without this the classifier learns typography rather than style.
• Length-matched. A classifier given passage length alone scores 0.4875, which is chance.
• Size: 58,006 passages, 29,003 per class. The splits are 46,644 train, 5,462 validation and 5,900 test, grouped by source volume and by author.
• Cross-generator holdout. Five models were held out of all three splits before training: gemini-3.1-pro-preview, gemma-4-31b-it, cohere/command-a, amazon/nova-pro-v1 and openai/gpt-5.6-sol. Of their 1,854 frozen generations, 1,766 survive length-matching and are paired with 1,766 authentic passages (n = 3,532). Three of the five come from families absent from training entirely. So the transfer claim is "held-out models, mostly held-out families".
• The benchmark's reserved source volumes are excluded from training.
• The training text is not redistributed, because some sources are still in copyright.

Training procedure

• microsoft/deberta-v3-large with num_labels=1 and BCEWithLogitsLoss. It is a one-logit binary classifier, not a regressor.
• 3 epochs, batch size 32, learning rate 2e-5, weight decay 0.01, warmup ratio 0.1, max length 256 tokens.
• The checkpoint was kept at epoch 2 by validation accuracy (val accuracy 0.876, AUROC 0.936).

Evaluation

Evaluation	n	Accuracy	F1	AUROC
Held-out test split	5,900	0.881	0.875	0.941
Cross-generator holdout, length-matched	3,532	0.901	0.901	0.964

Accuracy on the test split's negatives, by elicitation modality:

Modality	n	Accuracy
Constrained generation	777	0.988
Infill	1,140	0.890
Few-shot	210	0.871
Paraphrase	304	0.694
Continuation	486	0.586

Accuracy on the test split's negatives, for selected generator families:

Family	n	Accuracy
openai	320	0.741
meta-llama	203	0.660
talkie	97	0.227
nousresearch	35	0.114

A check that the detector is not reading vocabulary coverage instead of style: the within-class rank correlation between out-of-vocabulary rate and P(synthetic) is near zero (ρ = −0.06 and −0.12).

Limitations and biases

• Continuation and paraphrase are hard. The detector is much better at recognising text written to a prompt than text that continues or rewrites a real passage. The headline accuracy reflects the modality mix of the training negatives.
• Models trained on period text slip through. Talkie, which is trained only on pre-1931 text, is caught 23% of the time, and NousResearch fine-tunes 11% of the time (n = 35). Per-passage detection is exactly where such models evade it.
• False positives on genuine prose. About 9% of authentic passages are flagged as synthetic in the two large IDI collections, and 2–6% in the smaller COHA and fiction samples.
• Scope. The model is trained on English prose from 1831–1930 and passages of benchmark-answer length. Outside that period, language or length range its outputs have not been validated.
• Typography. The training text was normalized. Unnormalized input (curly quotes, em-dashes, line breaks) is out of distribution. Normalize first (see Usage).

A note on Goodhart's law

Publishing the judges makes them possible to train against. Any model trained, tuned, RL'd or selected against these instruments, or against models derived from them, no longer has valid Chronologic style scores. If you report Chronologic style scores for such a model, say that it was optimized against the judges.

Usage

(a) Pipeline. Quick inspection only; this skips the normalization the model was trained with.

from transformers import pipeline
clf = pipeline("text-classification", model="chronologic/chronologic-authenticity-deberta")
clf("It was a dark and stormy night; the rain fell in torrents.")
# [{'label': 'synthetic', 'score': ...}]   score = P(synthetic)

(b) Explicit code. Normalize first, using normalize_typography from stylejudge/normalize.py in the GitHub repository (standard library only):

import sys, torch
from transformers import AutoTokenizer, AutoModelForSequenceClassification

sys.path.insert(0, "Chronologic-EN/stylejudge")   # path to your clone
from normalize import normalize_typography

repo = "chronologic/chronologic-authenticity-deberta"
tok = AutoTokenizer.from_pretrained(repo)
model = AutoModelForSequenceClassification.from_pretrained(repo).eval()

texts = [normalize_typography(t) for t in ["...your passages..."]]
enc = tok(texts, padding=True, truncation=True, max_length=256, return_tensors="pt")
with torch.no_grad():
    p_synthetic = torch.sigmoid(model(**enc).logits.squeeze(-1))

(c) Benchmark scoring. For real Chronologic scores, use stylejudge/score_style.py (or the full modelasjudge/run_pipeline.py) from the GitHub repository. These apply the percentile layer against the reference corpus.

Citation

Preprint forthcoming.

@misc{underwood2026chronologicmeasuringlanguagemodels,
      title={Chronologic: Measuring Language Models' Ability to Represent the Past}, 
      author={Ted Underwood and Ziliang Qiu and Sarah Griebel and Laura K. Nelson and Edwin Roland and Wenyi Shang and Matthew Wilkens},
      year={2026},
      eprint={2609.23178},
      archivePrefix={arXiv},
      primaryClass={cs.CL},
      url={https://arxiv.org/abs/2609.23178}, 
}

Contact

Ted Underwood, University of Illinois Urbana-Champaign, tunder@illinois.edu.