chronologic-date-deberta

A DeBERTa-v3-large model that predicts the publication date of an English passage as a probability distribution over decades from the 1680s to the 2030s. It is one of the two style instruments in the Chronologic-EN benchmark. The companion model is chronologic-authenticity-deberta.

What the output means. The model outputs 36 logits, one per ten-year bin from 1680 to 2040. The softmax over them is a distribution over decades: label 1880s is the probability that the passage was published between 1880 and 1889. The model's year estimate is the probability-weighted mean of the bin midpoints (1685, 1695, …, 2035). The width of the distribution carries information too, which is why the benchmark keeps the whole distribution instead of the top label.

Intended use and out-of-scope use

This model is a model-level benchmark instrument. It measures how far a language model's answers, taken together, sit from matched authentic prose of the target period. It is not a tool for dating individual documents. Its error on a single passage is typically two decades or more (MAE about 23 years in the core period), and its target is publication date, not composition date.

Raw outputs are not Chronologic scores. The benchmark never reads the predicted date as a verdict. It ranks the model's error on an answer against the errors it makes on authentic prose. That step needs the code in the GitHub repository and the scored reference corpus (the CHRONOLOGIC_DATA directory). A number you compute from this model alone can't be compared with published Period Fidelity scores.

How the benchmark uses it

For each answer the benchmark takes the signed residual (predicted mean year minus the question's target year). It converts that residual to a percentile among the residuals of authentic passages from within ±10 years of the target date, in the same length bin, scored by the same frozen model. So the model's own biases cancel, and a model writing indistinguishably from period prose would produce uniform percentiles. Period Fidelity (0–100) is the Wasserstein-1 distance of those percentiles from uniform, rescaled against the distance that resampled authentic passages themselves produce (a pseudo-model baseline). So 100 means indistinguishable from genuine prose of the target period.

The benchmark also applies temperature scaling (T = 1.0551, fit on held-out authentic text) and abstains on sentence fragments (see Usage).

Training data

• Roster: 4,878 volumes, 150 per decade across the 1831–1930 core and 100 per decade elsewhere. Sources:
  • IDI (Harvard's Institutional Data Initiative), used first
  • ECCO before 1800, and OAPEN after 1925, where IDI falls short
  • the Chicago Novel Corpus, capped at 30% of twentieth-century fiction
  • COHA, uncapped, for genre diversity
• A MARC date-reliability filter drops volumes with unreliable dates (continuing resources, questionable dates), leaving 4,325 volumes. Hand corrections to publication dates are folded in from catalog records only.
• Passages: 66,000, exactly 2,000 per decade from the 1700s to the 2020s. Their length distribution is matched to real benchmark answers.
• Splits: 52,651 train, 6,434 validation and 6,915 test, grouped by volume and by author.
• Decades outside 1831–1930 are there so the model can say "this reads like 1770" or "this reads like 1990". The benchmark has to detect drift in both directions. We don't claim accuracy there.
• The benchmark's reserved source volumes are excluded from training.
• The training text is not redistributed, because some sources (COHA, the Chicago Novel Corpus, OAPEN) are in copyright.

Training procedure

• microsoft/deberta-v3-large with num_labels=36, a softmax over ten-year bins from 1680 to 2040. The bin grid is stored on the config as date_grid_lo (1680), date_grid_hi (2040), date_bin_width (10) and date_sigma (15).
• Soft-target cross-entropy. A passage published in 1883 is trained against a Gaussian bump centred on 1883 with σ = 15 years, not a one-hot label. So missing by one bin costs far less than missing by ten. This is an "ordinal softmax" in that loose sense only. It is not a regressor, and not a cumulative-link or CORAL ordinal model.
• Max length 256 tokens.
• The checkpoint was kept at epoch 2, the validation-loss minimum (loss 2.545).

Evaluation

CRPS (continuous ranked probability score) is the primary metric. It scores the whole predicted distribution, is expressed in years, and reduces to absolute error for a point prediction. All figures are at T = 1.

Evaluation	n	CRPS (yr)	MAE (yr)	Other
Test split, 1830–1930 core	2,032	16.69	23.09	cov@50 0.63, cov@90 0.95, mean signed residual −4.3 yr
Test split, all decades	6,915	20.13	26.94	cov@90 0.90, R² 0.795
Reserved benchmark volumes	606	17.66	24.20	NLL 2.50

"cov@50" and "cov@90" are the share of passages whose true date falls inside the central 50% and 90% intervals of the predicted distribution.

Comparison with the lexical predecessor (tf-idf plus multinomial logistic regression, same soft-target objective):

• On identical splits, CRPS is 20.13 against 44.46.
• On the reserved benchmark volumes, MAE / CRPS / NLL are 24.20 / 17.66 / 2.50 against 28.10 / 29.51 / 3.44.

Limitations and biases

• Early eighteenth century. The 1700s and 1710s fit badly (MAE 89 and 106 years), from a pool of only seventeen usable volumes.
• Residual bias. Mean signed residuals across the 1830s–1920s run from +4 to −10 years, except the 1900s at −23. The lexical predecessor's pull toward the centre of the range (+28 years at the 1830s to −43 at the 1930s) is mostly gone, but not entirely.
• Publication date, not composition date. Reprints, posthumous editions and late-published manuscripts are labelled by publication.
• Fragments. The model is trained on complete sentences. It is out of distribution on sub-sentential fragments, and the benchmark abstains on them. A raw model call does not abstain.
• Typography. The training text was normalized. Unnormalized input (curly quotes, em-dashes, line breaks) is out of distribution. Normalize first (see Usage).

A note on Goodhart's law

Publishing the judges makes them possible to train against. Any model trained, tuned, RL'd or selected against these instruments, or against models derived from them, no longer has valid Chronologic style scores. If you report Chronologic style scores for such a model, say that it was optimized against the judges.

Usage

(a) Pipeline. Quick inspection only. It skips normalization, temperature and the fragment check.

from transformers import pipeline
clf = pipeline("text-classification", model="chronologic/chronologic-date-deberta", top_k=5)
clf("The railway had lately come to the town, and with it a new class of traveller.")
# [[{'label': '1860s', 'score': ...}, {'label': '1870s', 'score': ...}, ...]]

(b) Explicit code. Normalize with normalize_typography from stylejudge/normalize.py in the GitHub repository (standard library only), skip fragments, apply the benchmark's temperature, and take the mean year:

import sys, torch
from transformers import AutoTokenizer, AutoModelForSequenceClassification

sys.path.insert(0, "Chronologic-EN/stylejudge")   # path to your clone
from normalize import normalize_typography
from measure_length_distribution import is_fragment   # needs nltk

repo = "chronologic/chronologic-date-deberta"
tok = AutoTokenizer.from_pretrained(repo)
model = AutoModelForSequenceClassification.from_pretrained(repo).eval()
cfg = model.config
midpoints = torch.arange(cfg.date_grid_lo, cfg.date_grid_hi, cfg.date_bin_width) + cfg.date_bin_width / 2
T = 1.0551   # the benchmark's temperature (t_nll in stylejudge/e3_temperature_fit.json)

texts = [normalize_typography(t) for t in ["...your passages..."]]
texts = [t for t in texts if not is_fragment(t)]   # the benchmark abstains on fragments
enc = tok(texts, padding=True, truncation=True, max_length=256, return_tensors="pt")
with torch.no_grad():
    probs = torch.softmax(model(**enc).logits / T, dim=-1)
mean_year = probs @ midpoints

is_fragment flags a passage whose first alphabetic character is lowercase and that is at most one sentence long.

(c) Benchmark scoring. For real Chronologic scores, use stylejudge/score_style.py (or the full modelasjudge/run_pipeline.py) from the GitHub repository. These apply the percentile layer against the reference corpus.

Citation

Preprint forthcoming.

@misc{underwood2026chronologicmeasuringlanguagemodels,
      title={Chronologic: Measuring Language Models' Ability to Represent the Past}, 
      author={Ted Underwood and Ziliang Qiu and Sarah Griebel and Laura K. Nelson and Edwin Roland and Wenyi Shang and Matthew Wilkens},
      year={2026},
      eprint={2609.23178},
      archivePrefix={arXiv},
      primaryClass={cs.CL},
      url={https://arxiv.org/abs/2609.23178}, 
}

Contact

Ted Underwood, University of Illinois Urbana-Champaign, tunder@illinois.edu.