README.md

---
license: mit
base_model: microsoft/deberta-v3-large
library_name: transformers
language: en
pipeline_tag: text-classification
tags:
  - digital-humanities
  - historical-text
  - ai-text-detection
  - deberta-v3
  - chronologic
metrics:
  - accuracy
  - f1
  - roc_auc
widget:
  - text: "One evening of late summer, before the nineteenth century had reached one-third of its span, a young man and woman, the latter carrying a child, were approaching the large village of Weydon-Priors, in Upper Wessex, on foot."
    example_title: "Authentic (Hardy, 1886)"
  - text: "The fog clung to the cobblestones like a shroud, and I could not shake the feeling that the very gaslights were watching my every step as I hurried toward the old manor."
    example_title: "Pastiche"
model-index:
  - name: chronologic-authenticity-deberta
    results:
      - task:
          type: text-classification
          name: Authentic period prose vs. LLM imitation
        dataset:
          type: chronologic-authenticity-test
          name: Held-out test split (grouped by volume and author)
        metrics:
          - type: accuracy
            value: 0.881
          - type: f1
            value: 0.875
          - type: roc_auc
            value: 0.941
      - task:
          type: text-classification
          name: Authentic period prose vs. LLM imitation
        dataset:
          type: chronologic-authenticity-holdout
          name: Cross-generator holdout (5 unseen models, length-matched)
        metrics:
          - type: accuracy
            value: 0.901
          - type: f1
            value: 0.901
          - type: roc_auc
            value: 0.964
---

# chronologic-authenticity-deberta

A DeBERTa-v3-large classifier that separates authentic English prose from 1831–1930
from language-model imitation of it. It is one of the two style instruments in the
[Chronologic-EN](https://github.com/Historical-AI-Lab/Chronologic-EN) benchmark.
The companion model is
[chronologic-date-deberta](https://huggingface.co/chronologic/chronologic-date-deberta).

**What the output means.** The model has a single logit. `sigmoid(logit)` is
P(synthetic), the probability that the passage was written by a language model. A low
score means the passage reads as genuine period prose. A high score means it reads as
machine imitation. The inference widget shows this as the `synthetic` label.

## Intended use and out-of-scope use

This model is a **model-level benchmark instrument**. It measures how far a language
model's answers, taken together, sit from matched authentic prose. It is **not** a
passage-level AI-text detector. Don't use it to decide whether a single document, essay
or submission was machine-written. Single-passage scores are noisy. It flags about 9% of
genuine period prose as synthetic in some collections. And single-passage scoring is
exactly where fine-tuned models slip through (see Limitations).

**Raw outputs are not Chronologic scores.** The benchmark never reads this probability
as a verdict. It turns the probability into a percentile against a reference corpus of
authentic passages, and that step needs the code in the GitHub repository and the scored
reference corpus (the `CHRONOLOGIC_DATA` directory). A number you compute from this
model alone can't be compared with published Authenticity Fidelity scores.

## How the benchmark uses it

Each answer's detector output is converted to a percentile among authentic passages
from within ±10 years of the question's target date, in the same length bin, scored by
the same frozen model. A model writing indistinguishably from period prose would
produce uniform percentiles. Authenticity Fidelity (0–100) is the mean percentile,
rescaled against the value that resampled authentic passages themselves produce
(a pseudo-model baseline). So 100 means indistinguishable from genuine prose of the
target period.

## Training data

- **Positives:** passages of English prose published 1831–1930, drawn from the same
  volume roster as [chronologic-date-deberta](https://huggingface.co/chronologic/chronologic-date-deberta),
  whose card describes the sources (IDI, COHA, the Chicago Novel Corpus and others).
- **Negatives** are elicited *from the positives themselves*, so topic and content stay
  roughly constant and style carries the signal. There are five elicitation modalities:
  - 40% infill of a gap between real bookending sentences
  - 20% continuation of a real passage
  - 10% paraphrase
  - 20% generation constrained by a title, a date and a one-sentence summary
  - 10% few-shot imitation
- The pool holds 29,681 newly elicited generations. It also reuses earlier infill
  generations and adds generations from Talkie, a model trained only on pre-1931 text,
  run locally. Generators come from a deliberately diverse set of model families,
  including base models.
- **Normalization.** Both classes go through the same typography folding (quotes,
  dashes, whitespace), after side-specific cleaning: OCR repair on the historical side,
  and Markdown/JSON stripping on the generated side. Without this the classifier learns
  typography rather than style.
- **Length-matched.** A classifier given passage length alone scores 0.4875, which is
  chance.
- **Size:** 58,006 passages, 29,003 per class. The splits are 46,644 train, 5,462
  validation and 5,900 test, grouped by source volume and by author.
- **Cross-generator holdout.** Five models were held out of all three splits before
  training: `gemini-3.1-pro-preview`, `gemma-4-31b-it`, `cohere/command-a`,
  `amazon/nova-pro-v1` and `openai/gpt-5.6-sol`. Of their 1,854 frozen generations,
  1,766 survive length-matching and are paired with 1,766 authentic passages (n = 3,532).
  Three of the five come from families absent from training entirely. So the transfer
  claim is "held-out models, mostly held-out families".
- The benchmark's reserved source volumes are excluded from training.
- The training text is not redistributed, because some sources are still in copyright.

## Training procedure

- `microsoft/deberta-v3-large` with `num_labels=1` and `BCEWithLogitsLoss`. It is a
  one-logit binary classifier, not a regressor.
- 3 epochs, batch size 32, learning rate 2e-5, weight decay 0.01, warmup ratio 0.1,
  max length 256 tokens.
- The checkpoint was kept at epoch 2 by validation accuracy (val accuracy 0.876, AUROC
  0.936).

## Evaluation

| Evaluation | n | Accuracy | F1 | AUROC |
|---|---|---|---|---|
| Held-out test split | 5,900 | 0.881 | 0.875 | 0.941 |
| Cross-generator holdout, length-matched | 3,532 | 0.901 | 0.901 | 0.964 |

Accuracy on the test split's negatives, by elicitation modality:

| Modality | n | Accuracy |
|---|---|---|
| Constrained generation | 777 | 0.988 |
| Infill | 1,140 | 0.890 |
| Few-shot | 210 | 0.871 |
| Paraphrase | 304 | 0.694 |
| **Continuation** | 486 | **0.586** |

Accuracy on the test split's negatives, for selected generator families:

| Family | n | Accuracy |
|---|---|---|
| openai | 320 | 0.741 |
| meta-llama | 203 | 0.660 |
| **talkie** | 97 | **0.227** |
| **nousresearch** | 35 | **0.114** |

A check that the detector is not reading vocabulary coverage instead of style: the
within-class rank correlation between out-of-vocabulary rate and P(synthetic) is near
zero (ρ = −0.06 and −0.12).

## Limitations and biases

- **Continuation and paraphrase are hard.** The detector is much better at recognising
  text written to a prompt than text that continues or rewrites a real passage. The
  headline accuracy reflects the modality mix of the training negatives.
- **Models trained on period text slip through.** Talkie, which is trained only on
  pre-1931 text, is caught 23% of the time, and NousResearch fine-tunes 11% of the time
  (n = 35). Per-passage detection is exactly where such models evade it.
- **False positives on genuine prose.** About 9% of authentic passages are flagged as
  synthetic in the two large IDI collections, and 2–6% in the smaller COHA and fiction
  samples.
- **Scope.** The model is trained on English prose from 1831–1930 and passages of
  benchmark-answer length. Outside that period, language or length range its outputs
  have not been validated.
- **Typography.** The training text was normalized. Unnormalized input (curly quotes,
  em-dashes, line breaks) is out of distribution. Normalize first (see Usage).

## A note on Goodhart's law

Publishing the judges makes them possible to train against. Any model trained, tuned,
RL'd or selected against these instruments, or against models derived from them, no
longer has valid Chronologic style scores. If you report Chronologic style scores for
such a model, say that it was optimized against the judges.

## Usage

**(a) Pipeline.** Quick inspection only; this skips the normalization the model was
trained with.

```python
from transformers import pipeline
clf = pipeline("text-classification", model="chronologic/chronologic-authenticity-deberta")
clf("It was a dark and stormy night; the rain fell in torrents.")
# [{'label': 'synthetic', 'score': ...}]   score = P(synthetic)
```

**(b) Explicit code.** Normalize first, using `normalize_typography` from
`stylejudge/normalize.py` in the [GitHub repository](https://github.com/Historical-AI-Lab/Chronologic-EN)
(standard library only):

```python
import sys, torch
from transformers import AutoTokenizer, AutoModelForSequenceClassification

sys.path.insert(0, "Chronologic-EN/stylejudge")   # path to your clone
from normalize import normalize_typography

repo = "chronologic/chronologic-authenticity-deberta"
tok = AutoTokenizer.from_pretrained(repo)
model = AutoModelForSequenceClassification.from_pretrained(repo).eval()

texts = [normalize_typography(t) for t in ["...your passages..."]]
enc = tok(texts, padding=True, truncation=True, max_length=256, return_tensors="pt")
with torch.no_grad():
    p_synthetic = torch.sigmoid(model(**enc).logits.squeeze(-1))
```

**(c) Benchmark scoring.** For real Chronologic scores, use `stylejudge/score_style.py`
(or the full `modelasjudge/run_pipeline.py`) from the GitHub repository. These apply the
percentile layer against the reference corpus.

## Citation

Preprint forthcoming.

```bibtex
@misc{underwood2026chronologicmeasuringlanguagemodels,
      title={Chronologic: Measuring Language Models' Ability to Represent the Past}, 
      author={Ted Underwood and Ziliang Qiu and Sarah Griebel and Laura K. Nelson and Edwin Roland and Wenyi Shang and Matthew Wilkens},
      year={2026},
      eprint={2609.23178},
      archivePrefix={arXiv},
      primaryClass={cs.CL},
      url={https://arxiv.org/abs/2609.23178}, 
}
```

## Contact

Ted Underwood, University of Illinois Urbana-Champaign, tunder@illinois.edu.