Upload continuous-time NewsBERT (full fine-tune, 1 epoch)

README.md

@@ -0,0 +1,104 @@
+---
+base_model: TextMachineProject/NewsBERT_1800-1920
+library_name: transformers
+tags:
+  - bert
+  - masked-language-modeling
+  - historical-nlp
+  - temporal-language-model
+---
+
+# NewsBERT_1800-1920-Temporal
+
+A fine-tuned version of [TextMachineProject/NewsBERT_1800-1920](https://huggingface.co/TextMachineProject/NewsBERT_1800-1920), conditioned on each item's publication year as a continuous input. A single model was trained across the full 1800-1920 span of the original NewsBERT_1800-1920's dataset, with each training example's year injected as a sinusoidal (Fourier) feature added to the token embeddings. Nearby years produce nearby embeddings by construction, so the model can be queried at any year in-range.
+
+## Architecture
+
+- **Base**: `TextMachineProject/NewsBERT_1800-1920` (BERT-base architecture)
+- **Time conditioning**: a small `ContinuousTimeEmbedding` module maps a
+  normalized year to a `hidden_size`-dimensional vector via sinusoidal
+  features at 24 log-spaced frequencies, spanning:
+  - lowest frequency: 1 cycle over the full 120-year corpus span
+  - highest frequency: 1 cycle per 5 years
+- This vector is added to every token's input embedding, then passed
+  through a `LayerNorm` before entering the transformer stack (this re-normalization step
+  was added after an earlier version without it let the time signal's magnitude
+  dominate the token embeddings it was added to, which suppressed learning).
+
+You need the accompanying `continuous_time_embedding.py` in this repo to
+load this model, as it is **not** loadable via a plain
+`AutoModelForMaskedLM.from_pretrained(...)` call, since the time-injection
+logic is outside the standard BERT forward pass.
+
+## Training data
+
+9.28M items (`text`, `year`) from the LwM and HMD collections (see [TextMachineProject/NewsBERT_1800-1920](https://huggingface.co/TextMachineProject/NewsBERT_1800-1920)), spanning 1800-1920, split into
+overlapping 126-token windows (stride 96), i.e. 94.0M training windows.
+
+## Training procedure
+
+- 1 epoch, batch size 256, linear LR decay, 500 warmup steps
+- Learning rate: `2e-5` for the base model's existing (pretrained) layers, 
+and a higher rate of `1e-4` for the ContinuousTimeEmbedding module 
+since that module is new and starts from scratch
+- bf16 mixed precision
+
+### Hardware / environment
+
+### Hardware / environment
+
+This research utilised Queen Mary's Apocrita HPC facility, supported by
+QMUL Research-IT. http://doi.org/10.5281/zenodo.438045
+
+- Single GPU per training job
+- GPU: NVIDIA A100-PCIE-40GB (40GB VRAM)
+- CUDA: 12.6 (as built into the `torch` version used)
+- Python 3.11.7 (GCC 12.2.0 build)
+- `torch` 2.11.0+cu126
+- `transformers` 5.12.1
+
+## Usage
+
+```python
+from continuous_time_embedding import load_continuous_time_model
+import torch.nn.functional as F
+import math
+
+tokenizer, model = load_time_aware_model("npedrazzini/NewsBERT_1800-1920-Temporal")
+device = next(model.parameters()).device
+
+def get_target_word_probability(sentence, target_word, year):
+    target_ids = tokenizer.encode(target_word, add_special_tokens=False)
+    mask_str = " ".join([tokenizer.mask_token] * len(target_ids))
+    masked_sentence = sentence.replace(target_word, mask_str, 1)
+
+    enc = tokenizer(masked_sentence, return_tensors="pt").to(device)
+    mask_positions = (enc["input_ids"][0] == tokenizer.mask_token_id).nonzero(as_tuple=True)[0].tolist()
+
+    embeddings_module = model.get_input_embeddings_module()
+    tok_embeds = embeddings_module(enc["input_ids"])
+    time_vec = model.time_embed(torch.tensor([float(year)]).to(device)).unsqueeze(1)
+    tok_embeds = model.post_inject_norm(tok_embeds + time_vec)
+
+    out = model.model(inputs_embeds=tok_embeds, attention_mask=enc["attention_mask"])
+
+    logprob_sum = 0.0
+    for pos, tgt_id in zip(mask_positions, target_ids):
+        log_probs = F.log_softmax(out.logits[0, pos].float(), dim=-1)
+        logprob_sum += log_probs[tgt_id].item()
+
+    return math.exp(logprob_sum) if len(target_ids) == 1 else None, logprob_sum
+
+# Example: how does P("train") in this sentence change across years?
+sentence = "The train arrived at the station on time"
+for year in [1800, 1830, 1860, 1890, 1920]:
+    prob, logprob_sum = get_target_word_probability(sentence, "train", year)
+    print(f"year={year}: P(train)={prob:.4f}" if prob else f"year={year}: logprob_sum={logprob_sum:.4f}")
+```
+
+## Known limitations
+
+- Trained for 1 epoch only. Treat results as preliminary.
+- No year/decade-reweighting was used for this specific run, so the model's
+  competence will not be uniform across the full 1800-1920 span and will reflect
+  whatever the skewed training corpus's year distribution is.

continuous_time_embedding.py

@@ -0,0 +1,117 @@
+"""
+Companion module for loading the continuous-time NewsBERT model.
+
+This model is not a plain AutoModelForMaskedLM. It wraps a full
+fine-tuned BERT with a continuous sinusoidal time embedding injected at the
+input layer. You need this file (or an equivalent copy of these two classes)
+to load and query it; `AutoModelForMaskedLM.from_pretrained(...)` alone will
+not work.
+
+Usage:
+    from continuous_time_embedding import load_continuous_time_model
+
+    tokenizer, model = load_continuous_time_model("npedrazzini/NewsBERT_1800-1920-Temporal")
+"""
+
+import os
+import math
+import torch
+import torch.nn as nn
+from transformers import AutoTokenizer, AutoModelForMaskedLM
+from huggingface_hub import snapshot_download
+
+MIN_YEAR = 1800.0
+MAX_YEAR = 1920.0
+MIN_PERIOD_YEARS = 5.0
+N_TIME_FREQS = 24
+
+
+class ContinuousTimeEmbedding(nn.Module):
+    """Sinusoidal (Fourier) features over normalized year, projected to
+    hidden_size. Nearby years produce nearby embeddings by construction.
+    Frequency band: lowest = 1 cycle over the full 1800-1920 span, highest =
+    1 cycle per MIN_PERIOD_YEARS (5 years) -- see training script / model
+    card for the reasoning."""
+
+    def __init__(self, hidden_size, n_freqs=N_TIME_FREQS, min_year=MIN_YEAR, max_year=MAX_YEAR,
+                 min_period_years=MIN_PERIOD_YEARS):
+        super().__init__()
+        self.min_year = min_year
+        self.max_year = max_year
+        span_years = max_year - min_year
+        low_freq_per_year = 1.0 / span_years
+        high_freq_per_year = 1.0 / min_period_years
+        freqs_per_year = torch.exp(torch.linspace(
+            math.log(low_freq_per_year), math.log(high_freq_per_year), n_freqs
+        ))
+        angular_freqs = freqs_per_year * span_years * 2 * math.pi
+        self.register_buffer("freqs", angular_freqs)
+        self.proj = nn.Linear(2 * n_freqs, hidden_size)
+
+    def forward(self, years: torch.Tensor) -> torch.Tensor:
+        t = (years - self.min_year) / (self.max_year - self.min_year)
+        t = t.clamp(0.0, 1.0).unsqueeze(-1)
+        angles = t * self.freqs
+        feats = torch.cat([torch.sin(angles), torch.cos(angles)], dim=-1)
+        return self.proj(feats)
+
+
+class ContinuousTimeBertForMLM(nn.Module):
+    """Full fine-tuned BERT + continuous time embedding, injected into every
+    token's input embedding, re-normalized via LayerNorm before entering the
+    transformer stack."""
+
+    def __init__(self, model, hidden_size, n_time_freqs=N_TIME_FREQS,
+                 min_year=MIN_YEAR, max_year=MAX_YEAR, inject_mode="all_tokens"):
+        super().__init__()
+        self.model = model
+        self.time_embed = ContinuousTimeEmbedding(hidden_size, n_time_freqs, min_year, max_year)
+        assert inject_mode in ("all_tokens", "cls_only")
+        self.inject_mode = inject_mode
+        self.post_inject_norm = nn.LayerNorm(hidden_size)
+
+    def get_input_embeddings_module(self):
+        return self.model.bert.embeddings
+
+    def forward(self, input_ids, attention_mask, years, labels=None):
+        embeddings_module = self.get_input_embeddings_module()
+        tok_embeds = embeddings_module(input_ids)
+        time_vec = self.time_embed(years).unsqueeze(1)
+
+        if self.inject_mode == "all_tokens":
+            tok_embeds = self.post_inject_norm(tok_embeds + time_vec)
+        else:
+            tok_embeds = tok_embeds.clone()
+            tok_embeds[:, 0, :] = self.post_inject_norm(tok_embeds[:, 0, :] + time_vec.squeeze(1))
+
+        return self.model(inputs_embeds=tok_embeds, attention_mask=attention_mask, labels=labels)
+
+    def save_pretrained(self, save_dir):
+        os.makedirs(save_dir, exist_ok=True)
+        self.model.save_pretrained(os.path.join(save_dir, "full_model"))
+        torch.save(self.time_embed.state_dict(), os.path.join(save_dir, "time_embed.pt"))
+        torch.save(self.post_inject_norm.state_dict(), os.path.join(save_dir, "post_inject_norm.pt"))
+
+    @classmethod
+    def load_pretrained(cls, save_dir, hidden_size=768, **kwargs):
+        model = AutoModelForMaskedLM.from_pretrained(os.path.join(save_dir, "full_model"))
+        obj = cls(model, hidden_size, **kwargs)
+        obj.time_embed.load_state_dict(torch.load(os.path.join(save_dir, "time_embed.pt"), map_location="cpu"))
+        obj.post_inject_norm.load_state_dict(torch.load(os.path.join(save_dir, "post_inject_norm.pt"), map_location="cpu"))
+        return obj
+
+
+def load_continuous_time_model(repo_id_or_path, device=None, **kwargs):
+    """Convenience loader. Works with either a local checkpoint directory or
+    a Hugging Face Hub repo id (downloads it locally first)."""
+    if os.path.isdir(repo_id_or_path):
+        local_dir = repo_id_or_path
+    else:
+        local_dir = snapshot_download(repo_id_or_path)
+
+    tokenizer = AutoTokenizer.from_pretrained(os.path.join(local_dir, "full_model"))
+    model = ContinuousTimeBertForMLM.load_pretrained(local_dir, **kwargs)
+
+    device = device or ("cuda" if torch.cuda.is_available() else "cpu")
+    model.to(device).eval()
+    return tokenizer, model

full_model/config.json

@@ -0,0 +1,30 @@
+{
+  "add_cross_attention": false,
+  "architectures": [
+    "BertForMaskedLM"
+  ],
+  "attention_probs_dropout_prob": 0.1,
+  "bos_token_id": null,
+  "classifier_dropout": null,
+  "dtype": "float32",
+  "eos_token_id": null,
+  "gradient_checkpointing": false,
+  "hidden_act": "gelu",
+  "hidden_dropout_prob": 0.1,
+  "hidden_size": 768,
+  "initializer_range": 0.02,
+  "intermediate_size": 3072,
+  "is_decoder": false,
+  "layer_norm_eps": 1e-12,
+  "max_position_embeddings": 512,
+  "model_type": "bert",
+  "num_attention_heads": 12,
+  "num_hidden_layers": 12,
+  "pad_token_id": 0,
+  "position_embedding_type": "absolute",
+  "tie_word_embeddings": true,
+  "transformers_version": "5.12.1",
+  "type_vocab_size": 2,
+  "use_cache": true,
+  "vocab_size": 30522
+}

full_model/model.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:177baf21c1b2d907a1ce6a2e831e22334f11038576aa91b25e6becfc9029e690
+size 438080896

optimizer.pt

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:54a8d18ccefc04f1469723e885ed4801bdc00f70f587c708083f5dd0c296f026
+size 876600395

post_inject_norm.pt

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:e782d66c99a643197f1ec655cbc2f23a4d5f3efabc3d8a69d9afc06adb054e65
+size 8173

rng_state.pth

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:6d6b98a51dcacc8dd6d5b6f856d73c38d9abf7918ceaf8c5c7fefdcf7134062e
+size 14645

scheduler.pt

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:89940e4161de6ed7f9b35686e39169e5a582679d61badafe238e6e4fafe390bb
+size 1465

time_embed.pt

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:02f20ee39ce2f0b8759379fe6e0ea93ed8c0540ccad77becc6ab4f310268da7d
+size 152828