README.md

---
license: mit
language:
- pl
base_model: FacebookAI/xlm-roberta-large
library_name: transformers
pipeline_tag: token-classification
tags:
- polish
- pii
- ner
- xlm-roberta
- hybrid
---

# NERGAL 1.0.3

**Named Entity Recognition with Grounded Additive Labels**

SlayerLab hybrid PII cleaner for Polish. Not a chat model. Not a drop-in `pipeline("token-classification")`.

## TL;DR

Python rules do the identifiers they can prove. A transformer NER head adds phone and other PII spans the regex misses. The cleaner **unions** the two on the original text, then replaces hits with `[Telefon]` or `[PII]`.

- **Version:** `1.0.3` (`hybrid.json`, `CHANGELOG.md`)
- **Ground:** `scrub_pii` regex (SHA256 `f32d5c54…`)
- **Additive labels:** XLM-RoBERTa-large token classifier, BIO tags `phone` / `pii`, threshold 0.95
- **This snapshot:** seed `202609160`, **epoch 5** of a seven-epoch schedule

## What NERGAL detects — and what it does not

NERGAL masks **contact details and selected identifiers in Polish text**. It is not a general-purpose anonymizer: names, postal addresses and other personal information can remain in the output. Its corpus-masking policy also includes public, institutional and company contacts and identifiers.

### Detection scope

These are target categories, not a guarantee that every occurrence or format is detected.

| Category | Values in scope | Replacement |
|---|---|---|
| Phone contacts | Phone, fax and SMS contact numbers, including foreign, emergency, short/service and vanity numbers; extensions and number alternatives | `[Telefon]` |
| Email | Email addresses, including recoverable broken or incomplete addresses | `[PII]` |
| Personal identifiers | PESEL, passport and identity-document numbers | `[PII]` |
| Organization identifiers | NIP/VAT, REGON, KRS, GEMI, LEI and equivalent foreign company registration numbers; labelled DUNS, BDO and RPWDL register-book numbers | `[PII]` |
| Financial identifiers | Bank/account numbers, including Polish accounts and foreign IBANs | `[PII]` |
| Property identifiers | Land-register (księga wieczysta, KW) numbers | `[PII]` |
| Electronic contacts and access | e-Doręczenia and ePUAP addresses, explicitly labelled numeric access PINs (including `pin=` URL values), GG account IDs | `[PII]` |

Context matters: a number that resembles a phone or identifier is not automatically in scope. Rules use labels, format checks and, for some unlabelled identifiers, checksums; the model adds contextual detections. Coverage varies by category, and the aggregate benchmark below does not establish recall for every category.

### Outside detection scope

NERGAL is not designed to remove:

- **Personal names**, including private individuals and public officials; organization names.
- **Postal/street addresses, dates of birth and ages.**
- **Social-media handles, ordinary URLs and filenames.** An in-scope value inside a URL, such as a labelled numeric PIN, can still be masked.
- **Vehicle registration plates and generic serial, model or version codes.**
- **Document, case, article, funding and procurement references**, including procedure UUIDs; ISBNs, ORCIDs, TERYT codes, EAN/GTIN product codes and CNIL website-registration references.
- Prices, list numbers, generic labels without values, clearly fictitious examples and already-redacted placeholders.

These are intended exclusions; false positives can still mask some of this content.

### Known gaps in 1.0.3

Unlabelled phones and identifiers, unusual formatting and damaged text can escape detection. **VINs and obfuscated emails** (such as `name (at) domain.pl`) are approved annotation targets, but that approval alone does not establish reliable support in the released 1.0.3 model. Do not rely on it to remove them consistently.

## Versions

841-dev, union at 0.95, 354 gold spans. Same weights and API is a patch; new capability is minor; API, threshold, or weight recipe is major. A release that changes these numbers updates `hybrid.json` `eval` and this table.

| Version | Whole /354 | Residual | Rules FP | Union FP | Char P | Char R | What changed |
|---|---:|---:|---:|---:|---:|---:|---|
| 1.0.0 | 323 | 25 | 133 | 133 | 97.76% | 95.95% | First Hub snapshot |
| 1.0.1 | 323 | 25 | 98 | 123 | 97.93% | 95.95% | Prefix-only glued-email trim |
| 1.0.2 | 324 | 24 | 98 | 123 | 97.93% | 96.12% | Labelled country-area phone fix |
| **1.0.3** | **324** | **24** | **98** | **123** | **97.93%** | **96.12%** | Label-note, e-Delivery and registry rules; placeholder and card fixes |

## 841-dev

Tables use one development split: 841 passages, 215 with gold PII, **354 spans** (169 phone, 185 other PII). It is the `dev` side of a 4,500-passage labelled tranche (3,655 train / 841 dev). Sources match Dynaword (EUR-Lex, HPLT, Wikipedia, parliamentary and government text, plus smaller news/literary slices). Labels mix unchanged silver with human review.

The files contain real identifiers, so they are not released with the weights.

## Why XLM-R

GLiNER, HerBERT-large, and XLM-R-large were trained on the same split and unioned with the same regex. Plot: diagnostic threshold 0.50; selection used the full threshold grid. GLiNER covers more at 0.50 and then dumps precision. XLM-R is the architecture we kept.

![Primary three-model curves](figures/primary-three-model-curves.png)

## Why epoch 5

Fresh XLM-R, seven epochs. 133 epoch/threshold combinations. Epoch 5 at 0.95 was the only point that both beat the historical GLiNER∪regex incumbent on coverage and introduced zero new false-mask characters. Epoch 7 covers more PII (334/354) but adds 15 new false characters.

![Seven-epoch XLM-R curves](figures/xlmr-seven-epoch-curves.png)

| Epoch | Covered @ 0.95 /354 | Residual passages | False characters | New false vs incumbent |
|---|---:|---:|---:|---:|
| 1 | 272 | 59 | 175 | 42 |
| 2 | 291 | 49 | 141 | 8 |
| 3 | 317 | 30 | 140 | 7 |
| 4 | 320 | 28 | 143 | 10 |
| **5** | **323** | **25** | **133** | **0** |
| 6 | 328 | 20 | 147 | 14 |
| 7 | 334 | 16 | 148 | 15 |

## Compared with other systems

Same 841-dev split, threshold **0.95**. **Naked** is the transformer alone. **∪ regex** is that model unioned with the current rules. Character scores are gold vs masked characters.

| System | Mode | Whole /354 | Residual | False chars | Char P | Char R |
|---|---|---:|---:|---:|---:|---:|
| Regex (`scrub_pii`) | rules | 259 | 70 | 98 | 98.13% | 85.04% |
| GLiNER 2.5-multi zero-shot | naked | 81 | 188 | 970 | 56.98% | 21.22% |
| GLiNER 2.5-multi zero-shot | ∪ regex | 273 | 61 | 1,068 | 83.33% | 88.16% |
| Historical GLiNER email12 | naked | 249 | 70 | 10 | 99.81% | 85.39% |
| Historical GLiNER email12 | ∪ regex | 290 | 50 | 108 | 98.13% | 93.76% |
| XLM-R epoch 5 | naked | 298 | 42 | 57 | 98.96% | 89.38% |
| **NERGAL 1.0.3** | **∪ regex** | 324 | 24 | 123 | 97.93% | 96.12% |

Zero-shot [GLiNER 2.5](https://huggingface.co/fastino/gliner2.5-multi-v1) is not competitive here, especially on non-phone PII (7/185 whole vs 160 naked / 179 union). Fine-tuned historical GLiNER is the precise naked baseline (10 false characters) and still trails XLM-R on coverage. NERGAL is XLM-R epoch 5 plus the regex: 145/169 phone, 179/185 other PII. Exact-span precision 86.34%, recall 89.27%, F1 87.78%.

Trained GLiNER, HerBERT-large, and XLM-R-large were also compared on this split (plot above). GLiNER’s 0.50 coverage lead is the precision collapse in that figure; no GLiNER or HerBERT operating point passed the content-preservation gate.

## Extra seeds

Historical seed-comparison results, before the 1.0.2 parser fix.

| Seed | Whole /354 | False chars | New false vs historical union |
|---|---:|---:|---:|
| 202609160 (selected weights) | 323 | 123 | 0 |
| 202609161 | 322 | 134 | 1 |
| 202609162 | 316 | 151 | 18 |

Seed 160 is the published weights. 161 and 162 were confirmation runs of the same recipe.

## Load

This repo is the PII island: `scrub_pii.py` plus `nergal.py`. `pipeline("token-classification")` will not match. Regex runs on the original text, the model adds spans at 0.95, then the two are unioned and replaced with `[Telefon]` / `[PII]`. Text that already holds `[PII]` / `[Telefon]` is scrubbed as usual, but an identifier with a placeholder inside it or right before it can be missed.

```python
from pathlib import Path
from huggingface_hub import snapshot_download

root = Path(snapshot_download("SlayerLab/NERGAL"))
import sys
sys.path.insert(0, str(root))
from nergal import Nergal

nergal = Nergal.from_pretrained(root, local_files_only=True)
masked, counts = nergal.scrub(text)
```

`hybrid.json` records version `1.0.3`, threshold 0.95, gap ids `250002` / `250003`, the 841-dev `eval` block, and two weight hashes: `model_safetensors_sha256` for the published file and `source_checkpoint_sha256` for the training checkpoint it was packed from. `test_nergal.py` is synthetic (no corpus text). From this snapshot: `python -m unittest test_nergal`.

Base weights: [`FacebookAI/xlm-roberta-large`](https://huggingface.co/FacebookAI/xlm-roberta-large) revision `c23d21b0620b635a76227c604d44e43a9f0ee389` (MIT).