Upload int8-weight-only release artifact

.gitattributes

@@ -33,3 +33,7 @@ saved_model/**/* filter=lfs diff=lfs merge=lfs -text
 *.zip filter=lfs diff=lfs merge=lfs -text
 *.zst filter=lfs diff=lfs merge=lfs -text
 *tfevents* filter=lfs diff=lfs merge=lfs -text
+figures/evaluation-test-1-1.png filter=lfs diff=lfs merge=lfs -text
+figures/evaluation-train-1-1.png filter=lfs diff=lfs merge=lfs -text
+model_quantized.onnx_data filter=lfs diff=lfs merge=lfs -text
+tokenizer.json filter=lfs diff=lfs merge=lfs -text

README.md

@@ -0,0 +1,408 @@
+---
+language:
+- ko
+- en
+license: apache-2.0
+library_name: onnxruntime
+pipeline_tag: token-classification
+inference: false
+base_model: openai/privacy-filter
+tags:
+- onnx
+- onnxruntime
+- int8
+- weight-only
+- token-classification
+- ner
+- pii
+- privacy
+- pii-masking
+- korean
+- finance
+- bioes
+- viterbi
+- mixture-of-experts
+datasets:
+- BCCard/pii-masking-openpii-finance
+metrics:
+- f1
+- precision
+- recall
+---
+
+# 1. Overview
+This repository provides the INT8 weight-only ONNX Runtime serving artifact for a Korean/English PII detection model for the finance domain. The model was built by full fine-tuning [`openai/privacy-filter`](https://huggingface.co/openai/privacy-filter) - a 1.4B-parameter MoE with 50M active parameters - on synthetic finance-domain PII data. It tags 18 PII entity types using 73
+BIOES token classes at token level.
+
+The model is designed as the NER layer of a multi-layer PII-masking gateway in front of LLM services. It should be deployed with a deterministic regex backstop for fully structured identifiers and must not be treated as a standalone anonymization, compliance or privacy guarantee. High-sensitivity deployments require in-domain evaluation and ongoing monitoring.
+
+This repository does not contain PyTorch weights. `AutoModelForTokenClassification.from_pretrained()`
+and the standard Transformers token-classification pipeline therefore do not load this artifact. Use ONNX Runtime as shown in [Usage](#13-usage).
+
+The serving chain is:
+
+```text
+text
+-> tokenizer
+-> INT8 weight-only ONNX graph
+-> FP32 logits
+-> constrained BIOES Viterbi
+-> character-offset spans
+-> whitespace boundary refinement
+```
+
+## 1.1. TL;DR
+* **Base model**: [`openai/privacy-filter`](https://huggingface.co/openai/privacy-filter) — 1.4B-parameter MoE (128 experts, 50M active), 8 layers, hidden 640, bidirectional banded attention (±128), o200k tokenizer
+* **Domain / Language**: Finance (BC Card — cards, accounts, national IDs, customer service text) / Korean + English
+* **Task**: Token classification (BIOES) → character-offset PII spans → masking
+* **Labels (18)**: `PERSON, RRN, FRN, CARD_NUMBER, ACCOUNT_NUMBER, SECRET, USER_ID, EMAIL, PHONE, PASSPORT, DRIVER_LICENSE, GENERIC_ID, ADDRESS, ZIPCODE, DATE, CARD_EXPIRY, CVC, IPIN`
+* **Method**: Full fine-tuning (all parameters incl. experts & router) with a re-initialized 73-class head (rows copied from the base head by taxonomy mapping)
+* **Decoding**: **constrained BIOES Viterbi** (not per-token argmax) + whitespace span refinement — the bundled `viterbi_calibration.json` exposes precision↔recall operating-point biases without retraining
+* **Format**: INT8 weight-only ONNX (`MatMulNBits`, `QMoE`, `GatherBlockQuantized`) with FP32 activations and logits; graph + external tensor data + tokenizer + label taxonomy + Viterbi calibration sidecar
+* **Sequence length**: trained on sequences ≤768 tokens — chunk longer inputs
+* **Intended use**
+  - In-house **PII masking gateway** (detect → mask before text reaches an LLM)
+  - Korean-centric finance text with mixed English (IDs, e-mails, card numbers)
+
+## 1.2. Label Taxonomy (N=18)
+The 18 labels re-map the upstream ai4privacy source labels to the granularity a Korean financial masking policy needs - merging fragments into single spans (`GIVENNAME`/`SURNAME` → `PERSON`, `CITY`/`STREET`/`BUILDINGNUM` → `ADDRESS`) and adding Korea-specific classes absent upstream (`RRN`, `FRN`, `IPIN`, `CARD_EXPIRY`, `CVC`, `SECRET`). `data source` records the row-source buckets in which each label occurs: `ko` means `openpii-1.5m-ko`, `en` means `openpii-1.5m-en`, and `domain` means locally synthesized rows.
+
+| label | description | data source |
+|-------|-------------|-------------|
+| `PERSON` | full name (surname + given, single span) | ko, en, domain |
+| `RRN` | resident registration number (Korea) | ko, domain |
+| `FRN` | foreign registration number | domain |
+| `CARD_NUMBER` | credit/debit card PAN | ko, en, domain |
+| `ACCOUNT_NUMBER` | bank account number | ko, domain |
+| `SECRET` | auth secret (password / API key / token) | ko, domain |
+| `USER_ID` | online member ID | ko, en, domain |
+| `EMAIL` | email address | ko, en, domain |
+| `PHONE` | phone number (mobile / landline) | ko, en, domain |
+| `PASSPORT` | passport number | ko, en, domain |
+| `DRIVER_LICENSE` | driver's license number | ko, en, domain |
+| `GENERIC_ID` | generic identifier (no KO counterpart) | ko, en, domain |
+| `ADDRESS` | address (city / street / building, single span) | ko, en, domain |
+| `ZIPCODE` | postal code | ko, en, domain |
+| `DATE` | date / time | ko, en, domain |
+| `CARD_EXPIRY` | card expiry date | domain |
+| `CVC` | card verification code | domain |
+| `IPIN` | I-PIN number | domain |
+
+Each entity type has `B-`, `I-`, `E-` and `S-` boundary classes, plus the background class `O`. This yields 73 output classes. The bundled `label-taxonomy.yaml` and `config.json` must remain in the same label order.
+
+## 1.3. Usage
+Install the CPU runtime and client dependencies:
+
+```bash
+pip install "onnxruntime>=1.28,<1.29" "huggingface-hub>=1.5" "transformers>=5.6" numpy
+```
+
+The following example downloads the repository and runs the INT8 ONNX graph. To reproduce the validated output, use the functions in [Viterbi Reference Implementation](#43-viterbi-reference-implementation)
+to apply constrained BIOES decoding and recover refined character-offset spans. Independent token argmax is not equivalent to the validated decoding chain.
+
+```python
+import json
+from pathlib import Path
+
+import numpy as np
+import onnxruntime as ort
+
+from huggingface_hub import snapshot_download
+from transformers import AutoTokenizer
+
+model_id = "BCCard/MoAI-Privacy-Filter-INT8"
+model_dir = Path(snapshot_download(repo_id=model_id))
+tokenizer = AutoTokenizer.from_pretrained(model_dir)
+config = json.loads((model_dir / "config.json").read_text(encoding="utf-8"))
+labels = tuple(
+    config["id2label"][str(index)]
+    for index in range(len(config["id2label"]))
+)
+calibration = json.loads(
+    (model_dir / "viterbi_calibration.json").read_text(encoding="utf-8")
+)
+biases = calibration["operating_points"]["default"]["biases"]
+
+session = ort.InferenceSession(
+    str(model_dir / "model_quantized.onnx"),
+    providers=["CPUExecutionProvider"],
+)
+text = "고객 모아이님(000000-0000000)께서 010-0000-0000로 연락 요청하셨습니다."
+encoding = tokenizer(
+    text,
+    return_offsets_mapping=True,
+    add_special_tokens=False,
+    return_tensors="np",
+)
+offsets = encoding.pop("offset_mapping")[0].tolist()
+feeds = {
+    model_input.name: np.asarray(encoding[model_input.name], dtype=np.int64)
+    for model_input in session.get_inputs()
+}
+logits = session.run(["logits"], feeds)[0]
+length = int(feeds["attention_mask"][0].sum())
+
+# Copy constrained_viterbi() and decode_spans() from Section 4.3.
+path = constrained_viterbi(logits[0, :length], labels, biases)
+tags = [labels[class_id] for class_id in path]
+spans = decode_spans(tags, offsets[:length], text)
+
+print(logits.shape)
+print(spans)
+```
+
+Output from the INT8 graph:
+
+```text
+(1, 30, 73)
+[
+  {'start': 3, 'end': 6, 'label': 'PERSON'},
+  {'start': 8, 'end': 22, 'label': 'RRN'},
+  {'start': 26, 'end': 39, 'label': 'PHONE'}
+]
+```
+
+The offsets use Python's half-open character interval `[start, end)`. Masking is downstream policy logic. For example, the spans above can produce:
+
+```text
+고객 [PERSON]님([RRN])께서 [PHONE]로 연락 요청하셨습니다.
+```
+
+For batches, enable right padding and pass only `input_ids` and `attention_mask` to the graph. `offset_mapping` stays outside ONNX and is used only to map decoded token tags back to the original text. Inputs are INT64 and the graph always returns FP32 logits.
+
+
+## 1.4. Training Data
+| Dataset | Role | Size |
+|---------|------|------|
+| (Public) [BCCard/pii-masking-openpii-finance](https://huggingface.co/datasets/BCCard/pii-masking-openpii-finance) (v2) | Training / validation | ~58.5k train rows · ~14.5k validation rows |
+| (Private) BCCard/pii-masking-openpii-finance-test (v2) | Golden Set (release gate; never used for training/tuning) | 2,000 rows (ko 1,460 / en 540) |
+
+* Sources: curated Korean subset of `ai4privacy/pii-masking-openpii-1.5m` (label taxonomy remapped, name spans merged & naturalized) + finance-domain synthetic templates + **~30% English replay** (forgetting guard)
+* Hard-example design baked into v2: surface-similar non-PII decoys (FP suppression), label-confusion pairs in one sentence (RRN↔FRN, DRIVER_LICENSE↔GENERIC_ID), weak-context true PII (FN suppression), long-span address boundary variants
+* All values are synthetic; validity-pattern collisions with real identifiers are removed at generation time (e.g. card numbers are forced to fail Luhn)
+
+
+## 1.5. Training Procedure
+| Item | Value |
+|------|-------|
+| Method | Full fine-tuning (1.4B params — experts and router included) |
+| Head | 33-class base head → 73-class head, initialized by copying base rows via taxonomy mapping |
+| Loss | Token-level cross-entropy |
+| Batch | effective 16 (per-device × world × accum), fixed across hardware layouts |
+| LR / scheduler | 1e-4 / linear decay, warmup 3% |
+| Optimizer | AdamW (fused), weight decay 0.0, max_grad_norm 1.0 |
+| Epochs | 5 — best checkpoint by validation span micro-F1, **decoded with the same constrained Viterbi as deployment** |
+| Precision | FP32 master weights + BF16 autocast; MoE router/experts explicitly kept FP32 during compute |
+| Hardware | 1× NVIDIA H100 (~5h) |
+
+<div align="center">
+  <img src="figures/evaluation-train-1-1.png" alt="Training loss, learning-rate and gradient-norm curves for the v1 and v2 models" >
+</div>
+
+<div align="center">
+  <img src="figures/evaluation-test-1-1.png" alt="Training-time validation metric curves for the v1 and v2 models" >
+</div>
+
+<br>
+
+# 2. Evaluation
+## 2.1. INT8 Validation Results
+The INT8 artifact was evaluated on all 14,543 v2 validation rows with the same tokenizer, actual character offsets, constrained Viterbi decoder and whitespace refinement used in the serving example.
+
+| Metric | INT8 ONNX |
+|---|---:|
+| strict micro F1 | 0.9599 |
+| macro F1 | 0.9607 |
+| ko strict micro F1 | 0.9560 |
+| ko macro F1 | 0.9571 |
+| en strict micro F1 | 0.9689 |
+| en macro F1 | 0.9641 |
+| masking coverage - diagnostic | 0.9980 |
+| ko masking coverage - diagnostic | 0.9987 |
+| en masking coverage - diagnostic | 0.9967 |
+
+Masking coverage is the proportion of gold PII characters covered by the union of predicted spans, regardless of predicted label. It is a diagnostic value and was not used as the INT8 pass or fail gate.
+
+## 2.2. INT8 Parity Against FP32 ONNX
+| Metric | FP32 ONNX | INT8 ONNX | Delta |
+|---|---:|---:|---:|
+| strict micro F1 | 0.959801 | 0.959886 | +0.000085 |
+| macro F1 | 0.960264 | 0.960662 | +0.000398 |
+| masking coverage - diagnostic | 0.997960 | 0.997980 | +0.000020 |
+| ko strict micro F1 | 0.955986 | 0.956047 | +0.000061 |
+| en strict micro F1 | 0.968768 | 0.968910 | +0.000142 |
+
+INT8 predictions reached strict micro F1 0.998568 against FP32 predictions and matched exactly on 14,429 of 14,543 rows. Overall PERSON recall changed by -0.000276 and ADDRESS recall changed by +0.000116. All thresholds fixed before the full run passed. Small positive quality deltas do not show that INT8 is intrinsically more accurate. Quantization moved a small number of boundary decisions in both directions.
+
+## 2.3. Limitations and Deployment Guidance
+* The graph uses Microsoft contrib operators. ONNX Runtime 1.28.0 with `CPUExecutionProvider` was validated. Confirm operator placement and fallback behavior before selecting another provider.
+* The M4 validation run was a functional observation, not a serving benchmark. Measure warmup, latency percentiles, throughput, RSS and peak memory on target hardware.
+* Independent token argmax can create invalid BIOES paths and does not reproduce the reported metrics. Use constrained Viterbi decoding.
+* Weak-context Korean person names remain the main known miss channel. Names without honorifics, particles or nearby role cues require particular monitoring.
+* Alphanumeric identifiers can swap among `USER_ID`, `SECRET`, `GENERIC_ID` and `ACCOUNT_NUMBER`. The value may still be masked even when the semantic label is wrong.
+* Training and evaluation use synthetic data. Robustness to real customer text, slang, OCR noise and previously unseen credential formats has not been established.
+* The label policy is fixed to the 18 types above. Changing taxonomy or boundary policy requires fine-tuning.
+* Inputs beyond 768 tokens were not part of this release validation. Chunk longer inputs with overlap and reconcile spans at chunk boundaries.
+* Use deterministic pattern rules as a backstop for structured identifiers and retain review paths for high-sensitivity workflows.
+
+<br>
+
+# 3. Future Work
+* Benchmark the INT8 artifact on the actual serving hardware and execution provider.
+* Improve weak-context PERSON coverage and collect privacy-safe shadow-mode failure patterns for a future data revision.
+
+<br>
+
+# 4. Meta Info
+## 4.1. Citation
+```bibtex
+@misc{bccard2026moaiprivacyfilter,
+  title        = {MoAI Privacy Filter INT8: A Korean Finance-Domain PII Detection Model},
+  author       = {BC Card AX Team},
+  year         = {2026},
+  howpublished = {https://huggingface.co/BCCard/MoAI-Privacy-Filter-INT8},
+  note         = {INT8 weight-only ONNX artifact of a full fine-tune of openai/privacy-filter}
+}
+```
+
+## 4.2. See Also
+* Base model - [`openai/privacy-filter`](https://huggingface.co/openai/privacy-filter)
+* Parent BF16 model - [`BCCard/MoAI-Privacy-Filter`](https://huggingface.co/BCCard/MoAI-Privacy-Filter)
+* Training dataset - [`BCCard/pii-masking-openpii-finance`](https://huggingface.co/datasets/BCCard/pii-masking-openpii-finance)
+* Source data attribution - `ai4privacy/pii-masking-openpii-1.5m` under CC-BY-4.0
+
+## 4.3. Viterbi Reference Implementation
+These functions apply the BIOES transition constraints and calibration biases used for validation, then recover character-offset spans with whitespace boundary refinement. Keep `config.json`, `label-taxonomy.yaml` and `viterbi_calibration.json` in their bundled label order and configuration.
+
+<details>
+<summary>Show the constrained Viterbi and span-decoding reference code</summary>
+
+```python
+import numpy as np
+
+
+def transition_bias(
+    from_prefix: str,
+    to_prefix: str,
+    biases: dict[str, float],
+) -> float:
+    if from_prefix == "O" and to_prefix == "O":
+        return biases["transition_bias_background_stay"]
+    if from_prefix == "O" and to_prefix in {"B", "S"}:
+        return biases["transition_bias_background_to_start"]
+    if from_prefix in {"E", "S"} and to_prefix == "O":
+        return biases["transition_bias_end_to_background"]
+    if from_prefix in {"E", "S"} and to_prefix in {"B", "S"}:
+        return biases["transition_bias_end_to_start"]
+    if from_prefix in {"B", "I"} and to_prefix == "I":
+        return biases["transition_bias_inside_to_continue"]
+    if from_prefix in {"B", "I"} and to_prefix == "E":
+        return biases["transition_bias_inside_to_end"]
+    raise ValueError(f"Unsupported transition family: {from_prefix} -> {to_prefix}")
+
+
+def constrained_viterbi(
+    logits: np.ndarray,
+    labels: tuple[str, ...],
+    biases: dict[str, float],
+) -> list[int]:
+    scores = np.asarray(logits, dtype=np.float32)
+    num_classes = len(labels)
+    parsed = [
+        ("O", None) if tag == "O" else tuple(tag.split("-", 1))
+        for tag in labels
+    ]
+    start_scores = np.full(num_classes, -np.inf, dtype=np.float32)
+    end_scores = np.full(num_classes, -np.inf, dtype=np.float32)
+    transition_scores = np.full(
+        (num_classes, num_classes),
+        -np.inf,
+        dtype=np.float32,
+    )
+
+    for class_id, (tag_prefix, _) in enumerate(parsed):
+        if tag_prefix in {"O", "B", "S"}:
+            start_scores[class_id] = 0.0
+        if tag_prefix in {"O", "E", "S"}:
+            end_scores[class_id] = 0.0
+
+    for from_id, (from_prefix, from_label) in enumerate(parsed):
+        for to_id, (to_prefix, to_label) in enumerate(parsed):
+            allowed = (
+                from_prefix in {"O", "E", "S"}
+                and to_prefix in {"O", "B", "S"}
+            ) or (
+                from_prefix in {"B", "I"}
+                and to_prefix in {"I", "E"}
+                and from_label == to_label
+            )
+            if allowed:
+                transition_scores[from_id, to_id] = transition_bias(
+                    from_prefix,
+                    to_prefix,
+                    biases,
+                )
+
+    delta = start_scores + scores[0]
+    backpointers = np.zeros((scores.shape[0], num_classes), dtype=np.int64)
+    for token_id in range(1, scores.shape[0]):
+        candidates = delta[:, None] + transition_scores
+        backpointers[token_id] = np.argmax(candidates, axis=0)
+        delta = np.max(candidates, axis=0) + scores[token_id]
+
+    last_class = int(np.argmax(delta + end_scores))
+    path = [last_class]
+    for token_id in range(scores.shape[0] - 1, 0, -1):
+        path.append(int(backpointers[token_id, path[-1]]))
+    return list(reversed(path))
+
+
+def decode_spans(
+    tags: list[str],
+    offsets: list[list[int]],
+    text: str,
+) -> list[dict[str, int | str]]:
+    spans = []
+    current = None
+
+    def flush() -> None:
+        nonlocal current
+        if current is None:
+            return
+        start = current["start"]
+        end = current["end"]
+        while start < end and text[start].isspace():
+            start += 1
+        while end > start and text[end - 1].isspace():
+            end -= 1
+        if start < end:
+            spans.append(
+                {
+                    "start": start,
+                    "end": end,
+                    "label": current["label"],
+                }
+            )
+        current = None
+
+    for tag, (start, end) in zip(tags, offsets, strict=True):
+        if tag == "O":
+            flush()
+            continue
+        tag_prefix, label = tag.split("-", 1)
+        if tag_prefix in {"B", "S"} or current is None or current["label"] != label:
+            flush()
+            current = {
+                "start": start,
+                "end": end,
+                "label": label,
+            }
+        else:
+            current["end"] = end
+        if tag_prefix in {"E", "S"}:
+            flush()
+    flush()
+    return spans
+```
+
+</details>

config.json

@@ -0,0 +1,202 @@
+{
+  "architectures": [
+    "OpenAIPrivacyFilterForTokenClassification"
+  ],
+  "attention_bias": true,
+  "attention_dropout": 0.0,
+  "bos_token_id": null,
+  "classifier_dropout": 0.0,
+  "default_n_ctx": 128000,
+  "eos_token_id": 199999,
+  "head_dim": 64,
+  "hidden_act": "silu",
+  "hidden_size": 640,
+  "id2label": {
+    "0": "O",
+    "1": "B-PERSON",
+    "2": "I-PERSON",
+    "3": "E-PERSON",
+    "4": "S-PERSON",
+    "5": "B-RRN",
+    "6": "I-RRN",
+    "7": "E-RRN",
+    "8": "S-RRN",
+    "9": "B-FRN",
+    "10": "I-FRN",
+    "11": "E-FRN",
+    "12": "S-FRN",
+    "13": "B-CARD_NUMBER",
+    "14": "I-CARD_NUMBER",
+    "15": "E-CARD_NUMBER",
+    "16": "S-CARD_NUMBER",
+    "17": "B-ACCOUNT_NUMBER",
+    "18": "I-ACCOUNT_NUMBER",
+    "19": "E-ACCOUNT_NUMBER",
+    "20": "S-ACCOUNT_NUMBER",
+    "21": "B-SECRET",
+    "22": "I-SECRET",
+    "23": "E-SECRET",
+    "24": "S-SECRET",
+    "25": "B-USER_ID",
+    "26": "I-USER_ID",
+    "27": "E-USER_ID",
+    "28": "S-USER_ID",
+    "29": "B-EMAIL",
+    "30": "I-EMAIL",
+    "31": "E-EMAIL",
+    "32": "S-EMAIL",
+    "33": "B-PHONE",
+    "34": "I-PHONE",
+    "35": "E-PHONE",
+    "36": "S-PHONE",
+    "37": "B-PASSPORT",
+    "38": "I-PASSPORT",
+    "39": "E-PASSPORT",
+    "40": "S-PASSPORT",
+    "41": "B-DRIVER_LICENSE",
+    "42": "I-DRIVER_LICENSE",
+    "43": "E-DRIVER_LICENSE",
+    "44": "S-DRIVER_LICENSE",
+    "45": "B-GENERIC_ID",
+    "46": "I-GENERIC_ID",
+    "47": "E-GENERIC_ID",
+    "48": "S-GENERIC_ID",
+    "49": "B-ADDRESS",
+    "50": "I-ADDRESS",
+    "51": "E-ADDRESS",
+    "52": "S-ADDRESS",
+    "53": "B-ZIPCODE",
+    "54": "I-ZIPCODE",
+    "55": "E-ZIPCODE",
+    "56": "S-ZIPCODE",
+    "57": "B-DATE",
+    "58": "I-DATE",
+    "59": "E-DATE",
+    "60": "S-DATE",
+    "61": "B-CARD_EXPIRY",
+    "62": "I-CARD_EXPIRY",
+    "63": "E-CARD_EXPIRY",
+    "64": "S-CARD_EXPIRY",
+    "65": "B-CVC",
+    "66": "I-CVC",
+    "67": "E-CVC",
+    "68": "S-CVC",
+    "69": "B-IPIN",
+    "70": "I-IPIN",
+    "71": "E-IPIN",
+    "72": "S-IPIN"
+  },
+  "initial_context_length": 4096,
+  "initializer_range": 0.02,
+  "intermediate_size": 640,
+  "label2id": {
+    "B-ACCOUNT_NUMBER": 17,
+    "B-ADDRESS": 49,
+    "B-CARD_EXPIRY": 61,
+    "B-CARD_NUMBER": 13,
+    "B-CVC": 65,
+    "B-DATE": 57,
+    "B-DRIVER_LICENSE": 41,
+    "B-EMAIL": 29,
+    "B-FRN": 9,
+    "B-GENERIC_ID": 45,
+    "B-IPIN": 69,
+    "B-PASSPORT": 37,
+    "B-PERSON": 1,
+    "B-PHONE": 33,
+    "B-RRN": 5,
+    "B-SECRET": 21,
+    "B-USER_ID": 25,
+    "B-ZIPCODE": 53,
+    "E-ACCOUNT_NUMBER": 19,
+    "E-ADDRESS": 51,
+    "E-CARD_EXPIRY": 63,
+    "E-CARD_NUMBER": 15,
+    "E-CVC": 67,
+    "E-DATE": 59,
+    "E-DRIVER_LICENSE": 43,
+    "E-EMAIL": 31,
+    "E-FRN": 11,
+    "E-GENERIC_ID": 47,
+    "E-IPIN": 71,
+    "E-PASSPORT": 39,
+    "E-PERSON": 3,
+    "E-PHONE": 35,
+    "E-RRN": 7,
+    "E-SECRET": 23,
+    "E-USER_ID": 27,
+    "E-ZIPCODE": 55,
+    "I-ACCOUNT_NUMBER": 18,
+    "I-ADDRESS": 50,
+    "I-CARD_EXPIRY": 62,
+    "I-CARD_NUMBER": 14,
+    "I-CVC": 66,
+    "I-DATE": 58,
+    "I-DRIVER_LICENSE": 42,
+    "I-EMAIL": 30,
+    "I-FRN": 10,
+    "I-GENERIC_ID": 46,
+    "I-IPIN": 70,
+    "I-PASSPORT": 38,
+    "I-PERSON": 2,
+    "I-PHONE": 34,
+    "I-RRN": 6,
+    "I-SECRET": 22,
+    "I-USER_ID": 26,
+    "I-ZIPCODE": 54,
+    "O": 0,
+    "S-ACCOUNT_NUMBER": 20,
+    "S-ADDRESS": 52,
+    "S-CARD_EXPIRY": 64,
+    "S-CARD_NUMBER": 16,
+    "S-CVC": 68,
+    "S-DATE": 60,
+    "S-DRIVER_LICENSE": 44,
+    "S-EMAIL": 32,
+    "S-FRN": 12,
+    "S-GENERIC_ID": 48,
+    "S-IPIN": 72,
+    "S-PASSPORT": 40,
+    "S-PERSON": 4,
+    "S-PHONE": 36,
+    "S-RRN": 8,
+    "S-SECRET": 24,
+    "S-USER_ID": 28,
+    "S-ZIPCODE": 56
+  },
+  "max_position_embeddings": 131072,
+  "model_type": "openai_privacy_filter",
+  "num_attention_heads": 14,
+  "num_experts_per_tok": 4,
+  "num_hidden_layers": 8,
+  "num_key_value_heads": 2,
+  "num_local_experts": 128,
+  "onnx_config": {
+    "activation_dtype": "float32",
+    "block_size": 32,
+    "external_data": "model_quantized.onnx_data",
+    "model_file": "model_quantized.onnx",
+    "operator_profile": "MatMulNBits + QMoE + GatherBlockQuantized",
+    "output_dtype": "float32",
+    "weight_bits": 8,
+    "weight_only": true
+  },
+  "output_router_logits": false,
+  "pad_token_id": 199999,
+  "rms_norm_eps": 1e-05,
+  "rope_parameters": {
+    "beta_fast": 32.0,
+    "beta_slow": 1.0,
+    "factor": 32.0,
+    "original_max_position_embeddings": 4096,
+    "rope_theta": 150000.0,
+    "rope_type": "yarn",
+    "truncate": false
+  },
+  "router_aux_loss_coef": 0.001,
+  "sliding_window": 128,
+  "tie_word_embeddings": false,
+  "transformers_version": "5.13.1",
+  "use_cache": false,
+  "vocab_size": 200064
+}

label-taxonomy.yaml

@@ -0,0 +1,242 @@
+# =============================================================================
+# Label Mapping - ai4privacy 1.5m 라벨 -> 내부 학습 라벨 매핑 테이블
+#
+# * 역할: 모델 학습 라벨 = 로깅 스키마 키 = 정책 테이블 키의 단일 소스
+#   (docs/handoff/2026-07-03-pii-masking-architecture-qna.md [13])
+# * 역산 근거: playbooks/privacy_filter/privacy-filter-policy.md (표 1 정형 패턴 / 표 2 마스킹 기준)
+# * 실측 근거: docs/handoff/2026-07-03-ai4privacy-1p5m-ko-audit.md (v0 초안 + ko 서브셋 감사)
+# * v0 -> v1 변경: 주소 처리 확정(단일+서브마스킹) / FRN 신설 / USER_ID·ACCOUNT_NUMBER
+#   드롭 철회 / 여권·운전면허 분리 유지 / regex 전담 항목 분리
+# * 2026-07-21: AGE·GENDER 드롭 확정 (표1·표2 무근거 - 정책 역산 원칙 관철, N=17 -> 15)
+# * 2026-07-21: 표2 문맥 항목 3종 모델 승격 (CARD_EXPIRY·CVC·IPIN - 문맥 검출은 모델 레인
+#   본령, 키워드 regex는 이중 레인 병행. N=15 -> 18, 61 -> 73클래스)
+# =============================================================================
+
+version: v1
+date: 2026-07-21
+basis:
+  policy: playbooks/privacy_filter/privacy-filter-policy.md
+  audit: docs/handoff/2026-07-03-ai4privacy-1p5m-ko-audit.md
+
+# -----------------------------------------------------------------------------
+# 1. 모델 학습 라벨 (N=18 -> BIOES 4N+1 = 73클래스)
+#    * data_source: 라벨 표본의 출처 리스트 - ko(1.5m-ko 정제본) / en(1.5m-en 리플레이) /
+#      domain(도메인 부트스트랩 대기 - 헤드는 유사 라벨 행 복사로 init)
+#    * RRN의 1.5m 유래분은 ko 한정 (en TAXNUM은 GENERIC_ID fallback) - domain은 증강 방식 B
+#    * en 분포 실측 (2026-07-13, 163,740행/1,247,393스팬): 상위 19종 집중 - PASSWORD 0,
+#      ACCOUNTNUM 1, USERNAME 20 -> "희소 라벨은 en 담당" 가설 기각, [domain] 재배정
+# -----------------------------------------------------------------------------
+model_labels:
+  - name: PERSON
+    description: 성명 (성+이름 병합 단일 스팬)
+    policy_ref: "표2 성명 (김*용 - 첫·끝 글자 제외) / 표2 성명(영문) (앞 4자리 노출) - 치환 시 스크립트(한글/영문)로 규칙 분기, full-span 경계 필수"
+    data_source: [ko, en]
+    head_init_base: private_person
+  - name: RRN
+    description: 주민등록번호
+    policy_ref: "표1 주민번호 / 표2 주민등록번호 (뒤 7자리 마스킹)"
+    data_source: [ko, domain]     # domain = 증강 방식 B 신규 생성분 (전략문 §3.3.1)
+    head_init_base: account_number
+  - name: FRN
+    description: 외국인등록번호 (신설 - 1.5m 소스 0건)
+    policy_ref: "표1 외국인등록번호 ([5-8] 시작) / 표2 주민등록번호(외국인등록번호 포함) (뒤 7자리)"
+    data_source: [domain]
+    head_init_from: RRN
+  - name: CARD_NUMBER
+    description: 신용/체크카드 번호
+    policy_ref: "표1 카드번호 / 표2 카드번호 (7~12번째 자리, PCI-DSS)"
+    data_source: [ko, en, domain] # domain = 증강 방식 B 신규 생성분 (전략문 §3.3.1)
+    head_init_base: account_number
+  - name: ACCOUNT_NUMBER
+    description: 계좌번호 (v0 드롭 철회 - 표2 마스킹 기준 존재)
+    policy_ref: "표1 계좌번호 (전화번호 흡수 위험 - recognizer 우선순위 필요) / 표2 계좌번호 (뒤 5자리)"
+    data_source: [domain]          # ko 5·en 1건 실측 - 합성 증강 필수 (은행별 포맷 규칙 생성 용이)
+    head_init_base: account_number
+  - name: SECRET
+    description: 인증 시크릿 (비밀번호·API 키·토큰 통합 - 카드/회원/ISP 구분은 문맥 불가 + 액션 동일)
+    policy_ref: "표2 카드비밀번호·온라인 회원 패스워드·ISP비밀번호 (처리 금지 -> 게이트웨이에서는 LLM 노출 금지 = 전체 마스킹으로 해석)"
+    data_source: [domain]          # 1.5m 전무 (ko 4·en 0 실측) - 도메인/합성 증강 전담
+    head_init_base: secret         # base 8종 중 secret 행 정확 복사 (인접 아닌 직계 상속)
+  - name: USER_ID
+    description: 온라인 회원 ID (v0 드롭 철회 - 표2 마스킹 기준 존재)
+    policy_ref: "표2 온라인 회원 ID (앞 2자리 제외)"
+    data_source: [domain]          # ko 4·en 20건 실측 - 증강 필수
+    head_init_base: account_number
+  - name: EMAIL
+    description: 이메일 주소
+    policy_ref: "표1 이메일 / 표2 이메일주소 (ID 앞 2자리 제외 마스킹)"
+    data_source: [ko, en]
+    head_init_base: private_email
+  - name: PHONE
+    description: 전화번호 (휴대폰/일반전화 통합 - 치환 단계에서 프리픽스로 재분류)
+    policy_ref: "표1 휴대폰번호·전화번호 / 표2 기본 뒤 6자리 공통 - 내부망 채널 한정 휴대폰
+      뒤 4자리 완화 (치환 시 값 프리픽스로 휴대폰 판별 + 채널 축 분기)"
+    data_source: [ko, en]
+    head_init_base: private_phone
+  - name: PASSPORT
+    description: 여권번호 (분리 유지 - 마스킹 기준 상이 + ko 3,211건)
+    policy_ref: "표1 여권번호 / 표2 여권번호 (뒤 4자리)"
+    data_source: [ko, en]
+    head_init_base: account_number
+  - name: DRIVER_LICENSE
+    description: 운전면허번호 (분리 유지 - 마스킹 기준 상이 + ko 4,092건)
+    policy_ref: "표1 운전면허번호 / 표2 운전면허번호 (중간 6자리)"
+    data_source: [ko, en]
+    head_init_base: account_number
+  - name: GENERIC_ID
+    description: 범용 식별자 (한국 대응물 없는 ID류 - recall-first로 O 대신 유지)
+    policy_ref: "audit §5 (SOCIALNUM 10자리·IDCARDNUM 영숫자)"
+    data_source: [ko, en]
+    head_init_base: account_number
+  - name: ADDRESS
+    description: 주소 (시·도로명·건물번호 통합 스팬 - 치환 단계에서 숫자 서브마스킹)
+    policy_ref: "표2 주소 (지번: 읍/면/동 미만 숫자 / 도로명: 건물번호·상세주소 숫자 마스킹 -> 스팬 내 규칙 치환)"
+    data_source: [ko, en]
+    head_init_base: private_address
+  - name: ZIPCODE
+    description: 우편번호 (분리 유지 - keep 토글 세분성 전제)
+    policy_ref: "handoff [10] ZIPCODE keep 실증"
+    data_source: [ko, en]
+    head_init_base: private_address
+  - name: DATE
+    description: 날짜·시각 (생년월일 미분리 - ko 데이터 DOB 0건 + 과잉 마스킹 준수 인정 조항)
+    policy_ref: "표2 생년월일 (노출 금지 - DATE 전체 마스킹으로 초과 준수)"
+    data_source: [ko, en]
+    head_init_base: private_date
+  - name: CARD_EXPIRY
+    description: 카드유효기한 (표2 문맥 항목 승격 2026-07-21 - 키워드 regex 이중 레인 병행)
+    policy_ref: "표2 카드유효기한 (**/** 전체 마스킹)"
+    data_source: [domain]          # 1.5m 원천 0건 - 증강 전담 (PAN 동반 문맥 생성)
+    head_init_base: private_date
+  - name: CVC
+    description: 카드검증코드 (CVC/CVV/CAV 통합 - 표2 문맥 항목 승격 2026-07-21)
+    policy_ref: "표2 카드검증코드 (저장·출력 금지 -> 게이트웨이 전체 마스킹 해석)"
+    data_source: [domain]          # 1.5m 원천 0건 - 증강 전담 (단독 생성 금지 - PAN 동반 필수)
+    head_init_base: account_number
+  - name: IPIN
+    description: I-PIN 번호 (표2 문맥 항목 승격 2026-07-21 - 값 규격 pending)
+    policy_ref: "표2 I-PIN (뒤 5자리 - 하이픈 무관 숫자 기준)"
+    data_source: [domain]          # 1.5m 원천 0건 - 규격 확정 전 파일럿 수량만
+    head_init_base: account_number
+
+# -----------------------------------------------------------------------------
+# 2. 소스 라벨 매핑 (ai4privacy 1.5m -> 내부 라벨)
+#    * "O" = 학습에서 비엔티티 처리 (드롭 리스트 - 정책팀 컨펌 후 확정)
+#    * 조건부 매핑은 locale + value_pattern 안전장치 동반
+# -----------------------------------------------------------------------------
+source_mapping: # <source_mapping>
+  GIVENNAME: PERSON
+  SURNAME: PERSON
+  TITLE: O                        # 군/양/gun/yang 직역 존칭 88% - 전량 드롭
+  TAXNUM:
+    label: RRN  # <conditional_mapping>
+    condition:
+      locale: ko
+      value_pattern: '^\d{6}-[1-4]\d{6}$'   # 실측 3,975/3,975 정합 - 안전장치
+    fallback: GENERIC_ID          # 비정합 값 / 타 로케일(en 리플레이)은 세무 ID -> 범용 ID
+  SOCIALNUM: GENERIC_ID
+  IDCARDNUM: GENERIC_ID
+  DRIVERLICENSENUM: DRIVER_LICENSE
+  PASSPORTNUM: PASSPORT
+  TELEPHONENUM: PHONE
+  EMAIL: EMAIL
+  CREDITCARDNUMBER: CARD_NUMBER
+  ACCOUNTNUM: ACCOUNT_NUMBER
+  CITY: ADDRESS
+  STREET: ADDRESS
+  BUILDINGNUM: ADDRESS
+  ZIPCODE: ZIPCODE
+  DATE: DATE
+  TIME: DATE
+  AGE: O                          # 2026-07-21 드롭 확정 - 표1·표2 무근거 (quasi-identifier 존치안 폐기)
+  GENDER: O                       # 〃
+  SEX: O                          # 〃 (GENDER 중복 소스)
+  USERNAME: USER_ID
+  PASSWORD: SECRET                # ko 4·en 0건 - 시드 미미, 본 표본은 도메인/합성 증강
+  # 롱테일 (ko <= 40건, 학습 라벨 부적격) - 정책팀 컨펌 대기 드롭 리스트
+  ORGANISATION: O
+  URL: O
+  AMOUNT: O
+  COUNTRY: O
+  CURRENCY: O
+  BANKNAME: O
+  TIMEZONE: O
+  SALARY: O
+  IPV4: O                         # 고객 IP는 표2 항목이나 ko 2건 - 1차 regex(표1 IP주소) 전담
+  JOBTITLE: O
+  HOSPITALNAME: O
+  ALLERGIES: O
+  WEIGHT: O                       # en 전용 라벨 (ko 0건) - 2026-07-13 en 전수 검증에서 발견, 정책 외
+  HEIGHT: O                       # validation 전용 희소 라벨 (ko 1건 + en 1건) - 2026-07-22 refine fail-fast로 발견, 정책 외
+
+# -----------------------------------------------------------------------------
+# 3. 병합 규칙 (정제 스크립트 2단계 - 인접 스팬 병합, ko·en 공통)
+#    * 병합은 언어 공통 (en "John Smith"도 단일 스팬이어야 표2 영문 성명 규칙 성립)
+#    * ko 전용은 병합이 아니라 그 다음의 "자연화"(성+명 붙여쓰기 - 전��문 §3.4.2 (1) 2단계)
+#    * 순서 강제 금지: ko 주소는 큰 -> 작, en 주소는 작 -> 큰 역순 - 인접성만 조건
+#
+# * gap_allowed : 두 엔티티 사이의 인식·허용이 가능한 구조 및 패턴
+#     - whitespace (공백·탭)
+#     - punct (구두점 : `, . - · / ( )` 등 문장 부호)
+# -----------------------------------------------------------------------------
+merge_rules:  # <merge_rules>
+  person:
+    source_labels: [GIVENNAME, SURNAME]
+    max_gap_chars: 4
+    gap_allowed: whitespace_or_punct
+    output_label: PERSON
+  address:
+    source_labels: [CITY, STREET, BUILDINGNUM]
+    max_gap_chars: 4
+    gap_allowed: whitespace_or_punct
+    output_label: ADDRESS
+
+# -----------------------------------------------------------------------------
+# 4. 1차 regex 전담 항목 (모델 라벨 제외 - 로깅 스키마 키에는 포함)
+#    * 표1 정형 패턴 recognizer가 검출, 치환 정책은 표2 준용
+# -----------------------------------------------------------------------------
+regex_only:
+  - key: CI
+    policy_ref: "표1 CI (86자+`==` 고정, lookaround 경계 - 검수 보강 2026-07-21) / 표2 CI (앞 7자리 노출)"
+    # SHA-512 base64 동형 과탐 허용 - 정밀도 필요시 ci·연계정보 키워드 가점
+  - key: IP_ADDRESS
+    policy_ref: "표1 IP주소 (옥텟 0~255 엄격형 - 검수 보강 2026-07-21) / 표2 고객의 IP주소 (앞 3자리 = 첫 옥텟 마스킹 해석, IPv4 한정)"
+
+# -----------------------------------------------------------------------------
+# 4-1. 1차 regex 병행 (이중 레인) - 모델 라벨이면서 키워드 regex도 병행
+#    * 2026-07-21 모델 승격 3종: 키워드 regex(고정밀 저recall - 명시 문형) + 모델(무키워드
+#      맥락 recall) union - 문맥 의존 검출은 모델 레인의 존재 이유
+#    * 패턴 엔진 전제: Python re (lookbehind 사용 - RE2/Hyperscan 계열 비호환)
+#    * 치환은 값 캡처그룹만 (키워드 보존 = 추적성)
+# -----------------------------------------------------------------------------
+regex_assist:
+  - key: CARD_EXPIRY
+    patterns:
+      keyword: '(?i)(유효\s*기[간한]|expir\w*|valid\s*thru)\D{0,12}(0[1-9]|1[0-2])\s*[/.\-]\s*((?:20)?\d\d)(?!\d)'
+      pan_adjacent: '(0[1-9]|1[0-2])\s*/\s*\d{2}(?!\d)'   # 카드번호 매치 직후 \D{0,20} 윈도우 내에서만 적용
+  - key: CVC
+    patterns:
+      keyword: '(?i)(\b(?:cv[vc]2?|security\s*code)\b|카드\s*검증\s*(?:번호|코드|값)?|보안\s*코드)\W{0,6}(?!(?:19|20)\d\d(?!\d))(\d{3,4})(?!\d)'
+      # CID·CSC 키워드 기본 제외 (correlation ID 충돌). 정밀도 옵션: PAN co-occurrence 게이트
+  - key: IPIN
+    patterns:
+      keyword: '(?i)(아이핀|i[-\s]?pin)\s*(?:번호|no\.?)?\D{0,8}(\d{6}[-\s]?\d{7})(?!\d)'
+      # 주민번호 규칙 선행 적용 후 잔여분만 IPIN. 값 규격(13자리 가정) 확인 필요 (pending)
+
+# -----------------------------------------------------------------------------
+# 5. 스코프 외 - 텍스트 게이트웨이가 다루지 않는 표2 항목 (선언적 기록)
+# -----------------------------------------------------------------------------
+out_of_scope:
+  - key: VIDEO_PERSONAL_INFO
+    policy_ref: "표2 개인영상정보 (검정 목적 외 처리 금지)"
+    reason: "텍스트 엔티티가 아닌 모달리티(이미지·영상) - 멀티모달 입력 경로가 열리면
+      텍스트 필터를 우회하므로 별도 통제(입력 차단 or 비전 필터) 필요"
+
+# -----------------------------------------------------------------------------
+# 6. 미결 - 정책팀 컨펌 항목, TBD
+# -----------------------------------------------------------------------------
+pending_confirmation:
+  - "드롭 리스트(O 매핑) 전체 = 비PII 선언 - 특히 ORGANISATION·AMOUNT·SALARY"
+  - "IPIN 값 규격 (13자리 가정) - 제공기관 계약 확인 (CVC 검출 규칙은 2026-07-21 키워드 동반 패턴 채택으로 해소)"
+  - "SECRET 신설 채택 (2026-07-13) - 표2 처리금지류의 '게이트웨이 전체 마스킹' 해석은
+    정책팀 사후 컨펌 대상"

model_quantized.onnx

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:2f9e709e0e9825dcd2d42e7f7bad9e2dcc7087cc3962db46787b9a4b123f5f74
+size 151493

model_quantized.onnx_data

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:3c86220d2388b60740a7ec824dd94e6f3152804e96f758d9c2b9a34dd5c97886
+size 1618085352

tokenizer.json

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:0614fe83cadab421296e664e1f48f4261fa8fef6e03e63bb75c20f38e37d07d3
+size 27868174

tokenizer_config.json

@@ -0,0 +1,13 @@
+{
+  "backend": "tokenizers",
+  "eos_token": "<|endoftext|>",
+  "is_local": false,
+  "local_files_only": false,
+  "model_input_names": [
+    "input_ids",
+    "attention_mask"
+  ],
+  "model_max_length": 128000,
+  "pad_token": "<|endoftext|>",
+  "tokenizer_class": "TokenizersBackend"
+}

viterbi_calibration.json

@@ -0,0 +1,14 @@
+{
+  "operating_points": {
+    "default": {
+      "biases": {
+        "transition_bias_background_stay": 0.0,
+        "transition_bias_background_to_start": 0.0,
+        "transition_bias_end_to_background": 0.0,
+        "transition_bias_end_to_start": 0.0,
+        "transition_bias_inside_to_continue": 0.0,
+        "transition_bias_inside_to_end": 0.0
+      }
+    }
+  }
+}