Seeded M/M' fixture and the gpt2 calibration band for Assay

README.md

@@ -0,0 +1,103 @@
+---
+license: mit
+library_name: transformers
+pipeline_tag: text-generation
+tags:
+  - trustworthy-ml
+  - model-verification
+  - model-substitution
+  - research-fixture
+  - gpt2
+---
+
+# Assay reproducibility bundle — the seeded M / M' pair and a calibrated band
+
+This is **not a model you should generate text with.** It is the fixture that makes
+[Assay](https://github.com/NagaYu/assay)'s model-substitution experiments reproducible
+offline, in seconds, with no training and no downloads of anything large.
+
+- **Code**: [github.com/NagaYu/assay](https://github.com/NagaYu/assay)
+- **Demo**: [NagaYu/assay-demo](https://huggingface.co/spaces/NagaYu/assay-demo)
+- **Receipt corpus**: [NagaYu/assay-receipts](https://huggingface.co/datasets/NagaYu/assay-receipts)
+
+## Contents
+
+| Path | What it is |
+|---|---|
+| `tiny_pair/M/` | The "real" model **M**: a seeded, randomly-initialized GPT-2, 4 layers, hidden 64, vocab 256 |
+| `tiny_pair/Mp/` | The "cheap" substitute **M'**: 2 layers, same hidden size and vocab, with an untied and permuted output head |
+| `calibration.json` | The sketch protocol, the tolerance band calibrated on **gpt2**, and the demo HMAC key |
+
+## Why M' has a permuted head
+
+This detail matters more than it looks. A randomly-initialized GPT-2 is dominated by the
+residual copy of the last prompt token, so two independently seeded models both greedily echo
+that token — they are *behaviourally identical*, and a "substitution" between them is one
+nobody could notice in the output. Permuting the tied embedding does not help either: the
+echo circuit reads and writes through the same matrix, so the permutation cancels.
+
+So M' here has its output head **untied and permuted**. It stays a well-formed model, it
+produces genuinely different text, and the substitution becomes something worth detecting.
+Anything that reads the emitted tokens — including Assay's generation-consistency check and
+the forgery it defends against — is meaningless without this.
+
+## Using it
+
+```python
+from transformers import GPT2LMHeadModel
+from huggingface_hub import snapshot_download
+
+path = snapshot_download("NagaYu/assay-tiny-pair")
+M  = GPT2LMHeadModel.from_pretrained(f"{path}/tiny_pair/M").eval()
+Mp = GPT2LMHeadModel.from_pretrained(f"{path}/tiny_pair/Mp").eval()
+```
+
+The identical pair is also constructed on the fly by the library, with no download at all:
+
+```python
+from assay.hooks import tiny_random_pair
+M, Mp = tiny_random_pair(seed=0)   # bit-identical to the weights published here
+```
+
+Both are published because the copy here pins the fixture: if the constructor ever changes,
+these weights still reproduce the numbers in the paper-style README and the receipt corpus.
+
+## `calibration.json`
+
+The tolerance band Assay's verifier compares against, measured on honest **gpt2** traffic
+with injected numerical jitter, plus the sketch protocol parameters:
+
+```json
+{
+  "sketch": {"seed": ..., "layers": [3, 6, 8, 11], "n_positions": 6, "sketch_dim": 32},
+  "calibration": {"mu": {...}, "sigma": {...}, "gen_mismatch_rate": ...},
+  "verifier": {"deep_check_rate": 0.25, "alpha": 0.01}
+}
+```
+
+`gen_mismatch_rate` is the rate at which an honest greedy provider's emitted token differs
+from M's own argmax at an inspected position — the null rate of the check that catches a
+forged receipt. It is calibrated rather than assumed to be zero, because numerical jitter can
+flip a near-tie.
+
+The HMAC key in that file is published deliberately: without it the signatures in the receipt
+corpus cannot be checked. It is a demo key for a research prototype, not a secret, and not a
+TEE attestation key.
+
+## Intended use and limits
+
+Intended for reproducing and probing substitution-detection experiments. These weights are
+random — M and M' have never been trained and their text is meaningless. Do not use them for
+anything other than the verification experiments they exist for.
+
+## Citation
+
+```bibtex
+@software{assay2026,
+  title  = {Assay: per-request internal receipts for model-substitution detection},
+  author = {Nagao, Yuta},
+  year   = {2026},
+  url    = {https://github.com/NagaYu/assay},
+  note   = {Research prototype}
+}
+```

calibration.json

@@ -0,0 +1,42 @@
+{
+  "sketch": {
+    "seed": 20260718,
+    "layers": [
+      3,
+      6,
+      8,
+      11
+    ],
+    "n_positions": 6,
+    "sketch_dim": 32,
+    "rank_top_m": 8
+  },
+  "calibration": {
+    "mu": {
+      "3": -7.309343497798924,
+      "6": -5.587953915026016,
+      "8": -5.450299438360374,
+      "11": -5.6453210104866685
+    },
+    "sigma": {
+      "3": 0.212757760417348,
+      "6": 0.1316980161516204,
+      "8": 0.0888338944598564,
+      "11": 0.11255746540264383
+    },
+    "n_samples": 48,
+    "gen_mismatch_rate": 0.01
+  },
+  "verifier": {
+    "deep_check_rate": 0.25,
+    "alpha": 0.01
+  },
+  "generation": {
+    "gen_len": 12,
+    "decoding": "greedy",
+    "jitter_sigma_rel": 0.0003
+  },
+  "model_m": "gpt2",
+  "hmac_demo_key": "assay-demo-key-0123456789abcdef!",
+  "signature_note": "HMAC-SHA256 over Receipt.canonical_bytes(); simulates a TEE attestation"
+}

tiny_pair/M/config.json

@@ -0,0 +1,34 @@
+{
+  "activation_function": "gelu_new",
+  "add_cross_attention": false,
+  "architectures": [
+    "GPT2LMHeadModel"
+  ],
+  "attn_pdrop": 0.0,
+  "bos_token_id": 50256,
+  "dtype": "float32",
+  "embd_pdrop": 0.0,
+  "eos_token_id": 50256,
+  "initializer_range": 0.02,
+  "layer_norm_epsilon": 1e-05,
+  "model_type": "gpt2",
+  "n_embd": 64,
+  "n_head": 4,
+  "n_inner": null,
+  "n_layer": 4,
+  "n_positions": 128,
+  "pad_token_id": null,
+  "reorder_and_upcast_attn": false,
+  "resid_pdrop": 0.0,
+  "scale_attn_by_inverse_layer_idx": false,
+  "scale_attn_weights": true,
+  "summary_activation": null,
+  "summary_first_dropout": 0.1,
+  "summary_proj_to_labels": true,
+  "summary_type": "cls_index",
+  "summary_use_proj": true,
+  "tie_word_embeddings": true,
+  "transformers_version": "5.16.1",
+  "use_cache": true,
+  "vocab_size": 256
+}

tiny_pair/M/generation_config.json

@@ -0,0 +1,9 @@
+{
+  "_from_model_config": true,
+  "bos_token_id": 50256,
+  "eos_token_id": 50256,
+  "output_attentions": false,
+  "output_hidden_states": false,
+  "transformers_version": "5.16.1",
+  "use_cache": true
+}

tiny_pair/M/model.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:e7e968601697cc5f0931a660abbb2680912e8a88cac6844b01a93eacaafac6fd
+size 903456

tiny_pair/Mp/config.json

@@ -0,0 +1,34 @@
+{
+  "activation_function": "gelu_new",
+  "add_cross_attention": false,
+  "architectures": [
+    "GPT2LMHeadModel"
+  ],
+  "attn_pdrop": 0.0,
+  "bos_token_id": 50256,
+  "dtype": "float32",
+  "embd_pdrop": 0.0,
+  "eos_token_id": 50256,
+  "initializer_range": 0.02,
+  "layer_norm_epsilon": 1e-05,
+  "model_type": "gpt2",
+  "n_embd": 64,
+  "n_head": 4,
+  "n_inner": null,
+  "n_layer": 2,
+  "n_positions": 128,
+  "pad_token_id": null,
+  "reorder_and_upcast_attn": false,
+  "resid_pdrop": 0.0,
+  "scale_attn_by_inverse_layer_idx": false,
+  "scale_attn_weights": true,
+  "summary_activation": null,
+  "summary_first_dropout": 0.1,
+  "summary_proj_to_labels": true,
+  "summary_type": "cls_index",
+  "summary_use_proj": true,
+  "tie_word_embeddings": false,
+  "transformers_version": "5.16.1",
+  "use_cache": true,
+  "vocab_size": 256
+}

tiny_pair/Mp/generation_config.json

@@ -0,0 +1,9 @@
+{
+  "_from_model_config": true,
+  "bos_token_id": 50256,
+  "eos_token_id": 50256,
+  "output_attentions": false,
+  "output_hidden_states": false,
+  "transformers_version": "5.16.1",
+  "use_cache": true
+}

tiny_pair/Mp/model.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:fd537928301388ee870cca1ebcdf400f2646972747eba57798de9b529c2d10d2
+size 566952