zero v1 — M7 lookup table, run p35w-2m-s2500

README.md

@@ -0,0 +1,171 @@
+---
+license: mit
+language: en
+library_name: numpy
+tags:
+  - retrieval
+  - sentence-similarity
+  - asymmetric-dual-encoder
+  - edge
+  - quantized
+base_model: NovaSearch/stella_en_400M_v5
+pipeline_tag: feature-extraction
+---
+
+# zero — a query encoder with no transformer in it
+
+`zero` is the query side of an **asymmetric dual encoder**. Documents are indexed once, in the
+cloud, by a large frozen encoder ([`NovaSearch/stella_en_400M_v5`](https://huggingface.co/NovaSearch/stella_en_400M_v5),
+1024-d). Queries are encoded on the edge by **this table**: 30,522 × 1024 int8 rows and one
+pooling rule. Encoding a query is a gather and a weighted sum — **no transformer, no matmul,
+no GPU**. Sub-millisecond per query on one CPU core; the whole query asset is **31.8 MB**.
+
+It was distilled from stella so that its output lands in stella's document space. Cosine
+similarity against stella document vectors is the score.
+
+> **Research preview.** This model **missed its own release bar** — see [Results](#results).
+> It is published so the architecture can be tested, not as a recommended drop-in retriever.
+> Read the results section before using it for anything.
+
+## Usage
+
+The query side needs `numpy` and `tokenizers`. That is the entire runtime.
+
+```python
+# pip install numpy tokenizers huggingface_hub
+from huggingface_hub import snapshot_download
+import sys
+
+d = snapshot_download("DylanCouzon/zero-query-encoder-v1")          # ~94 MB
+sys.path.insert(0, d)
+from zero_encoder import ZeroQueryEncoder
+
+enc = ZeroQueryEncoder(d, variant="int8")           # or "fp16"
+q = enc.encode(["how do mrna vaccines work?"])      # (1, 1024), L2-normalized
+```
+
+Documents are encoded by the frozen teacher — **pin the revision**, the table is only valid
+against this exact document space:
+
+```python
+# pip install sentence-transformers
+from sentence_transformers import SentenceTransformer
+import numpy as np
+
+doc_model = SentenceTransformer(
+    "NovaSearch/stella_en_400M_v5",
+    revision="ffeb2b7ee715c226d4ffe5e4619f7dbb48624c20",
+    trust_remote_code=True,
+    # required unless xformers is installed; also the pinned setting the table was distilled under
+    config_kwargs={"use_memory_efficient_attention": False, "unpad_inputs": False},
+)
+docs = [
+    "mRNA vaccines deliver a strand of messenger RNA encoding a viral antigen.",
+    "The Treaty of Westphalia ended the Thirty Years' War in 1648.",
+]
+D = doc_model.encode(docs, normalize_embeddings=True)   # no prefix on the document side
+
+scores = q @ D.T
+print(docs[int(np.argmax(scores))])
+```
+
+That asymmetry is the point: `doc_model` runs once per document, in the cloud. `enc` runs on
+every query, on the device, and costs almost nothing.
+
+### The rule, if you reimplement it
+
+Tokenize with the bundled WordPiece tokenizer (`add_special_tokens=True`, truncate at 512,
+**no padding**, no prefix). A token appearing `c` times in the query carries **total weight
+`sqrt(c)`**, not `c`. Sum the rows, divide by the weight sum, L2-normalize. An empty or
+near-zero-norm bag falls back to the normalized `[CLS]` row (id **101**; row 0 is `[PAD]`).
+`zero_encoder.py` is 80 lines and is the reference. `config.json` carries the rule and its
+fingerprint (`adb24fb2e8cad66f`).
+
+## Files
+
+| file | what |
+|---|---|
+| `model.npz` | `rows_int8` (30522×1024) + `int8_scale`, and `rows_fp16` for reference |
+| `config.json` | the frozen preprocessing rule, teacher pin, document-encoder spec, shas |
+| `zero_encoder.py` | the whole query path — numpy + tokenizers, no torch |
+| `tokenizer.json`, `vocab.txt`, … | stella's WordPiece tokenizer, copied at the pinned revision |
+
+Per-token learned weights are **folded into the rows**, so the int8 artifact is self-contained.
+`int8` is the variant every published number below was measured on; it was measured
+quality-free against `fp16` (upper bound **0.00013** nDCG@10).
+
+## Results
+
+nDCG@10, exact search (no ANN, so recall is not a confound), on six BEIR datasets. These are
+**confirmatory** numbers from a single pre-registered run against frozen comparator vectors —
+the system was frozen (sha `a7007b1a…`) before any of them were observed.
+
+| system | arguana | fiqa | nfcorpus | scidocs | scifact | trec-covid | **avg-6** |
+|---|---|---|---|---|---|---|---|
+| **`zero` (int8)** | 0.5916 | 0.3728 | 0.3124 | 0.1677 | 0.6101 | 0.5490 | **0.4339** |
+| `zero` + BM25, fused | 0.5975 | 0.4026 | 0.3497 | 0.1881 | 0.7068 | 0.7018 | **0.4911** |
+| BM25 alone | 0.4878 | 0.2532 | 0.3180 | 0.1565 | 0.6791 | 0.6099 | 0.4174 |
+| stella, symmetric (the teacher ceiling) | 0.6369 | 0.5536 | 0.4134 | 0.2395 | 0.7796 | 0.8234 | 0.5744 |
+
+The three registered comparisons, paired bootstrap + sign-flip, Holm-corrected across the family:
+
+| | Δ | 95% CI | verdict |
+|---|---|---|---|
+| `zero` > LightRetriever dense (0.4583) | **−0.0243** | [−0.0405, −0.0086] | **miss, resolved below the bar** |
+| `zero` > BM25 (0.4174) | +0.0165 | [+0.0017, +0.0311] | CI passes, multiplicity does not — unresolved |
+| fused > OpenSearch sparse (0.4868) | +0.0043 | [−0.0063, +0.0151] | **statistical tie** |
+
+**What that means.** A pure lookup table retains **75.5%** of its teacher's quality on these six
+sets at ~1/1000 of the query-side cost, and beats BM25 on average — but it does **not** beat a
+comparable small dense query encoder, and the honest headline is the miss. The one bright spot
+is fusion: `zero` + BM25 is a **statistical tie with OpenSearch's learned sparse retriever**
+while its query side remains a table lookup plus token counts.
+
+### Caveats you should read before trusting a number
+
+- **Teacher contamination.** stella discloses **ArguAna** and **FiQA** in its training data —
+  exactly this system's two strongest datasets. On the four sets with no disclosed overlap
+  (nfcorpus, scidocs, scifact, trec-covid), `zero` is **below BM25** (−0.0311 [−0.0517, −0.0109]).
+- **Distribution.** Training was Wikipedia- and e-commerce-shaped. Retention is 0.915 on
+  in-distribution development sets and 0.755 on these six. Expect the low number out of domain.
+- **English only**, 512 wordpieces, WordPiece-30522 vocab. Out-of-vocabulary terms degrade to
+  their subword rows; heavily fragmented queries are where the gap with a real encoder is widest.
+- The table is a **bag of tokens**. Word order, negation and syntax are not represented at all.
+
+## Costs
+
+| | |
+|---|---|
+| query asset (int8 rows + scales + tokenizer) | **31.8 MB** |
+| query encode, batch 1, one CPU core | **0.38 ms** (`zero_encoder.py` measures ~0.07 ms) |
+| hydration (cold load to first query) | **0.22 s** |
+| document index, 1024-d fp16 | 2.05 GB per 1M documents |
+| document index, 1024-d int8 | 1.02 GB per 1M documents |
+
+For reference at the document side: LightRetriever 3.07, OpenSearch sparse 1.40,
+bge-small 0.77 GB/1M. `zero`'s document index is not cheap — the trade is all on the query side.
+
+## Training
+
+L2 regression of the table's pooled output onto stella's query embeddings, over 340,850
+approved pairs plus 220,632 query-text-only rows. Sources: **Amazon ESCI** (Apache-2.0),
+**FEVER**, **HotpotQA**, **SQuAD**, **NQ-open**, **TriviaQA**, **Mr. TyDi (en)**.
+
+**MS MARCO is permanently excluded** from this lineage — its terms forbid commercial use.
+(Measured cost of that exclusion: +0.0058 [−0.0015, +0.0131] avg-6 had it been included, which
+still misses the bar. The gap is architectural, not licensing.)
+
+### Attribution
+
+NQ, SQuAD, HotpotQA, FEVER and Mr. TyDi derive from Wikipedia and are **CC BY-SA** (3.0/4.0).
+Amazon ESCI is Apache-2.0. The teacher, `NovaSearch/stella_en_400M_v5`, is MIT.
+
+## Citation / provenance
+
+```
+run_id             p35w-2m-s2500
+table sha256       a7007b1a6af120b976f093fd69ddcb5001996ec0b84b5864b4fd25d7af878abf
+teacher            NovaSearch/stella_en_400M_v5 @ ffeb2b7ee715c226d4ffe5e4619f7dbb48624c20
+preproc            prefix="" · add_special_tokens · max_length=512 · pool_mode=sqrt
+preproc fingerprint adb24fb2e8cad66f
+```

config.json

@@ -0,0 +1,40 @@
+{
+ "dim": 1024,
+ "document_encoder": {
+  "cls_id": 101,
+  "config_kwargs": {
+   "unpad_inputs": false,
+   "use_memory_efficient_attention": false
+  },
+  "dim": 1024,
+  "doc_prefix": "",
+  "max_length": 512,
+  "name": "stella-400M-v5",
+  "pooling": "mean",
+  "post_dense": "2_Dense_1024",
+  "query_prefix": "Instruct: Given a web search query, retrieve relevant passages that answer the query.\nQuery: ",
+  "repo": "NovaSearch/stella_en_400M_v5",
+  "revision": "ffeb2b7ee715c226d4ffe5e4619f7dbb48624c20",
+  "tokenizer_id": "bert-wordpiece-30522",
+  "vocab": 30522
+ },
+ "fallback_token_id": 101,
+ "learned_weights": false,
+ "model_type": "zero-lookup-table",
+ "preproc": {
+  "add_special_tokens": true,
+  "max_length": 512,
+  "pool_mode": "sqrt",
+  "prefix": ""
+ },
+ "preproc_fingerprint": "adb24fb2e8cad66f",
+ "recommended_variant": "int8",
+ "run_id": "p35w-2m-s2500",
+ "similarity": "cosine",
+ "source_table_sha256": "a7007b1a6af120b976f093fd69ddcb5001996ec0b84b5864b4fd25d7af878abf",
+ "teacher": "NovaSearch/stella_en_400M_v5",
+ "teacher_revision": "ffeb2b7ee715c226d4ffe5e4619f7dbb48624c20",
+ "version": "v1",
+ "vocab": 30522,
+ "weights_folded": true
+}

model.npz

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:a7007b1a6af120b976f093fd69ddcb5001996ec0b84b5864b4fd25d7af878abf
+size 93886950

special_tokens_map.json

@@ -0,0 +1,37 @@
+{
+  "cls_token": {
+    "content": "[CLS]",
+    "lstrip": false,
+    "normalized": false,
+    "rstrip": false,
+    "single_word": false
+  },
+  "mask_token": {
+    "content": "[MASK]",
+    "lstrip": false,
+    "normalized": false,
+    "rstrip": false,
+    "single_word": false
+  },
+  "pad_token": {
+    "content": "[PAD]",
+    "lstrip": false,
+    "normalized": false,
+    "rstrip": false,
+    "single_word": false
+  },
+  "sep_token": {
+    "content": "[SEP]",
+    "lstrip": false,
+    "normalized": false,
+    "rstrip": false,
+    "single_word": false
+  },
+  "unk_token": {
+    "content": "[UNK]",
+    "lstrip": false,
+    "normalized": false,
+    "rstrip": false,
+    "single_word": false
+  }
+}

tokenizer_config.json

@@ -0,0 +1,62 @@
+{
+  "added_tokens_decoder": {
+    "0": {
+      "content": "[PAD]",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "100": {
+      "content": "[UNK]",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "101": {
+      "content": "[CLS]",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "102": {
+      "content": "[SEP]",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "103": {
+      "content": "[MASK]",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    }
+  },
+  "clean_up_tokenization_spaces": true,
+  "cls_token": "[CLS]",
+  "do_lower_case": true,
+  "mask_token": "[MASK]",
+  "max_length": 8000,
+  "model_max_length": 32768,
+  "pad_to_multiple_of": null,
+  "pad_token": "[PAD]",
+  "pad_token_type_id": 0,
+  "padding_side": "right",
+  "sep_token": "[SEP]",
+  "stride": 0,
+  "strip_accents": null,
+  "tokenize_chinese_chars": true,
+  "tokenizer_class": "BertTokenizer",
+  "truncation_side": "right",
+  "truncation_strategy": "longest_first",
+  "unk_token": "[UNK]"
+}

zero_encoder.py

@@ -0,0 +1,89 @@
+"""Standalone query encoder for the `zero` lookup table. numpy + tokenizers, no torch.
+
+This is the whole query path. It is a vocab x dim table of vectors: tokenize the query,
+gather one row per token, take a count-saturated weighted mean, L2 normalize. There is no
+transformer and no matrix multiply -- encoding a query is a gather and a sum.
+
+The output lives in the document space of the frozen teacher (NovaSearch/stella_en_400M_v5,
+revision pinned in config.json), so it is only meaningful against document vectors produced
+by that exact encoder. Cosine similarity is the score.
+
+Conformance: this file reproduces the frozen training-time query path (m7src/table.py
+`encode_pooled`) to < 1e-5 max-abs on the release fixtures; see m11/release/test_conformance.py.
+"""
+import json
+from pathlib import Path
+
+import numpy as np
+from tokenizers import Tokenizer
+
+EPS = 1e-6
+
+
+class ZeroQueryEncoder:
+    """The released query encoder.
+
+    variant: "int8" is the artifact the published numbers were measured on (31 MB);
+             "fp16" is the same table before quantization (62 MB), included for reference --
+             int8 was measured quality-free against it (upper bound 0.00013 nDCG@10).
+    """
+
+    def __init__(self, model_dir, variant="int8"):
+        d = Path(model_dir)
+        self.config = json.loads((d / "config.json").read_text())
+        pre = self.config["preproc"]
+        if pre["pool_mode"] != "sqrt" or pre["prefix"] != "" or not pre["add_special_tokens"]:
+            raise ValueError(f"this file implements the frozen M7 rule only, got {pre}")
+        self.max_length = int(pre["max_length"])
+        self.fallback_id = int(self.config["fallback_token_id"])
+
+        z = np.load(d / "model.npz")
+        if variant == "int8":
+            self.rows = z["rows_int8"].astype(np.float32) * z["int8_scale"][:, None]
+        elif variant == "fp16":
+            self.rows = z["rows_fp16"].astype(np.float32)
+        else:
+            raise ValueError(f"variant must be 'int8' or 'fp16', got {variant!r}")
+        self.variant = variant
+
+        self.tokenizer = Tokenizer.from_file(str(d / "tokenizer.json"))
+        self.tokenizer.enable_truncation(max_length=self.max_length)
+        # stella's tokenizer.json ships with padding-to-512 enabled. Padding would put ~500
+        # [PAD] rows into every bag; the frozen path (transformers, padding off) never sees one.
+        self.tokenizer.no_padding()
+        self._fallback = self._normalize(self.rows[self.fallback_id])
+
+    @property
+    def dim(self):
+        return self.rows.shape[1]
+
+    @staticmethod
+    def _normalize(v):
+        n = float(np.linalg.norm(v))
+        if n <= EPS:                       # degenerate row: fall back to e_0
+            e0 = np.zeros_like(v)
+            e0[0] = 1.0
+            return e0
+        return v / n
+
+    def encode(self, texts):
+        """texts: str or list[str] -> float32 array (n, dim), L2-normalized."""
+        if isinstance(texts, str):
+            texts = [texts]
+        out = np.empty((len(texts), self.dim), dtype=np.float32)
+        for i, enc in enumerate(self.tokenizer.encode_batch(texts)):
+            out[i] = self._encode_ids(enc.ids)
+        return out
+
+    def _encode_ids(self, ids):
+        if not ids:
+            return self._fallback
+        uniq, counts = np.unique(np.asarray(ids, dtype=np.int64), return_counts=True)
+        # count saturation: a token seen c times carries TOTAL weight sqrt(c), not c. The
+        # denominator cancels under the final L2 normalize; it is kept so the intermediate
+        # stays in the released rule's range and the degeneracy threshold means the same thing.
+        w = np.sqrt(counts, dtype=np.float32)
+        vec = (self.rows[uniq] * w[:, None]).sum(0) / max(float(w.sum()), EPS)
+        if float(np.linalg.norm(vec)) <= EPS:
+            return self._fallback
+        return self._normalize(vec).astype(np.float32)