Add general-purpose baseline, paired significance test, and eval protocol

README.md

@@ -20,8 +20,7 @@ A 42.6M-parameter bi-encoder for retrieving **Linux kernel C** from
 natural-language queries. Queries and code go through the same encoder, with no
 prefix or instruction prompt.
 
-512-dim output, mean pooling, L2-normalised. `max_seq_length` is **320** and
-should not be raised — longer inputs degrade retrieval. Chunk longer functions.
+512-dim output, mean pooling, L2-normalised — score with cosine similarity.
 
 ## Usage
 
@@ -37,26 +36,85 @@ emb = model.encode([query, code], normalize_embeddings=True)
 print(emb @ emb.T)          # cosine similarity
 ```
 
+### Sequence length
+
+`max_seq_length` is **320**, and should not be raised even though
+`max_position_embeddings` is 512. Pretraining saw 512 tokens but the retrieval
+stage trained at 320, so positions 320–511 are undertrained; feeding longer
+inputs measurably degrades ranking. Chunk longer functions instead.
+
 ## Results
 
-Retrieval over the whole kernel — **914,554 candidate chunks**:
+Two different protocols. Read them separately — the candidate pools differ by
+two orders of magnitude, so the numbers are **not** comparable across tables.
+
+### 1. Open retrieval — the whole kernel
+
+Every `.c`/`.h` chunk of Linux v7.1-rc5 as the index (**914,554 candidates**),
+**N = 400** held-out queries, one correct answer each.
 
-| | dense | hybrid (+ BM25) |
+| | this model | + BM25 fusion |
 |---|---|---|
-| recall@1 | 0.8125 | **0.9050** |
-| recall@5 | 0.9375 | **0.9725** |
-| recall@10 | 0.9575 | **0.9775** |
-| recall@50 | 0.9850 | **0.9950** |
-| MRR | 0.8682 | **0.9374** |
-| median rank | 1 | 1 |
+| recall@1 | **0.8125** | 0.9050 |
+| recall@5 | 0.9375 | 0.9725 |
+| recall@10 | 0.9575 | 0.9775 |
+| recall@50 | 0.9850 | 0.9950 |
+| MRR | 0.8682 | 0.9374 |
 
-The correct function ranks first out of 914,554 candidates 90% of the time.
+**The model alone ranks the correct function first 81% of the time out of
+914,554 candidates** (95% CI ±3.8pp at N=400).
 
-Against a lexical baseline on a held-out set:
+The second column is reciprocal-rank fusion of this model with BM25. It is
+higher, but part of that lift is BM25's, so the 0.8125 figure is the one that
+belongs to this model.
 
-| | BM25 | this model |
-|---|---|---|
-| accuracy@1 | 0.7115 | **0.9225** |
-| NDCG@10 | 0.8297 | **0.9659** |
+### 2. Closed set — against other encoders
+
+**N = 2,000** held-out queries, 4,000 candidates, where each distractor is a
+sibling function from the same source file as the answer.
+
+| | params | accuracy@1 | NDCG@10 |
+|---|---|---|---|
+| BM25 (lexical) | — | 0.7115 | 0.8297 |
+| `jina-embeddings-v2-base-code` | 161M | 0.9035 | 0.9541 |
+| **this model** | **42.6M** | **0.9230** | **0.9661** |
+
+Scored with the identical query set, candidate pool and metric code. The
+general-purpose code embedder was given a 512-token budget — more than this
+model's 320.
+
+The +2.0pp accuracy@1 margin over `jina-embeddings-v2-base-code` is significant
+under a paired McNemar test: p = 0.003, 95% CI on the paired difference
++0.7pp to +3.4pp (N = 2,000). A 42.6M domain model edging out a 161M
+general-purpose one — at 3.8× fewer parameters — is the result worth having.
+
+## How the evaluation set was built
+
+This matters for reading the numbers above.
+
+Queries are **kernel-doc comments written by kernel developers**, not generated
+by a language model — so they are not distribution-matched to the training data
+by construction. They are held out from training. The **symbol name is stripped
+from the query**: left in, `kmalloc_node` would appear in both the query and the
+answer's signature and the task would collapse to identifier matching.
+
+Training used InfoNCE over in-batch negatives (batch size 24, softmax scale
+20.0), with hard negatives drawn as sibling functions from the same file.
+
+## Limitations
+
+- **Domain-specific.** Linux kernel C only. Not a general-purpose code or text
+  embedding model; do not expect transfer to other languages or codebases.
+- **Short context** — 320 tokens. Long functions must be chunked.
+- **kernel-doc phrasing.** Queries are developer-written documentation, which is
+  more precise than typical end-user questions. Expect lower accuracy on casual
+  or ambiguous phrasing.
+- **Single held-out split**, no seed variance reported.
+
+## License
 
-Trained on Linux kernel C only; not a general-purpose code embedding model.
+GPL-2.0, matching the Linux kernel corpus it was trained on. Whether a GPL
+training corpus propagates to model weights is legally unsettled; this is the
+conservative reading, chosen deliberately rather than by default. If you need
+different terms for commercial use, treat that as an open question to resolve
+with counsel rather than an answered one.