Remove internal artifact MODEL_CARD.md
Browse files- MODEL_CARD.md +0 -80
MODEL_CARD.md
DELETED
|
@@ -1,80 +0,0 @@
|
|
| 1 |
-
# CODEBERT_EMBED_V2-8f13d02d
|
| 2 |
-
|
| 3 |
-
Produced by Bonacci Foundry. Run `CODEBERT_EMBED_V2-8f13d02d`, spec digest `sha256:8f13d02db956b9db40326791fef0b1269417403422b298df43a80bb7ac072be6`.
|
| 4 |
-
|
| 5 |
-
Every number in this card is read from the run's own artifacts. Nothing here was typed by hand.
|
| 6 |
-
|
| 7 |
-
## What this is
|
| 8 |
-
|
| 9 |
-
- **Objective**: embedding
|
| 10 |
-
- **Base model**: `/home/ubuntu/base_encoder` (apache-2.0)
|
| 11 |
-
- **Method**: full
|
| 12 |
-
- **Delivery terms**: owned_outright
|
| 13 |
-
- **Training**: 1,673 steps, final loss 0.5576
|
| 14 |
-
|
| 15 |
-
## Data
|
| 16 |
-
|
| 17 |
-
- 856,743 training pairs after preparation, of which 50,000 were sampled for the corpus gate
|
| 18 |
-
- source `code_pairs_fn` (files), licence permissive-only (see Lineage)
|
| 19 |
-
## Lineage
|
| 20 |
-
|
| 21 |
-
This model is stage 2 of 2. Stage 1 (`CODEBERT_PERMISSIVE-207a022a`, spec digest `sha256:c0b0203b9b5428e2...`) was a masked-LM pretrain from scratch — no existing weights were used.
|
| 22 |
-
|
| 23 |
-
- **1,980,241 of 2,047,089 source files kept (96.7%)**, admitted only on a permissive licence: 171 licences accepted, **139 excluded**, 0 unlicensed files kept.
|
| 24 |
-
- **1,384,479 personal-data matches redacted** before packing. The trainer read the redacted copy.
|
| 25 |
-
- 3.97B tokens over 30,402 steps, final masked-LM loss 4.3267.
|
| 26 |
-
- 110.7M parameters, 32,000 vocabulary, bidirectional.
|
| 27 |
-
|
| 28 |
-
## Evaluation
|
| 29 |
-
|
| 30 |
-
Held-out retrieval over 2,200 documents, 200 queries:
|
| 31 |
-
|
| 32 |
-
| metric | this model | BM25 (word overlap) |
|
| 33 |
-
|---|---|---|
|
| 34 |
-
| recall @1 | 0.220 | 0.225 |
|
| 35 |
-
| recall @5 | 0.325 | 0.275 |
|
| 36 |
-
| recall @10 | 0.385 | 0.310 |
|
| 37 |
-
| MRR | 0.268 | 0.255 |
|
| 38 |
-
|
| 39 |
-
That is **1.05x** word overlap. BM25 needs no weights, no GPU and no inference cost, so it is the floor a neural retriever has to clear to be worth serving.
|
| 40 |
-
|
| 41 |
-
### CoIR (NDCG@10)
|
| 42 |
-
|
| 43 |
-
| task | this model | v1 (file-level pairs) | BM25 | UniXcoder | GTE-Base | E5-Base |
|
| 44 |
-
|---|---|---|---|---|---|---|
|
| 45 |
-
| CodeSearchNet-python | **87.84** | — | — | — | — | — |
|
| 46 |
-
| CodeSearchNet-go | **68.71** | 53.43 | — | — | — | — |
|
| 47 |
-
| CodeSearchNet-ruby | **56.52** | 40.68 | — | — | — | — |
|
| 48 |
-
| stackoverflow-qa | **55.35** | 58.40 | 56.80 | 44.67 | 62.71 | 86.86 |
|
| 49 |
-
| CodeSearchNet-php | **53.36** | — | — | — | — | — |
|
| 50 |
-
| CodeSearchNet-javascript | **50.41** | 38.73 | — | — | — | — |
|
| 51 |
-
| codefeedback-st | **48.58** | — | — | — | — | — |
|
| 52 |
-
| synthetic-text2sql | **35.73** | — | — | — | — | — |
|
| 53 |
-
| cosqa | **25.95** | 20.91 | 13.96 | 25.14 | 30.24 | 32.59 |
|
| 54 |
-
| codetrans-dl | **24.12** | 28.14 | 50.13 | 41.82 | 33.81 | 62.50 |
|
| 55 |
-
| codefeedback-mt | **21.78** | — | — | — | — | — |
|
| 56 |
-
| apps | **2.83** | 3.08 | — | — | — | — |
|
| 57 |
-
|
| 58 |
-
Measured with the `coir-eval` harness, mean pooling, 256 tokens, cosine similarity. `v1` is the previous version of this model, run on the same machine with the same harness; blanks are tasks that version did not complete before its spot instance was preempted.
|
| 59 |
-
|
| 60 |
-
BM25, UniXcoder, GTE-Base and E5-Base are quoted from Table 3 of the CoIR paper (arXiv:2407.02883v3). That table reports CodeSearchNet as one aggregate across six languages, while the figures here are per language, so those rows are left blank rather than compared to something they are not.
|
| 61 |
-
|
| 62 |
-
**No benchmark average is quoted.** Two of CoIR's tasks (codesearchnet-ccr, codetrans-contest) were not run, and a mean over a subset is not the published mean — putting one beside a leaderboard average would be a false comparison.
|
| 63 |
-
|
| 64 |
-
|
| 65 |
-
## Limitations and serving requirements
|
| 66 |
-
|
| 67 |
-
- **Pooling: mean over non-padding tokens, then L2 normalise.** This is the vector the model was trained to produce. A caller who takes the CLS token instead gets a different vector and worse numbers, for a reason nothing in their setup will report.
|
| 68 |
-
- No human probe set was evaluated.
|
| 69 |
-
|
| 70 |
-
## Checks that did not pass
|
| 71 |
-
|
| 72 |
-
Every check passed.
|
| 73 |
-
|
| 74 |
-
## Reproducing this run
|
| 75 |
-
|
| 76 |
-
```
|
| 77 |
-
foundry run jobspec.json --workdir .
|
| 78 |
-
```
|
| 79 |
-
|
| 80 |
-
_Generated 2026-08-16 04:47:20 UTC._
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|