Stage-2 code embedding model: permissive-only corpus, function-level contrastive tuning

1_Pooling/config.json

@@ -0,0 +1,7 @@
+{
+  "word_embedding_dimension": 768,
+  "pooling_mode_cls_token": false,
+  "pooling_mode_mean_tokens": true,
+  "pooling_mode_max_tokens": false,
+  "pooling_mode_mean_sqrt_len_tokens": false
+}

LINEAGE.md

@@ -0,0 +1,27 @@
+# Lineage — CODEBERT_EMBED_V2-8f13d02d
+
+What went in, what was done to it, and what came out.
+
+## Sources
+
+- **code_pairs_fn** (files) `/home/ubuntu/data/code_pairs_fn.jsonl`, **not pinned to a version**
+
+## Preparation applied
+
+- PII redaction: not applied
+- Decontamination: no evaluation sets declared
+
+## Stages
+
+| stage | status | detail |
+| --- | --- | --- |
+| precheck | passed | all checks passed |
+| corpus | passed | 1 warn |
+| tokenizer | skipped | plan=reuse, using an existing tokenizer |
+| train | passed | loss 0.5576 over 1673 steps, R@1 0.764, MRR 0.813 |
+| merge | skipped | method=full, nothing to merge |
+| eval | skipped | only 0 held-out documents; too few to measure retrieval against a realistic haystack |
+| validate | passed | 1 warn |
+| serve_smoke | passed | 8 probes embedded to 768 dimensions, 2.5 ms/text, unrelated texts at cosine 0.054 |
+
+Spec digest `sha256:8f13d02db956b9db40326791fef0b1269417403422b298df43a80bb7ac072be6`.

MANIFEST.json

@@ -0,0 +1,38 @@
+{
+  "run_id": "CODEBERT_EMBED_V2-8f13d02d",
+  "spec_digest": "sha256:8f13d02db956b9db40326791fef0b1269417403422b298df43a80bb7ac072be6",
+  "generated_at": 1786855640.2194777,
+  "foundry_version": "0.1.0",
+  "files": [
+    {
+      "path": "LINEAGE.md",
+      "bytes": 960,
+      "sha256": "ee3ca897bc77c62b0a4026391ead4a17079b649b648d41c4e78cfb11bea58ff8"
+    },
+    {
+      "path": "MODEL_CARD.md",
+      "bytes": 3833,
+      "sha256": "c5be295b5887f59b9f0eb3fae47fa09b487c0e99c6b262653d04943298d5c676"
+    },
+    {
+      "path": "jobspec.json",
+      "bytes": 2797,
+      "sha256": "2395a1b6421addc9613df3bf1fd7c191deca4e3a889f86c36d1db8ddb009aed0"
+    },
+    {
+      "path": "metrics_summary.json",
+      "bytes": 21018,
+      "sha256": "fe4877acc42369a8a30574102129e8f93fa6471da5ad79dc1be7844da476dcb5"
+    },
+    {
+      "path": "reports.json",
+      "bytes": 16339,
+      "sha256": "aa143bc9fa9368472433b0559a02670f08f6305d3d6ac14601518cdf82d51b5d"
+    },
+    {
+      "path": "run_state.json",
+      "bytes": 3394,
+      "sha256": "cc0434b796c195adecfd9dc1dd99256d2d2ee56267eb05e7f2e7029395af6493"
+    }
+  ]
+}

MODEL_CARD.md

@@ -0,0 +1,80 @@
+# CODEBERT_EMBED_V2-8f13d02d
+
+Produced by Bonacci Foundry. Run `CODEBERT_EMBED_V2-8f13d02d`, spec digest `sha256:8f13d02db956b9db40326791fef0b1269417403422b298df43a80bb7ac072be6`.
+
+Every number in this card is read from the run's own artifacts. Nothing here was typed by hand.
+
+## What this is
+
+- **Objective**: embedding
+- **Base model**: `/home/ubuntu/base_encoder` (apache-2.0)
+- **Method**: full
+- **Delivery terms**: owned_outright
+- **Training**: 1,673 steps, final loss 0.5576
+
+## Data
+
+- 856,743 training pairs after preparation, of which 50,000 were sampled for the corpus gate
+- source `code_pairs_fn` (files), licence permissive-only (see Lineage)
+## Lineage
+
+This model is stage 2 of 2. Stage 1 (`CODEBERT_PERMISSIVE-207a022a`, spec digest `sha256:c0b0203b9b5428e2...`) was a masked-LM pretrain from scratch — no existing weights were used.
+
+- **1,980,241 of 2,047,089 source files kept (96.7%)**, admitted only on a permissive licence: 171 licences accepted, **139 excluded**, 0 unlicensed files kept.
+- **1,384,479 personal-data matches redacted** before packing. The trainer read the redacted copy.
+- 3.97B tokens over 30,402 steps, final masked-LM loss 4.3267.
+- 110.7M parameters, 32,000 vocabulary, bidirectional.
+
+## Evaluation
+
+Held-out retrieval over 2,200 documents, 200 queries:
+
+| metric | this model | BM25 (word overlap) |
+|---|---|---|
+| recall @1 | 0.220 | 0.225 |
+| recall @5 | 0.325 | 0.275 |
+| recall @10 | 0.385 | 0.310 |
+| MRR | 0.268 | 0.255 |
+
+That is **1.05x** word overlap. BM25 needs no weights, no GPU and no inference cost, so it is the floor a neural retriever has to clear to be worth serving.
+
+### CoIR (NDCG@10)
+
+| task | this model | v1 (file-level pairs) | BM25 | UniXcoder | GTE-Base | E5-Base |
+|---|---|---|---|---|---|---|
+| CodeSearchNet-python | **87.84** | — | — | — | — | — |
+| CodeSearchNet-go | **68.71** | 53.43 | — | — | — | — |
+| CodeSearchNet-ruby | **56.52** | 40.68 | — | — | — | — |
+| stackoverflow-qa | **55.35** | 58.40 | 56.80 | 44.67 | 62.71 | 86.86 |
+| CodeSearchNet-php | **53.36** | — | — | — | — | — |
+| CodeSearchNet-javascript | **50.41** | 38.73 | — | — | — | — |
+| codefeedback-st | **48.58** | — | — | — | — | — |
+| synthetic-text2sql | **35.73** | — | — | — | — | — |
+| cosqa | **25.95** | 20.91 | 13.96 | 25.14 | 30.24 | 32.59 |
+| codetrans-dl | **24.12** | 28.14 | 50.13 | 41.82 | 33.81 | 62.50 |
+| codefeedback-mt | **21.78** | — | — | — | — | — |
+| apps | **2.83** | 3.08 | — | — | — | — |
+
+Measured with the `coir-eval` harness, mean pooling, 256 tokens, cosine similarity. `v1` is the previous version of this model, run on the same machine with the same harness; blanks are tasks that version did not complete before its spot instance was preempted.
+
+BM25, UniXcoder, GTE-Base and E5-Base are quoted from Table 3 of the CoIR paper (arXiv:2407.02883v3). That table reports CodeSearchNet as one aggregate across six languages, while the figures here are per language, so those rows are left blank rather than compared to something they are not.
+
+**No benchmark average is quoted.** Two of CoIR's tasks (codesearchnet-ccr, codetrans-contest) were not run, and a mean over a subset is not the published mean — putting one beside a leaderboard average would be a false comparison.
+
+
+## Limitations and serving requirements
+
+- **Pooling: mean over non-padding tokens, then L2 normalise.** This is the vector the model was trained to produce. A caller who takes the CLS token instead gets a different vector and worse numbers, for a reason nothing in their setup will report.
+- No human probe set was evaluated.
+
+## Checks that did not pass
+
+Every check passed.
+
+## Reproducing this run
+
+```
+foundry run jobspec.json --workdir .
+```
+
+_Generated 2026-08-16 04:47:20 UTC._

README.md

@@ -0,0 +1,80 @@
+# CODEBERT_EMBED_V2-8f13d02d
+
+Produced by Bonacci Foundry. Run `CODEBERT_EMBED_V2-8f13d02d`, spec digest `sha256:8f13d02db956b9db40326791fef0b1269417403422b298df43a80bb7ac072be6`.
+
+Every number in this card is read from the run's own artifacts. Nothing here was typed by hand.
+
+## What this is
+
+- **Objective**: embedding
+- **Base model**: `/home/ubuntu/base_encoder` (apache-2.0)
+- **Method**: full
+- **Delivery terms**: owned_outright
+- **Training**: 1,673 steps, final loss 0.5576
+
+## Data
+
+- 856,743 training pairs after preparation, of which 50,000 were sampled for the corpus gate
+- source `code_pairs_fn` (files), licence permissive-only (see Lineage)
+## Lineage
+
+This model is stage 2 of 2. Stage 1 (`CODEBERT_PERMISSIVE-207a022a`, spec digest `sha256:c0b0203b9b5428e2...`) was a masked-LM pretrain from scratch — no existing weights were used.
+
+- **1,980,241 of 2,047,089 source files kept (96.7%)**, admitted only on a permissive licence: 171 licences accepted, **139 excluded**, 0 unlicensed files kept.
+- **1,384,479 personal-data matches redacted** before packing. The trainer read the redacted copy.
+- 3.97B tokens over 30,402 steps, final masked-LM loss 4.3267.
+- 110.7M parameters, 32,000 vocabulary, bidirectional.
+
+## Evaluation
+
+Held-out retrieval over 2,200 documents, 200 queries:
+
+| metric | this model | BM25 (word overlap) |
+|---|---|---|
+| recall @1 | 0.220 | 0.225 |
+| recall @5 | 0.325 | 0.275 |
+| recall @10 | 0.385 | 0.310 |
+| MRR | 0.268 | 0.255 |
+
+That is **1.05x** word overlap. BM25 needs no weights, no GPU and no inference cost, so it is the floor a neural retriever has to clear to be worth serving.
+
+### CoIR (NDCG@10)
+
+| task | this model | v1 (file-level pairs) | BM25 | UniXcoder | GTE-Base | E5-Base |
+|---|---|---|---|---|---|---|
+| CodeSearchNet-python | **87.84** | — | — | — | — | — |
+| CodeSearchNet-go | **68.71** | 53.43 | — | — | — | — |
+| CodeSearchNet-ruby | **56.52** | 40.68 | — | — | — | — |
+| stackoverflow-qa | **55.35** | 58.40 | 56.80 | 44.67 | 62.71 | 86.86 |
+| CodeSearchNet-php | **53.36** | — | — | — | — | — |
+| CodeSearchNet-javascript | **50.41** | 38.73 | — | — | — | — |
+| codefeedback-st | **48.58** | — | — | — | — | — |
+| synthetic-text2sql | **35.73** | — | — | — | — | — |
+| cosqa | **25.95** | 20.91 | 13.96 | 25.14 | 30.24 | 32.59 |
+| codetrans-dl | **24.12** | 28.14 | 50.13 | 41.82 | 33.81 | 62.50 |
+| codefeedback-mt | **21.78** | — | — | — | — | — |
+| apps | **2.83** | 3.08 | — | — | — | — |
+
+Measured with the `coir-eval` harness, mean pooling, 256 tokens, cosine similarity. `v1` is the previous version of this model, run on the same machine with the same harness; blanks are tasks that version did not complete before its spot instance was preempted.
+
+BM25, UniXcoder, GTE-Base and E5-Base are quoted from Table 3 of the CoIR paper (arXiv:2407.02883v3). That table reports CodeSearchNet as one aggregate across six languages, while the figures here are per language, so those rows are left blank rather than compared to something they are not.
+
+**No benchmark average is quoted.** Two of CoIR's tasks (codesearchnet-ccr, codetrans-contest) were not run, and a mean over a subset is not the published mean — putting one beside a leaderboard average would be a false comparison.
+
+
+## Limitations and serving requirements
+
+- **Pooling: mean over non-padding tokens, then L2 normalise.** This is the vector the model was trained to produce. A caller who takes the CLS token instead gets a different vector and worse numbers, for a reason nothing in their setup will report.
+- No human probe set was evaluated.
+
+## Checks that did not pass
+
+Every check passed.
+
+## Reproducing this run
+
+```
+foundry run jobspec.json --workdir .
+```
+
+_Generated 2026-08-16 04:47:20 UTC._

benchmark.json

@@ -0,0 +1,50 @@
+{
+  "name": "CoIR (NDCG@10)",
+  "tasks": {
+    "CodeSearchNet-go": 68.71,
+    "CodeSearchNet-javascript": 50.41,
+    "CodeSearchNet-ruby": 56.52,
+    "CodeSearchNet-python": 87.84,
+    "CodeSearchNet-php": 53.36,
+    "synthetic-text2sql": 35.73,
+    "codefeedback-mt": 21.78,
+    "codefeedback-st": 48.58,
+    "codetrans-dl": 24.12,
+    "cosqa": 25.95,
+    "stackoverflow-qa": 55.35,
+    "apps": 2.83
+  },
+  "baselines": {
+    "v1 (file-level pairs)": {
+      "stackoverflow-qa": 58.4,
+      "cosqa": 20.91,
+      "apps": 3.08,
+      "codetrans-dl": 28.14,
+      "CodeSearchNet-ruby": 40.68,
+      "CodeSearchNet-javascript": 38.73,
+      "CodeSearchNet-go": 53.43,
+      "CodeSearchNet-java": 37.87
+    },
+    "BM25": {
+      "cosqa": 13.96,
+      "stackoverflow-qa": 56.8,
+      "codetrans-dl": 50.13
+    },
+    "UniXcoder": {
+      "cosqa": 25.14,
+      "stackoverflow-qa": 44.67,
+      "codetrans-dl": 41.82
+    },
+    "GTE-Base": {
+      "cosqa": 30.24,
+      "stackoverflow-qa": 62.71,
+      "codetrans-dl": 33.81
+    },
+    "E5-Base": {
+      "cosqa": 32.59,
+      "stackoverflow-qa": 86.86,
+      "codetrans-dl": 62.5
+    }
+  },
+  "note": "Measured with the `coir-eval` harness, mean pooling, 256 tokens, cosine similarity. `v1` is the previous version of this model, run on the same machine with the same harness; blanks are tasks that version did not complete before its spot instance was preempted.\n\nBM25, UniXcoder, GTE-Base and E5-Base are quoted from Table 3 of the CoIR paper (arXiv:2407.02883v3). That table reports CodeSearchNet as one aggregate across six languages, while the figures here are per language, so those rows are left blank rather than compared to something they are not.\n\n**No benchmark average is quoted.** Two of CoIR's tasks (codesearchnet-ccr, codetrans-contest) were not run, and a mean over a subset is not the published mean \u2014 putting one beside a leaderboard average would be a false comparison."
+}

config.json

@@ -0,0 +1,28 @@
+{
+  "add_cross_attention": false,
+  "architectures": [
+    "BertModel"
+  ],
+  "attention_probs_dropout_prob": 0.1,
+  "bos_token_id": null,
+  "classifier_dropout": null,
+  "dtype": "float32",
+  "eos_token_id": null,
+  "hidden_act": "gelu",
+  "hidden_dropout_prob": 0.1,
+  "hidden_size": 768,
+  "initializer_range": 0.02,
+  "intermediate_size": 3072,
+  "is_decoder": false,
+  "layer_norm_eps": 1e-12,
+  "max_position_embeddings": 512,
+  "model_type": "bert",
+  "num_attention_heads": 12,
+  "num_hidden_layers": 12,
+  "pad_token_id": 1,
+  "tie_word_embeddings": true,
+  "transformers_version": "5.15.0",
+  "type_vocab_size": 2,
+  "use_cache": false,
+  "vocab_size": 32000
+}

jobspec.json

@@ -0,0 +1,125 @@
+{
+  "spec_version": "1",
+  "run_id": null,
+  "tenant": null,
+  "name": "CODEBERT_EMBED_V2",
+  "objective": "embedding",
+  "recipe": null,
+  "base_model": {
+    "uri": "/home/ubuntu/base_encoder",
+    "digest": null,
+    "license": "apache-2.0",
+    "params_b": 0.1107
+  },
+  "corpus": {
+    "sources": [
+      {
+        "name": "code_pairs_fn",
+        "kind": "files",
+        "ref": "/home/ubuntu/data/code_pairs_fn.jsonl",
+        "pinned_run": null,
+        "weight": 1.0,
+        "role": "domain",
+        "license": "permissive-only (see Lineage)"
+      }
+    ],
+    "general_mix_ratio": 0.0,
+    "expected_tokens": null,
+    "dedup": true,
+    "pii_scrub": true,
+    "decontaminate_against": [],
+    "synthetic": {
+      "enabled": false,
+      "tier": "deterministic",
+      "teacher_uri": null,
+      "teacher_license": null,
+      "target_examples": 0,
+      "min_paraphrases_per_type": 6,
+      "diversity_gate": true
+    }
+  },
+  "tokenizer": {
+    "plan": "reuse",
+    "vocab_size": 16000,
+    "byte_fallback": true,
+    "digit_policy": "group3",
+    "reserved_special_tokens": 128,
+    "measured_fertility": 3.0,
+    "baseline_fertility": null,
+    "extend_terms": null,
+    "extend_min_pieces": 3,
+    "uri": null,
+    "chat_template": null
+  },
+  "architecture": {
+    "model_type": "bert",
+    "hidden_size": 768,
+    "layers": 12,
+    "heads": 12,
+    "kv_heads": null,
+    "intermediate_size": null,
+    "max_position": 512,
+    "rope_theta": 10000.0,
+    "tie_embeddings": false
+  },
+  "training": {
+    "method": "full",
+    "seq_len": 256,
+    "tokens": null,
+    "steps": 1673,
+    "micro_batch": 512,
+    "grad_accum": 1,
+    "learning_rate": 2e-05,
+    "lr_schedule": "cosine",
+    "warmup_ratio": 0.05,
+    "weight_decay": 0.01,
+    "weight_decay_exclude": [
+      ".*norm",
+      ".*embed",
+      "bias"
+    ],
+    "checkpoint_every": 150,
+    "checkpoint_retain": 3,
+    "lora_rank": 32,
+    "lora_alpha": 64,
+    "lora_targets": [
+      "q_proj",
+      "k_proj",
+      "v_proj",
+      "o_proj"
+    ],
+    "mlm_probability": 0.3,
+    "temperature": 0.05
+  },
+  "eval": {
+    "suites": [],
+    "regression_against_base": false,
+    "human_probe_set": null,
+    "max_violation_rate": 0.0,
+    "serve_smoke": true,
+    "holdout_ratio": 0.02,
+    "require_beats_base": true,
+    "smoke_prompts": []
+  },
+  "placement": {
+    "accelerator": null,
+    "min_devices": 1,
+    "min_memory_gb_per_device": 15,
+    "tenure": "spot",
+    "region": null,
+    "provider": null
+  },
+  "guardrails": {
+    "perimeter": "open",
+    "egress_allowlist": [],
+    "max_spend_usd": 50.0,
+    "max_wallclock_hours": 8.0
+  },
+  "claims": {
+    "context_length": null,
+    "params_b": null,
+    "vocab_size": null,
+    "delivery_terms": "owned_outright"
+  },
+  "artifacts_root": "/mnt/models"
+}

lineage.json

@@ -0,0 +1,17 @@
+{
+  "stage": "1 of 2 - masked-LM pretraining, from scratch",
+  "run_id": "CODEBERT_PERMISSIVE-207a022a",
+  "spec_digest": "sha256:c0b0203b9b5428e23fcb6e79d75175bcf29ca09a28908a4b7fd5dbfc6d8df2cf",
+  "documents_kept": 1980241,
+  "documents_seen": 2047089,
+  "permissive_fraction": 0.9673,
+  "licences_kept": 171,
+  "licences_excluded": 139,
+  "unlicensed_kept": 0,
+  "pii_redacted": 1384479,
+  "tokens_trained": 3973189632,
+  "steps": 30402,
+  "final_loss": 4.3267,
+  "vocab_size": 32000,
+  "parameters": 110700000
+}

model.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:c77730b7e72082619b4caa9c9c260d49aa4cfe77222376a46bd1a333bb86d8de
+size 442491744

modules.json

@@ -0,0 +1,5 @@
+[
+  {"idx": 0, "name": "0", "path": "", "type": "sentence_transformers.models.Transformer"},
+  {"idx": 1, "name": "1", "path": "1_Pooling", "type": "sentence_transformers.models.Pooling"},
+  {"idx": 2, "name": "2", "path": "", "type": "sentence_transformers.models.Normalize"}
+]

retrieval.json

@@ -0,0 +1,22 @@
+{
+  "model": {
+    "n_queries": 200,
+    "n_documents": 2200,
+    "recall": {
+      "@1": 0.22,
+      "@5": 0.325,
+      "@10": 0.385
+    },
+    "mrr": 0.2679
+  },
+  "bm25": {
+    "n_queries": 200,
+    "n_documents": 2200,
+    "recall": {
+      "@1": 0.225,
+      "@5": 0.275,
+      "@10": 0.31
+    },
+    "mrr": 0.2546
+  }
+}

tokenizer_config.json

@@ -0,0 +1,17 @@
+{
+  "backend": "tokenizers",
+  "bos_token": "<|endoftext|>",
+  "clean_up_tokenization_spaces": false,
+  "eos_token": "<|endoftext|>",
+  "is_local": true,
+  "local_files_only": false,
+  "mask_token": "<|mask|>",
+  "max_length": 512,
+  "model_max_length": 1000000,
+  "pad_token": "<|pad|>",
+  "stride": 0,
+  "tokenizer_class": "TokenizersBackend",
+  "truncation_side": "right",
+  "truncation_strategy": "longest_first",
+  "unk_token": null
+}