Clarify held-out-source fertility methodology

README.md

@@ -176,8 +176,8 @@ was fractionally lower at checkpoint 3564.
 The tokenizer is an in-place, primarily **Latin-script Uzbek** retrofit rather
 than a vocabulary extension. The initial 20,000-document figure was measured on
 training-source `uz-crawl`, so we replaced it with a larger corpus-stratified
-test: 118,832 independently sourced documents plus a separate 100,000-document
-in-domain control. Documents were selected with deterministic SHA-256 bottom-k
+test: 118,832 held-out-source documents plus a separate 100,000-document
+training-source control. Documents were selected with deterministic SHA-256 bottom-k
 sampling (seed `20260825`), exact duplicates were excluded from the selected
 sample, tiny texts were filtered, and raw source text was tokenized without
 apostrophe normalization.
@@ -186,11 +186,11 @@ apostrophe normalization.
 
 | Corpus | Status | Documents | Words | NeuronAI-4B | Qwen3.5-4B | Reduction (95% CI) |
 | --- | --- | ---: | ---: | ---: | ---: | ---: |
-| Community OSCAR Uzbek | Out-of-domain web | 100,000 | 7,618,770 | **2.0304** | 3.3639 | **39.64%** (39.57–39.71%) |
-| Uzbek legal corpus | Out-of-domain legal | 18,832 | 2,534,566 | **2.3747** | 2.9705 | **20.06%** (19.55–20.57%) |
-| uz-crawl | In-domain control | 100,000 | 20,825,680 | **2.3206** | 3.3224 | **30.15%** (30.02–30.30%) |
+| Community OSCAR Uzbek | Held-out web source | 100,000 | 7,618,770 | **2.0304** | 3.3639 | **39.64%** (39.57–39.71%) |
+| Uzbek legal corpus | Held-out legal source/domain | 18,832 | 2,534,566 | **2.3747** | 2.9705 | **20.06%** (19.55–20.57%) |
+| uz-crawl | Training-source control | 100,000 | 20,825,680 | **2.3206** | 3.3224 | **30.15%** (30.02–30.30%) |
 
-Across the two out-of-domain sources combined, the tokenizer uses **35.19%
+Across the two held-out sources combined, the tokenizer uses **35.19%
 fewer tokens overall**. The script split is essential: it uses **40.90% fewer
 tokens on Latin-dominant text**, but only **0.15% fewer on Cyrillic-dominant
 text**. NeuronAI-4B should therefore be described as Latin-Uzbek optimized, not
@@ -198,8 +198,10 @@ equally optimized for both Uzbek scripts.
 
 The paired intervals use 5,000 bootstrap replicates over 1,000 deterministic
 document buckets. OSCAR may still have incidental overlap with other public web
-corpora; the legal corpus, which does not appear in the tokenizer or training
-source manifests, is the cleanest domain holdout in this test. Full results and
+corpora and was previously checked in a post-hoc weak-token coverage analysis,
+but it contributed no tokenizer-training rows. The legal corpus does not appear
+in the tokenizer or training source manifests and is the cleanest source-and-domain
+holdout in this test. Full results and
 script/length breakdowns: [`fertility_large_20260825.json`](fertility_large_20260825.json)
 and [`fertility_large_20260825.md`](fertility_large_20260825.md).
 

fertility_large_20260825.json

@@ -17,7 +17,7 @@
   },
   "corpora": {
     "OSCAR Uzbek": {
-      "domain_status": "out-of-domain web",
+      "domain_status": "held-out web source",
       "source": "xkas2001/uzbek-language-dataset, community-oscar-uzbek/all_metadata_text.txt",
       "sampling": {
         "records_scanned": 4762756,
@@ -239,7 +239,7 @@
       }
     },
     "Uzbek legal corpus": {
-      "domain_status": "out-of-domain legal",
+      "domain_status": "held-out legal source/domain",
       "source": "sukhrobnurali/uzbek-legal-corpus-v1, articles",
       "sampling": {
         "records_scanned": 21021,
@@ -438,7 +438,7 @@
       }
     },
     "uz-crawl control": {
-      "domain_status": "in-domain control",
+      "domain_status": "training-source control",
       "source": "tahrirchi/uz-crawl local training source",
       "sampling": {
         "records_scanned": 1618035,

fertility_large_20260825.md

@@ -7,13 +7,14 @@ The confidence interval is a paired 5,000-replicate bootstrap over 1,000 determi
 
 | Corpus | Domain status | Documents | Words | NeuronAI-4B | Qwen3.5-4B | Token reduction (95% CI) |
 | --- | --- | ---: | ---: | ---: | ---: | ---: |
-| OSCAR Uzbek | out-of-domain web | 100,000 | 7,618,770 | 2.0304 | 3.3639 | 39.64% (39.57%–39.71%) |
-| Uzbek legal corpus | out-of-domain legal | 18,832 | 2,534,566 | 2.3747 | 2.9705 | 20.06% (19.55%–20.57%) |
-| uz-crawl control | in-domain control | 100,000 | 20,825,680 | 2.3206 | 3.3224 | 30.15% (30.02%–30.30%) |
+| OSCAR Uzbek | held-out web source | 100,000 | 7,618,770 | 2.0304 | 3.3639 | 39.64% (39.57%–39.71%) |
+| Uzbek legal corpus | held-out legal source/domain | 18,832 | 2,534,566 | 2.3747 | 2.9705 | 20.06% (19.55%–20.57%) |
+| uz-crawl control | training-source control | 100,000 | 20,825,680 | 2.3206 | 3.3224 | 30.15% (30.02%–30.30%) |
 
 ## Interpretation
 
-- The out-of-domain corpora are the fairness readout; the in-domain corpus is retained only as a control.
+- The held-out-source corpora are the fairness readout; the training-source corpus is retained only as a control.
 - Exact web overlap cannot be ruled out for OSCAR because both corpora ultimately derive from public web data.
-- The legal corpus is absent from the tokenizer and CPT source manifests and is the cleanest domain holdout in this run.
+- OSCAR was checked in a post-hoc weak-token coverage analysis but contributed no tokenizer-training rows.
+- The legal corpus is absent from the tokenizer and CPT source manifests and is the cleanest source-and-domain holdout in this run.
 - Results use raw source text without apostrophe normalization or other model-specific preprocessing.