Opt-in --bid-items mode: TF-IDF also trained on 2,842 real TxDOT bid items (item-held-out .489 -> .608)

IMPROVEMENT-PLAN.md

@@ -124,3 +124,62 @@ full sections; add an `other/<division>` fallback and score it honestly; revisit
 - More synthetic line-item augmentation (tested: `mf-0.3` — no line-item transfer, manual regression).
 - More TF-IDF feature engineering (already saturated; gains below the CI).
 - Reporting sub-CI improvements as wins on the model card.
+
+---
+
+## Progress — P0.1 acquisition started (2026-10-08)
+
+**Tooling added (this stage).**
+- `scripts/compare.py` — the gate the plan was missing: **paired bootstrap CI of the difference + exact
+  McNemar**, per set, with a `LIFT / DROP / no detectable change` verdict. `cloud/stats.py` had Wilson CIs
+  but no significance test, which is how sub-CI changes kept reading as wins.
+- `scripts/build_manual_eval_v2.py` — section eval over **every** local manual, with a train/val
+  leakage guard (drops any chunk whose text appears in the corpus).
+- `scripts/import_bid_tabulation.py` — pulls real line items from public bid tabulations.
+
+**What the sources actually hold (the hard part).**
+- The local AEC-Bench / ConstructBench cache has 18 documents, but **only 2 are project manuals**
+  (`spec.pdf` ×2 → Wenatchee 651 pp, Macon 807 pp); the rest are drawings, sheets and submittals. Building
+  over all 18 → **153 sections / 6,890 chunks, 0 leaked** — so the section eval is capped at ±8 pts by the
+  *source*, not by code. **≥400 sections needs more public project manuals.**
+- No public MasterFormat-labelled dataset exists: Hugging Face dataset search returns **0** for
+  `masterformat` / `construction specifications` / `CSI division`; Kaggle has none either.
+- Public bid tabulations are real but are labelled with **agency** codes, not CSI: `aniket0712/bid-tabulations`
+  (TxDOT, 960,791 rows) has `BID ITEM DESCRIPTION` + `SPECIFICATION CODE` (TxDOT's own numbering). Useful as
+  weak-label **training** text (P0.2) and as a **candidate pool** for labelling — not as a true-label eval.
+
+**Produced.** `eval/line_items_v2_candidates.tsv` — **2,000 unique real line items** (deduped, cleaned) with
+their source spec code and a blank `true_group` for a reviewer to fill. That is the automated half of P0.1's
+"≥1,000 real line items"; the labels are the human half.
+
+**Conclusion.** Both halves of P0.1 are blocked on **data and labelling, not modelling**: more public
+project manuals for sections, and human labels (or a licensed source) for line items. The mf-0.5 Kaggle run
+already confirmed the point — `val` rose (+2.3) while line items (−1.6) and sections (−1.3) fell, i.e. the
+auxiliary coarse-to-fine loss bought nothing on the target distribution.
+
+---
+
+## Result 2026-10-13 — P0.2 first pass: real bid items in the TF-IDF scorer (`runs/bidtab/`, NOT published)
+
+Data: full TxDOT bid tabulation (960k rows -> 13,239 unique item texts, `data/raw/txdot_bid_items_unique.jsonl`)
+labelled by the eval crosswalk + 45 training-only codes (`data/txdot_crosswalk_train.json`, titles taken from
+TxDOT's own SPECIFICATION DESCRIPTION column). After dropping every eval text and every eval text up to digits:
+2,842 training items, 19 groups. Weight x3 chosen on a dev split of training-only codes. Script:
+`scripts/bidtab_experiment.py`, results `runs/bidtab-exp.json`. Transformer unchanged (mf-0.2, blend w 0.25).
+
+| eval | published ensemble | + bid items | paired |
+|---|---:|---:|---|
+| TxDOT, **item numbers held out** (5-fold grouped CV), n 790 | 0.489 [.454,.523] | **0.608** [.573,.641] | 104 vs 10, p < 0.001 |
+| TxDOT capped <= 20/item, n 529 | 0.369 [.329,.411] | **0.492** [.449,.534] | 73 vs 8, p < 0.001 |
+| hand-labelled building line items, n 194 | 0.691 | 0.701 | 3 vs 1, p 0.63 |
+| manual chunks, n 6,890 | 0.326 | 0.329 | 65 vs 47, p 0.11 |
+| whole sections top-1 / division, n 153 | 0.620 / 0.804 | 0.613 / 0.791 | -1 / -2 sections |
+| UFGS val, n 11,598 | 0.5875 | 0.5836 | 61 vs 106, p < 0.001 |
+
+TxDOT with item numbers *seen* (texts deduped) is 0.953 — memorisation of item types, not a headline number.
+
+Gate check (§3): lift on real line items is large and significant on the agency set but not measurable on the
+194-item building set; **section division -1.3 pts misses the "within 1 pt" gate by 2 sections** (noise-level,
+but the gate was set in advance) and UFGS val drops 0.4 pts (significant). Decision left to the owner:
+ship as default, ship as an opt-in "bid-item" TF-IDF, or first enlarge the section eval (P0.1) to tell a
+2-section move from noise.

README.md

@@ -240,6 +240,36 @@ non-overlapping intervals. It is still well below the 70 % on hand-written build
 bid items are out of domain for a UFGS-trained model, and these labels come from an item-number crosswalk
 (reviewed, not per-item), covering civil work only.
 
+### Bid-item mode (`--bid-items`, opt-in)
+
+`ensemble/tfidf_bid.joblib` is the same TF-IDF scorer **also trained on 2,842 real TxDOT bid items**: the full
+public bid tabulation (960k rows → 13,239 unique descriptions), labelled through the crosswalk above plus 45
+training-only item codes (`txdot_crosswalk_train.json`, mapped from TxDOT's own specification titles). Every
+evaluation description — and every description equal to one after removing numbers ("TREE REMOVAL (4" - 12" DIA)"
+vs "(12" - 18" DIA)") — was removed from training. Bid items are weighted ×3, chosen on a dev split of
+training-only codes.
+
+```bash
+python ensemble/predict_ensemble.py --bid-items "CL C CONC (RAIL FOUNDATION)" "SET (TY II) (24 IN) (RCP) (6: 1) (P)"
+```
+
+The honest number is **item-held-out**: 5-fold cross-validation grouped by TxDOT item number, so each evaluation
+row is scored by a model that saw no bid item of its type (new item types, not paraphrases). Paired against the
+default on the same rows:
+
+| Ensemble on 790 TxDOT bid items | default | `--bid-items` | changed |
+|---|---:|---:|---|
+| top-1, item types held out | 0.489 [0.454, 0.523] | **0.608 [0.573, 0.641]** | 104 fixed, 10 broken (McNemar p < 0.001) |
+| top-1 capped ≤ 20/item (n = 529) | 0.369 [0.329, 0.411] | **0.492 [0.449, 0.534]** | 73 fixed, 8 broken |
+
+The shipped `tfidf_bid.joblib` has seen these item *types* (not these descriptions); on them it reaches 0.95 —
+that is memorised item types, not a measure of quality on new agencies' items.
+
+**Why it is opt-in.** On building text it is a wash or slightly worse: 194 hand-labelled line items 0.691 → 0.701
+(3 vs 1 changed), manual chunks 0.326 → 0.329, whole sections 0.620 → 0.613 top-1 and 0.804 → 0.791 division
+(2 of 153 sections), UFGS val 0.588 → 0.584. Use it for agency / DOT bid tabulations, keep the default for
+specifications and building estimates.
+
 ## Calibration and abstention
 
 The ensemble is already close to calibrated; a single **temperature** `T = 0.90` (fit on `val`) lowers the
@@ -342,9 +372,10 @@ manual-chunk and whole-section accuracy. Constructelligence's proprietary models
 - `predict.py` — CLI example: batching, `--top-k`, `--divisions`, `--json`, stdin/file input, `--onnx`.
 - `onnx/` — ONNX fp32 and int8 exports.
 - `ensemble/` — `tfidf.joblib` (word + char n-grams, ~112 MB), `blend.json`, `predict_ensemble.py`: the
-  transformer + TF-IDF line-item ensemble.
+  transformer + TF-IDF line-item ensemble; `tfidf_bid.joblib` (~115 MB) is the opt-in `--bid-items` scorer.
 - `abbrev.py` — bid-item abbreviation expansion used by both predictors (`--no-expand` to disable).
 - `txdot_crosswalk.json` — TxDOT item number → MasterFormat group table behind the bid-item evaluation.
+- `txdot_crosswalk_train.json` — 45 further TxDOT codes used only to label `--bid-items` training data.
 - `IMPROVEMENT-PLAN.md` — the prioritised roadmap (enlarged eval, data, domain-adaptive pretraining, distillation).
 - `CITATION.cff`, `requirements.txt`.
 

ensemble/predict_ensemble.py

@@ -10,6 +10,7 @@ manual-chunk accuracy). This script reproduces the blend. The weight is chosen o
     python predict_ensemble.py "EPDM membrane roofing" "4000 psi concrete slab on grade"
     python predict_ensemble.py --top-k 5 --divisions "12\" RCP storm drain pipe"
     python predict_ensemble.py --json --file items.txt > out.json
+    python predict_ensemble.py --bid-items "CL C CONC (RAIL FOUNDATION)" "RE PM W/RET REQ TY I (W)6\"(SLD)"   # agency bid tabs
 
     # default TF-IDF/results live beside this script; point --model at the Hub or a local run
     python predict_ensemble.py --model constructelligence/masterformat-classifier \
@@ -98,10 +99,14 @@ def main():
     ap.add_argument("--file")
     ap.add_argument("--json", action="store_true")
     ap.add_argument("--no-expand", action="store_true", help="do not expand bid-item abbreviations first")
+    ap.add_argument("--bid-items", action="store_true",
+                    help="agency bid-tab mode: use tfidf_bid.joblib (also trained on 2,842 real TxDOT bid items)")
     a = ap.parse_args()
+    if a.bid_items and a.tfidf == str(HERE / "tfidf.joblib"):
+        a.tfidf = str(HERE / "tfidf_bid.joblib")   # same blend weight (blend.json) as the default scorer
 
     if a.weight is None:
-        bf = Path(a.tfidf).with_name("blend.json")
+        bf = HERE / "blend.json" if a.bid_items else Path(a.tfidf).with_name("blend.json")
         a.weight = json.loads(bf.read_text())["weight"] if bf.exists() else 0.2
     w = a.weight
 

ensemble/tfidf_bid.joblib

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:ac597727d5baf9e563c1f2d5ecc771de8f437ba65fa1fa7d9b1e25e967c5a4f6
+size 115060552

metrics.json

@@ -70,5 +70,71 @@
     "division": 0.57
    }
   }
+ },
+ "txdot_bid_items_bidmode": {
+  "_doc": "ensemble/tfidf_bid.joblib (opt-in --bid-items); item-held-out = 5-fold CV grouped by TxDOT item number; paired vs default ensemble; building-set regression with the shipped model",
+  "item_held_out": {
+   "n": 790,
+   "top1": 0.6076,
+   "ci": [
+    0.573,
+    0.641
+   ],
+   "vs_published": {
+    "new_right_old_wrong": 104,
+    "old_right_new_wrong": 10,
+    "p": 0.0
+   }
+  },
+  "item_held_out_capped": {
+   "n": 529,
+   "top1": 0.4915,
+   "ci": [
+    0.449,
+    0.534
+   ],
+   "vs_published": {
+    "new_right_old_wrong": 73,
+    "old_right_new_wrong": 8,
+    "p": 0.0
+   }
+  },
+  "default_same_rows": {
+   "n": 790,
+   "top1": 0.4886,
+   "ci": [
+    0.454,
+    0.523
+   ]
+  },
+  "items_seen_optimistic": 0.9532,
+  "building_regression": {
+   "val": {
+    "default": 0.5875,
+    "bid_mode": 0.5836
+   },
+   "line_items": {
+    "default": 0.6907,
+    "bid_mode": 0.701
+   },
+   "manuals": {
+    "default": 0.3261,
+    "bid_mode": 0.3287
+   },
+   "manual_sections": {
+    "default": {
+     "sections": 153,
+     "top1": 0.62,
+     "division": 0.803921568627451
+    },
+    "bid_mode": {
+     "sections": 153,
+     "top1": 0.6133333333333333,
+     "division": 0.7908496732026143
+    }
+   }
+  },
+  "train_items": 2842,
+  "bid_weight": 3.0
  }
-}
+}

txdot_crosswalk_train.json

@@ -0,0 +1,50 @@
+{
+ "_doc": "TRAINING-ONLY additions to eval/txdot_crosswalk.json: TxDOT specification codes -> MasterFormat level-2 group, for labelling bid items from the full TxDOT bid tabulation (Kaggle aniket0712/bid-tabulations). Titles are TxDOT's own SPECIFICATION DESCRIPTION strings from that file, not recalled from memory. Same rules as the eval crosswalk: only titles that map to one of the 171 groups without doubt; maintenance (7xx), temporary work-zone, signs and signals (34 40 00 is not a class), mixed scopes (400 structure excavation, 423 retaining walls, 450 railing, 434 bearings, 476 jacking/boring, 533 rumble strips, 305 RAP, 401 flowable fill) and all 4-digit special specifications are left out. Grouping follows the eval crosswalk's own choices (erosion control -> 31 20 00 as item 506; removal of existing work -> 02 40 00 as items 104/496; vehicle barriers -> 34 70 00 as item 540). The eval file's 790 rows keep their original labels. Built 2026-10-13.",
+ "items": {
+  "112": ["SUBGRADE WIDENING", "31 20 00"],
+  "134": ["BACKFILLING PAVEMENT EDGES", "31 20 00"],
+  "158": ["SPECIALIZED EXCAVATION WORK", "31 20 00"],
+  "169": ["SOIL RETENTION BLANKETS", "31 20 00"],
+  "160": ["TOPSOIL", "32 90 00"],
+  "161": ["COMPOST", "32 90 00"],
+  "162": ["SODDING FOR EROSION CONTROL", "32 90 00"],
+  "166": ["FERTILIZER", "32 90 00"],
+  "180": ["WILDFLOWER SEEDING", "32 90 00"],
+  "192": ["LANDSCAPE PLANTING", "32 90 00"],
+  "170": ["IRRIGATION SYSTEM", "32 80 00"],
+  "251": ["REWORKING BASE COURSES", "32 10 00"],
+  "276": ["CEMENT TREATMENT (PLANT-MIXED)", "31 30 00"],
+  "310": ["PRIME COAT", "32 10 00"],
+  "315": ["FOG SEAL", "32 10 00"],
+  "334": ["HOT-MIX COLD-LAID ASPHALT CONCRETE PAVEMENT", "32 10 00"],
+  "340": ["DENSE-GRADED HOT-MIX ASPHALT (SMALL QUANTITY)", "32 10 00"],
+  "341": ["DENSE-GRADED HOT-MIX ASPHALT", "32 10 00"],
+  "342": ["PERMEABLE FRICTION COURSE", "32 10 00"],
+  "344": ["SUPERPAVE MIXTURES", "32 10 00"],
+  "346": ["STONE-MATRIX ASPHALT", "32 10 00"],
+  "347": ["THIN OVERLAY MIXTURES (TOM)", "32 10 00"],
+  "348": ["THIN BONDED FRICTION COURSES", "32 10 00"],
+  "407": ["STEEL PILING", "31 60 00"],
+  "409": ["PRESTRESSED CONCRETE PILING", "31 60 00"],
+  "427": ["SURFACE FINISHES FOR CONCRETE", "03 30 00"],
+  "442": ["METAL FOR STRUCTURES", "05 10 00"],
+  "459": ["GABIONS AND GABION MATTRESSES", "31 30 00"],
+  "460": ["CORRUGATED METAL PIPE", "33 40 00"],
+  "467": ["SAFETY END TREATMENT", "33 40 00"],
+  "468": ["THERMOPLASTIC PIPE CULVERTS AND DRAINS", "33 40 00"],
+  "471": ["FRAMES, GRATES, RINGS AND COVERS", "33 40 00"],
+  "474": ["LINEAR DRAINS", "33 40 00"],
+  "481": ["PIPE OR DRAINS", "33 40 00"],
+  "556": ["PIPE UNDERDRAINS", "33 40 00"],
+  "514": ["PERMANENT CONCRETE TRAFFIC BARRIER", "34 70 00"],
+  "543": ["CABLE BARRIER SYSTEM", "34 70 00"],
+  "542": ["REMOVING METAL BEAM GUARD FENCE", "02 40 00"],
+  "528": ["COLORED TEXTURED CONCRETE AND LANDSCAPE PAVERS", "32 10 00"],
+  "536": ["CONCRETE MEDIANS AND DIRECTIONAL ISLANDS", "32 10 00"],
+  "552": ["WIRE FENCE", "32 30 00"],
+  "613": ["HIGH MAST ILLUMINATION POLES", "26 50 00"],
+  "614": ["HIGH MAST ILLUMINATION ASSEMBLIES", "26 50 00"],
+  "621": ["TRAY CABLE", "26 00 00"],
+  "628": ["ELECTRICAL SERVICES", "26 00 00"]
+ }
+}