IMPROVEMENT-PLAN.md

# MasterFormat classifier — improvement plan

**Date:** 2026-10-05 · **Owner:** Constructelligence
**Published:** [`constructelligence/masterformat-classifier`](https://huggingface.co/constructelligence/masterformat-classifier)
**Current:** `mf-0.2` transformer + `ensemble/` (word+char TF-IDF blend).

| metric | published value | 95 % CI | n |
|---|---:|---|---:|
| line-item top-1 (hand-labelled estimates) | 0.702 | **[0.633, 0.762]** | 194 (191 in-taxonomy) |
| whole-section top-1 | 0.620 | **[0.540, 0.694]** | 153 |
| whole-section division | 0.804 | ±~6 pts | 153 |
| UFGS val top-1 | 0.588 | [0.579, 0.596] | 11,598 |
| manual-chunk top-1 | 0.330 | [0.319, 0.341] | 6,890 |

---

## 1. The binding constraint is evidence, not modelling

The two metrics that matter for the product — **hand-labelled line items** and **whole real spec sections** —
come from tiny samples. Their confidence intervals are **±6–8 points**, which means:

- the recent `0.686 → 0.702` ensemble change is **statistically indistinguishable from noise**;
- any model change smaller than ~10 points on line items cannot be validated on this eval;
- the tight, large-sample metric (`val`, ±0.9 pts) is **in-domain UFGS text**, not the target distribution —
  a model can improve `val` a lot (see `mf-0.3`: 0.466 → 0.556) while getting *worse* on real manuals.

**Consequence:** the next gains are gated on **a bigger, real-world evaluation set**, not on architecture.
Chasing sub-CI improvements (more TF-IDF features, more seeds) is a trap we have already fallen into once.

## 2. Where the headroom actually is

1. **Line items (estimating).** TF-IDF dominates; the transformer adds little. True ceiling unknown because
   the eval is underpowered. Needs *real* labelled items, not synthetic augmentation (`mf-0.3` proved
   synthetic terse augmentation does not transfer: line items stayed 0.597 and manuals regressed).
2. **Section-level *top-1*** (0.62) is the weak spot, while *division* (0.81) is strong → most residual error
   is **confusion within a division** (e.g. `03 30 00` vs `03 50 00`). A coarse-to-fine or contrastive
   approach targets exactly this.
3. **Out-of-taxonomy** sections only score at division level; there is no `other`/hierarchical fallback.
4. **No level-3 / full 8-digit section codes** — real spec books use them.

## 3. Definition of done (gates for any new model)

| Gate | Target | How to verify |
|---|---|---|
| Powered eval | line items ≥ 1,000; sections ≥ 400 | Wilson CI ≤ ±3 pts |
| Measurable lift | ≥ +3 pts line-item top-1 vs published, 95 % CI non-overlapping | paired bootstrap / McNemar |
| No regression | manuals + section division within 1 pt | fixed eval sets |
| Calibrated | ECE ≤ 0.05 after temperature scaling | reliability curve |
| Useful abstention | ≥ 90 % precision at ≥ 50 % coverage on line items | coverage–precision curve |
| Reproducible | one `cloud/run_kaggle.py --run <name>` + seed | identical metrics ± noise |

## 4. Plan

### P0 — Make the measurement trustworthy, then feed it real data

**P0.1 — Enlarge the held-out eval (do first).**
*Why:* everything downstream is unverifiable until this exists. *How:* assemble ≥1,000 real estimate line
items and ≥400 real spec sections with true MasterFormat labels from public sources (§5); store as
`eval/line_items_v2.tsv`, `eval/manuals_v2.jsonl`; report Wilson CIs + paired bootstrap for every comparison.
*Done when:* CIs ≤ ±3 pts and the published vs candidate comparison reports a p-value. *Effort:* M.

**P0.2 — Acquire real line-item training labels (not just eval).**
*Why:* the transformer's line-item ceiling is a data problem, not a capacity problem. *How:* mine public bid
tabulations / agency item catalogs; map agency item codes to MasterFormat where a crosswalk exists; treat the
rest as weak/self-supervised signal. Keep a strict train/eval split by project. *Done when:* ≥10k labelled
items, none from the eval projects. *Effort:* L.

**P0.3 — Domain-adaptive pretraining corpus.**
*Why:* the encoder only ever saw UFGS. *How:* collect public construction prose (specs, bid tabs, RFIs, submittal
logs) beyond UFGS — VA design manuals, state DOT standard specs, Corps/UFC, NASA. Continued-MLM the encoder,
then fine-tune. *Done when:* mf-0.5 beats the current model on the *enlarged* eval with non-overlapping CI.
*Effort:* L (compute: 1–2 GPU-days).

**P0.4 — Label space: level-3 sections + explicit out-of-taxonomy handling.**
*Why:* real specs use 8-digit codes and many sections outside the 171 groups. *How:* extend the taxonomy to
full sections; add an `other/<division>` fallback and score it honestly; revisit `group_of()`. *Effort:* M.

### P1 — Modelling (only once P0.1 is in place)

- **P1.1 Distillation** from the TF-IDF/ensemble teacher using **out-of-fold** soft targets (never on-data
  teacher labels) into the transformer. Replaces the current brittle log-prob blend with one model.
- **P1.2 Coarse-to-fine / hierarchical head** (division → group), directly attacking within-division confusion
  and giving graceful out-of-taxonomy fallback.
- **P1.3 Retrieval + rerank:** kNN over labelled embeddings for the top-k, plus a cross-encoder to reorder
  them. Often the cheapest way to absorb new labelled data as it arrives.
- **P1.4 Calibration + abstention:** temperature scaling on `val`; per-class thresholds; a `REVIEW` queue.
  Ship this regardless of accuracy — it converts a probabilistic classifier into a usable one.
- **P1.5 Multi-task:** shared encoder with group + division + section heads; consistency regularisation between
  them.

### P2 — Serving / ops

- **P2.1 Routing policy:** transformer for whole sections / long text, ensemble for terse items (already the
  evidence-backed split).
- **P2.2 Quantization-aware / int8-calibrated ONNX** (the int8 export currently drifts; add QAT or
  per-channel calibration and a browser benchmark).
- **P2.3 Golden-file CI** on the enlarged eval; nightly regression that fails on a gate breach.
- **P2.4 Metrics dashboard** with coverage/precision so abstention is tuned in production.

## 5. Data sourcing shortlist (public unless noted)

| Source | Use | Note |
|---|---|---|
| UFGS + UFC | train (in use) | US federal, public domain |
| State DOT standard specs (Caltrans, TxDOT, WSDOT, MnDOT…) | pretrain / train | mostly public |
| VA design manuals & specifications | pretrain / train | US federal, public |
| NASA / DoD / Corps specs | pretrain | public |
| Public **bid tabulations** (agency open-data portals; city/state capital projects) | line-item text + weak labels | item number + description; codes vary by agency |
| SAM.gov / FPDS contract line items | weak labels | huge, noisy |
| Public BOQ / cost datasets (e.g. Kaggle construction cost) | line items | check licence |
| RSMeans / Gordian cost codes | labels | **proprietary** — licence required |

## 6. Sequenced milestones

1. **M1 (now):** publish the evaluation caveat + `cloud/stats.py` (CIs, abstention). Freeze `mf-0.2`+ensemble
   as the reference. ✅ started in this change.
2. **M2:** enlarged eval (P0.1) from public bid tabs + manuals; re-score the reference.
3. **M3:** domain-adaptive pretrain + retrain (`mf-0.5`) on P0.2/P0.3 data; validate with CIs (P0.3).
4. **M4:** distillation + hierarchical head (`mf-0.6`); pick the best single model (P1.1/P1.2).
5. **M5:** calibration/abstention + routing in serving; publish with a coverage–precision card (P1.4, P2.1).

## 7. Explicitly not doing

- More synthetic line-item augmentation (tested: `mf-0.3` — no line-item transfer, manual regression).
- More TF-IDF feature engineering (already saturated; gains below the CI).
- Reporting sub-CI improvements as wins on the model card.

---

## Progress — P0.1 acquisition started (2026-10-08)

**Tooling added (this stage).**
- `scripts/compare.py` — the gate the plan was missing: **paired bootstrap CI of the difference + exact
  McNemar**, per set, with a `LIFT / DROP / no detectable change` verdict. `cloud/stats.py` had Wilson CIs
  but no significance test, which is how sub-CI changes kept reading as wins.
- `scripts/build_manual_eval_v2.py` — section eval over **every** local manual, with a train/val
  leakage guard (drops any chunk whose text appears in the corpus).
- `scripts/import_bid_tabulation.py` — pulls real line items from public bid tabulations.

**What the sources actually hold (the hard part).**
- The local AEC-Bench / ConstructBench cache has 18 documents, but **only 2 are project manuals**
  (`spec.pdf` ×2 → Wenatchee 651 pp, Macon 807 pp); the rest are drawings, sheets and submittals. Building
  over all 18 → **153 sections / 6,890 chunks, 0 leaked** — so the section eval is capped at ±8 pts by the
  *source*, not by code. **≥400 sections needs more public project manuals.**
- No public MasterFormat-labelled dataset exists: Hugging Face dataset search returns **0** for
  `masterformat` / `construction specifications` / `CSI division`; Kaggle has none either.
- Public bid tabulations are real but are labelled with **agency** codes, not CSI: `aniket0712/bid-tabulations`
  (TxDOT, 960,791 rows) has `BID ITEM DESCRIPTION` + `SPECIFICATION CODE` (TxDOT's own numbering). Useful as
  weak-label **training** text (P0.2) and as a **candidate pool** for labelling — not as a true-label eval.

**Produced.** `eval/line_items_v2_candidates.tsv` — **2,000 unique real line items** (deduped, cleaned) with
their source spec code and a blank `true_group` for a reviewer to fill. That is the automated half of P0.1's
"≥1,000 real line items"; the labels are the human half.

**Conclusion.** Both halves of P0.1 are blocked on **data and labelling, not modelling**: more public
project manuals for sections, and human labels (or a licensed source) for line items. The mf-0.5 Kaggle run
already confirmed the point — `val` rose (+2.3) while line items (−1.6) and sections (−1.3) fell, i.e. the
auxiliary coarse-to-fine loss bought nothing on the target distribution.

---

## Result 2026-10-13 — P0.2 first pass: real bid items in the TF-IDF scorer (`runs/bidtab/`, NOT published)

Data: full TxDOT bid tabulation (960k rows -> 13,239 unique item texts, `data/raw/txdot_bid_items_unique.jsonl`)
labelled by the eval crosswalk + 45 training-only codes (`data/txdot_crosswalk_train.json`, titles taken from
TxDOT's own SPECIFICATION DESCRIPTION column). After dropping every eval text and every eval text up to digits:
2,842 training items, 19 groups. Weight x3 chosen on a dev split of training-only codes. Script:
`scripts/bidtab_experiment.py`, results `runs/bidtab-exp.json`. Transformer unchanged (mf-0.2, blend w 0.25).

| eval | published ensemble | + bid items | paired |
|---|---:|---:|---|
| TxDOT, **item numbers held out** (5-fold grouped CV), n 790 | 0.489 [.454,.523] | **0.608** [.573,.641] | 104 vs 10, p < 0.001 |
| TxDOT capped <= 20/item, n 529 | 0.369 [.329,.411] | **0.492** [.449,.534] | 73 vs 8, p < 0.001 |
| hand-labelled building line items, n 194 | 0.691 | 0.701 | 3 vs 1, p 0.63 |
| manual chunks, n 6,890 | 0.326 | 0.329 | 65 vs 47, p 0.11 |
| whole sections top-1 / division, n 153 | 0.620 / 0.804 | 0.613 / 0.791 | -1 / -2 sections |
| UFGS val, n 11,598 | 0.5875 | 0.5836 | 61 vs 106, p < 0.001 |

TxDOT with item numbers *seen* (texts deduped) is 0.953 — memorisation of item types, not a headline number.

Gate check (§3): lift on real line items is large and significant on the agency set but not measurable on the
194-item building set; **section division -1.3 pts misses the "within 1 pt" gate by 2 sections** (noise-level,
but the gate was set in advance) and UFGS val drops 0.4 pts (significant). Decision left to the owner:
ship as default, ship as an opt-in "bid-item" TF-IDF, or first enlarge the section eval (P0.1) to tell a
2-section move from noise.