Bid-item shorthand expansion (abbrev.py) in both predictors + 790-item TxDOT bid-item evaluation

README.md

@@ -75,6 +75,10 @@ MasterFormat codes without a human choosing from a 171-row list.
 > division level. Because the transformer is trained on specification prose, short estimate line items are its
 > weak spot; the repo therefore also ships a **transformer + TF-IDF ensemble** (`ensemble/`) that lifts
 > line-item top-1 to **70.2 %** and improves manual-chunk and section accuracy; see *Ensemble*.
+>
+> **Update (Oct 2026): bid-item shorthand.** Both predictors now expand agency bid abbreviations
+> (`CONC BOX CULV` → *concrete box culvert*) before classifying. On **790 real TxDOT bid items** the ensemble
+> goes from **19.9 % to 48.9 %** top-1 with no change on any earlier evaluation; see *Agency bid-item shorthand*.
 
 MasterFormat is a registered trademark of CSI / CSC. This project is independent and not affiliated with or
 endorsed by them.
@@ -203,6 +207,39 @@ the weights on the 194-item test itself and does not generalise. The transformer
 section *division* accuracy (0.810). A reasonable production split is the transformer for whole sections, the
 ensemble for terse line items.
 
+## Agency bid-item shorthand
+
+Public bid tabulations write items in a terse shorthand the transformer never saw in spec prose:
+`RE PM W/RET REQ TY I (W)6"(SLD)(060MIL)`, `CL C CONC (RAIL FOUNDATION)`, `CONC BOX CULV (5 FT X 2 FT)`. Read
+literally, the model sends `CL B CONC` to *Structured Cabling* and `RE PM` to *Electrical Utilities*.
+`abbrev.py` expands 82 common US bid abbreviations (`CONC`, `PAV`, `CULV`, `REINF`, `ASPH`, `W/`…) before
+scoring — **all-caps tokens only**, so prose is left alone, and the original text is appended so nothing is lost.
+`predict.py` and `ensemble/predict_ensemble.py` do this by default; `--no-expand` turns it off. The dictionary is
+written from general bid-item conventions, not fitted to the evaluation below.
+
+**Evaluation: 790 TxDOT bid items.** Real item descriptions from Texas DOT public bid tabulations, labelled by
+`txdot_crosswalk.json` — a table mapping 41 TxDOT standard-specification item numbers to MasterFormat groups
+(17 groups, Divisions 01–03, 26, 31–34). Items that could belong to more than one group (repairs, temporary
+work-zone items, signs/signals, special provisions) are left out. *Capped* scores keep at most 20 rows per
+TxDOT item, so pavement markings (207 rows) cannot dominate. 95 % Wilson intervals.
+
+| Scorer | top-1 | top-1, capped (n = 529) | top-3 | division |
+|---|---:|---:|---:|---:|
+| mf-0.2 (transformer), raw text | 0.125 [0.104, 0.150] | 0.161 | 0.239 | 0.225 |
+| mf-0.2, abbreviations expanded | 0.270 [0.240, 0.302] | 0.272 | 0.546 | 0.352 |
+| TF-IDF, raw text | 0.266 [0.236, 0.298] | 0.246 | 0.451 | 0.339 |
+| TF-IDF, expanded | 0.484 [0.449, 0.518] | 0.348 | 0.629 | 0.559 |
+| Ensemble, raw text | 0.199 [0.172, 0.228] | 0.223 [0.190, 0.260] | 0.416 | 0.285 |
+| **Ensemble, expanded (default)** | **0.489 [0.454, 0.523]** | **0.369 [0.329, 0.411]** | **0.671** | **0.570** |
+
+On the earlier sets the expansion changes 1 of 194 line items and 39 of 6,890 manual chunks: line items stay
+at 0.702, manual chunks at 0.330, and whole-section top-1 moves 0.620 → 0.627.
+
+**Reading it.** The expansion is the largest single gain this model has had on real estimate text, with
+non-overlapping intervals. It is still well below the 70 % on hand-written building line items: highway
+bid items are out of domain for a UFGS-trained model, and these labels come from an item-number crosswalk
+(reviewed, not per-item), covering civil work only.
+
 ## Calibration and abstention
 
 The ensemble is already close to calibrated; a single **temperature** `T = 0.90` (fit on `val`) lowers the
@@ -223,8 +260,8 @@ enough to file unattended. Reproduce with `python cloud/stats.py`.
 
 Further gains are **evidence-bound**, not architecture-bound (see the repo's `IMPROVEMENT-PLAN.md`):
 
-1. **Enlarge the eval** (≥1,000 line items, ≥400 sections) so changes are measurable at ±3 pts — currently
-   line-item changes below ~10 pts cannot be validated.
+1. **Enlarge the eval** (≥1,000 line items, ≥400 sections) so changes are measurable at ±3 pts. *Progress:*
+   790 crosswalk-labelled TxDOT bid items (±3.5 pts, civil only); building-estimate items are still 194.
 2. **Real line-item training data** (public bid tabulations, agency item catalogs) — the transformer's
    line-item ceiling is a data problem; synthetic augmentation was tested and did not transfer (`mf-0.3`).
 3. **Domain-adaptive pretraining** on construction prose beyond UFGS, then retrain.
@@ -261,7 +298,8 @@ Further gains are **evidence-bound**, not architecture-bound (see the repo's `IM
 - **Out-of-taxonomy inputs** (sections whose level-2 group is not among the 171) can only be scored at
   division level.
 - **Short, terse line items are the hardest inputs**; division (2-digit) accuracy is consistently higher than
-  group (6-digit) accuracy.
+  group (6-digit) accuracy. Agency shorthand is expanded by `abbrev.py`, but abbreviations outside its
+  dictionary, and highway work generally, remain weak (48.9 % top-1 on TxDOT bid items).
 - **Quantized ONNX drifts.** The int8 export is smaller but less faithful than `mf-0.1`'s; prefer fp32.
 
 ## Bias, risks and safety
@@ -305,6 +343,8 @@ manual-chunk and whole-section accuracy. Constructelligence's proprietary models
 - `onnx/` — ONNX fp32 and int8 exports.
 - `ensemble/` — `tfidf.joblib` (word + char n-grams, ~112 MB), `blend.json`, `predict_ensemble.py`: the
   transformer + TF-IDF line-item ensemble.
+- `abbrev.py` — bid-item abbreviation expansion used by both predictors (`--no-expand` to disable).
+- `txdot_crosswalk.json` — TxDOT item number → MasterFormat group table behind the bid-item evaluation.
 - `IMPROVEMENT-PLAN.md` — the prioritised roadmap (enlarged eval, data, domain-adaptive pretraining, distillation).
 - `CITATION.cff`, `requirements.txt`.
 

abbrev.py

@@ -0,0 +1,46 @@
+"""Expand standard construction bid-item abbreviations before classification.
+
+Agency bid tabulations and estimates write items in a terse shorthand ("CONC BOX CULV (5 FT X 2 FT)",
+"RE PM W/RET REQ TY I") that a model trained on spec prose cannot read. This is a general dictionary of
+common US bid-item abbreviations (DOT and municipal), written from those conventions, not fitted to an eval set.
+Whole ALL-CAPS tokens only; the original text is kept after the expansion so nothing is lost.
+"""
+import re
+
+ABBREV = {
+    "ACP": "asphalt concrete pavement", "AGGR": "aggregate", "AGG": "aggregate", "ASPH": "asphalt",
+    "ASM": "assembly", "ASSM": "assembly", "ASSY": "assembly", "ATTEN": "attenuator", "BARR": "barricade",
+    "BKFL": "backfill", "BM": "beam", "BRK": "broken line", "BS": "base", "CEM": "cement", "CL": "class",
+    "CMP": "corrugated metal pipe", "COMPL": "complete", "CONC": "concrete", "CONDR": "conductor",
+    "CONDT": "conduit", "CONST": "construction", "CONT": "control", "CRCP": "continuously reinforced concrete pavement",
+    "CULV": "culvert", "CUSH": "cushion", "DEL": "delineator", "DIA": "diameter", "DRV": "driveway",
+    "ELEC": "electrical", "EMBK": "embankment", "EXC": "excavation", "EXCAV": "excavation", "EXST": "existing",
+    "EXIST": "existing", "FBR": "fiber", "FEN": "fence", "FL": "flexible", "GD": "guard", "GR": "grade",
+    "HMA": "hot mix asphalt", "IL": "illumination", "ILL": "illumination", "INSTL": "install", "INST": "install",
+    "LT": "light", "MATL": "material", "MLCH": "mulch", "MRK": "marking", "MRKR": "marker", "MTL": "metal",
+    "PAV": "pavement", "PVMT": "pavement", "PERM": "permanent", "PM": "pavement marking", "PREFAB": "prefabricated",
+    "PRESTR": "prestressed", "RC": "reinforced concrete", "RCP": "reinforced concrete pipe", "RD": "roadway",
+    "RDWY": "roadway", "REFL": "reflectorized", "REINF": "reinforced", "REM": "remove", "REMOV": "remove",
+    "RET": "retroreflectivity", "ROW": "right of way", "RT": "right", "SEDMT": "sediment", "STAB": "stabilized",
+    "STR": "structure", "STRUCT": "structure", "SW": "sidewalk", "SWK": "sidewalk", "TEMP": "temporary",
+    "TRT": "treatment", "TREAT": "treatment", "TY": "type", "TRANS": "transition", "VEH": "vehicle",
+    "W/": "with ", "WW": "wingwall", "HW": "headwall",
+}
+_TOK = re.compile(r"[Ww]/(?=[A-Za-z ])|[A-Za-z]+")
+
+
+def expand(text: str) -> str:
+    """'CONC BOX CULV (5 FT X 2 FT)' -> 'concrete BOX culvert (5 FT X 2 FT) | CONC BOX CULV (5 FT X 2 FT)'."""
+    hit = False
+
+    def sub(m):
+        nonlocal hit
+        tok = m.group(0)
+        # shorthand is written in capitals: prose words ("row", "cl") are left alone
+        w = ABBREV.get(tok.upper()) if tok.upper() == tok else None
+        if w:
+            hit = True
+            return w
+        return m.group(0)
+    out = _TOK.sub(sub, text)
+    return f"{out} | {text}" if hit else text

ensemble/predict_ensemble.py

@@ -23,6 +23,8 @@ from pathlib import Path
 import numpy as np
 
 HERE = Path(__file__).resolve().parent
+sys.path.insert(0, str(HERE.parent))
+from abbrev import expand  # noqa: E402  bid-item shorthand ("CONC BOX CULV") -> words the models know
 MODEL = "constructelligence/masterformat-classifier"
 
 DIVISIONS = {
@@ -95,6 +97,7 @@ def main():
     ap.add_argument("--divisions", action="store_true")
     ap.add_argument("--file")
     ap.add_argument("--json", action="store_true")
+    ap.add_argument("--no-expand", action="store_true", help="do not expand bid-item abbreviations first")
     a = ap.parse_args()
 
     if a.weight is None:
@@ -113,7 +116,8 @@ def main():
 
     tr, tf = Transformer(a.model), Tfidf(a.tfidf)
     assert tf.classes == list(range(len(tr.labels))), "TF-IDF classes do not align with the transformer labels"
-    lp = norm(w * tr.logprobs(texts) + (1 - w) * tf.logprobs(texts))   # blend of log-probs, renormalised
+    mt = texts if a.no_expand else [expand(t) for t in texts]
+    lp = norm(w * tr.logprobs(mt) + (1 - w) * tf.logprobs(mt))   # blend of log-probs, renormalised
 
     results = []
     for text, row in zip(texts, lp):

metrics.json

@@ -21,5 +21,54 @@
   "sections": 153,
   "top1": 0.6066666666666667,
   "division": 0.8104575163398693
+ },
+ "txdot_bid_items": {
+  "_doc": "790 TxDOT bid items labelled via txdot_crosswalk.json; capped = <=20 rows per TxDOT item (n=529); 95% Wilson CI",
+  "transformer": {
+   "raw": {
+    "top1": 0.125,
+    "top1_ci": [
+     0.104,
+     0.15
+    ],
+    "capped_top1": 0.161
+   },
+   "expanded": {
+    "top1": 0.27,
+    "top1_ci": [
+     0.24,
+     0.302
+    ],
+    "capped_top1": 0.272,
+    "top3": 0.546,
+    "division": 0.352
+   }
+  },
+  "ensemble": {
+   "raw": {
+    "top1": 0.199,
+    "top1_ci": [
+     0.172,
+     0.228
+    ],
+    "capped_top1": 0.223,
+    "top3": 0.416,
+    "division": 0.285
+   },
+   "expanded": {
+    "top1": 0.489,
+    "top1_ci": [
+     0.454,
+     0.523
+    ],
+    "capped_top1": 0.369,
+    "capped_top1_ci": [
+     0.329,
+     0.411
+    ],
+    "top3": 0.671,
+    "division": 0.57
+   }
+  }
  }
-}
+}

predict.py

@@ -17,6 +17,10 @@ Without arguments it classifies one built-in example.
 import argparse
 import json
 import sys
+from pathlib import Path
+
+sys.path.insert(0, str(Path(__file__).resolve().parent))
+from abbrev import expand  # noqa: E402  bid-item shorthand ("CONC BOX CULV") -> words the model knows
 
 MODEL = "constructelligence/masterformat-classifier"
 
@@ -101,9 +105,9 @@ def onnx_scorer(path, top_k):
     return score
 
 
-def predict(texts, score, top_k, want_div):
+def predict(texts, score, top_k, want_div, model_texts=None):
     results = []
-    for text, preds in zip(texts, score(texts)):
+    for text, preds in zip(texts, score(model_texts or texts)):
         pl = []
         for p in preds:
             code, name = split_label(p["label"])
@@ -124,6 +128,8 @@ def main(argv=None):
     ap.add_argument("--divisions", action="store_true", help="also show the level-1 (division) roll-up")
     ap.add_argument("--file", help="read newline-separated inputs from a file")
     ap.add_argument("--json", action="store_true", help="emit JSON instead of a table")
+    ap.add_argument("--no-expand", action="store_true",
+                    help="do not expand bid-item abbreviations (CONC, PAV, CULV, ...) before classifying")
     a = ap.parse_args(argv)
 
     texts = list(a.text)
@@ -138,7 +144,7 @@ def main(argv=None):
         ap.error("no input text")
 
     score = onnx_scorer(a.onnx, a.top_k) if a.onnx else hf_scorer(a.model, a.top_k)
-    results = predict(texts, score, a.top_k, a.divisions)
+    results = predict(texts, score, a.top_k, a.divisions, None if a.no_expand else [expand(t) for t in texts])
 
     if a.json:
         json.dump(results, sys.stdout, indent=2, ensure_ascii=False)

txdot_crosswalk.json

@@ -0,0 +1,46 @@
+{
+ "_doc": "TxDOT standard-specification item number -> MasterFormat level-2 group (the classifier's 171 classes). Labels for eval/line_items_txdot.tsv come from this table, not from per-item judgement. Only items whose TxDOT title maps to one group without doubt are listed; left out on purpose: repair/maintenance items that could equally sit in a maintenance group (351, 354, 361, 438, 446, 677, 678, 7xx maintenance specs), temporary work-zone items (512, 662), signs and signals (636, 644, 680: MasterFormat 34 40 00 is not one of the 171 classes), 400 (mixes structure excavation with cut-and-restore paving), and every 4-digit special specification (title unknown). Built 2026-10-11; review before relying on it.",
+ "items": {
+  "100": ["Preparing Right of Way", "31 10 00"],
+  "104": ["Removing Concrete", "02 40 00"],
+  "105": ["Removing Stabilized Base and Asphalt Pavement", "02 40 00"],
+  "110": ["Excavation", "31 20 00"],
+  "132": ["Embankment", "31 20 00"],
+  "164": ["Seeding for Erosion Control", "32 90 00"],
+  "247": ["Flexible Base", "32 10 00"],
+  "260": ["Lime Treatment (Road-Mixed)", "31 30 00"],
+  "275": ["Cement Treatment (Road-Mixed)", "31 30 00"],
+  "316": ["Seal Coat", "32 10 00"],
+  "360": ["Concrete Pavement", "32 10 00"],
+  "416": ["Drilled Shaft Foundations", "31 60 00"],
+  "420": ["Concrete Substructures", "03 30 00"],
+  "422": ["Concrete Superstructures", "03 30 00"],
+  "425": ["Precast Prestressed Concrete Structural Members", "03 40 00"],
+  "429": ["Concrete Structure Repair", "03 00 00"],
+  "432": ["Riprap", "31 30 00"],
+  "462": ["Concrete Box Culverts and Storm Drains", "33 40 00"],
+  "464": ["Reinforced Concrete Pipe", "33 40 00"],
+  "465": ["Junction Boxes, Manholes, and Inlets", "33 40 00"],
+  "466": ["Headwalls and Wingwalls", "33 40 00"],
+  "496": ["Removing Structures", "02 40 00"],
+  "500": ["Mobilization", "01 70 00"],
+  "502": ["Barricades, Signs, and Traffic Handling", "01 50 00"],
+  "506": ["Temporary Erosion, Sedimentation, and Environmental Controls", "31 20 00"],
+  "529": ["Concrete Curb, Gutter, and Combined Curb and Gutter", "32 10 00"],
+  "530": ["Intersections, Driveways, and Turnouts", "32 10 00"],
+  "531": ["Sidewalks", "32 10 00"],
+  "540": ["Metal Beam Guard Fence", "34 70 00"],
+  "544": ["Guardrail End Treatments", "34 70 00"],
+  "545": ["Crash Cushion Attenuators", "34 70 00"],
+  "550": ["Chain Link Fence", "32 30 00"],
+  "610": ["Roadway Illumination Assemblies", "26 50 00"],
+  "618": ["Conduit", "26 00 00"],
+  "620": ["Electrical Conductors", "26 00 00"],
+  "624": ["Ground Boxes", "26 00 00"],
+  "658": ["Delineator and Object Marker Assemblies", "34 70 00"],
+  "666": ["Retroreflectorized Pavement Markings", "32 10 00"],
+  "668": ["Prefabricated Pavement Markings", "32 10 00"],
+  "672": ["Raised Pavement Markers", "32 10 00"],
+  "752": ["Tree and Brush Removal", "31 10 00"]
+ }
+}