README.md

---
language:
- en
license: mit
library_name: transformers
pipeline_tag: text-classification
base_model: BAAI/bge-small-en-v1.5
tags:
- masterformat
- masterformat-classifier
- csi-masterformat
- construction
- construction-technology
- text-classification
- sequence-classification
- bert
- onnx
- safetensors
- tfidf
- ensemble
- specs
- spec-writing
- specifications
- takeoff
- estimating
- cost-code
- ufgs
- public-domain
metrics:
- accuracy
model-index:
- name: masterformat-classifier
  results:
  - task:
      type: text-classification
      name: MasterFormat level-2 group classification
    dataset:
      type: ufgs-val-v2
      name: UFGS held-out text units (11,598)
      split: validation
    metrics:
    - type: accuracy
      value: 0.466
      name: Top-1 accuracy (mf-0.2)
widget:
- text: EPDM membrane roofing
  example_title: Membrane roofing
- text: Wet pipe sprinkler system, light hazard
  example_title: Fire suppression
- text: 8" CMU wall, grout filled at 32" o.c.
  example_title: Unit masonry
- text: Addressable fire alarm system, devices and panel
  example_title: Fire alarm
---

# MasterFormat classifier (mf-0.2) — construction spec & line-item classification

> **Constructelligence is developing frontier construction-AI models.** This classifier is the *lite*, open
> tier — released for research and evaluation; the company's flagship production models are a separate,
> larger tier (see [constructelligence.co](https://constructelligence.co)).

A BERT text classifier that maps **construction line items, specification paragraphs, submittals and
section titles** to one of **171 MasterFormat level-2 groups** (e.g. `03 30 00 Cast-in-Place Concrete`,
`23 30 00 HVAC Air Distribution`) across **32 divisions**. It is the machine-learning counterpart to a
cost-code lookup table: hand it "EPDM membrane roofing" and it returns the MasterFormat group, ranked.
English text, one label per input.

Built for estimating, takeoff, spec-writing and RAG pipelines that need to tag free text with CSI
MasterFormat codes without a human choosing from a 171-row list.

> **Completed GPU fine-tune.** `mf-0.2` is the full run that `mf-0.1` (step 700) was an early checkpoint of:
> **6,132 steps / 12 epochs** on a Kaggle T4 over the rebuilt v2 dataset. Top-1 accuracy on the v2 UFGS
> validation split is **46.6 %** — **2.3×** `mf-0.1` (20.0 %). Line-item accuracy on 194 hand-labelled
> estimate items is **59.7 %** (was 28.3 %), and whole-section accuracy is **60.7 %** with **81.0 %** at
> division level. Because the transformer is trained on specification prose, short estimate line items are its
> weak spot; the repo therefore also ships a **transformer + TF-IDF ensemble** (`ensemble/`) that lifts
> line-item top-1 to **70.2 %** and improves manual-chunk and section accuracy; see *Ensemble*.
>
> **Update (Oct 2026): bid-item shorthand.** Both predictors now expand agency bid abbreviations
> (`CONC BOX CULV` → *concrete box culvert*) before classifying. On **790 real TxDOT bid items** the ensemble
> goes from **19.9 % to 48.9 %** top-1 with no change on any earlier evaluation; see *Agency bid-item shorthand*.

MasterFormat is a registered trademark of CSI / CSC. This project is independent and not affiliated with or
endorsed by them.

## Quick start

```python
from transformers import pipeline

clf = pipeline("text-classification", model="constructelligence/masterformat-classifier", top_k=3)
for p in clf("EPDM membrane roofing"):
    print(p["label"], round(p["score"], 3))
# 07 50 00 Membrane Roofing 0.863
# 07 10 00 Dampproofing and Waterproofing 0.068
# 07 30 00 Steep Slope Roofing 0.047
```

Batched, with the level-1 division roll-up and JSON output:

```bash
pip install -r requirements.txt        # transformers + torch
python predict.py --top-k 5 --divisions "Addressable fire alarm system, devices and panel"
echo "12\" RCP storm drain pipe" | python predict.py -
python predict.py --file items.txt --json > out.json
```

ONNX (no torch, ~34 MB int8):

```bash
pip install onnxruntime transformers
python predict.py --onnx onnx/model_quantized.onnx "Wet pipe sprinkler system, light hazard"
```

The encoder is English-only; inputs should be English.

### ONNX exports

`onnx/model.onnx` (fp32, 134 MB) and `onnx/model_quantized.onnx` (dynamic int8, 34 MB) are exported with
dynamic batch and sequence axes, in the layout `transformers.js` / Optimum expect — inputs `input_ids`,
`attention_mask`, `token_type_ids`, output `logits`. The fp32 export matches PyTorch on a 4-text smoke set
(max |Δ logit| 1e-5, argmax agreement 4/4). Dynamic int8 quantization shifts the logits more than it did for
`mf-0.1` — the better-trained head is more confident — so verify the quantized model on your own inputs;
the fp32 export is the safer default. Reproduce with `python scripts/export_onnx.py runs/mf-0.2`.

## Model

- **Base:** [`BAAI/bge-small-en-v1.5`](https://huggingface.co/BAAI/bge-small-en-v1.5) (BERT, 33M parameters, 384-d, MIT).
- **Head:** 171-way sequence-classification layer; `id2label` / `label2id` are in [`config.json`](config.json).
- **Recipe:** `label smoothing 0.05`, max length 64, batch 128, 12 epochs (6,132 steps), fp16 autocast,
  AdamW with 6 % linear warmup; body LR 5e-5, head LR 1e-3; embeddings and the first 4 encoder layers frozen.
- **Checkpoint:** `train_state.json` records `{"step": 6132, "val_acc": 0.4661}`.
- **Trained on:** Kaggle T4, ~16.5 min wall-clock, from `BAAI/bge-small-en-v1.5` (not resumed from `mf-0.1`).

## Results

All numbers are measured in this project on held-out data. The `line_items`, `manuals` and `manual_sections`
sets are **fixed**, so those columns are directly comparable for every scorer. `val` is the **v2** UFGS split
(11,598 units).

**v2 validation split**

| Scorer | val top-1 | val top-3 | val division |
|---|---:|---:|---:|
| **mf-0.2 (step 6132, published)** | 0.466 | 0.641 | 0.593 |
| mf-0.3 (v3 data, unfrozen) | **0.556** | **0.710** | **0.662** |
| mf-0.4 (v2 data, unfrozen) | 0.491 | 0.657 | 0.609 |
| mf-0.1 (step 700) | 0.200 | 0.372 | 0.356 |
| TF-IDF (word + char n-grams) | 0.571 | 0.732 | 0.670 |

**Fixed held-out sets (line items · manual chunks · whole sections)**

| Scorer | line-item top-1 | manual-chunk top-1 | manual-section top-1 | manual-section division |
|---|---:|---:|---:|---:|
| **mf-0.2 (step 6132, published)** | 0.597 | 0.280 | 0.607 | **0.810** |
| mf-0.3 (v3 data, unfrozen) | 0.597 | 0.259 | 0.547 | 0.765 |
| mf-0.4 (v2 data, unfrozen) | **0.618** | 0.270 | 0.567 | 0.817 |
| mf-0.1 (step 700) | 0.283 | 0.176 | 0.287 | 0.588 |
| TF-IDF (word + char n-grams) | 0.696 | 0.287 | 0.587 | 0.752 |
| **Ensemble mf-0.2 + TF-IDF (w = 0.25)** | **0.702** | **0.330** | **0.620** | 0.804 |

`line_items` = 194 hand-labelled estimate line items; `manuals` = 6,890 chunks from 153 sections of two real
commercial project manuals; out-of-taxonomy gold labels (e.g. `22 40 00`) count only toward the division
score, so top-1 is over in-taxonomy items only.

**Evidence caveat.** The line-item and section sets are small. At n = 191 the 95 % Wilson interval on the
line-item top-1 is **[0.633, 0.762] (±6.4 pts)**, and whole-section is ±~8 pts — so differences of a few
points between models here are **noise**. Treat the `val` and `manual-chunk` rows (±1 pt) as the measurable
ones, and the line-item/section rows as indicative. Enlarging these eval sets is the top item on the roadmap.

**Reading it:** `mf-0.2` is the best scorer overall at section-level *division* accuracy (0.810). On short
line items a lexical TF-IDF model is stronger (0.696 alone), so the shipped **transformer + TF-IDF ensemble**
reaches **0.702** and is the production candidate; see *Ensemble*.

## Ensemble (closing the line-item gap)

The transformer is trained on specification prose, so on **short, terse estimate line items** a lexical
TF-IDF model is still stronger (0.696 vs 0.597 top-1). To close that gap without giving up the transformer's
long-text accuracy, this repo ships a **log-probability ensemble** of `mf-0.2` and a TF-IDF + SGD scorer:

```
p = softmax( 0.25 · log_softmax(mf-0.2)  +  0.75 · log_softmax(tfidf) )
```

The 0.25 weight is chosen on the UFGS **validation** split — not on the line-item test set. `ensemble/tfidf.joblib`
holds the fitted vectorizer (word 1–2 grams, 100k features, plus `char_wb` 3–5 grams, 150k features, `min_df`
2/3, sublinear tf) and SGD logistic classifier; `ensemble/blend.json` records the weight and all metrics;
`ensemble/predict_ensemble.py` runs the blend.

| Scorer | val top-1 | val division | line-item top-1 | manual-chunk top-1 | manual-section top-1 | manual-section division |
|---|---:|---:|---:|---:|---:|---:|
| mf-0.2 (transformer) | 0.466 | 0.593 | 0.597 | 0.280 | 0.607 | **0.810** |
| TF-IDF (word + char n-grams) | 0.571 | 0.670 | 0.696 | 0.287 | 0.587 | 0.752 |
| **Ensemble (w = 0.25)** | **0.588** | **0.689** | **0.702** | **0.330** | **0.620** | 0.804 |

```bash
pip install transformers torch scikit-learn joblib
python ensemble/predict_ensemble.py --model constructelligence/masterformat-classifier \
    --tfidf ensemble/tfidf.joblib --top-k 3 --divisions "4000 psi concrete slab on grade"
```

**Honest caveat.** The ensemble beats TF-IDF alone mainly on the transformer's strengths — validation top-1
(0.588 vs 0.571), manual-chunk top-1 (0.330 vs 0.287) and whole-section top-1 (0.620 vs 0.587) — while line
items are nearly saturated by TF-IDF (0.702 vs 0.696). Adding bge-small embeddings does **not** help once the
blend weight is chosen on `val` (its weight goes to ~0); an earlier 0.712 line-item figure came from tuning
the weights on the 194-item test itself and does not generalise. The transformer remains the best scorer at
section *division* accuracy (0.810). A reasonable production split is the transformer for whole sections, the
ensemble for terse line items.

## Agency bid-item shorthand

Public bid tabulations write items in a terse shorthand the transformer never saw in spec prose:
`RE PM W/RET REQ TY I (W)6"(SLD)(060MIL)`, `CL C CONC (RAIL FOUNDATION)`, `CONC BOX CULV (5 FT X 2 FT)`. Read
literally, the model sends `CL B CONC` to *Structured Cabling* and `RE PM` to *Electrical Utilities*.
`abbrev.py` expands 82 common US bid abbreviations (`CONC`, `PAV`, `CULV`, `REINF`, `ASPH`, `W/`…) before
scoring — **all-caps tokens only**, so prose is left alone, and the original text is appended so nothing is lost.
`predict.py` and `ensemble/predict_ensemble.py` do this by default; `--no-expand` turns it off. The dictionary is
written from general bid-item conventions, not fitted to the evaluation below.

**Evaluation: 790 TxDOT bid items.** Real item descriptions from Texas DOT public bid tabulations, labelled by
`txdot_crosswalk.json` — a table mapping 41 TxDOT standard-specification item numbers to MasterFormat groups
(17 groups, Divisions 01–03, 26, 31–34). Items that could belong to more than one group (repairs, temporary
work-zone items, signs/signals, special provisions) are left out. *Capped* scores keep at most 20 rows per
TxDOT item, so pavement markings (207 rows) cannot dominate. 95 % Wilson intervals.

| Scorer | top-1 | top-1, capped (n = 529) | top-3 | division |
|---|---:|---:|---:|---:|
| mf-0.2 (transformer), raw text | 0.125 [0.104, 0.150] | 0.161 | 0.239 | 0.225 |
| mf-0.2, abbreviations expanded | 0.270 [0.240, 0.302] | 0.272 | 0.546 | 0.352 |
| TF-IDF, raw text | 0.266 [0.236, 0.298] | 0.246 | 0.451 | 0.339 |
| TF-IDF, expanded | 0.484 [0.449, 0.518] | 0.348 | 0.629 | 0.559 |
| Ensemble, raw text | 0.199 [0.172, 0.228] | 0.223 [0.190, 0.260] | 0.416 | 0.285 |
| **Ensemble, expanded (default)** | **0.489 [0.454, 0.523]** | **0.369 [0.329, 0.411]** | **0.671** | **0.570** |

On the earlier sets the expansion changes 1 of 194 line items and 39 of 6,890 manual chunks: line items stay
at 0.702, manual chunks at 0.330, and whole-section top-1 moves 0.620 → 0.627.

**Reading it.** The expansion is the largest single gain this model has had on real estimate text, with
non-overlapping intervals. It is still well below the 70 % on hand-written building line items: highway
bid items are out of domain for a UFGS-trained model, and these labels come from an item-number crosswalk
(reviewed, not per-item), covering civil work only.

### Bid-item mode (`--bid-items`, opt-in)

`ensemble/tfidf_bid.joblib` is the same TF-IDF scorer **also trained on 2,842 real TxDOT bid items**: the full
public bid tabulation (960k rows → 13,239 unique descriptions), labelled through the crosswalk above plus 45
training-only item codes (`txdot_crosswalk_train.json`, mapped from TxDOT's own specification titles). Every
evaluation description — and every description equal to one after removing numbers ("TREE REMOVAL (4" - 12" DIA)"
vs "(12" - 18" DIA)") — was removed from training. Bid items are weighted ×3, chosen on a dev split of
training-only codes.

```bash
python ensemble/predict_ensemble.py --bid-items "CL C CONC (RAIL FOUNDATION)" "SET (TY II) (24 IN) (RCP) (6: 1) (P)"
```

The honest number is **item-held-out**: 5-fold cross-validation grouped by TxDOT item number, so each evaluation
row is scored by a model that saw no bid item of its type (new item types, not paraphrases). Paired against the
default on the same rows:

| Ensemble on 790 TxDOT bid items | default | `--bid-items` | changed |
|---|---:|---:|---|
| top-1, item types held out | 0.489 [0.454, 0.523] | **0.608 [0.573, 0.641]** | 104 fixed, 10 broken (McNemar p < 0.001) |
| top-1 capped ≤ 20/item (n = 529) | 0.369 [0.329, 0.411] | **0.492 [0.449, 0.534]** | 73 fixed, 8 broken |

The shipped `tfidf_bid.joblib` has seen these item *types* (not these descriptions); on them it reaches 0.95 —
that is memorised item types, not a measure of quality on new agencies' items.

**Why it is opt-in.** On building text it is a wash or slightly worse: 194 hand-labelled line items 0.691 → 0.701
(3 vs 1 changed), manual chunks 0.326 → 0.329, whole sections 0.620 → 0.613 top-1 and 0.804 → 0.791 division
(2 of 153 sections), UFGS val 0.588 → 0.584. Use it for agency / DOT bid tabulations, keep the default for
specifications and building estimates.

## Calibration and abstention

The ensemble is already close to calibrated; a single **temperature** `T = 0.90` (fit on `val`) lowers the
expected calibration error on `val` from **0.060 to 0.022**. More useful in production: send the
lowest-confidence predictions to a human instead of filing them.

Coverage → precision on the 194 line items (predictions sorted by confidence):

| auto-filed (coverage) | 100 % | 90 % | 80 % | 70 % | 50 % | 30 % |
|---|---:|---:|---:|---:|---:|---:|
| precision | 0.69 | 0.75 | 0.81 | **0.85** | **0.91** | 0.97 |

Routing the least-confident **30 %** to review lifts precision to **91 %**; routing 50 % gives **97 %**. So
the model is usable today as an **assist that flags its own uncertainty**, even though it is not accurate
enough to file unattended. Reproduce with `python cloud/stats.py`.

## Roadmap

Further gains are **evidence-bound**, not architecture-bound (see the repo's `IMPROVEMENT-PLAN.md`):

1. **Enlarge the eval** (≥1,000 line items, ≥400 sections) so changes are measurable at ±3 pts. *Progress:*
   790 crosswalk-labelled TxDOT bid items (±3.5 pts, civil only); building-estimate items are still 194.
2. **Real line-item training data** (public bid tabulations, agency item catalogs) — the transformer's
   line-item ceiling is a data problem; synthetic augmentation was tested and did not transfer (`mf-0.3`).
3. **Domain-adaptive pretraining** on construction prose beyond UFGS, then retrain.
4. **Distillation + hierarchical (division → group) head** to beat the TF-IDF blend with a single model.
5. **Level-3 / full-section codes** and an explicit out-of-taxonomy fallback.

## Intended use

- **Use it for:** tagging construction text with a candidate MasterFormat group (top-3 shown), an
  auto-classification step for estimating or spec workflows, and as a strong base to fine-tune or distil.
- **Do not use it for:** unattended production takeoff, bid pricing, code compliance, or anything where a
  wrong cost code has financial or contractual consequences. Keep a human in the loop.
- **Not a substitute for review.** Classifies text content only — if the input already contains a
  MasterFormat number, read the number instead.

## Training data and taxonomy

- **Source:** UFGS (Unified Facilities Guide Specifications) `.SEC` files — US federal works in the
  **public domain**. Paragraphs and titles are parsed into labelled text units; boilerplate shared by
  multiple sections is dropped, cross-references are stripped so section numbers cannot leak labels, and
  long paragraphs are cut on sentence boundaries into 8–60-word windows.
- **Split:** held out by a hash of the normalised source text (10 %), so a paragraph and its augmentations
  stay on the same side. **65,496 train / 11,598 val** rows (the v2 set).
- **Taxonomy:** 171 level-2 groups over 32 MasterFormat divisions; group numbers follow the MasterFormat
  numbering convention and the short names are this project's own (see `config.json`).

## Limitations

- **Below the ensemble on line items.** 59.7 % vs 70.2 % on 194 hand-labelled estimate items; do not deploy
  as the sole classifier.
- **Domain skew.** UFGS over-represents heavy-civil, water/wastewater and process work relative to commercial
  building estimates; the training set is class-balanced, so raw predictions do not reflect building-project
  priors.
- **Out-of-taxonomy inputs** (sections whose level-2 group is not among the 171) can only be scored at
  division level.
- **Short, terse line items are the hardest inputs**; division (2-digit) accuracy is consistently higher than
  group (6-digit) accuracy. Agency shorthand is expanded by `abbrev.py`, but abbreviations outside its
  dictionary, and highway work generally, remain weak (48.9 % top-1 on TxDOT bid items).
- **Quantized ONNX drifts.** The int8 export is smaller but less faithful than `mf-0.1`'s; prefer fp32.

## Bias, risks and safety

- **Estimating bias.** Class-balanced training over a public-domain federal corpus does not represent any
  particular firm's cost structure or regional practice. Do not treat output as a standard or an authority.
- **Trademark.** MasterFormat is a registered trademark of CSI / CSC; this model is not endorsed by them and
  its group names are the project's own short descriptions, not CSI's official titles.
- **Privacy.** The model runs locally; no input text leaves your machine unless you call a hosted endpoint.

## FAQ

**What is MasterFormat?** The CSI/CSC MasterFormat is the North American standard for organising construction
specifications and cost data into numbered divisions and sections. This model predicts the **level-2 group**
(a 6-digit code such as `03 30 00 Cast-in-Place Concrete`), not the full section number.

**How is this different from `mf-0.1`?** `mf-0.1` was step 700 of an interrupted CPU run; `mf-0.2` is the
completed 12-epoch GPU fine-tune on the rebuilt dataset. Same architecture, ~2.3× the validation accuracy.

**Can it classify a whole specification section?** Yes — average the model's log-probabilities over a
section's chunks. On two real project manuals that gives 60.7 % top-1 and 81.0 % at division level.

**Can it read a MasterFormat number out of the text?** No. It classifies the description. If the number is
already present, parse it directly.

**Does it work offline / in the browser?** Yes — the ONNX exports are intended for `onnxruntime` and
`transformers.js`.

**Is a better model available?** Yes — this repo ships a **transformer + TF-IDF ensemble** (`ensemble/`) that
scores **70.2 % top-1 on hand-labelled line items** (vs 59.7 % for the transformer alone) and improves
manual-chunk and whole-section accuracy. Constructelligence's proprietary models are at
[constructelligence.co](https://constructelligence.co).

## Files

- `model.safetensors`, `config.json`, `tokenizer.json`, `tokenizer_config.json`, `vocab.txt`,
  `special_tokens_map.json` — standard `transformers` checkpoint.
- `train_state.json` — step and validation accuracy of the saved checkpoint.
- `metrics.json` — full held-out evaluation (`val`, `line_items`, `manuals`, `manual_sections`).
- `predict.py` — CLI example: batching, `--top-k`, `--divisions`, `--json`, stdin/file input, `--onnx`.
- `onnx/` — ONNX fp32 and int8 exports.
- `ensemble/` — `tfidf.joblib` (word + char n-grams, ~112 MB), `blend.json`, `predict_ensemble.py`: the
  transformer + TF-IDF line-item ensemble; `tfidf_bid.joblib` (~115 MB) is the opt-in `--bid-items` scorer.
- `abbrev.py` — bid-item abbreviation expansion used by both predictors (`--no-expand` to disable).
- `txdot_crosswalk.json` — TxDOT item number → MasterFormat group table behind the bid-item evaluation.
- `txdot_crosswalk_train.json` — 45 further TxDOT codes used only to label `--bid-items` training data.
- `IMPROVEMENT-PLAN.md` — the prioritised roadmap (enlarged eval, data, domain-adaptive pretraining, distillation).
- `CITATION.cff`, `requirements.txt`.

## Citation

```bibtex
@misc{constructelligence_masterformat_classifier,
  title        = {MasterFormat Classifier (mf-0.2): construction spec and line-item classification},
  author       = {Constructelligence},
  year         = {2026},
  howpublished = {\url{https://huggingface.co/constructelligence/masterformat-classifier}},
  note         = {Fine-tuned from BAAI/bge-small-en-v1.5; 171-way MasterFormat level-2 classifier}
}
```

## Licence and attribution

Released under the **MIT licence**, matching the base model. UFGS source text is public domain. MasterFormat
is a registered trademark of CSI / CSC; this project is not affiliated with or endorsed by them.
Constructelligence's production models are available at [constructelligence.co](https://constructelligence.co).