hung-k-nguyen commited on
Commit
4576ade
Β·
verified Β·
1 Parent(s): 5031089

Update v2 numbers (retrained head + fuser)

Browse files
Files changed (1) hide show
  1. README.md +10 -9
README.md CHANGED
@@ -40,14 +40,15 @@ in-house model and a cloud frontier VLM reference.
40
  <!-- RESULTS-TABLE:START β€” filled from leaderboard/leaderboard.json by the scoring run -->
41
  | Benchmark (macro-F1) | `document-classification-v2` | best generalist | best cloud VLM |
42
  | --- | ---: | ---: | ---: |
43
- | **DocLayNet** (page types) | **0.88** | 0.63 | 0.83 |
44
  | **Forms** | **1.00** | 0.11 | 1.00 |
45
- | **Tobacco** (doc types) | 0.69 | 0.43 | **0.85** |
46
- | **OOD** (unseen doc types) | **0.97** | β€” | β€” |
47
- | **OOV** (synonym wording) | **0.80** | β€” | β€” |
48
 
49
- Latency: **~8 docs/s on an A40** (p50 106 ms / p95 194 ms). The open-weight
50
- [`v1`](https://huggingface.co/nutrientdocs/document-classification-v1) runs ~3–7 docs/s on the same GPU.
 
51
  <!-- RESULTS-TABLE:END -->
52
 
53
  Every model β€” ours and cloud β€” is scored by the same open macro-F1 scorer; full per-model ranking on the
@@ -55,10 +56,10 @@ Every model β€” ours and cloud β€” is scored by the same open macro-F1 scorer; f
55
 
56
  - **Matches/leads the cloud on visual document-type tracks** (DocLayNet, Forms) at **zero per-request API
57
  cost** β€” and runs on a single GPU.
58
- - **Trails on Tobacco** (0.69 vs 0.85): that track rewards *reading* fine header text (memo vs letter vs
59
  email) β€” a large VLM reads it; an embedding model can't.
60
- - **OOD robustness:** on document types absent from training (invoices, handwriting, charts, tables), 0.97
61
- macro-F1. **Open-vocab (OOV):** under never-seen synonym label wording, 0.80 β€” matches the *concept*, not
62
  your exact string. (Cloud VLMs train on ~all data, so these held-out axes aren't reported for them.)
63
 
64
  ## Intended use & limits
 
40
  <!-- RESULTS-TABLE:START β€” filled from leaderboard/leaderboard.json by the scoring run -->
41
  | Benchmark (macro-F1) | `document-classification-v2` | best generalist | best cloud VLM |
42
  | --- | ---: | ---: | ---: |
43
+ | **DocLayNet** (page types) | **0.97** | 0.63 | 0.83 |
44
  | **Forms** | **1.00** | 0.11 | 1.00 |
45
+ | **Tobacco** (doc types) | 0.74 | 0.43 | **0.85** |
46
+ | **OOD** (unseen doc types) | **0.95** | β€” | β€” |
47
+ | **OOV** (synonym wording) | **0.83** | β€” | β€” |
48
 
49
+ Latency (fused image+text, A40): **~4.9 pages/s** on text-heavy pages; the image branch alone runs ~8 docs/s
50
+ when no OCR is present. The open-weight
51
+ [`v1`](https://huggingface.co/nutrientdocs/document-classification-v1) runs ~5.7 pages/s fused on the same GPU.
52
  <!-- RESULTS-TABLE:END -->
53
 
54
  Every model β€” ours and cloud β€” is scored by the same open macro-F1 scorer; full per-model ranking on the
 
56
 
57
  - **Matches/leads the cloud on visual document-type tracks** (DocLayNet, Forms) at **zero per-request API
58
  cost** β€” and runs on a single GPU.
59
+ - **Trails on Tobacco** (0.74 vs 0.85): that track rewards *reading* fine header text (memo vs letter vs
60
  email) β€” a large VLM reads it; an embedding model can't.
61
+ - **OOD robustness:** on document types absent from training (invoices, handwriting, charts, tables), 0.95
62
+ macro-F1. **Open-vocab (OOV):** under never-seen synonym label wording, 0.83 β€” matches the *concept*, not
63
  your exact string. (Cloud VLMs train on ~all data, so these held-out axes aren't reported for them.)
64
 
65
  ## Intended use & limits