Update v2 numbers (retrained head + fuser)
Browse files
README.md
CHANGED
|
@@ -40,14 +40,15 @@ in-house model and a cloud frontier VLM reference.
|
|
| 40 |
<!-- RESULTS-TABLE:START β filled from leaderboard/leaderboard.json by the scoring run -->
|
| 41 |
| Benchmark (macro-F1) | `document-classification-v2` | best generalist | best cloud VLM |
|
| 42 |
| --- | ---: | ---: | ---: |
|
| 43 |
-
| **DocLayNet** (page types) | **0.
|
| 44 |
| **Forms** | **1.00** | 0.11 | 1.00 |
|
| 45 |
-
| **Tobacco** (doc types) | 0.
|
| 46 |
-
| **OOD** (unseen doc types) | **0.
|
| 47 |
-
| **OOV** (synonym wording) | **0.
|
| 48 |
|
| 49 |
-
Latency: **~
|
| 50 |
-
|
|
|
|
| 51 |
<!-- RESULTS-TABLE:END -->
|
| 52 |
|
| 53 |
Every model β ours and cloud β is scored by the same open macro-F1 scorer; full per-model ranking on the
|
|
@@ -55,10 +56,10 @@ Every model β ours and cloud β is scored by the same open macro-F1 scorer; f
|
|
| 55 |
|
| 56 |
- **Matches/leads the cloud on visual document-type tracks** (DocLayNet, Forms) at **zero per-request API
|
| 57 |
cost** β and runs on a single GPU.
|
| 58 |
-
- **Trails on Tobacco** (0.
|
| 59 |
email) β a large VLM reads it; an embedding model can't.
|
| 60 |
-
- **OOD robustness:** on document types absent from training (invoices, handwriting, charts, tables), 0.
|
| 61 |
-
macro-F1. **Open-vocab (OOV):** under never-seen synonym label wording, 0.
|
| 62 |
your exact string. (Cloud VLMs train on ~all data, so these held-out axes aren't reported for them.)
|
| 63 |
|
| 64 |
## Intended use & limits
|
|
|
|
| 40 |
<!-- RESULTS-TABLE:START β filled from leaderboard/leaderboard.json by the scoring run -->
|
| 41 |
| Benchmark (macro-F1) | `document-classification-v2` | best generalist | best cloud VLM |
|
| 42 |
| --- | ---: | ---: | ---: |
|
| 43 |
+
| **DocLayNet** (page types) | **0.97** | 0.63 | 0.83 |
|
| 44 |
| **Forms** | **1.00** | 0.11 | 1.00 |
|
| 45 |
+
| **Tobacco** (doc types) | 0.74 | 0.43 | **0.85** |
|
| 46 |
+
| **OOD** (unseen doc types) | **0.95** | β | β |
|
| 47 |
+
| **OOV** (synonym wording) | **0.83** | β | β |
|
| 48 |
|
| 49 |
+
Latency (fused image+text, A40): **~4.9 pages/s** on text-heavy pages; the image branch alone runs ~8 docs/s
|
| 50 |
+
when no OCR is present. The open-weight
|
| 51 |
+
[`v1`](https://huggingface.co/nutrientdocs/document-classification-v1) runs ~5.7 pages/s fused on the same GPU.
|
| 52 |
<!-- RESULTS-TABLE:END -->
|
| 53 |
|
| 54 |
Every model β ours and cloud β is scored by the same open macro-F1 scorer; full per-model ranking on the
|
|
|
|
| 56 |
|
| 57 |
- **Matches/leads the cloud on visual document-type tracks** (DocLayNet, Forms) at **zero per-request API
|
| 58 |
cost** β and runs on a single GPU.
|
| 59 |
+
- **Trails on Tobacco** (0.74 vs 0.85): that track rewards *reading* fine header text (memo vs letter vs
|
| 60 |
email) β a large VLM reads it; an embedding model can't.
|
| 61 |
+
- **OOD robustness:** on document types absent from training (invoices, handwriting, charts, tables), 0.95
|
| 62 |
+
macro-F1. **Open-vocab (OOV):** under never-seen synonym label wording, 0.83 β matches the *concept*, not
|
| 63 |
your exact string. (Cloud VLMs train on ~all data, so these held-out axes aren't reported for them.)
|
| 64 |
|
| 65 |
## Intended use & limits
|