| --- |
| license: other |
| license_name: nutrient-commercial |
| pipeline_tag: zero-shot-image-classification |
| language: |
| - en |
| tags: |
| - zero-shot-image-classification |
| - image-classification |
| - document-ai |
| - document-image-classification |
| - open-vocabulary |
| - clip |
| datasets: |
| - nutrientdocs/doc-openvocab-benchmark |
| metrics: |
| - accuracy |
| --- |
| |
| # doc-img-classification · _commercial_ |
|
|
| **Classify any document or figure image against labels you choose at runtime.** `doc-img-classification` is a |
| document-adapted dual-encoder (image ↔ text) that scores an image against an *arbitrary* set of text |
| labels and returns the best match — invoices, resumes, scientific papers, financial reports, receipts, |
| charts, signatures, screenshots, and whatever label you type next. No fixed class list, no retraining to |
| add a category: it is **open-vocabulary**. |
|
|
| It is the document specialist in a field of generalists. General open-vocab models (CLIP, SigLIP) were |
| trained on web photos; this one is adapted to documents and figures — and it shows on the |
| [leaderboard](https://huggingface.co/spaces/nutrientdocs/doc-openvocab-leaderboard). **Weights are |
| commercial** (not downloadable here); this page is a spec + scorecard. |
|
|
| - 🎯 **Try it:** [doc-openvocab-demo](https://huggingface.co/spaces/nutrientdocs/doc-openvocab-demo) |
| - 🏆 **Leaderboard:** [doc-openvocab-leaderboard](https://huggingface.co/spaces/nutrientdocs/doc-openvocab-leaderboard) |
| - 📊 **Benchmark:** [doc-openvocab-benchmark](https://huggingface.co/datasets/nutrientdocs/doc-openvocab-benchmark) |
|
|
| ## Results |
|
|
| On the open [doc-openvocab-benchmark](https://huggingface.co/datasets/nutrientdocs/doc-openvocab-benchmark) |
| — document & figure images scored against a ~48-label open vocabulary — against the strongest general |
| open-vocab image classifiers. Top-1 accuracy; higher is better. |
|
|
| <!-- RESULTS-TABLE:START — filled from leaderboard/leaderboard.json by the scoring run (docs/PUBLISH.md Phase B) --> |
| | Track | `doc-img-classification` | OpenAI CLIP L/14 | LAION CLIP-H/14 | SigLIP2 SO400M | |
| | --- | ---: | ---: | ---: | ---: | |
| | **Broad** (48-label open vocab) | **.880** | .726 | .654 | .801 | |
| | **Specialized** (document categories) | **.712** | .463 | .250 | .536 | |
| | **Synonym** (paraphrase robustness) | .730 | **.738** | .672 | .689 | |
| <!-- RESULTS-TABLE:END --> |
|
|
| The point of the benchmark: this is the open-vocabulary task — labels supplied at inference, not baked |
| into a head — which a fixed-label classifier structurally cannot do. Among models that *can* do it, the |
| document-adapted one leads on document and figure imagery. Full ranking on the |
| [leaderboard](https://huggingface.co/spaces/nutrientdocs/doc-openvocab-leaderboard). |
|
|
| **Concepts, not strings.** On the *synonym* track the same images are scored against held-out paraphrases |
| of every label ("invoice" → "bill", "manuals" → "user guide"). Accuracy holds — the model matches the |
| *concept*, so your exact wording doesn't have to match ours. |
|
|
| ## Calibrating the score |
|
|
| Contrastive training maximizes *ranking*, which can leave the raw similarity overconfident. For a score |
| you can threshold, apply **temperature scaling** — divide the logits by a fitted `T` before the sigmoid. |
| It is monotonic, so ranking/accuracy is untouched and only the confidence values are repaired. On our |
| serving distribution the fitted temperature drives pairwise calibration error from **0.027 → 0.002**, and |
| it ships folded directly into the exported graph. Re-fit `T` if your input pipeline changes. |
|
|
| ## Intended use & limits |
|
|
| - **Use it for:** zero-shot classification of document page images and figure/zone crops in a free-label |
| setting — the caller supplies the candidate labels. Built for the Nutrient Maestro SDK |
| image-classification contract (a single combined ONNX graph, on-prem). |
| - **Limits:** optimized for **document and figure** imagery; general natural-image zero-shot ability is |
| preserved but not the focus. English label strings are the primary target. |
|
|
| ## License & data |
|
|
| The model **weights** are offered under a commercial Nutrient license — deployed on-prem, so your |
| documents never leave your infrastructure. The training set is not redistributed. The **evaluation** data |
| is public and reproducible: |
| [doc-openvocab-benchmark](https://huggingface.co/datasets/nutrientdocs/doc-openvocab-benchmark) |
| (CC-BY-SA-4.0, test-only). |
|
|
| > ### 📩 Get access |
| > |
| > `doc-img-classification` is commercial and its weights are not downloadable here. To run it on-prem — |
| > open-vocabulary, calibrated, private — **contact Nutrient: |
| > [nutrient.io/contact-sales](https://www.nutrient.io/contact-sales/).** |
|
|
| ## About the author |
|
|
| <a href="https://nutrient.io/"> |
| <img src="https://avatars2.githubusercontent.com/u/1527679?v=3&s=200" height="80" /> |
| </a> |
|
|
| This project is maintained and funded by [Nutrient](https://nutrient.io/) - The deterministic document infrastructure enterprises run their highest-stakes workflows on: replayable output, clear exceptions, and full audit trails on the messy, regulated documents where AI alone breaks. |
|
|
