document-classification-v1 — open-weight open-vocab document classifier (ONNX)

.gitattributes

@@ -0,0 +1,37 @@
+*.7z filter=lfs diff=lfs merge=lfs -text
+*.arrow filter=lfs diff=lfs merge=lfs -text
+*.bin filter=lfs diff=lfs merge=lfs -text
+*.bz2 filter=lfs diff=lfs merge=lfs -text
+*.ckpt filter=lfs diff=lfs merge=lfs -text
+*.ftz filter=lfs diff=lfs merge=lfs -text
+*.gz filter=lfs diff=lfs merge=lfs -text
+*.h5 filter=lfs diff=lfs merge=lfs -text
+*.joblib filter=lfs diff=lfs merge=lfs -text
+*.lfs.* filter=lfs diff=lfs merge=lfs -text
+*.mlmodel filter=lfs diff=lfs merge=lfs -text
+*.model filter=lfs diff=lfs merge=lfs -text
+*.msgpack filter=lfs diff=lfs merge=lfs -text
+*.npy filter=lfs diff=lfs merge=lfs -text
+*.npz filter=lfs diff=lfs merge=lfs -text
+*.onnx filter=lfs diff=lfs merge=lfs -text
+*.ot filter=lfs diff=lfs merge=lfs -text
+*.parquet filter=lfs diff=lfs merge=lfs -text
+*.pb filter=lfs diff=lfs merge=lfs -text
+*.pickle filter=lfs diff=lfs merge=lfs -text
+*.pkl filter=lfs diff=lfs merge=lfs -text
+*.pt filter=lfs diff=lfs merge=lfs -text
+*.pth filter=lfs diff=lfs merge=lfs -text
+*.rar filter=lfs diff=lfs merge=lfs -text
+*.safetensors filter=lfs diff=lfs merge=lfs -text
+saved_model/**/* filter=lfs diff=lfs merge=lfs -text
+*.tar.* filter=lfs diff=lfs merge=lfs -text
+*.tar filter=lfs diff=lfs merge=lfs -text
+*.tflite filter=lfs diff=lfs merge=lfs -text
+*.tgz filter=lfs diff=lfs merge=lfs -text
+*.wasm filter=lfs diff=lfs merge=lfs -text
+*.xz filter=lfs diff=lfs merge=lfs -text
+*.zip filter=lfs diff=lfs merge=lfs -text
+*.zst filter=lfs diff=lfs merge=lfs -text
+*tfevents* filter=lfs diff=lfs merge=lfs -text
+modules/bge-text/tokenizer.json filter=lfs diff=lfs merge=lfs -text
+modules/omni-image/tokenizer.json filter=lfs diff=lfs merge=lfs -text

README.md

@@ -0,0 +1,89 @@
+---
+license: apache-2.0
+pipeline_tag: zero-shot-image-classification
+language:
+  - en
+tags:
+  - zero-shot-image-classification
+  - image-classification
+  - document-ai
+  - open-vocabulary
+  - open-weights
+datasets:
+  - nutrientdocs/document-classification-benchmark
+---
+
+# document-classification-v1 — open-weight
+
+**An open-weight, open-vocabulary document classifier you can download and run.** Supply any set of text
+labels at inference; the model scores a document image against them by calibrated cosine and returns a
+per-label match probability. No fixed class list, no per-class training.
+
+The **open-weight** sibling of the commercial flagship
+[`document-classification-v2`](https://huggingface.co/nutrientdocs/document-classification-v2). It ships as
+two self-contained **ONNX** graphs — an image tower and a text tower — that you run with `onnxruntime`.
+`embed_dim: 1024`; classification `p = sigmoid(scale·cos + bias)` (calibration in
+`modules/omni-image/config.json`).
+
+- 🎯 **Try it:** [document-classification-demo](https://huggingface.co/spaces/nutrientdocs/document-classification-demo)
+- 🏆 **Leaderboard:** [document-classification-leaderboard](https://huggingface.co/spaces/nutrientdocs/document-classification-leaderboard)
+- 📊 **Benchmark:** [document-classification-benchmark](https://huggingface.co/datasets/nutrientdocs/document-classification-benchmark)
+- 🏵️ **Flagship (commercial):** [document-classification-v2](https://huggingface.co/nutrientdocs/document-classification-v2)
+
+## Results (macro-F1, zero-shot)
+
+| Benchmark | **v1 (open)** | v2 (commercial) | best cloud VLM |
+| --- | ---: | ---: | ---: |
+| DocLayNet | **0.89** | 0.88 | 0.83 |
+| Forms | 0.80 | 1.00 | 1.00 |
+| Tobacco | 0.62 | 0.69 | 0.85 |
+| OOD (unseen types) | 0.87 | 0.97 | — |
+| OOV (synonym wording) | 0.74 | 0.80 | — |
+
+Every entry is scored by the same open scorer — full ranking, plus a **generalist zero-shot baseline** and
+each cloud model, on the
+[leaderboard](https://huggingface.co/spaces/nutrientdocs/document-classification-leaderboard). v1 leads on the
+visual document-type track (DocLayNet) as a free download; like all embedding models it trails large VLMs on
+Tobacco (a read-the-header task). ~**3–7 docs/s on an A40** (image branch).
+
+## Usage (ONNX)
+
+```python
+import numpy as np, onnxruntime as ort, json
+from transformers import AutoProcessor, AutoTokenizer
+from huggingface_hub import hf_hub_download
+
+R = "nutrientdocs/document-classification-v1"
+img_sess = ort.InferenceSession(hf_hub_download(R, "modules/omni-image/image_model.onnx"))
+txt_sess = ort.InferenceSession(hf_hub_download(R, "modules/omni-image/text_model.onnx"))
+cal = json.load(open(hf_hub_download(R, "modules/omni-image/config.json")))["calibration"]
+proc = AutoProcessor.from_pretrained(R, subfolder="modules/omni-image")   # bundled preprocessor
+tok  = AutoTokenizer.from_pretrained(R, subfolder="modules/omni-image")   # bundled tokenizer (right-pad + attention_mask)
+
+from PIL import Image
+labels = ["invoice", "letter", "memo", "form", "scientific article", "resume"]
+pix = proc(images=[Image.open("doc.png").convert("RGB")], return_tensors="np")["pixel_values"].astype(np.float16)
+ie = img_sess.run(["image_emb"], {"pixel_values": pix})[0]                      # [1, 1024] L2
+enc = tok(labels, padding=True, truncation=True, max_length=64, return_tensors="np")
+te = txt_sess.run(["text_emb"], {"input_ids": enc["input_ids"].astype(np.int64),
+                                 "attention_mask": enc["attention_mask"].astype(np.int64)})[0]  # [N,1024] L2
+cos = (ie @ te.T)[0]
+probs = 1 / (1 + np.exp(-(cal["scale"] * cos + cal["bias"])))
+print(dict(zip(labels, probs.round(3).tolist())))
+```
+
+## What's in this repo
+- `modules/omni-image/{image_model.onnx, text_model.onnx}` — the image + text towers (fp16, `onnxruntime`).
+- `modules/omni-image/{config.json, preprocessor_config.json, tokenizer.json}` — calibration + the
+  preprocessor and tokenizer needed to run them. That's it — nothing else required.
+
+Open weights under **Apache-2.0** — free to download and run. For the higher-accuracy commercial flagship
+(on-prem, calibrated), see [`document-classification-v2`](https://huggingface.co/nutrientdocs/document-classification-v2).
+
+## About the author
+
+<a href="https://nutrient.io/">
+  <img src="https://avatars2.githubusercontent.com/u/1527679?v=3&s=200" height="80" />
+</a>
+
+This project is maintained and funded by [Nutrient](https://nutrient.io/) - The deterministic document infrastructure enterprises run their highest-stakes workflows on: replayable output, clear exceptions, and full audit trails on the messy, regulated documents where AI alone breaks.

modules/omni-image/config.json

@@ -0,0 +1,9 @@
+{
+  "embed_dim": 1024,
+  "calibration": {
+    "scale": 39.22854489999006,
+    "bias": -7.550847457627121
+  },
+  "format": "opaque-merged",
+  "n_tensors": 1202
+}

modules/omni-image/image_model.onnx

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:2a53c51167fe784f2a079d93f05ab14ab3e8663f4b65867fb81e8871d99b5793
+size 867778973

modules/omni-image/omni-calibration.json

@@ -0,0 +1,14 @@
+{
+  "model": "checkpoints/omni-stage2-merged",
+  "split": "test",
+  "label_set": "canonical",
+  "n": 5056,
+  "scale": 39.22854489999006,
+  "bias": -7.550847457627121,
+  "nll_pair_before": 0.7197994543067421,
+  "nll_pair_after": 0.08995073535632168,
+  "ece_pair_before": 0.3980247445107305,
+  "ece_pair_after": 0.0032327794736899413,
+  "ece_top1_before": 0.18871585317257003,
+  "ece_top1_after": 0.21049625066654812
+}

modules/omni-image/preprocessor_config.json

@@ -0,0 +1,23 @@
+{
+  "do_convert_rgb": null,
+  "do_normalize": true,
+  "do_rescale": true,
+  "do_resize": true,
+  "image_mean": [
+    0.5,
+    0.5,
+    0.5
+  ],
+  "image_processor_type": "SiglipImageProcessor",
+  "image_std": [
+    0.5,
+    0.5,
+    0.5
+  ],
+  "resample": 2,
+  "rescale_factor": 0.00392156862745098,
+  "size": {
+    "height": 512,
+    "width": 512
+  }
+}

modules/omni-image/text_model.onnx

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:5b039657d9e56b39ebec1517b834a5f2c3c4e4a75e3574fe071938b803fa6004
+size 1310191414

modules/omni-image/tokenizer.json

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:93623af029cdc69b87f2864d3b2cc2424fdf16684f15e139b5b9d08ec34ced91
+size 11423701

modules/omni-image/tokenizer_config.json

@@ -0,0 +1,15 @@
+{
+  "add_prefix_space": false,
+  "backend": "tokenizers",
+  "bos_token": null,
+  "clean_up_tokenization_spaces": false,
+  "eos_token": "<|im_end|>",
+  "errors": "replace",
+  "is_local": false,
+  "local_files_only": false,
+  "model_max_length": 131072,
+  "pad_token": "<|endoftext|>",
+  "split_special_tokens": false,
+  "tokenizer_class": "Qwen2Tokenizer",
+  "unk_token": null
+}