v1 re-scored heuristic + fused usage; v2 col

README.md

@@ -34,42 +34,58 @@ two self-contained **ONNX** graphs — an image tower and a text tower — that
 
 | Benchmark | **v1 (open)** | v2 (commercial) | best cloud VLM |
 | --- | ---: | ---: | ---: |
-| DocLayNet | **0.89** | 0.88 | 0.83 |
-| Forms | 0.80 | 1.00 | 1.00 |
-| Tobacco | 0.62 | 0.69 | 0.85 |
-| OOD (unseen types) | 0.87 | 0.97 | — |
-| OOV (synonym wording) | 0.74 | 0.80 | — |
+| DocLayNet | 0.75 | **0.97** | 0.83 |
+| Forms | 0.80 | **1.00** | 1.00 |
+| Tobacco | 0.61 | 0.74 | **0.85** |
+| OOD (unseen types) | 0.86 | **0.95** | — |
+| OOV (synonym wording) | 0.73 | **0.83** | — |
 
 Every entry is scored by the same open scorer — full ranking, plus a **generalist zero-shot baseline** and
 each cloud model, on the
-[leaderboard](https://huggingface.co/spaces/nutrientdocs/document-classification-leaderboard). v1 leads on the
-visual document-type track (DocLayNet) as a free download; like all embedding models it trails large VLMs on
-Tobacco (a read-the-header task). ~**3–7 docs/s on an A40** (image branch).
+[leaderboard](https://huggingface.co/spaces/nutrientdocs/document-classification-leaderboard). v1 is the
+free, open-weight sibling: it trails the commercial [`v2`](https://huggingface.co/nutrientdocs/document-classification-v2)
+and the large cloud VLMs on accuracy, but it's Apache-2.0 and downloadable. Like all embedding models it
+trails VLMs most on Tobacco (a read-the-header task). ~**5.7 pages/s on an A40** (fused image+text).
 
 ## Usage (ONNX)
 
 ```python
 import numpy as np, onnxruntime as ort, json
-from transformers import AutoProcessor, AutoTokenizer
+from transformers import AutoImageProcessor, AutoTokenizer
 from huggingface_hub import hf_hub_download
+from PIL import Image
 
 R = "nutrientdocs/document-classification-v1"
-img_sess = ort.InferenceSession(hf_hub_download(R, "modules/omni-image/image_model.onnx"))
-txt_sess = ort.InferenceSession(hf_hub_download(R, "modules/omni-image/text_model.onnx"))
+img_sess = ort.InferenceSession(hf_hub_download(R, "modules/omni-image/image_model.onnx"))   # SigLIP image tower
+txt_sess = ort.InferenceSession(hf_hub_download(R, "modules/omni-image/text_model.onnx"))     # Qwen text tower
 cal = json.load(open(hf_hub_download(R, "modules/omni-image/config.json")))["calibration"]
-proc = AutoProcessor.from_pretrained(R, subfolder="modules/omni-image")   # bundled preprocessor
-tok  = AutoTokenizer.from_pretrained(R, subfolder="modules/omni-image")   # bundled tokenizer (right-pad + attention_mask)
+proc = AutoImageProcessor.from_pretrained(R, subfolder="modules/omni-image")   # SigLIP image processor
+tok  = AutoTokenizer.from_pretrained(R, subfolder="modules/omni-image")         # Qwen tokenizer
 
-from PIL import Image
 labels = ["invoice", "letter", "memo", "form", "scientific article", "resume"]
+calib = lambda cos: 1 / (1 + np.exp(-(cal["scale"] * cos + cal["bias"])))
+
+def embed_text(texts, maxlen):
+    e = tok(texts, padding=True, truncation=True, max_length=maxlen, return_tensors="np")
+    return txt_sess.run(["text_emb"], {"input_ids": e["input_ids"].astype(np.int64),
+                                       "attention_mask": e["attention_mask"].astype(np.int64)})[0]  # [.,1024] L2
+
+lab = embed_text(labels, 64)                                              # label embeds, once
+
+# --- image branch: page image vs labels (image ONNX has batch=1; loop+pool for multi-page) ---
 pix = proc(images=[Image.open("doc.png").convert("RGB")], return_tensors="np")["pixel_values"].astype(np.float16)
-ie = img_sess.run(["image_emb"], {"pixel_values": pix})[0]                      # [1, 1024] L2
-enc = tok(labels, padding=True, truncation=True, max_length=64, return_tensors="np")
-te = txt_sess.run(["text_emb"], {"input_ids": enc["input_ids"].astype(np.int64),
-                                 "attention_mask": enc["attention_mask"].astype(np.int64)})[0]  # [N,1024] L2
-cos = (ie @ te.T)[0]
-probs = 1 / (1 + np.exp(-(cal["scale"] * cos + cal["bias"])))
-print(dict(zip(labels, probs.round(3).tolist())))
+ie  = img_sess.run(["image_emb"], {"pixel_values": pix})[0]              # [1,1024] L2
+image_probs = calib((ie @ lab.T)[0])                                     # [N]
+
+# --- text branch: the page's OCR text vs labels (up to ~2048 tokens) ---
+doc_text = open("doc.txt").read()
+text_probs = calib((embed_text([doc_text], 2048) @ lab.T)[0])            # [N]
+
+# --- reliability fusion: weight each branch by how DECISIVE it is (top1-top2 margin) ---
+margin = lambda p: float(np.partition(p, -2)[-1] - np.partition(p, -2)[-2])
+wi, wt = margin(image_probs), margin(text_probs); s = wi + wt + 1e-9
+fused = (wi / s) * image_probs + (wt / s) * text_probs
+print(dict(zip(labels, fused.round(3).tolist())))
 ```
 
 ## What's in this repo