Card: MODA Pro Lite+ — results, recipe, serving_ann.py; process detail removed

README.md

@@ -1,110 +1,117 @@
 ---
 license: apache-2.0
-library_name: open_clip
-pipeline_tag: zero-shot-image-classification
 tags:
-- fashion
-- retrieval
-- text-to-image
-- siglip2
-- e-commerce
-- moda
-datasets:
-- Marqo/marqo-GS-10M
-base_model: timm/ViT-B-16-SigLIP2-384
-model-index:
-- name: MODA Pro Lite
-  results:
-  - task: {type: text-to-image-retrieval, name: Text-to-Image Retrieval}
-    dataset: {type: kaggle-fashion, name: KAGL (full corpus, 44K)}
-    metrics:
-    - {type: map@10, value: 0.31851, name: MAP@10}
-  - task: {type: text-to-image-retrieval, name: Text-to-Image Retrieval}
-    dataset: {type: polyvore, name: Polyvore (full corpus, 94K)}
-    metrics:
-    - {type: map@10, value: 0.39970, name: MAP@10}
-  - task: {type: text-to-image-retrieval, name: Text-to-Image Retrieval}
-    dataset: {type: atlas, name: Atlas (full corpus, 78K)}
-    metrics:
-    - {type: map@10, value: 0.19451, name: MAP@10}
-  - task: {type: text-to-image-retrieval, name: Text-to-Image Retrieval}
-    dataset: {type: fashion200k, name: Fashion200K (full corpus, 202K)}
-    metrics:
-    - {type: map@10, value: 0.18023, name: MAP@10}
+  - fashion
+  - retrieval
+  - text-to-image
+  - open_clip
+  - siglip2
+pipeline_tag: feature-extraction
+library_name: open_clip
 ---
 
-# MODA Pro Lite
+# MODA Pro Lite+
+
+**The strongest open system at ≤250M parameters on catalogue and title search.**
 
-**A 213M open-weights fashion encoder that beats [MODA](https://huggingface.co/HopitAI/moda-fashionsiglip-multiview-203m) — the strongest open fashion retrieval system — on catalog search, at full corpus and with statistical significance, from one checkpoint with no serving machinery.**
+MODA Pro Lite+ is a 213M fashion retrieval encoder (SigLIP2-base-384 backbone,
+fashion-vocabulary build) served with a calibrated multi-view recipe. Both ship here:
+the weights, and `serving_ann.py` which implements the recipe.
 
-By [Hopit AI](https://hopit.ai). Weights are open (Apache-2.0); the training and data pipeline are proprietary.
+The encoder on its own is **MODA Pro Lite**; with the recipe it is **MODA Pro Lite+**.
+Same weights, same repository, one extra step at index time and none at query time.
+Numbers for both are below.
 
-| vs MODA, full corpus | MODA Pro Lite | gain |
-|---|--:|--:|
-| KAGL (44K products) | **0.3185** vs 0.2890 | **+10.2%** |
-| Polyvore (94K) | **0.3997** vs 0.3726 | **+7.3%** |
-| ZooClaw-Fashion short | **0.4994** vs 0.4775 | **+4.6%** |
+## Results
 
-The KAGL and Polyvore gains are significant under a paired bootstrap (10,000 resamples, 95% CI excludes zero); on Atlas the difference is positive but inconclusive, and on Fashion200K and In-Shop MODA is significantly better — see the full table. Every number in this card is a **full-corpus** measurement (no subsampled galleries) through one shared harness, alongside FashionSigLIP, SigLIP-SO400M, and ZooClaw under identical protocol. Full six-system table: [benchmark page](https://hopit-ai.github.io/Moda/).
+MAP@10, full corpus, all ground-truth queries, one evaluator (`pytrec_eval map_cut.10`).
+`MODA` is FashionSigLIP with its own serving recipe, shown for reference.
 
-## The MODA family
+| benchmark | MODA | Pro Lite (bare) | **Pro Lite+** (with recipe) |
+|---|---:|---:|---:|
+| KAGL | 0.2887 | 0.3055 | **0.3201** |
+| Polyvore | 0.3726 | 0.3952 | **0.4049** |
+| Atlas | 0.1862 | 0.1814 | **0.1904** |
+| Fashion200K | **0.1946** | 0.1758 | 0.1846 |
+| DeepFashion In-Shop | **0.1642** | 0.0930 | 0.1026 |
+| DeepFashion Multimodal | **0.0147** | 0.0118 | 0.0133 |
 
-| model | what it is | availability |
-|---|---|---|
-| [**MODA**](https://huggingface.co/HopitAI/moda-fashionsiglip-multiview-203m) | frozen FashionSigLIP + zero-parameter multi-view serving recipe | open source + open weights |
-| **MODA Pro Lite** (this model) | trained 213M encoder — the drop-in single-checkpoint option | **open weights** |
-| **MODA Pro** | our hosted retrieval system. Rank 1 or 2 on **9 of 10** benchmark cells; +6.9% mean over MODA on the six academic sets | closed · hosted by Hopit AI |
+**Pro Lite+ leads the ≤250M class on KAGL, Polyvore and Atlas** — +10.9% over MODA on KAGL,
++8.7% on Polyvore, both significant under a paired bootstrap (10,000 resamples).
 
-**Which to pick:** already running MODA's recipe → keep MODA (it still leads on caption-style benchmarks). Want one checkpoint, standard CLIP usage, strongest catalog/title search → **Pro Lite**. Want the strongest overall → [MODA Pro](https://hopit.ai) (hosted).
+The recipe is worth +2.5% to +12.8% over the bare encoder on every benchmark, and costs
+nothing at query time: the views are fused into a single vector before indexing.
 
-## Usage
+**Where this model is weak, stated plainly.** Pro Lite is tuned for short catalogue titles.
+On long natural-language descriptions it trails FashionSigLIP substantially — DeepFashion
+In-Shop queries average 75 words, and Pro Lite+ scores 0.1026 there against MODA's 0.1642.
+If your queries are descriptions rather than titles, use
+[MODA Duo](https://huggingface.co/HopitAI/moda-duo), which routes per query.
+
+## Serving cost
+
+```
+stored vectors per item : 1
+ANN queries per search  : 1
+image forwards at index : 3x   offline, paid once
+text forwards per query : 2x   negligible beside the ANN probe
+```
+
+The recipe is a rule for *what you encode*, not a model change. Views are combined into one
+unit vector before indexing, so nearest-neighbour search costs exactly what the bare encoder
+costs — same index, same probe, no extra routes and no re-ranking.
+
+## Use
+
+```bash
+pip install open_clip_torch pillow numpy hnswlib
+python serving_ann.py --demo
+```
 
 ```python
-import torch, open_clip
-from PIL import Image
+from serving_ann import load, encode_images, encode_queries, build_index, search
+
+enc   = load()                          # open_clip, this repo's weights
+docs  = encode_images(catalogue, enc)   # (n, 768) float32, one vector per item
+index = build_index(docs)               # hnswlib, cosine via inner product
+qry   = encode_queries(["black leather ankle boots"], enc)
+ids, scores = search(index, qry, k=10)
+```
+
+Bare encoder, if you would rather not use the recipe:
+
+```python
+import open_clip, torch
 
 model, _, preprocess = open_clip.create_model_and_transforms("hf-hub:HopitAI/moda-pro-lite")
 tokenizer = open_clip.get_tokenizer("hf-hub:HopitAI/moda-pro-lite")
 model.eval()
 
-image = preprocess(Image.open("product.jpg")).unsqueeze(0)
-text = tokenizer(["red floral maxi dress", "slim-fit denim jacket"])
-
-with torch.no_grad(), torch.autocast("cuda"):
-    img_emb = model.encode_image(image)
-    txt_emb = model.encode_text(text)
-    img_emb /= img_emb.norm(dim=-1, keepdim=True)
-    txt_emb /= txt_emb.norm(dim=-1, keepdim=True)
-    scores = txt_emb @ img_emb.T
+with torch.no_grad():
+    image = torch.nn.functional.normalize(model.encode_image(preprocess(img).unsqueeze(0)), dim=-1)
+    text  = torch.nn.functional.normalize(model.encode_text(tokenizer(["black leather ankle boots"])), dim=-1)
+    score = (text @ image.T).item()
 ```
 
-Plain bi-encoder: one 768-d vector per image, one per query, cosine similarity. Works with any vector database; no reranker, no prompt template required.
-
-## What's inside (summary)
+768-d embeddings, cosine similarity, one vector per item. Index them in any vector database.
 
-- **Base:** SigLIP2-base-384 (`timm/ViT-B-16-SigLIP2-384`), full fine-tune of both towers.
-- **Size — 213,159,938 params:** the base's 256K-token multilingual vocabulary is trimmed to the 44,646 pieces fashion text actually uses (BPE merge-closure preserved). Verified: **100.00% bit-identical** outputs on 12,000 benchmark queries and 6,000 held-out fashion texts vs the untrimmed 375M model — every benchmark number in this card transfers exactly. Out-of-vocabulary input (emoji, non-Latin scripts, rare typos) re-segments gracefully into kept pieces (cosine 0.82–0.97 to full model), never a silent failure.
-- **Data:** 93K commercial product images (from [Marqo-GS-10M](https://huggingface.co/datasets/Marqo/marqo-GS-10M), Apache-2.0), re-annotated by a frontier VLM with graded relevance labels (99.9% coverage).
-- **Training:** a proprietary fine-tuning recipe; the pipeline and annotated dataset are not released.
-- **Distinctive property:** unlike typical fashion fine-tunes, Pro Lite does **not** collapse out of domain — it adds fashion skill while keeping generalist behavior (image-tower drift from base: 0.36%).
+## The recipe
 
-## Evaluation protocol
+```
+document = normalize(official + 0.25 * square_pad + 0.25 * foreground_pad)
+query    = normalize(raw      + 0.25 * "a photo of {query}")
+```
 
-Measured on six academic fashion benchmarks (DeepFashion In-Shop, DeepFashion Multimodal, Fashion200K, Polyvore, Atlas, KAGL) at **full corpus** — no gallery subsampling — plus H&M (10K, MRR) and ZooClaw-Fashion (12K, MAP@10), all through one open harness with identical preprocessing per model. Where Pro Lite loses (instance retrieval, caption-style sets), the card for [MODA](https://huggingface.co/HopitAI/moda-fashionsiglip-multiview-203m) shows the stronger pick — see the [full comparison](https://hopit-ai.github.io/Moda/) rather than a cherry-picked subset.
+`serving_ann.py` implements it. Zero added parameters, one stored vector.
 
-## Using this in production
+## Evaluation
 
-MODA Pro Lite is free to use under Apache-2.0. If you want the hosted MODA Pro,
-help evaluating on your own catalog, or a pilot, [book a call](https://calendly.com/arkid_/new-meeting?back=1) — we bring
-specific examples from your category and a proposed pilot shape.
+All figures are full corpus, all ground-truth queries, MAP@10 under one evaluator
+(`pytrec_eval map_cut.10`), float32. Per-query results and confidence intervals are in the
+[repository](https://github.com/hopit-ai/Moda).
 
-## Citation
+## Related
 
-```bibtex
-@misc{modaprolite2026,
-  title  = {MODA Pro Lite: a fashion encoder that does not collapse out of domain},
-  author = {Hopit AI},
-  year   = {2026},
-  url    = {https://huggingface.co/HopitAI/moda-pro-lite}
-}
-```
+- [MODA Duo](https://huggingface.co/HopitAI/moda-duo) — routes each query to Pro Lite+ or MODA by its shape; beats both on a mixed workload.
+- [MODA](https://huggingface.co/HopitAI/moda-fashionsiglip-multiview-203m) — FashionSigLIP with a serving recipe. Stronger on long descriptions.
+- [MODA-SigLIP-Distilled](https://huggingface.co/HopitAI/moda-fashion-distilled) — image-to-image retrieval.

serving_ann.py

@@ -0,0 +1,173 @@
+"""MODA Pro Lite+ (moda-pro-lite with its calibrated serving harness) -- retrieval with ANN, end to end.
+
+The harness below was selected on held-out development data only (OpenVTON + GLAMI); no target benchmark was touched during selection.
+It adds ZERO parameters and stores ONE vector per item.
+
+Serving cost
+------------
+  stored vectors per item : 1
+  ANN queries per search  : 1
+  image forwards at index : 3x (paid once, offline)
+  text forwards per query : 2x (cheap next to the ANN probe)
+
+The harness is a recipe for WHAT YOU ENCODE, not a model change: the views are
+combined into a single unit vector before indexing, so nearest-neighbour search
+costs exactly what it costs for the bare model. No extra routes, no re-ranking.
+
+    pip install open_clip_torch pillow numpy hnswlib
+
+Example
+-------
+    python serving_ann.py --demo
+"""
+from __future__ import annotations
+
+import argparse
+import math
+from typing import Sequence
+
+import numpy as np
+import torch
+import torch.nn.functional as F
+from PIL import Image
+
+MODEL = "hf-hub:HopitAI/moda-pro-lite"
+IMAGE_MIX = {'official': 1.0, 'pad': 0.25, 'foreground_pad': 0.25}          # view -> weight, combined then L2-normalised
+PROMPT_MIX = {'raw': 1.0, 'photo': 0.25}        # prompt -> weight, combined then L2-normalised
+PROMPT_TEMPLATES = {
+    "raw": "{query}",
+    "photo": "a photo of {query}",
+    "product": "a fashion product photo of {query}",
+}
+
+
+# ---------------------------------------------------------------- views ----
+def square_pad(image: Image.Image, fill: int = 128) -> Image.Image:
+    image = image.convert("RGB")
+    side = max(image.size)
+    canvas = Image.new("RGB", (side, side), (fill, fill, fill))
+    canvas.paste(image, ((side - image.width) // 2, (side - image.height) // 2))
+    return canvas
+
+
+def center_square(image: Image.Image) -> Image.Image:
+    side = min(image.size)
+    left, top = (image.width - side) // 2, (image.height - side) // 2
+    return image.convert("RGB").crop((left, top, left + side, top + side))
+
+
+def foreground_square(image: Image.Image) -> Image.Image:
+    """Crop a near-white catalog border, then pad without distorting aspect."""
+    image = image.convert("RGB")
+    preview = image.copy()
+    preview.thumbnail((256, 256), Image.Resampling.BILINEAR)
+    mask = np.any(np.asarray(preview, dtype=np.uint8) < 242, axis=-1)
+    if float(mask.mean()) < 0.01:
+        return square_pad(image)
+    ys, xs = np.nonzero(mask)
+    sx, sy = image.width / preview.width, image.height / preview.height
+    left = max(0, math.floor(float(xs.min()) * sx))
+    right = min(image.width, math.ceil(float(xs.max() + 1) * sx))
+    top = max(0, math.floor(float(ys.min()) * sy))
+    bottom = min(image.height, math.ceil(float(ys.max() + 1) * sy))
+    mx, my = max(1, round((right - left) * 0.05)), max(1, round((bottom - top) * 0.05))
+    return square_pad(image.crop((max(0, left - mx), max(0, top - my),
+                                  min(image.width, right + mx), min(image.height, bottom + my))))
+
+
+VIEWS = {
+    "official": lambda im: im.convert("RGB"),
+    "pad": square_pad,
+    "pad_white": lambda im: square_pad(im, fill=255),
+    "center_crop": center_square,
+    "foreground_pad": foreground_square,
+}
+
+
+# -------------------------------------------------------------- encoding ----
+def load(device: str = "cpu"):
+    import open_clip
+    model, _, preprocess = open_clip.create_model_and_transforms(MODEL)
+    model.eval().to(device)
+    for p in model.parameters():
+        p.requires_grad = False
+    return model, preprocess, open_clip.get_tokenizer(MODEL), device
+
+
+@torch.inference_mode()
+def encode_images(images: Sequence[Image.Image], enc, batch_size: int = 32) -> np.ndarray:
+    """-> (n, 768) float32, unit norm. ONE vector per item."""
+    model, preprocess, _, device = enc
+    parts = {v: [] for v in IMAGE_MIX}
+    for s in range(0, len(images), batch_size):
+        chunk = images[s:s + batch_size]
+        for view in IMAGE_MIX:
+            px = torch.stack([preprocess(VIEWS[view](im)) for im in chunk]).to(device)
+            parts[view].append(F.normalize(model.encode_image(px).float(), dim=-1).cpu())
+    fused = sum(w * torch.cat(parts[v]) for v, w in IMAGE_MIX.items())
+    return F.normalize(fused, dim=-1).numpy().astype("float32")
+
+
+@torch.inference_mode()
+def encode_queries(texts: Sequence[str], enc, batch_size: int = 128) -> np.ndarray:
+    """-> (n, 768) float32, unit norm. ONE vector per query."""
+    model, _, tokenizer, device = enc
+    parts = {p: [] for p in PROMPT_MIX}
+    for s in range(0, len(texts), batch_size):
+        chunk = texts[s:s + batch_size]
+        for prompt in PROMPT_MIX:
+            rendered = [PROMPT_TEMPLATES[prompt].format(query=t) for t in chunk]
+            tok = tokenizer(rendered).to(device)
+            parts[prompt].append(F.normalize(model.encode_text(tok).float(), dim=-1).cpu())
+    fused = sum(w * torch.cat(parts[p]) for p, w in PROMPT_MIX.items())
+    return F.normalize(fused, dim=-1).numpy().astype("float32")
+
+
+# ------------------------------------------------------------------ ANN ----
+def build_index(vectors: np.ndarray, m: int = 32, ef_construction: int = 200):
+    """Cosine similarity over unit vectors is inner product, so use space='ip'."""
+    import hnswlib
+    index = hnswlib.Index(space="ip", dim=vectors.shape[1])
+    index.init_index(max_elements=len(vectors), ef_construction=ef_construction, M=m)
+    index.add_items(vectors, np.arange(len(vectors)))
+    return index
+
+
+def search(index, queries: np.ndarray, k: int = 10, ef: int = 64):
+    index.set_ef(max(ef, k))
+    ids, distances = index.knn_query(queries, k=k)
+    return ids, 1.0 - distances          # ip distance -> cosine similarity
+
+
+def search_exact(vectors: np.ndarray, queries: np.ndarray, k: int = 10):
+    """Ground truth, for checking ANN recall on your own corpus."""
+    sims = queries @ vectors.T
+    ids = np.argsort(-sims, axis=1)[:, :k]
+    return ids, np.take_along_axis(sims, ids, axis=1)
+
+
+def _demo() -> None:
+    enc = load()
+    corpus = [Image.new("RGB", (w, h), c) for w, h, c in
+              [(224, 300, "white"), (300, 224, "black"), (256, 256, "navy"),
+               (400, 200, "beige"), (200, 400, "maroon")]]
+    doc = encode_images(corpus, enc)
+    qry = encode_queries(["black leather ankle boots", "navy wool coat"], enc)
+    print(f"documents {doc.shape}  queries {qry.shape}  (one vector each)")
+    index = build_index(doc)
+    ann_ids, ann_scores = search(index, qry, k=3)
+    ex_ids, _ = search_exact(doc, qry, k=3)
+    agree = float(np.mean([len(set(a) & set(b)) / len(b) for a, b in zip(ann_ids, ex_ids)]))
+    print(f"ANN top-3 ids {ann_ids.tolist()}")
+    print(f"exact top-3 ids {ex_ids.tolist()}")
+    print(f"ANN/exact overlap@3: {agree:.3f} (1.000 expected on a corpus this small)")
+
+
+if __name__ == "__main__":
+    ap = argparse.ArgumentParser(description=__doc__)
+    ap.add_argument("--demo", action="store_true")
+    args = ap.parse_args()
+    if args.demo:
+        _demo()
+    else:
+        ap.print_help()