Model card: name as Hyper3-CLIP, add paper/code links, clarify objective and limitations

README.md

@@ -11,52 +11,77 @@ tags:
 - scratch-training
 ---
 
-# Hyper3-CLIP beta
+# Hyper3-CLIP
 
-Hyper3-CLIP beta is the hyper³labs ViT-B hyperbolic checkpoint.
+Hyper3-CLIP is a hyperbolic vision-language model from hyper³labs. Its 512-d
+image and text projections are lifted onto a Lorentz hyperboloid of learned
+curvature and trained with entailment objectives over a rule-based caption
+hierarchy, so that broader concepts and their more specific descriptions are
+related by explicit part-whole and parent-child terms rather than by cosine
+similarity alone.
 
-This repository publishes the raw PyTorch training checkpoint for the completed
-500k-step paper-scratch run. It is the scratch-trained paper checkpoint and is
-separate from the pretrained-base
-[Hyper3-CLIP v1](https://huggingface.co/hyper3labs/hyper3-clip-v1)
-SentenceTransformers release.
+This repository publishes the raw PyTorch checkpoint for the completed
+500k-step scratch run reported in the paper. It is separate from the
+pretrained-base [Hyper3-CLIP v1](https://huggingface.co/hyper3labs/hyper3-clip-v1)
+SentenceTransformers release, which is a different model line; do not mix
+results between the two.
+
+- Paper: [Hyper3-CLIP: Hierarchy-Conditioned Hyperbolic Vision-Language Training](https://arxiv.org/abs/2608.29313) (arXiv:2608.29313)
+- Code: [github.com/Hyper3Labs/hyper3-clip](https://github.com/Hyper3Labs/hyper3-clip)
+- Venue: oral at the [Beyond Euclidean](https://sites.google.com/view/beyondeuclidean/home) workshop, ECCV 2026
+
+## What the model does
+
+During training, each caption is decomposed into a small hierarchy of queries:
+the full caption, sentence fragments, localized part descriptions, and extracted
+phrases. Each query conditions an 8-head cross-attention pooling over the image
+patch tokens and instantiates its own visual node in the shared hyperbolic
+space. Those nodes are supervised through three entailment relations, alongside
+the base contrastive and grounded box-level entailment objective.
+
+Query-conditioned pooling is active only during training. At inference the model
+is a standard dual encoder and the pooling module adds no computation.
 
 ## Artifact
 
 - Checkpoint: `checkpoint_final.pt`
 - Config: `config.yaml`
 - Training metadata: `metadata.json`
-- Objective: `uncha` (see `config.yaml` for the full loss configuration)
-- Vision backbone: `vit_base_patch16_224`
-- Vision pretrained: `false`
-- Text model architecture/tokenizer: `openai/clip-vit-base-patch32`
-- Text pretrained: `false`
-- Embedding dimension: 512
-- Training steps: 500,000
-- Global batch size: 768
+- Base objective: `uncha` (`model.objective`), with query-conditioned pooling
+  supplied by the `model.beta_clip_*` keys and `data.beta_clip`; see
+  `config.yaml` for the full loss configuration
+- Vision backbone: `vit_base_patch16_224`, randomly initialized (`vision_pretrained: false`)
+- Text architecture/tokenizer: `openai/clip-vit-base-patch32`, randomly initialized (`text_pretrained: false`)
+- Embedding dimension: 512, Lorentz projection with learned curvature
+- Queries: at most 6 per image, at most 5 localized parts per image
+- Training steps: 500,000; global batch size 768; seed 31
+- Run id recorded in `eval/summary_wide.csv`: `hier_beta_argent_vit_b_paper_scratch_500k_s31`
 
 ## Evaluation
 
-The `eval/` directory includes the paper-comparable full benchmark table and the
-raw wide summary row used for the current model comparison.
+The `eval/` directory holds the paper-comparable full benchmark table and the
+raw wide summary row. Headline numbers from that evaluation:
 
-Headline row from the local full eval:
-
-- ImageNet top-1: 46.984%
 - COCO I2T/T2I R@10: 84.30 / 73.19
 - Flickr I2T/T2I R@10: 97.60 / 91.44
+- ImageNet top-1: 46.98%
 - WordNet hierarchy: TIE 3.1597, LCA 2.0786, Jaccard 0.8179
 - PEP AUC/AP: 96.07 / 69.36
 
-The checkpoint is strong on retrieval in the paper-comparable table, but weak on
-several flat/fine-grained zero-shot datasets such as Food101, CUB, Flowers102,
-Cars, and Aircraft. Treat this release as a research checkpoint, not a polished
-production model.
+Against the UNCHA checkpoint this model builds on, all eight reported COCO and
+Flickr R@5/R@10 retrieval settings improve, and multi-label mAP under a single
+shared evaluator improves on VOC and COCO. Two results in the paper do not
+improve and are worth knowing before you use this checkpoint: the ImageNet
+hierarchy metrics are competitive with UNCHA rather than better, and zero-shot
+classification is unusually sensitive to prompt wording, with large swings on
+fine-grained datasets such as Food-101, CUB, and Flowers-102. Treat this as a
+research checkpoint, not a polished production model.
 
 ## Loading
 
-This is a raw training checkpoint. Use the hyper³labs `hyper3-clip` codebase and
-the included `config.yaml` to instantiate the model, then load
+This is a raw training checkpoint. Use the hyper³labs
+[`hyper3-clip`](https://github.com/Hyper3Labs/hyper3-clip) codebase and the
+included `config.yaml` to instantiate the model, then load
 `checkpoint_final.pt`.
 
 ```python
@@ -66,6 +91,20 @@ checkpoint = torch.load("checkpoint_final.pt", map_location="cpu", weights_only=
 state_dict = checkpoint.get("model", checkpoint)
 ```
 
+## Citation
+
+```bibtex
+@inproceedings{mahmood2026hyper3clip,
+  title     = {Hyper3-CLIP: Hierarchy-Conditioned Hyperbolic Vision-Language Training},
+  author    = {Mahmood, Matin and Rueda-Toicen, Antonio and ElBassat, Mohamed and Elkerdany, Seifeldin and Wang, Weixing and de Melo, Gerard},
+  booktitle = {Proceedings of the ECCV 2026 Workshop on Beyond Euclidean: Hyperbolic Deep Learning for Computer Vision},
+  year      = {2026},
+  eprint    = {2608.29313},
+  archivePrefix = {arXiv},
+  primaryClass  = {cs.CV}
+}
+```
+
 ## License And Attribution
 
 The model materials in this repository are released under OpenMDW-1.0.