Model card: name as Hyper3-CLIP, add paper/code links, clarify objective and limitations

a7417bc verified about 14 hours ago

4.9 kB

license: openmdw-1.0
pipeline_tag: feature-extraction
tags:
  - vision-language
  - multimodal
  - image-text-retrieval
  - hyperbolic-embeddings
  - clip
  - research
  - scratch-training

Hyper3-CLIP

Hyper3-CLIP is a hyperbolic vision-language model from hyper³labs. Its 512-d image and text projections are lifted onto a Lorentz hyperboloid of learned curvature and trained with entailment objectives over a rule-based caption hierarchy, so that broader concepts and their more specific descriptions are related by explicit part-whole and parent-child terms rather than by cosine similarity alone.

This repository publishes the raw PyTorch checkpoint for the completed 500k-step scratch run reported in the paper. It is separate from the pretrained-base Hyper3-CLIP v1 SentenceTransformers release, which is a different model line; do not mix results between the two.

Paper: Hyper3-CLIP: Hierarchy-Conditioned Hyperbolic Vision-Language Training (arXiv:2608.29313)
Code: github.com/Hyper3Labs/hyper3-clip
Venue: oral at the Beyond Euclidean workshop, ECCV 2026

What the model does

During training, each caption is decomposed into a small hierarchy of queries: the full caption, sentence fragments, localized part descriptions, and extracted phrases. Each query conditions an 8-head cross-attention pooling over the image patch tokens and instantiates its own visual node in the shared hyperbolic space. Those nodes are supervised through three entailment relations, alongside the base contrastive and grounded box-level entailment objective.

Query-conditioned pooling is active only during training. At inference the model is a standard dual encoder and the pooling module adds no computation.

Artifact

Checkpoint: checkpoint_final.pt
Config: config.yaml
Training metadata: metadata.json
Base objective: uncha (model.objective), with query-conditioned pooling supplied by the model.beta_clip_* keys and data.beta_clip; see config.yaml for the full loss configuration
Vision backbone: vit_base_patch16_224, randomly initialized (vision_pretrained: false)
Text architecture/tokenizer: openai/clip-vit-base-patch32, randomly initialized (text_pretrained: false)
Embedding dimension: 512, Lorentz projection with learned curvature
Queries: at most 6 per image, at most 5 localized parts per image
Training steps: 500,000; global batch size 768; seed 31
Run id recorded in eval/summary_wide.csv: hier_beta_argent_vit_b_paper_scratch_500k_s31

Evaluation

The eval/ directory holds the paper-comparable full benchmark table and the raw wide summary row. Headline numbers from that evaluation:

COCO I2T/T2I R@10: 84.30 / 73.19
Flickr I2T/T2I R@10: 97.60 / 91.44
ImageNet top-1: 46.98%
WordNet hierarchy: TIE 3.1597, LCA 2.0786, Jaccard 0.8179
PEP AUC/AP: 96.07 / 69.36

Against the UNCHA checkpoint this model builds on, all eight reported COCO and Flickr R@5/R@10 retrieval settings improve, and multi-label mAP under a single shared evaluator improves on VOC and COCO. Two results in the paper do not improve and are worth knowing before you use this checkpoint: the ImageNet hierarchy metrics are competitive with UNCHA rather than better, and zero-shot classification is unusually sensitive to prompt wording, with large swings on fine-grained datasets such as Food-101, CUB, and Flowers-102. Treat this as a research checkpoint, not a polished production model.

Loading

This is a raw training checkpoint. Use the hyper³labs hyper3-clip codebase and the included config.yaml to instantiate the model, then load checkpoint_final.pt.

import torch

checkpoint = torch.load("checkpoint_final.pt", map_location="cpu", weights_only=False)
state_dict = checkpoint.get("model", checkpoint)

Citation

@inproceedings{mahmood2026hyper3clip,
  title     = {Hyper3-CLIP: Hierarchy-Conditioned Hyperbolic Vision-Language Training},
  author    = {Mahmood, Matin and Rueda-Toicen, Antonio and ElBassat, Mohamed and Elkerdany, Seifeldin and Wang, Weixing and de Melo, Gerard},
  booktitle = {Proceedings of the ECCV 2026 Workshop on Beyond Euclidean: Hyperbolic Deep Learning for Computer Vision},
  year      = {2026},
  eprint    = {2608.29313},
  archivePrefix = {arXiv},
  primaryClass  = {cs.CV}
}

License And Attribution

The model materials in this repository are released under OpenMDW-1.0. Redistributions should preserve NOTICE, LICENSE, and the model card when practical.

Please cite and link to the original hyper³labs model repository when publishing benchmarks, papers, derivative checkpoints, or public demos based on this model.