fusion-embedding-2-2b-preview

fusion-embedding-2-2b-preview is the second generation of Eximius Labs' unified multimodal embedding models: text, images, video, and audio in one vector space. It extends the first generation with modality-gated deep adapters — in-layer audio capacity added to a byte-frozen base. For the first-generation architecture, see fusion-embedding-1-2b-preview (that line is final at v0.3).

Sense packs extend the model to new sensors as separable, optional artifacts: Ember, the thermal sense, is the first (thermal infrared in, all other outputs bit-for-bit unchanged).

GitHub | fusion-embedding-1 | Technical report: in preparation

Model Overview

fusion-embedding-2 architecture: frozen Qwen3-VL-Embedding base with modality-gated adapters inside; frozen audio tower and trained FusionResampler on the audio branch; one shared embedding space

fusion-embedding-2-2b-preview embeds all four modalities with a Qwen3-VL-Embedding-2B base that is byte-identical to its original release — its text, image, and video behaviour (and benchmark scores) carry over exactly. Audio is added by training 60.6M parameters (~2.3% of the stack): a perceiver-resampler that translates frozen Qwen2.5-Omni audio-tower features into the base's input space, and — new in this generation — 28 gated adapters (44.2M) that give the frozen language model in-layer capacity to process audio. The adapters are active only while encoding audio; every other forward pass returns the frozen layers' output untouched, so the invariance is bitwise, not approximate (base_drift == 0 is asserted on every training run, and this model reproduces the base's text→image retrieval scores to four decimal places). Trained on 518K audio–caption pairs with a full-corpus frozen-text negative bank, it leads every unified embedding model we measured on audio↔text retrieval — ahead of ImageBind, LanguageBind, and Gemini Embedding 2 in both directions — and improves on fusion-embedding-1 v0.3 in 8 of 12 release-protocol cells, including every recorded text→audio direction. Audio↔image alignment is emergent (zero audio–image pairs in training).

Feature	Value
Parameters	~2.06B frozen base + 640M frozen audio tower; 60.6M trained
Modalities	text, image, video, audio
Supported tasks	`retrieval` (all modality pairs), `zero-shot classification`
Max input	254 text tokens · 30 s audio per window (up to 8 windows)
Embedding dimension	2048
Matryoshka dimensions	64, 128, 256, 512, 1024, 1536, 2048
Pooling strategy	Last-token pooling
Base model	Qwen/Qwen3-VL-Embedding-2B (byte-frozen)
Audio tower	Qwen/Qwen2.5-Omni-7B audio encoder (frozen)
Trained components	FusionResampler 16.4M + 28× gated adapters 44.2M
Distribution	~250 MB trained components; frozen towers download from their original repos

Training and Evaluation

Contrastive training (InfoNCE over the Matryoshka ladder, symmetric) against the frozen base's native chat-template text embeddings: 518,183 audio–caption pairs from six sources (73,716 clips with content-free metadata excluded), a full-corpus frozen-text negative bank, soft labels 0.3, false-negative masking 0.98, bf16, 3,900 steps at effective batch 1,024, then an in-domain fine-tune on an 86,394-pair AudioCaps pool: the original train split plus the 41,416 additional training clips introduced by AudioCaps 2.0 (v0.2: 1,600 steps at batch 128; v0.1 used the original 45K split for 400 steps). AudioCaps 2.0 keeps the original validation and test splits unchanged; we re-verified that its train pool is disjoint from them before ingestion. All evaluation-set audio (Clotho, ESC-50, UrbanSound8K, VGGSound, AudioCaps test/val) is excluded from training by ID blacklists at ingestion. A technical report is in preparation.

AudioCaps and AudioCaps 2.0 are by Kim et al. (AudioCaps: Generating Captions for Audios in The Wild, NAACL-HLT 2019; the 2.0 release, February 2025, in the AudioCaps repository) and are used for academic purposes per their terms; raw audio obtained via the authors' request form.

All numbers below use the release protocol (bf16 base precision, native chat-template text). Bold marks the better value per row/column.

Positioning: VGGSound-696 cross-modal retrieval versus trained parameters; the fusion-embedding family leads unified models on audio-text and leads the emergent audio-image cluster (ImageBind's supervised pair annotated)

Versus fusion-embedding-1 v0.3

Board / direction	fusion-embedding-1 v0.3	fusion-embedding-2 (this repo, v0.2)
AudioCaps A→T R@1	0.332	0.310
AudioCaps A→T R@10	0.741	0.759
AudioCaps T→A R@1	0.280	0.303
AudioCaps T→A R@10	0.746	0.785
Clotho (zero-shot) A→T R@1	0.135	0.135
Clotho (zero-shot) A→T R@10	0.433	0.414
Clotho (zero-shot) T→A R@1	0.136	0.146
Clotho (zero-shot) T→A R@10	0.460	0.485
VGGSound audio→text R@1	0.213	0.216
VGGSound audio→text R@10	0.625	0.645
VGGSound text→audio R@1	0.213	0.257
VGGSound text→audio R@10	0.645	0.697
VGGSound audio→image R@10 (emergent)	0.407	0.408

fusion-embedding-2 v0.2 takes the large majority of cells, with its largest gains on AudioCaps (the expanded fine-tuning pool) and in the text→audio direction (searching audio with a text query). fusion-embedding-1 v0.3 retains the AudioCaps A→T R@1 and Clotho A→T R@10 cells; emergent audio→image is at parity at this operating point (0.408 vs 0.407). The pre-fine-tune fusion-embedding-2 checkpoint scores 0.443 on that cell — the project record — and may be released separately as the emergent-alignment operating point.

Cross-modal retrieval — versus unified embedding models (VGGSound-AV, 696 pairs, chance R@10 = 0.014)

R@10 shown as audio-side → other / other → audio-side:

Model	audio↔image	audio↔text	text↔image
ImageBind-Huge	0.718 / 0.720	0.404 / 0.348	0.243 / 0.282
LanguageBind	0.365 / 0.415	0.547 / 0.331	0.221 / 0.283
Gemini Embedding 2 (API, 2026-07-09)	0.312 / 0.316	0.379 / 0.374	0.273 / 0.366
fusion-embedding-1-2b-preview v0.3	0.407 / 0.428	0.625 / 0.645	0.331 / 0.319
fusion-embedding-2-2b-preview (v0.2)	0.408 / 0.443	0.645 / 0.697	0.331 / 0.319

ImageBind trains directly on audio–image pairs, so that pair is its supervised direction; its audio–text alignment is emergent. LanguageBind trains audio against language; its audio↔image is emergent. Both fusion-embedding generations train on audio–text only; their audio–image alignment is emergent. All models evaluated with identical clips, frames, and scoring, using the released imagebind_huge checkpoint and revision-pinned LanguageBind checkpoints. Gemini Embedding 2 is Google's natively multimodal embedding API, evaluated at its documented default invocation on the date shown; API models may change after that date. fusion-embedding-2's text↔image cells are identical to fusion-embedding-1's by construction — text and images never touch the trained components — and this is verified: the v0.2 readout run reproduces fusion-embedding-1 v0.3's text→image scores to four decimal places.

Audio–text retrieval — versus specialist CLAP models

Specialist CLAP models fine-tune their text towers on audio captions — the direct trade this architecture declines in order to keep one shared space for all four modalities. They remain ahead on the audio-caption boards (e.g., AudioCaps T→A R@1: M2D-CLAP 41.4 vs 29.2 here); this model family is the strongest option we measured when one model must serve text, images, video, and audio together. See the fusion-embedding-1 card for the full CLAP comparison tables; fusion-embedding-2 improves on fusion-embedding-1 in the text→audio direction on every board.

MAEB (public leaderboard) — nine sound-event tasks

Both generations are on the public MAEB(beta) leaderboard through the mteb registry. The v0.2 rescore (revision 9451b840f0d1, mteb==2.18.x, identical protocol to the v0.1 submission):

Task	v0.2	v0.1
BeijingOpera	.907	.898
ClothoT2ARetrieval	.278	.285
GTZANAudioReranking	.731	.723
GTZANGenre	.665	.647
MACST2ARetrieval	.158	.158
RavdessZeroshot	.348	.304
SpeechCommandsZeroshotv0.02	.133	.141
UrbanSound8KT2ARetrieval	.010	.009
VehicleSoundClustering	.024	.018
Mean	.361	.354

Six of nine tasks improve; the two dips (Clotho task protocol, speech commands) are within the noise band of these task sizes. Speech-content and fine-grained classification remain the corpus's known weak axes — the training pool is sound-event-centric, and speech is a roadmap item, not a claim.

Changelog

v0.2 (2026-07-20): fine-tuned on the expanded AudioCaps 2.0 training pool (86,394 pairs, 1,600 steps). AudioCaps A→T R@10 0.743 → 0.759, T→A R@10 0.775 → 0.785; Clotho and VGGSound within noise of v0.1, emergent audio→image up ~1.5 points. All comparisons run against v0.1 under identical protocols. Non-audio outputs remain bit-for-bit the base model's (verified on this artifact).
v0.1 (2026-07-12): initial release (v0.1-preview tag; the revision pinned by the MTEB/MAEB leaderboard submission).

Usage

Requirements

fusion_embedding package: pip install git+https://github.com/Eximius-Labs/fusion-embedding
transformers>=4.46, torch (CUDA), torchvision, pillow, soundfile, librosa
~14 GB GPU memory at bf16

via inference.py (this repository)

from inference import FusionEmbedder

fe = FusionEmbedder.from_pretrained(
    "EximiusLabs/fusion-embedding-2-2b-preview",
    revision="v0.2-preview",   # pin a tag if you build on this model
)

a = fe.embed_audio("dog.wav")            # audio file or (array, sr=...)
t = fe.embed_text("a dog barks")         # uses the base's native chat template
i = fe.embed_image("dog.jpg")            # PIL image or path

print((a @ t).item(), (a @ i).item())    # cosine similarities in the shared space

# Matryoshka: pass dim= for smaller embeddings (64..2048)
t_small = fe.embed_text("a dog barks", dim=256)

The checkpoint contains the gated adapters and the loader refuses to run without them — an adapter checkpoint can never be silently executed as the first-generation architecture. All inputs use the base model's chat-template format; embedding quality is sensitive to this formatting, so use the templates provided by FusionEmbedder rather than constructing your own.

Cross-modal ranking tip

When ranking a gallery of one modality against queries of another, per-modality mean-centering of the gallery improves cross-modal recall by roughly two points across modality pairs:

gallery = FusionEmbedder.center(gallery_embeddings)

License

Code is Apache-2.0 (GitHub); model weights in this repository are CC BY-NC 4.0 (research preview). The frozen base and audio tower retain their original licenses.

Citation

To cite the model:

@software{fusion_embedding_2_2026,
  title  = {fusion-embedding-2},
  author = {Tonmoy, Abdul Basit},
  year   = {2026},
  url    = {https://huggingface.co/EximiusLabs/fusion-embedding-2-2b-preview}
}

To cite the paper:

@article{tonmoy2026fusion,
  title   = {Fusion Embedding: A Unified Embedding Space for Text, Image, Video, and Audio},
  author  = {Tonmoy, Abdul Basit and Hoque, Kazi Fardinul and Arham, Md. Shahrier Islam and Luthra, Arman},
  journal = {arXiv preprint arXiv:2607.18666},
  year    = {2026}
}