Add animated similarity sweep + PCA semantic-features section

cdc0044 1 day ago

14.2 kB

	---
	license: other
	license_name: embedl-models-community-licence-1.0
	license_link: https://github.com/embedl/embedl-models/blob/main/LICENSE
	base_model:
	- facebook/dinov3-vitb16-pretrain-lvd1689m
	quantized_from:
	- facebook/dinov3-vitb16-pretrain-lvd1689m
	tags:
	- image-classification
	- quantization
	- onnx
	- tensorrt
	- edge
	- embedl
	gated: true
	extra_gated_heading: "Access Embedl Dinov3 Vitb16"
	extra_gated_description: "To access this model, please review and accept the terms below. Your contact information is collected solely to manage access and, with your explicit consent, to notify you about updated or new optimized models from Embedl."
	extra_gated_button_content: "Agree and request access"
	extra_gated_prompt: "By requesting access you agree to the Embedl Models Community Licence and the upstream Dinov3 Vitb16 Pretrain Lvd1689M License"
	extra_gated_fields:
	Company: text
	I agree to the Embedl Models Community Licence and upstream Dinov3 Vitb16 Pretrain Lvd1689M License: checkbox
	I consent to being contacted by Embedl about products and services (optional): checkbox
	---
	<!-- embedl-banner:start -->
	<style>
	.embedl-btn-primary { transition: background 160ms ease, box-shadow 160ms ease; }
	.embedl-btn-primary:hover { background: #4FDCE4 !important; box-shadow: 0 8px 22px rgba(45,212,221,0.45) !important; }
	.embedl-btn-secondary { transition: background 160ms ease; }
	.embedl-btn-secondary:hover { background: rgba(45,212,221,0.15) !important; }
	.embedl-headline { font-size: clamp(11px, 2.15vw, 15px) !important; }
	.embedl-btn-primary, .embedl-btn-secondary {
	font-size: clamp(11px, 1.65vw, 13px) !important;
	padding: clamp(6px, 1.1vw, 9px) clamp(10px, 1.6vw, 14px) !important;
	}
	</style>
	<div style="background:radial-gradient(600px 220px at 0% 50%,rgba(45,212,221,0.22) 0%,rgba(45,212,221,0) 60%),radial-gradient(400px 180px at 100% 100%,rgba(45,212,221,0.10) 0%,rgba(45,212,221,0) 55%),linear-gradient(135deg,#0B1626 0%,#142338 100%);border:1px solid rgba(45,212,221,0.28);border-radius:12px;padding:22px 24px;margin:0 0 24px 0;color:#F2F6FA;box-shadow:0 4px 16px rgba(11,22,38,0.18);overflow:hidden;box-sizing:border-box;max-width:100%;">
	<table style="width:100%;border-collapse:collapse;border:0;background:transparent;">
	<tr style="background:transparent;">
	<td style="vertical-align:middle;border:0;padding:0;background:transparent;">
	<div style="display:inline-block;font-size:10px;letter-spacing:0.08em;text-transform:uppercase;font-weight:700;color:#2DD4DD;background:rgba(45,212,221,0.15);border:1px solid rgba(45,212,221,0.35);padding:4px 10px;border-radius:999px;margin-bottom:10px;white-space:nowrap;">Optimized by Embedl</div>
	<div class="embedl-headline" style="font-size:15px;font-weight:700;line-height:1.35;color:#F2F6FA;margin-bottom:4px;">Need to <span style="color:#2DD4DD;white-space:nowrap;">fine-tune</span>, hit <span style="color:#2DD4DD;white-space:nowrap;">performance targets</span>, or deploy on <span style="color:#2DD4DD;white-space:nowrap;">specific hardware</span>?</div>
	<div style="font-size:13px;color:#9BA7B5;">We've got you covered.</div>
	</td>
	<td width="1%" style="vertical-align:middle;border:0;padding:0 0 0 18px;white-space:nowrap;text-align:right;background:transparent;">
	<a href="https://www.embedl.com/models" class="embedl-btn-secondary" style="display:inline-block;font-size:13px;font-weight:600;padding:9px 14px;border-radius:6px;border:1px solid #2DD4DD;color:#2DD4DD;text-decoration:none;margin-right:8px;">Learn more</a>
	<a href="https://www.embedl.com/contact" class="embedl-btn-primary" style="display:inline-block;font-size:13px;font-weight:600;padding:9px 14px;border-radius:6px;border:1px solid #2DD4DD;background:#2DD4DD;color:#0B1626;text-decoration:none;box-shadow:0 6px 18px rgba(45,212,221,0.28);">Get in touch →</a>
	</td>
	</tr>
	</table>
	</div>
	<!-- embedl-banner:end -->

	# Embedl Dinov3 Vitb16 (Quantized for TensorRT)

	Deployable INT8-quantized version of [`facebook/dinov3-vitb16-pretrain-lvd1689m`](https://huggingface.co/facebook/dinov3-vitb16-pretrain-lvd1689m),
	optimized with [embedl-deploy](https://github.com/embedl/embedl-deploy)
	for low-latency NVIDIA TensorRT inference on edge GPUs.

	## Upstream Model

	<a href="https://hfviewer.com/facebook/dinov3-vitb16-pretrain-lvd1689m?utm_source=huggingface&utm_medium=embedded_model_card&utm_campaign=facebook__dinov3-vitb16-pretrain-lvd1689m_card" target="_blank" rel="noopener">
	<img
	src="https://hfviewer.com/api/card.svg?source=facebook%2Fdinov3-vitb16-pretrain-lvd1689m&v=20260501clipcard"
	alt="Open facebook/dinov3-vitb16-pretrain-lvd1689m in hfviewer"
	width="100%"
	/>
	</a>

	## Highlights

	- Mixed-precision INT8/FP16 quantization with hardware-aware
	optimizations from [embedl-deploy](https://github.com/embedl/embedl-deploy).
	- Drop-in replacement for `facebook/dinov3-vitb16-pretrain-lvd1689m` in TensorRT pipelines —
	same input shape (224×224), same output semantics.
	- Validated accuracy within 0.48 pp of the FP32 baseline on imagenette
	(see Accuracy table below).
	- Faster than `trtexec --best` on supported NVIDIA hardware (see Performance table below).
	- Includes both ONNX (for TensorRT) and PT2
	(`torch.export`-loadable) artifacts plus runnable inference scripts.

	## Quick Start

	```bash
	pip install huggingface_hub torch torchvision matplotlib pillow numpy
	python -c "from huggingface_hub import snapshot_download; snapshot_download('embedl/dinov3-quantized-tensorrt', local_dir='.')"
	python infer_trt.py --image path/to/image.jpg # TensorRT
	# or
	python infer_pt2.py --image path/to/image.jpg # pure PyTorch via torch.export
	```

	## Files

	\| File \| Purpose \|
	\|---\|---\|
	\| `embedl_dinov3-vitb16_int8.onnx` \| INT8-quantized ONNX with Q/DQ nodes — feed to TensorRT. \|
	\| `embedl_dinov3-vitb16_int8.pt2` \| INT8-quantized `torch.export` ExportedProgram. \|
	\| `dinov3-vitb16_int8.onnx.data` \| External weight data for the ONNX — must sit next to the `.onnx`. \|
	\| `infer_trt.py` \| Build a TRT engine from the ONNX and render a patch cosine-similarity map (requires TensorRT). \|
	\| `infer_pt2.py` \| Load the `.pt2` with `torch.export.load` and render a patch cosine-similarity map. \|

	## Example Inputs

	DINOv3 is a general-purpose vision encoder — it produces rich 768-dim CLS-token
	embeddings that transfer across tasks (retrieval, segmentation, depth estimation)
	without task-specific heads.

	<table>
	<tr>
	<td><img src="https://huggingface.co/datasets/embedl/documentation-images/resolve/main/dinov3-quantized-tensorrt/demo_cats.png" width="200"/></td>
	<td><img src="https://huggingface.co/datasets/embedl/documentation-images/resolve/main/dinov3-quantized-tensorrt/demo_dog.png" width="200"/></td>
	<td><img src="https://huggingface.co/datasets/embedl/documentation-images/resolve/main/dinov3-quantized-tensorrt/demo_church.png" width="200"/></td>
	</tr>
	<tr>
	<td align="center"><em>Cats (COCO val)</em></td>
	<td align="center"><em>English springer (imagenette val)</em></td>
	<td align="center"><em>Church (imagenette val)</em></td>
	</tr>
	</table>

	Each image is resized and center-cropped to 224×224, ImageNet-normalised, and fed
	through the encoder. The output is `last_hidden_state` of shape `[1, 201, 768]` —
	1 CLS token + 4 register tokens + 196 patch tokens. Use `output[:, 0, :]` as the
	image-level embedding.

	## Patch Cosine-Similarity Maps

	<img src="https://huggingface.co/datasets/embedl/documentation-images/resolve/main/dinov3-quantized-tensorrt/anchor_sweep_dog.gif" width="100%" alt="Animated: patch similarity to a moving anchor, FP16 vs Embedl INT8"/>

	*Move the anchor (×) and the similarity map reorganizes — identically in FP16 and
	Embedl INT8. Every frame is one already-computed forward pass; the encoder runs
	once per image.*

	DINOv3's key property: pick any anchor patch (red ×) and every other patch is
	coloured by cosine similarity to that anchor. Semantically related regions light
	up without any supervision — and the INT8 engines preserve this spatial
	structure (CLS cosine vs the FP16 reference ≈ 0.70–0.81 on the images below).

	<img src="https://huggingface.co/datasets/embedl/documentation-images/resolve/main/dinov3-quantized-tensorrt/patch_similarity_cats.png" width="100%" alt="Patch similarity — COCO cats"/>

	<img src="https://huggingface.co/datasets/embedl/documentation-images/resolve/main/dinov3-quantized-tensorrt/patch_similarity_dog.png" width="100%" alt="Patch similarity — English springer"/>

	<img src="https://huggingface.co/datasets/embedl/documentation-images/resolve/main/dinov3-quantized-tensorrt/patch_similarity_church.png" width="100%" alt="Patch similarity — church"/>

	*Each row: input (anchor = red ×) · TRT FP16 on Jetson AGX Orin · Embedl INT8 on
	Jetson AGX Orin · Embedl INT8 on NVIDIA L4, with per-engine GPU compute latency.*

	## Emergent Semantic Features (PCA)

	Projecting each patch feature onto the top-3 PCA components (as RGB) shows the
	encoder decomposing scenes into semantic regions with no supervision. The PCA
	basis is fitted on the FP16 reference and reused unchanged for the INT8
	engines — matching colors mean the quantized features live in the same geometry.

	<img src="https://huggingface.co/datasets/embedl/documentation-images/resolve/main/dinov3-quantized-tensorrt/pca_features.png" width="100%" alt="PCA of patch features: FP16 vs Embedl INT8 on Orin and L4"/>

	## Performance

	Latency measured with TensorRT + `trtexec`, GPU compute time only
	(`--noDataTransfers`), CUDA Graph + Spin Wait enabled, clocks locked
	(`nvpmodel -m 0 && jetson_clocks` on Jetson).

	<img src="https://huggingface.co/datasets/embedl/documentation-images/resolve/main/dinov3-quantized-tensorrt/dinov3-quantized-tensorrt__nvidia-l4__latency.svg" alt="Dinov3 Vitb16 latency on NVIDIA L4 GPU">

	<img src="https://huggingface.co/datasets/embedl/documentation-images/resolve/main/dinov3-quantized-tensorrt/dinov3-quantized-tensorrt__orin.mountain.view__latency.svg" alt="Dinov3 Vitb16 latency on NVIDIA Jetson AGX Orin">

	<img src="https://huggingface.co/datasets/embedl/documentation-images/resolve/main/dinov3-quantized-tensorrt/dinov3-quantized-tensorrt__orin.mountain.view__memory.svg" alt="Dinov3 Vitb16 peak memory on NVIDIA Jetson AGX Orin">

	### NVIDIA L4 GPU

	\| Configuration \| Mean Latency \| Speedup vs FP16 \|
	\|---\|---\|---\|
	\| TensorRT FP16 \| 1.50 ms \| 1.00x \|
	\| TensorRT --best (unconstrained) \| 1.49 ms \| 1.01x \|
	\| Embedl Deploy INT8 \| 1.13 ms \| 1.34x \|

	### NVIDIA Jetson AGX Orin

	\| Configuration \| Mean Latency \| Speedup vs FP16 \|
	\|---\|---\|---\|
	\| TensorRT FP16 \| 2.71 ms \| 1.00x \|
	\| TensorRT --best (unconstrained) \| 2.66 ms \| 1.02x \|
	\| Embedl Deploy INT8 \| 2.25 ms \| 1.20x \|


	## Accuracy

	Evaluated on the imagenette validation split. The quantized model
	retains nearly all of the FP32 accuracy with a small tolerance.

	\| Model \| knn_top1 \|
	\|---\|---\|
	\| `facebook/dinov3-vitb16-pretrain-lvd1689m` FP32 (ours) \| 99.75% \|
	\| Embedl Dinov3 Vitb16 INT8 \| 99.26% \|

	## Creating Your Own Optimized Models

	This artifact was produced with
	[embedl-deploy](https://github.com/embedl/embedl-deploy),
	Embedl's open-source PyTorch → TensorRT deployment library. You can
	apply the same workflow to your own models — see
	[the documentation](https://github.com/embedl/embedl-deploy#readme)
	for installation and usage.

	## License

	\| Component \| License \|
	\|---\|---\|
	\| Optimized model artifacts (this repo) \| [Embedl Models Community Licence v1.0](https://github.com/embedl/embedl-models/blob/main/LICENSE) — no redistribution as a hosted service \|
	\| Upstream architecture and weights \| [Dinov3 Vitb16 Pretrain Lvd1689M License](https://huggingface.co/facebook/dinov3-vitb16-pretrain-lvd1689m) \|

	## Contact

	We offer engineering support for on-prem/edge deployments and partner
	co-marketing opportunities. Reach out at
	[contact@embedl.com](mailto:contact@embedl.com), or open an issue on
	[GitHub](https://github.com/embedl/embedl-deploy).

	<!-- embedl-discord-banner:start -->
	<style>
	.embedl-discord-btn { transition: background 160ms ease, box-shadow 160ms ease; }
	.embedl-discord-btn:hover { background: #6C77F5 !important; box-shadow: 0 8px 22px rgba(88,101,242,0.55) !important; }
	</style>
	<div style="background:radial-gradient(600px 220px at 0% 50%,rgba(88,101,242,0.22) 0%,rgba(88,101,242,0) 60%),radial-gradient(400px 180px at 100% 100%,rgba(88,101,242,0.10) 0%,rgba(88,101,242,0) 55%),linear-gradient(135deg,#0B1626 0%,#142338 100%);border:1px solid rgba(88,101,242,0.35);border-radius:12px;padding:22px 24px;margin:24px 0 0 0;color:#F2F6FA;box-shadow:0 4px 16px rgba(11,22,38,0.18);overflow:hidden;box-sizing:border-box;max-width:100%;">
	<table style="width:100%;border-collapse:collapse;border:0;background:transparent;">
	<tr style="background:transparent;">
	<td style="vertical-align:middle;border:0;padding:0;background:transparent;">
	<div style="display:inline-block;font-size:10px;letter-spacing:0.08em;text-transform:uppercase;font-weight:700;color:#A5B4FC;background:rgba(88,101,242,0.18);border:1px solid rgba(88,101,242,0.45);padding:4px 10px;border-radius:999px;margin-bottom:10px;white-space:nowrap;">Community & support</div>
	<div style="font-size:15px;font-weight:700;line-height:1.35;color:#F2F6FA;margin-bottom:4px;">Need help with this model? Chat with the Embedl team and other engineers on <span style="color:#A5B4FC;white-space:nowrap;">Discord</span>.</div>
	<div style="font-size:13px;color:#9BA7B5;">Quantization gotchas, hardware questions, fine-tuning tips — bring them all.</div>
	</td>
	<td width="1%" style="vertical-align:middle;border:0;padding:0 0 0 18px;white-space:nowrap;text-align:right;background:transparent;">
	<a href="https://discord.gg/MTbMWdKqE" class="embedl-discord-btn" style="display:inline-block;font-size:13px;font-weight:600;padding:9px 14px;border-radius:6px;border:1px solid #5865F2;background:#5865F2;color:#FFFFFF;text-decoration:none;box-shadow:0 6px 18px rgba(88,101,242,0.35);">Join our Discord →</a>
	</td>
	</tr>
	</table>
	</div>
	<!-- embedl-discord-banner:end -->