Hugging Face's logo

embedl
/
dinov3-quantized-tensorrt

Image Classification
TensorRT
ONNX
quantization
edge
embedl
Model card Files
xet
 Community
2

How to reproduce this ONNX with the public PyPI embedl-deploy 0.8.1?

#2
by Jonnas0919 - opened
Discussion
Jonnas0919

Hi! I'm trying to reproduce this quantized model with the public embedl-deploy package (0.8.1, latest on PyPI) and the official docs workflow, but I can't get the same QDQ layout.

What I did:

  1. Load facebook/dinov3-vitb16-pretrain-lvd1689m via transformers.DINOv3ViTModel (224×224 input)
  2. transform(model, args, patterns=TENSORRT_PATTERNS)
  3. quantize(fused, args, config=QuantConfig(activation=INT8 symmetric per-tensor, weight=INT8 symmetric per-channel), forward_loop=calibration, freeze_weights=True)
  4. torch.onnx.export(..., opset 20)

Result: 98 QDQ pairs (61 weight + 24 LayerNorm output + 12 Linear output + 1 input).

Your ONNX: 120 QDQ pairs (72 weight + 24 LayerNorm output + 12 MatMul output + 12 GELU output).

Key differences I found by inspecting the official ONNX:

  • The official graph has quantization on GELU outputs and attention (attn@V) MatMul outputs — with QuantStub_66 etc. present in the initializers
  • In my exported graph, GELU is fused inside FusedLinearAct and attention is fused into FusedScaledDotProductAttention, so those outputs have no QDQ points
  • The official graph's attention seems not fused (RoPE / MatMul / Softmax nodes are explicit), while the default TENSORRT_PATTERNS fuses it

Question:
How was this ONNX generated? Is there a pattern-list configuration (e.g. skipping ScaledDotProductAttentionPattern / LinearActPattern) or a specific library version / internal tool that produces
the 120-QDQ layout? The public 0.8.1 with the documented workflow seems unable to reproduce it.

Thanks!
Magzimilian
 Embedl org

Hi!

Try to use the 0.8.0 version instead and see if that works.

BR
Maximilian
Jonnas0919

Hi!

Try to use the 0.8.0 version instead and see if that works.

BR
Maximilian

Hi Maximilian

Thank you very much for your reply and for suggesting we try 0.8.0. We appreciate you taking the time to look into this.

Following your suggestion, we downloaded embedl_deploy-0.8.0 from PyPI and ran the same workflow as before. To our surprise, 0.8.0 produces exactly the same result as 0.8.1: 170 QDQ pairs with the default QuantConfig, and the source code of the TensorRT modules appears identical between the two versions. So switching to 0.8.0 alone did not reproduce the official 120-QDQ layout on our side.

What did work: applying a skip configuration — excluding ScaledDotProductAttention and nn.Conv2d from quantization — on the default QuantConfig gives exactly 120 QDQ pairs, matching your
published layout.

One thing we could not yet confirm is whether the activation calibration in your pipeline is HISTOGRAM — that is still an assumption on our side, since with HISTOGRAM calibration, our accuracy matches your published model.

Could you confirm whether the official artifact was generated with the default QuantConfig plus some skip configuration? That would help us document the reproduction accurately.

Again, thank you for your help — the suggestion to try 0.8.0 was appreciated.

Thanks!

Sign up or log in to comment