README.md

---
license: apache-2.0
tags:
- executorch
- xnnpack
- pte
- on-device
- mask-generation
base_model:
- facebook/sam2.1-hiera-tiny
base_model_relation: quantized
---
# SAM 2.1 hiera-tiny — ExecuTorch XNNPACK (encoder + decoder)

Two .pte files (image-segment path: run the encoder once per image, the decoder
per click):

- `sam21_tiny_encoder_xnnpack_fp32.pte` (109.2 MB) — image (1,3,1024,1024) →
  image_embed (1,256,64,64), feat_s0 (1,32,256,256), feat_s1 (1,64,128,128)
- `sam21_tiny_decoder_xnnpack_fp32.pte` (24.7 MB) — (image_embed, feat_s0, feat_s1,
  points (1,1,N,2) fp32 pixel coords in 1024-space, labels (1,1,N) int64 1=fg/0=bg)
  → mask logits (1,1,3,256,256), iou scores (1,1,3)

Both graphs also ship in fp16, and both hold their accuracy there:

- `sam21_tiny_encoder_xnnpack_fp16.pte` (55.6 MB, corr 0.999993)
- `sam21_tiny_decoder_xnnpack_fp16.pte` (12.6 MB, corr 0.999999)

Every file takes and returns fp32 tensors, so precision is a file swap — the fp16
pair is 68.2 MB against 133.9 MB. Hiera is attention-heavy, which is why fp16 halves
it cleanly.

Dynamic int8 is measured and not published. It does shrink the encoder — 109.2 MB
to 29.1 MB at worst-output correlation 0.9958 — but the number that decides is what
the decoder makes of it. Feeding the int8 encoder's embeddings to the same fp32
decoder, over 42 clicks the fp32 build answered, the worst decoded mask comes back
at **0.9375 IoU** against the fp32 encoder's mask (median 0.9956), under the 0.95
bar every mask on this shelf has to clear.

If size is the binding constraint, look at
[EdgeTAM](https://huggingface.co/mlboydaisuke/EdgeTAM-ExecuTorch) — same output
contract, 19.7 MB encoder.

- **Source**: facebook/sam2.1-hiera-tiny (transformers Sam2Model)
- **License**: Apache-2.0
- **Preprocess**: RGB/255, imagenet norm (mean .485/.456/.406, std .229/.224/.225),
  resize 1024x1024
- **Postprocess**: pick argmax(iou) of the 3 mask logits, threshold > 0, upsample
  4x (256→1024) to image space. The prompt encoder is embedded in the decoder —
  pass raw click coordinates, no separate point-encoding code needed.

## Verification (Mac arm64, executorch 1.4.0, torch 2.13.0)

Parity vs torch fp32 eager (random input): all 5 outputs corr 1.000000
(worst max_abs_diff 2.2e-05). End-to-end two-.pte chain on a synthetic image:
mask IoU vs torch = 1.0000, iou_scores match to 6 decimals.

Median latency (Mac, reference only — ViT-heavy graphs are known to time poorly
on Mac XNNPACK; device numbers to follow): encoder 2506 ms, decoder 20 ms.

## Conversion notes

torch.export → to_edge_transform_and_lower(XnnpackPartitioner) → .pte, with:
- `.contiguous()` on the encoder outputs / decoder inputs (transformers emits
  channels_last; leaving it poisons XNNPACK's runtime shape propagation)
- identity `repeat_interleave(1, dim)` calls stripped (delegate mis-sizes their
  decomposition)
- constant `dense_prompt_embeddings` (no-mask embedding) baked as a buffer