README.md

---
license: apache-2.0
tags:
- executorch
- xnnpack
- pte
- on-device
- image-classification
base_model:
- google/vit-base-patch16-224
base_model_relation: quantized
---
# ViT-Base/16 ImageNet-1k — ExecuTorch

The original Vision Transformer, fine-tuned on ImageNet-1k: an image in, 1000 logits
out. The reference classifier most tutorials reach for, as four `.pte` files.

- **Source**: google/vit-base-patch16-224 — 86M parameters, ViT-Base/16, 12 layers,
  hidden 768, 224x224
- **License**: apache-2.0
- **Input**: `pixel_values` `[1, 3, 224, 224]` fp32 — RGB resized to 224x224, scaled to
  [0,1], then normalised with mean=(0.5, 0.5, 0.5) std=(0.5, 0.5, 0.5). **Not** the
  ImageNet statistics; read off this model's own `preprocessor_config.json`.
- **Output**: logits `[1, 1000]`, in the order of the repo's `config.json` `id2label`.
  Softmax is left to the caller.

## Variants

| build | file | size (MB) | Mac median (ms)* | labels kept | margin shift at the boundary (logits) |
|---|---|---|---|---|---|
| fp32 | `imgcls_vit_base_in1k_xnnpack_fp32.pte` | 346.4 | 34.9 | 24 of 24 | 0.0000 |
| fp16 | `imgcls_vit_base_in1k_xnnpack_fp16.pte` | 174.6 | 71.0 | 24 of 24 | 0.0117 |
| Core ML (fp16, iOS) | `imgcls_vit_base_in1k_coreml_all.pte` | 173.6 | **4.0** | 24 of 24 | 0.0547 |

\*Mac arm64, single process, median of 10. PyTorch eager fp32 on the same machine is
**27.5 ms**, so the Core ML build is **6.9x eager**, 100% delegated in one subgraph.
fp16 is slower than fp32 — XNNPACK emulates it — and is listed only because it halves
the file.

## The int8 build is not published, and correlation would have shipped it

Dynamic int8 converts, comes out at **89.7 MB**, runs at 31.7 ms, and reads
**correlation 0.999814** against fp32 eager. It keeps the fp32 label on all 24
photographs. Every number a conversion normally reports says ship it.

The number that decides says otherwise:

```
worst margin shift at the decision boundary   0.1856 logits
closest photograph's distance to a decision   0.0900 logits
```

The build's error is **twice the distance that separates the closest of these
photographs from being called something else**. It cannot be trusted to agree with fp32
on an image the model is not already sure about.

**What changed is the number of classes, not the quantisation.** The same recipe on
this shelf's two-class ViT head shifts the margin by 1.4795 logits and passes
comfortably — because with two classes the closest photograph sits **7.88 logits** from
the decision. Put a thousand classes in competition and the runner-up is usually a near
tie: median distance **1.09 logits**, minimum **0.09**. The error budget collapses by
roughly two orders of magnitude while the error itself does not.

So the honest form of the rule is not *int8 is fine on ViTs*. It is: **an error is
small or large only relative to the decision it has to survive, and a 1000-way decision
has almost no room.**

## How these were checked

Label agreement alone would not have separated any of these builds — all four keep
24 of 24. So each photograph is also walked along the gradient of its class margin
(the winner's lead over its closest rival) until the margin crosses zero, bisected onto
it, and the builds compared *there*. All 6 boundary inputs were reached for every build;
a walk that never leaves the class is reported as not measured rather than as a zero.

The boundary inputs are perturbed photographs, not natural ones. They exercise the
arithmetic where it decides something; they are not a claim about accuracy on real data.

## What is not tested here

The 24 photographs are this shelf's calibration set — street scenes, animals, people,
general views — not an ImageNet validation split. What is verified is **fidelity to the
fp32 model**, not the model's own top-1 accuracy. These files reproduce whatever the
upstream checkpoint does.

Worth knowing before you build on it: on these images the fp32 model's own confidence
runs from **0.022 to 0.501** (median). A 1000-class classifier is rarely sure, and a
caller that thresholds on probability should read that row before picking a threshold.

## Conversion

```bash
python convert/export_imgcls.py vit_base_in1k
python convert/check_imgcls.py vit_base_in1k int8
```

(conversion scripts: [executorch-models](https://github.com/john-rocky/executorch-models))