Remove MLX artifacts: 7.5-19.6x slower than Core ML on device
Browse filesMLX measured substantially slower than Core ML on an iPhone 16 for every convolutional vision model tested (efficientnet-v2-s 19.6x, yolo26-s 16.0x, yolo26-seg-s 7.5x, yolo26-pose-n 10.0x). MLX weight quantization also cannot shrink a conv model, since it only rewrites nn.Linear/embedding modules.
The .pte and config.json are removed; each mlx/ dir keeps a NOTES.md with the measurements. The exporter is retained in export-scripts so this can be re-tested against a future ExecuTorch/MLX release.
Earlier tags (v0.9.0 and below) are untouched and still contain the MLX artifacts.
- mlx/NOTES.md +50 -0
- mlx/config.json +0 -42
- mlx/efficientnet_v2_s_mlx_fp16.pte +0 -3
mlx/NOTES.md
ADDED
|
@@ -0,0 +1,50 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
# MLX is not shipped for this model
|
| 2 |
+
|
| 3 |
+
The MLX artifacts for this model were removed deliberately. The exporter in
|
| 4 |
+
`export-scripts` is kept so the decision can be re-tested against a future
|
| 5 |
+
ExecuTorch/MLX release, but no `.pte` or `config.json` is published here.
|
| 6 |
+
|
| 7 |
+
## Why
|
| 8 |
+
|
| 9 |
+
Measured on device (iPhone 16, ExecuTorch 1.4.1, Release build, 3 warmup runs +
|
| 10 |
+
15 timed runs, median):
|
| 11 |
+
|
| 12 |
+
| backend | precision | median latency |
|
| 13 |
+
|---|---|---|
|
| 14 |
+
| Core ML | fp16 | **2.4 ms** |
|
| 15 |
+
| MLX | fp16 | 47.0 ms |
|
| 16 |
+
|
| 17 |
+
MLX is **19.6x slower than Core ML** for EfficientNet-V2-S @384.
|
| 18 |
+
|
| 19 |
+
This is not specific to this model. MLX was measured behind Core ML on every
|
| 20 |
+
convolutional vision model tested, by 7.5x to 19.6x:
|
| 21 |
+
|
| 22 |
+
| model | Core ML | MLX | MLX penalty |
|
| 23 |
+
|---|---|---|---|
|
| 24 |
+
| efficientnet-v2-s @384 | 2.4 ms | 47.0 ms | 19.6x |
|
| 25 |
+
| yolo26-s @640 | 3.7 ms | 59.2 ms | 16.0x |
|
| 26 |
+
| yolo26-seg-s @640 | 11.0 ms | 82.9 ms | 7.5x |
|
| 27 |
+
| yolo26-pose-n @384 | 1.2 ms | 12.0 ms | 10.0x |
|
| 28 |
+
|
| 29 |
+
Two things compound the latency result:
|
| 30 |
+
|
| 31 |
+
1. **MLX quantization cannot shrink a conv model.**
|
| 32 |
+
`executorch.backends.mlx.llm.quantization.quantize_model_` only rewrites
|
| 33 |
+
`nn.Linear` and embedding modules. EfficientNet-V2-S holds just 6.0% of its
|
| 34 |
+
parameters (1,281,000 of 21,458,488) in its single `nn.Linear`, so 4-bit
|
| 35 |
+
weight quantization buys about 5% file size. `nvfp4` is a literal no-op: it
|
| 36 |
+
logs `Skipping classifier.1: weight shape [1000,1280] incompatible with
|
| 37 |
+
nvfp4 (group_size=16)` and emits a byte-identical artifact.
|
| 38 |
+
2. **Core ML quantizes the whole model.** Core ML's
|
| 39 |
+
`op_linear_quantizer_config` reaches the convolutions, so int8/per_channel
|
| 40 |
+
halves the artifact again (43.9 MB fp16 -> 22.1 MB) at identical latency.
|
| 41 |
+
|
| 42 |
+
## Scope
|
| 43 |
+
|
| 44 |
+
This applies to convolution-dominated vision models. It does **not** generalize:
|
| 45 |
+
|
| 46 |
+
- `rfdetr-nano` (conv + transformer) measured only **1.24x** behind Core ML.
|
| 47 |
+
- The `lfm2.5` text encoders measured **2.7-3.1x faster on MLX** than XNNPACK.
|
| 48 |
+
|
| 49 |
+
MLX suits matmul-heavy graphs. The right backend depends on model class, so
|
| 50 |
+
this file is not a statement about MLX in general.
|
mlx/config.json
DELETED
|
@@ -1,42 +0,0 @@
|
|
| 1 |
-
{
|
| 2 |
-
"$schema": "https://huggingface.co/software-mansion/react-native-executorch-spec/resolve/main/config.schema.json",
|
| 3 |
-
"model": "efficientnet_v2_s",
|
| 4 |
-
"family": "efficientnet",
|
| 5 |
-
"capabilities": [
|
| 6 |
-
"classification"
|
| 7 |
-
],
|
| 8 |
-
"backend": "mlx",
|
| 9 |
-
"license": "bsd-3-clause",
|
| 10 |
-
"variants": [
|
| 11 |
-
{
|
| 12 |
-
"file": "efficientnet_v2_s_mlx_fp16.pte",
|
| 13 |
-
"precision": "fp16",
|
| 14 |
-
"quantized": false,
|
| 15 |
-
"default": true,
|
| 16 |
-
"methods": {
|
| 17 |
-
"forward": {
|
| 18 |
-
"inputs": [
|
| 19 |
-
{
|
| 20 |
-
"shape": [
|
| 21 |
-
1,
|
| 22 |
-
3,
|
| 23 |
-
384,
|
| 24 |
-
384
|
| 25 |
-
],
|
| 26 |
-
"dtype": "float32"
|
| 27 |
-
}
|
| 28 |
-
],
|
| 29 |
-
"outputs": [
|
| 30 |
-
{
|
| 31 |
-
"shape": [
|
| 32 |
-
1,
|
| 33 |
-
1000
|
| 34 |
-
],
|
| 35 |
-
"dtype": "float32"
|
| 36 |
-
}
|
| 37 |
-
]
|
| 38 |
-
}
|
| 39 |
-
}
|
| 40 |
-
}
|
| 41 |
-
]
|
| 42 |
-
}
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
mlx/efficientnet_v2_s_mlx_fp16.pte
DELETED
|
@@ -1,3 +0,0 @@
|
|
| 1 |
-
version https://git-lfs.github.com/spec/v1
|
| 2 |
-
oid sha256:fc41b9a923b89b27f95efc43d59c9651a3fe713619f61dd1d870bdd0713d7faf
|
| 3 |
-
size 43509376
|
|
|
|
|
|
|
|
|
|
|
|