msluszniak commited on
Commit
ddffe13
·
verified ·
1 Parent(s): f22e1b8

Remove MLX artifacts: 7.5-19.6x slower than Core ML on device

Browse files

MLX measured substantially slower than Core ML on an iPhone 16 for every convolutional vision model tested (efficientnet-v2-s 19.6x, yolo26-s 16.0x, yolo26-seg-s 7.5x, yolo26-pose-n 10.0x). MLX weight quantization also cannot shrink a conv model, since it only rewrites nn.Linear/embedding modules.

The .pte and config.json are removed; each mlx/ dir keeps a NOTES.md with the measurements. The exporter is retained in export-scripts so this can be re-tested against a future ExecuTorch/MLX release.

Earlier tags (v0.9.0 and below) are untouched and still contain the MLX artifacts.

mlx/NOTES.md ADDED
@@ -0,0 +1,50 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ # MLX is not shipped for this model
2
+
3
+ The MLX artifacts for this model were removed deliberately. The exporter in
4
+ `export-scripts` is kept so the decision can be re-tested against a future
5
+ ExecuTorch/MLX release, but no `.pte` or `config.json` is published here.
6
+
7
+ ## Why
8
+
9
+ Measured on device (iPhone 16, ExecuTorch 1.4.1, Release build, 3 warmup runs +
10
+ 15 timed runs, median):
11
+
12
+ | backend | precision | median latency |
13
+ |---|---|---|
14
+ | Core ML | fp16 | **2.4 ms** |
15
+ | MLX | fp16 | 47.0 ms |
16
+
17
+ MLX is **19.6x slower than Core ML** for EfficientNet-V2-S @384.
18
+
19
+ This is not specific to this model. MLX was measured behind Core ML on every
20
+ convolutional vision model tested, by 7.5x to 19.6x:
21
+
22
+ | model | Core ML | MLX | MLX penalty |
23
+ |---|---|---|---|
24
+ | efficientnet-v2-s @384 | 2.4 ms | 47.0 ms | 19.6x |
25
+ | yolo26-s @640 | 3.7 ms | 59.2 ms | 16.0x |
26
+ | yolo26-seg-s @640 | 11.0 ms | 82.9 ms | 7.5x |
27
+ | yolo26-pose-n @384 | 1.2 ms | 12.0 ms | 10.0x |
28
+
29
+ Two things compound the latency result:
30
+
31
+ 1. **MLX quantization cannot shrink a conv model.**
32
+ `executorch.backends.mlx.llm.quantization.quantize_model_` only rewrites
33
+ `nn.Linear` and embedding modules. EfficientNet-V2-S holds just 6.0% of its
34
+ parameters (1,281,000 of 21,458,488) in its single `nn.Linear`, so 4-bit
35
+ weight quantization buys about 5% file size. `nvfp4` is a literal no-op: it
36
+ logs `Skipping classifier.1: weight shape [1000,1280] incompatible with
37
+ nvfp4 (group_size=16)` and emits a byte-identical artifact.
38
+ 2. **Core ML quantizes the whole model.** Core ML's
39
+ `op_linear_quantizer_config` reaches the convolutions, so int8/per_channel
40
+ halves the artifact again (43.9 MB fp16 -> 22.1 MB) at identical latency.
41
+
42
+ ## Scope
43
+
44
+ This applies to convolution-dominated vision models. It does **not** generalize:
45
+
46
+ - `rfdetr-nano` (conv + transformer) measured only **1.24x** behind Core ML.
47
+ - The `lfm2.5` text encoders measured **2.7-3.1x faster on MLX** than XNNPACK.
48
+
49
+ MLX suits matmul-heavy graphs. The right backend depends on model class, so
50
+ this file is not a statement about MLX in general.
mlx/config.json DELETED
@@ -1,42 +0,0 @@
1
- {
2
- "$schema": "https://huggingface.co/software-mansion/react-native-executorch-spec/resolve/main/config.schema.json",
3
- "model": "efficientnet_v2_s",
4
- "family": "efficientnet",
5
- "capabilities": [
6
- "classification"
7
- ],
8
- "backend": "mlx",
9
- "license": "bsd-3-clause",
10
- "variants": [
11
- {
12
- "file": "efficientnet_v2_s_mlx_fp16.pte",
13
- "precision": "fp16",
14
- "quantized": false,
15
- "default": true,
16
- "methods": {
17
- "forward": {
18
- "inputs": [
19
- {
20
- "shape": [
21
- 1,
22
- 3,
23
- 384,
24
- 384
25
- ],
26
- "dtype": "float32"
27
- }
28
- ],
29
- "outputs": [
30
- {
31
- "shape": [
32
- 1,
33
- 1000
34
- ],
35
- "dtype": "float32"
36
- }
37
- ]
38
- }
39
- }
40
- }
41
- ]
42
- }
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
mlx/efficientnet_v2_s_mlx_fp16.pte DELETED
@@ -1,3 +0,0 @@
1
- version https://git-lfs.github.com/spec/v1
2
- oid sha256:fc41b9a923b89b27f95efc43d59c9651a3fe713619f61dd1d870bdd0713d7faf
3
- size 43509376