msluszniak commited on
Commit
62fbc64
·
verified ·
1 Parent(s): 3735331

Remove MLX artifacts: slower than Core ML on device

Browse files

MLX measured slower than Core ML on an iPhone 16 for this model, at matched fp32 precision and against the shipped Core ML fp16 build. These exporters only ever declared an MLX fp32 variant, so fp32-vs-fp32 is the fair and conservative comparison, and MLX still loses it.

The .pte and config.json are removed; each mlx/ dir keeps a NOTES.md with this model's own measurements. The exporter is retained in export-scripts so this can be re-tested against a future ExecuTorch/MLX release.

MLX for these models was only ever published on main / v0.10.0; no tag at v0.9.0 or below carried it. v0.9.0 and below are untouched.

mlx/NOTES.md ADDED
@@ -0,0 +1,57 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ # MLX is not shipped for this model
2
+
3
+ The MLX artifacts for this model were removed deliberately. The exporter in
4
+ `export-scripts` is kept so the decision can be re-tested against a future
5
+ ExecuTorch/MLX release, but no `.pte` or `config.json` is published here.
6
+
7
+ ## Why
8
+
9
+ Measured on device (iPhone 16, ExecuTorch 1.4.1, Release build, 3 warmup runs +
10
+ 15 timed runs, median), on selfie-segmentation:
11
+
12
+ | backend | precision | median latency |
13
+ |---|---|---|
14
+ | Core ML | fp32 (precision matched) | 4.5 ms |
15
+ | MLX | fp32 (only precision offered) | 5.1 ms |
16
+ | Core ML | fp16 (shipped default) | **0.3 ms** |
17
+
18
+ MLX is **1.1x slower than Core ML at matched precision**, and **17.0x slower than
19
+ the Core ML build that actually ships**.
20
+
21
+ These exporters only ever declared an MLX fp32 variant, so fp32-vs-fp32 is the
22
+ fair comparison and it is deliberately the conservative one: MLX still loses it.
23
+
24
+ **Caveat specific to this model.** At matched precision MLX is only 1.1x behind
25
+ Core ML, which is near parity and the closest result in this study. MLX is not
26
+ performing badly here. It is removed because Core ML fp16 is exceptional on this
27
+ model (0.3 ms, a 15x gain over its own fp32), and fp16 is what ships. If Core ML
28
+ fp16 ever regressed for this model, MLX would be a reasonable fallback.
29
+
30
+ ## Context
31
+
32
+ MLX was measured behind Core ML on every convolutional vision model tested. The
33
+ margin varies widely with architecture, so each model carries its own number
34
+ rather than a blanket figure.
35
+
36
+ Two things compound the gap. MLX weight quantization cannot shrink a conv model:
37
+ `quantize_model_` only rewrites `nn.Linear` and embedding modules, and
38
+ EfficientNet-V2-S holds just 6.0% of its parameters in its single `nn.Linear`,
39
+ so 4-bit quantization buys about 5% file size. Core ML by contrast reaches the
40
+ convolutions.
41
+
42
+ ## Scope
43
+
44
+ This concerns convolution-dominated vision models. It does **not** generalize:
45
+
46
+ - `rfdetr-nano` (conv + transformer) measured only **1.24x** behind Core ML.
47
+ - The `lfm2.5` text encoders measured **2.7-3.1x faster on MLX** than XNNPACK.
48
+
49
+ MLX suits matmul-heavy graphs. The right backend depends on model class, so this
50
+ file is not a statement about MLX in general.
51
+
52
+ ## Availability history
53
+
54
+ MLX for this model was only ever published on `main` / `v0.10.0`. No earlier tag
55
+ (`v0.9.0` and below) carried an MLX artifact for it, so there is no pinned
56
+ revision from which these files can still be fetched. They remain recoverable
57
+ from this repository's git history.
mlx/config.json DELETED
@@ -1,76 +0,0 @@
1
- {
2
- "$schema": "https://huggingface.co/software-mansion/react-native-executorch-spec/resolve/main/config.schema.json",
3
- "model": "selfie_segmentation",
4
- "family": "selfie",
5
- "capabilities": [
6
- "semantic-segmentation"
7
- ],
8
- "backend": "mlx",
9
- "license": "apache-2.0",
10
- "variants": [
11
- {
12
- "file": "selfie_segmentation_mlx_fp32.pte",
13
- "precision": "fp32",
14
- "quantized": false,
15
- "default": true,
16
- "methods": {
17
- "forward": {
18
- "inputs": [
19
- {
20
- "shape": [
21
- 1,
22
- 3,
23
- 256,
24
- 256
25
- ],
26
- "dtype": "float32"
27
- }
28
- ],
29
- "outputs": [
30
- {
31
- "shape": [
32
- 1,
33
- 1,
34
- 256,
35
- 256
36
- ],
37
- "dtype": "float32"
38
- }
39
- ]
40
- }
41
- }
42
- },
43
- {
44
- "file": "selfie_segmentation_landscape_mlx_fp32.pte",
45
- "precision": "fp32",
46
- "quantized": false,
47
- "default": true,
48
- "methods": {
49
- "forward": {
50
- "inputs": [
51
- {
52
- "shape": [
53
- 1,
54
- 3,
55
- 144,
56
- 256
57
- ],
58
- "dtype": "float32"
59
- }
60
- ],
61
- "outputs": [
62
- {
63
- "shape": [
64
- 1,
65
- 1,
66
- 144,
67
- 256
68
- ],
69
- "dtype": "float32"
70
- }
71
- ]
72
- }
73
- }
74
- }
75
- ]
76
- }
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
mlx/selfie_segmentation_landscape_mlx_fp32.pte DELETED
@@ -1,3 +0,0 @@
1
- version https://git-lfs.github.com/spec/v1
2
- oid sha256:e682609e6d6d89d4cab614239fc3fc212ad31a70f5df6e19bd6b145a026e5106
3
- size 482944
 
 
 
 
mlx/selfie_segmentation_mlx_fp32.pte DELETED
@@ -1,3 +0,0 @@
1
- version https://git-lfs.github.com/spec/v1
2
- oid sha256:fbff091b26542dd8e69bf8ad49510a6a1b19f905190f596be82fefec5a6b7965
3
- size 482944