Remove MLX artifacts: slower than Core ML on device
Browse filesMLX measured slower than Core ML on an iPhone 16 for this model, at matched fp32 precision and against the shipped Core ML fp16 build. These exporters only ever declared an MLX fp32 variant, so fp32-vs-fp32 is the fair and conservative comparison, and MLX still loses it.
The .pte and config.json are removed; each mlx/ dir keeps a NOTES.md with this model's own measurements. The exporter is retained in export-scripts so this can be re-tested against a future ExecuTorch/MLX release.
MLX for these models was only ever published on main / v0.10.0; no tag at v0.9.0 or below carried it. v0.9.0 and below are untouched.
- mlx/NOTES.md +57 -0
- mlx/config.json +0 -76
- mlx/selfie_segmentation_landscape_mlx_fp32.pte +0 -3
- mlx/selfie_segmentation_mlx_fp32.pte +0 -3
mlx/NOTES.md
ADDED
|
@@ -0,0 +1,57 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
# MLX is not shipped for this model
|
| 2 |
+
|
| 3 |
+
The MLX artifacts for this model were removed deliberately. The exporter in
|
| 4 |
+
`export-scripts` is kept so the decision can be re-tested against a future
|
| 5 |
+
ExecuTorch/MLX release, but no `.pte` or `config.json` is published here.
|
| 6 |
+
|
| 7 |
+
## Why
|
| 8 |
+
|
| 9 |
+
Measured on device (iPhone 16, ExecuTorch 1.4.1, Release build, 3 warmup runs +
|
| 10 |
+
15 timed runs, median), on selfie-segmentation:
|
| 11 |
+
|
| 12 |
+
| backend | precision | median latency |
|
| 13 |
+
|---|---|---|
|
| 14 |
+
| Core ML | fp32 (precision matched) | 4.5 ms |
|
| 15 |
+
| MLX | fp32 (only precision offered) | 5.1 ms |
|
| 16 |
+
| Core ML | fp16 (shipped default) | **0.3 ms** |
|
| 17 |
+
|
| 18 |
+
MLX is **1.1x slower than Core ML at matched precision**, and **17.0x slower than
|
| 19 |
+
the Core ML build that actually ships**.
|
| 20 |
+
|
| 21 |
+
These exporters only ever declared an MLX fp32 variant, so fp32-vs-fp32 is the
|
| 22 |
+
fair comparison and it is deliberately the conservative one: MLX still loses it.
|
| 23 |
+
|
| 24 |
+
**Caveat specific to this model.** At matched precision MLX is only 1.1x behind
|
| 25 |
+
Core ML, which is near parity and the closest result in this study. MLX is not
|
| 26 |
+
performing badly here. It is removed because Core ML fp16 is exceptional on this
|
| 27 |
+
model (0.3 ms, a 15x gain over its own fp32), and fp16 is what ships. If Core ML
|
| 28 |
+
fp16 ever regressed for this model, MLX would be a reasonable fallback.
|
| 29 |
+
|
| 30 |
+
## Context
|
| 31 |
+
|
| 32 |
+
MLX was measured behind Core ML on every convolutional vision model tested. The
|
| 33 |
+
margin varies widely with architecture, so each model carries its own number
|
| 34 |
+
rather than a blanket figure.
|
| 35 |
+
|
| 36 |
+
Two things compound the gap. MLX weight quantization cannot shrink a conv model:
|
| 37 |
+
`quantize_model_` only rewrites `nn.Linear` and embedding modules, and
|
| 38 |
+
EfficientNet-V2-S holds just 6.0% of its parameters in its single `nn.Linear`,
|
| 39 |
+
so 4-bit quantization buys about 5% file size. Core ML by contrast reaches the
|
| 40 |
+
convolutions.
|
| 41 |
+
|
| 42 |
+
## Scope
|
| 43 |
+
|
| 44 |
+
This concerns convolution-dominated vision models. It does **not** generalize:
|
| 45 |
+
|
| 46 |
+
- `rfdetr-nano` (conv + transformer) measured only **1.24x** behind Core ML.
|
| 47 |
+
- The `lfm2.5` text encoders measured **2.7-3.1x faster on MLX** than XNNPACK.
|
| 48 |
+
|
| 49 |
+
MLX suits matmul-heavy graphs. The right backend depends on model class, so this
|
| 50 |
+
file is not a statement about MLX in general.
|
| 51 |
+
|
| 52 |
+
## Availability history
|
| 53 |
+
|
| 54 |
+
MLX for this model was only ever published on `main` / `v0.10.0`. No earlier tag
|
| 55 |
+
(`v0.9.0` and below) carried an MLX artifact for it, so there is no pinned
|
| 56 |
+
revision from which these files can still be fetched. They remain recoverable
|
| 57 |
+
from this repository's git history.
|
mlx/config.json
DELETED
|
@@ -1,76 +0,0 @@
|
|
| 1 |
-
{
|
| 2 |
-
"$schema": "https://huggingface.co/software-mansion/react-native-executorch-spec/resolve/main/config.schema.json",
|
| 3 |
-
"model": "selfie_segmentation",
|
| 4 |
-
"family": "selfie",
|
| 5 |
-
"capabilities": [
|
| 6 |
-
"semantic-segmentation"
|
| 7 |
-
],
|
| 8 |
-
"backend": "mlx",
|
| 9 |
-
"license": "apache-2.0",
|
| 10 |
-
"variants": [
|
| 11 |
-
{
|
| 12 |
-
"file": "selfie_segmentation_mlx_fp32.pte",
|
| 13 |
-
"precision": "fp32",
|
| 14 |
-
"quantized": false,
|
| 15 |
-
"default": true,
|
| 16 |
-
"methods": {
|
| 17 |
-
"forward": {
|
| 18 |
-
"inputs": [
|
| 19 |
-
{
|
| 20 |
-
"shape": [
|
| 21 |
-
1,
|
| 22 |
-
3,
|
| 23 |
-
256,
|
| 24 |
-
256
|
| 25 |
-
],
|
| 26 |
-
"dtype": "float32"
|
| 27 |
-
}
|
| 28 |
-
],
|
| 29 |
-
"outputs": [
|
| 30 |
-
{
|
| 31 |
-
"shape": [
|
| 32 |
-
1,
|
| 33 |
-
1,
|
| 34 |
-
256,
|
| 35 |
-
256
|
| 36 |
-
],
|
| 37 |
-
"dtype": "float32"
|
| 38 |
-
}
|
| 39 |
-
]
|
| 40 |
-
}
|
| 41 |
-
}
|
| 42 |
-
},
|
| 43 |
-
{
|
| 44 |
-
"file": "selfie_segmentation_landscape_mlx_fp32.pte",
|
| 45 |
-
"precision": "fp32",
|
| 46 |
-
"quantized": false,
|
| 47 |
-
"default": true,
|
| 48 |
-
"methods": {
|
| 49 |
-
"forward": {
|
| 50 |
-
"inputs": [
|
| 51 |
-
{
|
| 52 |
-
"shape": [
|
| 53 |
-
1,
|
| 54 |
-
3,
|
| 55 |
-
144,
|
| 56 |
-
256
|
| 57 |
-
],
|
| 58 |
-
"dtype": "float32"
|
| 59 |
-
}
|
| 60 |
-
],
|
| 61 |
-
"outputs": [
|
| 62 |
-
{
|
| 63 |
-
"shape": [
|
| 64 |
-
1,
|
| 65 |
-
1,
|
| 66 |
-
144,
|
| 67 |
-
256
|
| 68 |
-
],
|
| 69 |
-
"dtype": "float32"
|
| 70 |
-
}
|
| 71 |
-
]
|
| 72 |
-
}
|
| 73 |
-
}
|
| 74 |
-
}
|
| 75 |
-
]
|
| 76 |
-
}
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
mlx/selfie_segmentation_landscape_mlx_fp32.pte
DELETED
|
@@ -1,3 +0,0 @@
|
|
| 1 |
-
version https://git-lfs.github.com/spec/v1
|
| 2 |
-
oid sha256:e682609e6d6d89d4cab614239fc3fc212ad31a70f5df6e19bd6b145a026e5106
|
| 3 |
-
size 482944
|
|
|
|
|
|
|
|
|
|
|
|
mlx/selfie_segmentation_mlx_fp32.pte
DELETED
|
@@ -1,3 +0,0 @@
|
|
| 1 |
-
version https://git-lfs.github.com/spec/v1
|
| 2 |
-
oid sha256:fbff091b26542dd8e69bf8ad49510a6a1b19f905190f596be82fefec5a6b7965
|
| 3 |
-
size 482944
|
|
|
|
|
|
|
|
|
|
|
|