Document why MLX is not shipped: 2.9x slower than Core ML and non-finite output
Browse filesMeasured on an iPhone 16 (ET 1.4.1, 3 warmup + 15 timed). All four styles emit NaN/Inf while Core ML returns finite output on the identical input. 50 subgraphs from missing aten.flip, _native_batch_norm_legit.no_stats (InstanceNorm) and upsample_bilinear2d handlers.
- mlx/NOTES.md +56 -0
mlx/NOTES.md
ADDED
|
@@ -0,0 +1,56 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
# MLX is not shipped for this model
|
| 2 |
+
|
| 3 |
+
The MLX build was measured and rejected. The exporter in `export-scripts` is
|
| 4 |
+
kept so the decision can be re-tested against a future ExecuTorch/MLX release,
|
| 5 |
+
but no `.pte` or `config.json` is published here.
|
| 6 |
+
|
| 7 |
+
## Why: it is slower, and its output is not finite
|
| 8 |
+
|
| 9 |
+
Measured on device (iPhone 16, ExecuTorch 1.4.1, 3 warmup runs + 15 timed runs,
|
| 10 |
+
median, inputs allocated from each model's own schema and filled with 0.5). All
|
| 11 |
+
four styles share one architecture and differ only in weights, so all four were
|
| 12 |
+
run rather than measuring one and generalising:
|
| 13 |
+
|
| 14 |
+
| style | Core ML fp16 | MLX fp16 | MLX penalty | MLX output |
|
| 15 |
+
| --- | --- | --- | --- | --- |
|
| 16 |
+
| candy | **193.4 ms** | 564.5 ms | 2.92x slower | non-finite |
|
| 17 |
+
| mosaic | **194.7 ms** | 546.4 ms | 2.81x slower | non-finite |
|
| 18 |
+
| rain princess | **193.3 ms** | 566.7 ms | 2.93x slower | non-finite |
|
| 19 |
+
| udnie | **196.9 ms** | 574.7 ms | 2.92x slower | non-finite |
|
| 20 |
+
|
| 21 |
+
The latency alone would be reason enough, but the output settles it. Every MLX
|
| 22 |
+
build emitted NaN or Inf, while the Core ML build fed the identical input on the
|
| 23 |
+
same device in the same run returned finite output in all four cases. This
|
| 24 |
+
artifact does not merely run slowly, it produces garbage.
|
| 25 |
+
|
| 26 |
+
Size is not an argument either. MLX fp16 is 3.48 MB against Core ML fp16 at
|
| 27 |
+
3.79 MB, a 0.3 MB difference on a model of this size.
|
| 28 |
+
|
| 29 |
+
## Root cause: op coverage
|
| 30 |
+
|
| 31 |
+
The ExecuTorch MLX backend has no handler for three ops this architecture uses
|
| 32 |
+
everywhere, so the graph shatters into **50 subgraphs**:
|
| 33 |
+
|
| 34 |
+
| unsupported op | count | where it comes from |
|
| 35 |
+
| --- | --- | --- |
|
| 36 |
+
| `aten.flip` | 64 | the reflection pads |
|
| 37 |
+
| `aten._native_batch_norm_legit.no_stats` | 15 | InstanceNorm |
|
| 38 |
+
| `aten.upsample_bilinear2d` | 2 | the two upsampling stages |
|
| 39 |
+
|
| 40 |
+
Note the specific batch-norm overload. The MLX backend does register
|
| 41 |
+
`_native_batch_norm_legit_no_training`, but not the `no_stats` variant that
|
| 42 |
+
InstanceNorm lowers to, and a style transfer network is InstanceNorm in every
|
| 43 |
+
block.
|
| 44 |
+
|
| 45 |
+
This is the same class of failure as the SDXS text-to-image model, which
|
| 46 |
+
fragments into 28 subgraphs for want of `native_group_norm`, filed upstream as
|
| 47 |
+
https://github.com/pytorch/executorch/issues/22017.
|
| 48 |
+
|
| 49 |
+
## Why style transfer is not a special case
|
| 50 |
+
|
| 51 |
+
MLX runs on the Metal GPU; Core ML lowers to the ANE, which is purpose-built for
|
| 52 |
+
this kind of network. MLX was measured behind Core ML on every convolutional
|
| 53 |
+
vision model tested. Its real advantage is transformer workloads, where
|
| 54 |
+
weight-only quantization shrinks the artifact substantially: CLIP vision is 97%
|
| 55 |
+
`nn.Linear` and goes 351 MB to 98 MB. A style transfer network is pure
|
| 56 |
+
convolution and lands on the wrong side of that line.
|