msluszniak commited on
Commit
f3cedaa
·
verified ·
1 Parent(s): 5215a18

Document why MLX is not shipped: 2.9x slower than Core ML and non-finite output

Browse files

Measured on an iPhone 16 (ET 1.4.1, 3 warmup + 15 timed). All four styles emit NaN/Inf while Core ML returns finite output on the identical input. 50 subgraphs from missing aten.flip, _native_batch_norm_legit.no_stats (InstanceNorm) and upsample_bilinear2d handlers.

Files changed (1) hide show
  1. mlx/NOTES.md +56 -0
mlx/NOTES.md ADDED
@@ -0,0 +1,56 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ # MLX is not shipped for this model
2
+
3
+ The MLX build was measured and rejected. The exporter in `export-scripts` is
4
+ kept so the decision can be re-tested against a future ExecuTorch/MLX release,
5
+ but no `.pte` or `config.json` is published here.
6
+
7
+ ## Why: it is slower, and its output is not finite
8
+
9
+ Measured on device (iPhone 16, ExecuTorch 1.4.1, 3 warmup runs + 15 timed runs,
10
+ median, inputs allocated from each model's own schema and filled with 0.5). All
11
+ four styles share one architecture and differ only in weights, so all four were
12
+ run rather than measuring one and generalising:
13
+
14
+ | style | Core ML fp16 | MLX fp16 | MLX penalty | MLX output |
15
+ | --- | --- | --- | --- | --- |
16
+ | candy | **193.4 ms** | 564.5 ms | 2.92x slower | non-finite |
17
+ | mosaic | **194.7 ms** | 546.4 ms | 2.81x slower | non-finite |
18
+ | rain princess | **193.3 ms** | 566.7 ms | 2.93x slower | non-finite |
19
+ | udnie | **196.9 ms** | 574.7 ms | 2.92x slower | non-finite |
20
+
21
+ The latency alone would be reason enough, but the output settles it. Every MLX
22
+ build emitted NaN or Inf, while the Core ML build fed the identical input on the
23
+ same device in the same run returned finite output in all four cases. This
24
+ artifact does not merely run slowly, it produces garbage.
25
+
26
+ Size is not an argument either. MLX fp16 is 3.48 MB against Core ML fp16 at
27
+ 3.79 MB, a 0.3 MB difference on a model of this size.
28
+
29
+ ## Root cause: op coverage
30
+
31
+ The ExecuTorch MLX backend has no handler for three ops this architecture uses
32
+ everywhere, so the graph shatters into **50 subgraphs**:
33
+
34
+ | unsupported op | count | where it comes from |
35
+ | --- | --- | --- |
36
+ | `aten.flip` | 64 | the reflection pads |
37
+ | `aten._native_batch_norm_legit.no_stats` | 15 | InstanceNorm |
38
+ | `aten.upsample_bilinear2d` | 2 | the two upsampling stages |
39
+
40
+ Note the specific batch-norm overload. The MLX backend does register
41
+ `_native_batch_norm_legit_no_training`, but not the `no_stats` variant that
42
+ InstanceNorm lowers to, and a style transfer network is InstanceNorm in every
43
+ block.
44
+
45
+ This is the same class of failure as the SDXS text-to-image model, which
46
+ fragments into 28 subgraphs for want of `native_group_norm`, filed upstream as
47
+ https://github.com/pytorch/executorch/issues/22017.
48
+
49
+ ## Why style transfer is not a special case
50
+
51
+ MLX runs on the Metal GPU; Core ML lowers to the ANE, which is purpose-built for
52
+ this kind of network. MLX was measured behind Core ML on every convolutional
53
+ vision model tested. Its real advantage is transformer workloads, where
54
+ weight-only quantization shrinks the artifact substantially: CLIP vision is 97%
55
+ `nn.Linear` and goes 351 MB to 98 MB. A style transfer network is pure
56
+ convolution and lands on the wrong side of that line.