Core ML: pin compute_unit=CPU_ONLY, drop the fp32 variant
Browse filesThese graphs fail the Apple Neural Engine path (coremltools#2825 masked mean-pool over a dynamic sequence dim; clip-text additionally ANECCompile FAILED). Core ML still returned correct output because it silently fell back to CPU, but the failed ANE compile was paid on every call. Pinning compute_unit=CPU_ONLY skips the attempt.
Measured on an iPhone 16, fp16, 3 warmup + 15 timed runs, median:
all-MiniLM 13.7 ms -> 2.5 ms 5.5x
multi-qa 14.1 ms -> 2.5 ms 5.6x
paraphrase-L12 26.9 ms -> 4.2 ms 6.4x
distiluse 33.0 ms -> 8.0 ms 4.1x
clip-text 18.1 ms -> 4.9 ms 3.7x
The fp32 variant is removed. Once fp16 is pinned it is dominated on both axes: slower (all-MiniLM 4.5 vs 2.5 ms, clip-text 9.5 vs 4.9 ms) and twice the size. Output fidelity against fp32 was gated first: relative L2 4e-6 (all-MiniLM, multi-qa), 0.0010 (paraphrase), 0.0046 (distiluse).
v0.9.0 and below are untouched.
|
@@ -1,3 +1,3 @@
|
|
| 1 |
version https://git-lfs.github.com/spec/v1
|
| 2 |
-
oid sha256:
|
| 3 |
-
size
|
|
|
|
| 1 |
version https://git-lfs.github.com/spec/v1
|
| 2 |
+
oid sha256:9a08b1c452315aa2f25b4bc5b19ca688166a349e3ebe9d1ec6fc61b811ac571e
|
| 3 |
+
size 45460244
|
|
@@ -1,3 +0,0 @@
|
|
| 1 |
-
version https://git-lfs.github.com/spec/v1
|
| 2 |
-
oid sha256:5156e7c3908a8853c4493e942b70508246f7de7d2500259fcb188fc23d5f7e06
|
| 3 |
-
size 90545948
|
|
|
|
|
|
|
|
|
|
|
|
|
@@ -42,41 +42,6 @@
|
|
| 42 |
]
|
| 43 |
}
|
| 44 |
}
|
| 45 |
-
},
|
| 46 |
-
{
|
| 47 |
-
"file": "all_minilm_l6_v2_coreml_fp32.pte",
|
| 48 |
-
"precision": "fp32",
|
| 49 |
-
"quantized": false,
|
| 50 |
-
"default": false,
|
| 51 |
-
"methods": {
|
| 52 |
-
"forward": {
|
| 53 |
-
"inputs": [
|
| 54 |
-
{
|
| 55 |
-
"shape": [
|
| 56 |
-
1,
|
| 57 |
-
254
|
| 58 |
-
],
|
| 59 |
-
"dtype": "int64"
|
| 60 |
-
},
|
| 61 |
-
{
|
| 62 |
-
"shape": [
|
| 63 |
-
1,
|
| 64 |
-
254
|
| 65 |
-
],
|
| 66 |
-
"dtype": "int64"
|
| 67 |
-
}
|
| 68 |
-
],
|
| 69 |
-
"outputs": [
|
| 70 |
-
{
|
| 71 |
-
"shape": [
|
| 72 |
-
1,
|
| 73 |
-
384
|
| 74 |
-
],
|
| 75 |
-
"dtype": "float32"
|
| 76 |
-
}
|
| 77 |
-
]
|
| 78 |
-
}
|
| 79 |
-
}
|
| 80 |
}
|
| 81 |
]
|
| 82 |
}
|
|
|
|
| 42 |
]
|
| 43 |
}
|
| 44 |
}
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 45 |
}
|
| 46 |
]
|
| 47 |
}
|