msluszniak commited on
Commit
2a9233a
·
verified ·
1 Parent(s): 9c04f8c

Core ML: pin compute_unit=CPU_ONLY, drop the fp32 variant

Browse files

These graphs fail the Apple Neural Engine path (coremltools#2825 masked mean-pool over a dynamic sequence dim; clip-text additionally ANECCompile FAILED). Core ML still returned correct output because it silently fell back to CPU, but the failed ANE compile was paid on every call. Pinning compute_unit=CPU_ONLY skips the attempt.

Measured on an iPhone 16, fp16, 3 warmup + 15 timed runs, median:
all-MiniLM 13.7 ms -> 2.5 ms 5.5x
multi-qa 14.1 ms -> 2.5 ms 5.6x
paraphrase-L12 26.9 ms -> 4.2 ms 6.4x
distiluse 33.0 ms -> 8.0 ms 4.1x
clip-text 18.1 ms -> 4.9 ms 3.7x

The fp32 variant is removed. Once fp16 is pinned it is dominated on both axes: slower (all-MiniLM 4.5 vs 2.5 ms, clip-text 9.5 vs 4.9 ms) and twice the size. Output fidelity against fp32 was gated first: relative L2 4e-6 (all-MiniLM, multi-qa), 0.0010 (paraphrase), 0.0046 (distiluse).

v0.9.0 and below are untouched.

coreml/clip_vit_base_patch32_text_coreml_fp16.pte CHANGED
@@ -1,3 +1,3 @@
1
  version https://git-lfs.github.com/spec/v1
2
- oid sha256:444d96f3a26eb0ced6f3cc1e1cad96f491a93c10b749b53e3ac95a11362676c6
3
  size 127415056
 
1
  version https://git-lfs.github.com/spec/v1
2
+ oid sha256:7be060864a0d48002dfdab28b2927a8c4ef467bfb5a3a95a8093ec4b3327c0b9
3
  size 127415056
coreml/clip_vit_base_patch32_text_coreml_fp32.pte DELETED
@@ -1,3 +0,0 @@
1
- version https://git-lfs.github.com/spec/v1
2
- oid sha256:c2140d759fffac30e207a526732c648d5aaabb3316745e0b7c75f9f2f3ee49ac
3
- size 254181021