msluszniak commited on
Commit
7bbfab9
·
verified ·
1 Parent(s): f8a64a1

Core ML: pin compute_unit=CPU_ONLY, drop the fp32 variant

Browse files

These graphs fail the Apple Neural Engine path (coremltools#2825 masked mean-pool over a dynamic sequence dim; clip-text additionally ANECCompile FAILED). Core ML still returned correct output because it silently fell back to CPU, but the failed ANE compile was paid on every call. Pinning compute_unit=CPU_ONLY skips the attempt.

Measured on an iPhone 16, fp16, 3 warmup + 15 timed runs, median:
all-MiniLM 13.7 ms -> 2.5 ms 5.5x
multi-qa 14.1 ms -> 2.5 ms 5.6x
paraphrase-L12 26.9 ms -> 4.2 ms 6.4x
distiluse 33.0 ms -> 8.0 ms 4.1x
clip-text 18.1 ms -> 4.9 ms 3.7x

The fp32 variant is removed. Once fp16 is pinned it is dominated on both axes: slower (all-MiniLM 4.5 vs 2.5 ms, clip-text 9.5 vs 4.9 ms) and twice the size. Output fidelity against fp32 was gated first: relative L2 4e-6 (all-MiniLM, multi-qa), 0.0010 (paraphrase), 0.0046 (distiluse).

v0.9.0 and below are untouched.

coreml/config.json CHANGED
@@ -42,41 +42,6 @@
42
  ]
43
  }
44
  }
45
- },
46
- {
47
- "file": "multi_qa_minilm_l6_cos_v1_coreml_fp32.pte",
48
- "precision": "fp32",
49
- "quantized": false,
50
- "default": false,
51
- "methods": {
52
- "forward": {
53
- "inputs": [
54
- {
55
- "shape": [
56
- 1,
57
- 510
58
- ],
59
- "dtype": "int64"
60
- },
61
- {
62
- "shape": [
63
- 1,
64
- 510
65
- ],
66
- "dtype": "int64"
67
- }
68
- ],
69
- "outputs": [
70
- {
71
- "shape": [
72
- 1,
73
- 384
74
- ],
75
- "dtype": "float32"
76
- }
77
- ]
78
- }
79
- }
80
  }
81
  ]
82
  }
 
42
  ]
43
  }
44
  }
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
45
  }
46
  ]
47
  }
coreml/multi_qa_minilm_l6_cos_v1_coreml_fp16.pte CHANGED
@@ -1,3 +1,3 @@
1
  version https://git-lfs.github.com/spec/v1
2
- oid sha256:021acf4f26621d2605ea864bbf62c9183273b0a03a0585f9c07f81cad565e975
3
- size 45460224
 
1
  version https://git-lfs.github.com/spec/v1
2
+ oid sha256:2ea94be5d4413ad53c60998388882a215ef1bcf6597c8a1aa3094bcbefee330c
3
+ size 45460244
coreml/multi_qa_minilm_l6_cos_v1_coreml_fp32.pte DELETED
@@ -1,3 +0,0 @@
1
- version https://git-lfs.github.com/spec/v1
2
- oid sha256:dc5e6305a7d5a2118a008a18e37688b87a8c341c9d0c8673475e7f7cdbf9b4bd
3
- size 90545948