msluszniak commited on
Commit
c28547e
·
verified ·
1 Parent(s): 6ebb863

Core ML: pin compute_unit=CPU_ONLY, drop the fp32 variant

Browse files

These graphs fail the Apple Neural Engine path (coremltools#2825 masked mean-pool over a dynamic sequence dim; clip-text additionally ANECCompile FAILED). Core ML still returned correct output because it silently fell back to CPU, but the failed ANE compile was paid on every call. Pinning compute_unit=CPU_ONLY skips the attempt.

Measured on an iPhone 16, fp16, 3 warmup + 15 timed runs, median:
all-MiniLM 13.7 ms -> 2.5 ms 5.5x
multi-qa 14.1 ms -> 2.5 ms 5.6x
paraphrase-L12 26.9 ms -> 4.2 ms 6.4x
distiluse 33.0 ms -> 8.0 ms 4.1x
clip-text 18.1 ms -> 4.9 ms 3.7x

The fp32 variant is removed. Once fp16 is pinned it is dominated on both axes: slower (all-MiniLM 4.5 vs 2.5 ms, clip-text 9.5 vs 4.9 ms) and twice the size. Output fidelity against fp32 was gated first: relative L2 4e-6 (all-MiniLM, multi-qa), 0.0010 (paraphrase), 0.0046 (distiluse).

v0.9.0 and below are untouched.

coreml/config.json CHANGED
@@ -1,18 +1,16 @@
1
  {
2
  "$schema": "https://huggingface.co/software-mansion/react-native-executorch-spec/resolve/main/config.schema.json",
3
  "model": "paraphrase_multilingual_minilm_l12_v2",
4
- "family": "minilm",
5
  "capabilities": [
6
  "text-embedding"
7
  ],
8
  "backend": "coreml",
9
  "license": "apache-2.0",
10
- "tokenizer": "../tokenizer.json",
11
- "tokenizer_config": "../tokenizer_config.json",
12
  "variants": [
13
  {
14
- "file": "paraphrase_multilingual_minilm_l12_v2_coreml_fp32.pte",
15
- "precision": "fp32",
16
  "quantized": false,
17
  "default": true,
18
  "methods": {
@@ -21,35 +19,14 @@
21
  {
22
  "shape": [
23
  1,
24
- -1
25
  ],
26
  "dtype": "int64"
27
- }
28
- ],
29
- "outputs": [
30
- {
31
- "shape": [
32
- 1,
33
- 384
34
- ],
35
- "dtype": "float32"
36
- }
37
- ]
38
- }
39
- }
40
- },
41
- {
42
- "file": "paraphrase_multilingual_minilm_l12_v2_coreml_fp16.pte",
43
- "precision": "fp16",
44
- "quantized": false,
45
- "default": false,
46
- "methods": {
47
- "forward": {
48
- "inputs": [
49
  {
50
  "shape": [
51
  1,
52
- -1
53
  ],
54
  "dtype": "int64"
55
  }
@@ -67,4 +44,4 @@
67
  }
68
  }
69
  ]
70
- }
 
1
  {
2
  "$schema": "https://huggingface.co/software-mansion/react-native-executorch-spec/resolve/main/config.schema.json",
3
  "model": "paraphrase_multilingual_minilm_l12_v2",
4
+ "family": "sbert",
5
  "capabilities": [
6
  "text-embedding"
7
  ],
8
  "backend": "coreml",
9
  "license": "apache-2.0",
 
 
10
  "variants": [
11
  {
12
+ "file": "paraphrase_multilingual_minilm_l12_v2_coreml_fp16.pte",
13
+ "precision": "fp16",
14
  "quantized": false,
15
  "default": true,
16
  "methods": {
 
19
  {
20
  "shape": [
21
  1,
22
+ 126
23
  ],
24
  "dtype": "int64"
25
+ },
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
26
  {
27
  "shape": [
28
  1,
29
+ 126
30
  ],
31
  "dtype": "int64"
32
  }
 
44
  }
45
  }
46
  ]
47
+ }
coreml/paraphrase_multilingual_minilm_l12_v2_coreml_fp16.pte CHANGED
@@ -1,3 +1,3 @@
1
  version https://git-lfs.github.com/spec/v1
2
- oid sha256:da57435320e06764b5f479e9d44b99e663d5a8d45a5b0cf3742d07e36bbcf873
3
  size 235565340
 
1
  version https://git-lfs.github.com/spec/v1
2
+ oid sha256:ffec84b4e357bc843d65dcc612158d7883b19e7f533c15dd6e6c57d853d9f1ac
3
  size 235565340
coreml/paraphrase_multilingual_minilm_l12_v2_coreml_fp32.pte DELETED
@@ -1,3 +0,0 @@
1
- version https://git-lfs.github.com/spec/v1
2
- oid sha256:fd7e80c15a7cc5c905c9f5db232aeb742ef6054d7af05947113e25db34317354
3
- size 470488106