perf(coreml): run decode on the CPU, not the Neural Engine
Browse filesdecode is a single-token program whose cost on device is dispatch rather than arithmetic, so it was paying Neural Engine overhead on every call under ExecuTorch's ALL default. Pinning only decode to CPU_ONLY (encode stays on ALL) measures 6.7 ms/step against 20.0 on an iPhone 16.
On 14.5s of real speech the whole transcribe is 1.5-3.1x faster depending on size, with a byte-identical transcript: this removes overhead, it does not change numerics.
Exported from export-scripts MR !18.
coreml/whisper_small_coreml_fp16.pte
CHANGED
|
@@ -1,3 +1,3 @@
|
|
| 1 |
version https://git-lfs.github.com/spec/v1
|
| 2 |
-
oid sha256:
|
| 3 |
-
size
|
|
|
|
| 1 |
version https://git-lfs.github.com/spec/v1
|
| 2 |
+
oid sha256:847a5288dabc604b2f622c8e30303e5c2ce959712348aa1be53fa4296a68bf19
|
| 3 |
+
size 484941877
|