README.md

---
library_name: coreai
license: apache-2.0
base_model: ibm-granite/granite-embedding-97m-multilingual-r2
tags:
  - coreai
  - sentence-similarity
  - feature-extraction
  - apple-silicon
  - on-device
  - modernbert
language:
  - multilingual
  - ja
  - en
pipeline_tag: sentence-similarity
---

Core AI is Apple's on-device ML runtime in iOS 27 / macOS 27 and the successor to Core ML: PyTorch models are exported with Apple's `coreai-torch` (LLMs: `coreai.llm.export`) into `.aimodel` bundles that run on the GPU or the Neural Engine, e.g. Qwen3-8B 4-bit decodes at 94 tok/s on an M4 Max GPU, MLX 90 under the same protocol ([apple-silicon-llm-bench](https://github.com/john-rocky/apple-silicon-llm-bench), macOS 27 beta, 2026-06).

<!-- gen-cards:devicemark begin (managed by scripts/gen-cards + tools/devicemark_row.py — edit cards.json, not this block) -->
This model has no row on [DeviceMark](https://devicemark.github.io/), the on-device LLM leaderboard.
<!-- gen-cards:devicemark end -->

# Granite-Embedding-97M-Multilingual-R2 — Core AI export

Zoo card, recipe and gate transcript: [coreai-model-zoo/models/granite-embedding-97m](https://github.com/john-rocky/coreai-model-zoo/blob/main/models/granite-embedding-97m/README.md).

IBM's 97M-parameter **multilingual text embedder** — a ModernBERT encoder, 384-d CLS-pooled
unit vectors, Japanese and English among its languages — as a static `.aimodel` for macOS 27
and, ahead-of-time compiled, for the iPhone 17 Pro.
[`ibm-granite/granite-embedding-97m-multilingual-r2`](https://huggingface.co/ibm-granite/granite-embedding-97m-multilingual-r2)
(Apache-2.0, revision `835ad1408…`) is the **smallest embedder in this catalog** (390 MB fp32,
against 1.2 GB for EmbeddingGemma-300m and 1.1 GB for Qwen3-Embedding-0.6B) and its **first
encoder-architecture one** — every other embedder here is a causal decoder run as an encoder.
Its retrieval quality relative to those three was **not** measured here: the fixture set below
is a parity instrument (35 texts, 4 queries, 12 documents), not a benchmark.

**This is an encoder, not a generator** — one forward over the right-padded grid returns one
unit vector. No autoregressive loop, no KV cache, no LM head. It runs as a plain `.aimodel`
through raw `AIModel.run` (like the vision encoders), not the pipelined generate engine.

Architecture (`model_type: modernbert`): 12 layers, hidden 384, 12 heads × 32, GLU MLP 1536
(SiLU), vocabulary 180,000, biasless everything (attention, MLP, LayerNorm ε 1e-5). Global
attention at layers **0, 3, 6, 9** (RoPE θ 150,000); the other eight are **local**, a sliding
window of inclusive radius 64 (129 keys per interior query, RoPE θ 160,000). Layer 0 has no
attention pre-norm (the embedding LayerNorm serves). Pooling is CLS → L2 normalize, both in the
graph.

## Graph contract

```
input  "input_ids"       [1, S]    int32   right-padded to the grid S with 179935
input  "attention_mask"  [1, S]    int32   1 over real tokens, 0 over padding
output "embedding"       [1, 384]  fp32    CLS-pooled, L2-normalized
S = 128 or 512 (export-time choice); batch = 1
```

**Host recipe** — the tokenizer is the whole contract, and the stock one is not enough:
- **No prefix, no stripping, no normalization.** Query and document prompts are both empty in
  the checkpoint. Raw whitespace is kept: sentence-transformers strips text before tokenizing,
  the upstream README's `AutoTokenizer` path does not, and the two disagree on `"  東京駅から…\n"`.
  The reference is the raw path.
- Tokenize with the pinned `tokenizer.json`: regex `Split(Isolated)` → `ByteLevel` (no prefix
  space) → byte BPE with **`ignore_merges = true`** (a whole pre-token that is in the vocabulary
  wins; ` ક` is token 2999, not three). A BPE that ignores the flag tokenizes differently.
- Truncate the **body to S−2**, then wrap: `[CLS 179934] body… [SEP 179938]`, right-pad with
  **PAD 179935** and mask 0. Truncating after adding the specials loses SEP; padding with 0 is a
  different token. Both are silent.
- Similarity = dot product (unit vectors). Dimension truncation is not a property of this model.

[`conversion/granite_embedding/_granite_tokenizer.py`](https://github.com/john-rocky/coreai-model-zoo/blob/main/conversion/granite_embedding/_granite_tokenizer.py) is that recipe with no HF import, and
`host/GraniteTokenizer.swift` in this repo the same recipe in Foundation-only Swift; the gate
holds both to `AutoTokenizer` exactly (ids and masks) over **681 texts × 2 grids = 1,362 cases**
including every added token in five boundary contexts, and proves four mutations are caught
(pad 0 / lose SEP / strip / `ignore_merges=false`).

## Measured

**iPhone 17 Pro** (iPhone18,1), iOS 27.0 build **24A437**, the compiled `h18p` bundles loaded by
the native `AIModel` loader, GPU-preferred (MPSGraph/Metal plan). Every row: 35 HF texts, the
gate below, 105 warm samples, thermal state fair before and after, caches retained (so "first"
is process-first, not cache-cold). Peak footprint is the whole app process, tokenizer and file
hashing included. Measured 2026-09-19.

| Variant | S | Gate | Min cosine vs HF | Max \|err\| | Load | First after load | **Warm median** | Peak footprint |
|---|---:|---|---:|---:|---:|---:|---:|---:|
| fp32 | 128 | 35/35 | 0.999999999999407 | 1.97e-7 | 81 ms | 23.1 ms | **5.54 ms** | 640 MB |
| fp32 | 512 | 35/35 | 0.999999999999486 | 2.38e-7 | 586 ms | 39.9 ms | **20.99 ms** | 640 MB |
| w8 / fp32 table | 128 | 35/35 | 0.999410 | 5.67e-3 | 61 ms | 25.3 ms | 6.64 ms | 555 MB |
| w8 / fp32 table | 512 | 35/35 | 0.999410 | 5.67e-3 | 447 ms | 138.7 ms | 23.07 ms | 553 MB |

Each row matched 4/4 retrieval top-1s with 0 clear-pair flips and 0 repeat drift.

**Mac** (M4 Max, Mac16,9), macOS 27.0 build 26A428, the JIT `.aimodel`, GPU-preferred, fp32.
The driver refused to run while any foreign accelerator job was present; 105 warm samples.

| S | Gate | Min cosine vs HF | Max \|err\| | Load | First after load | **Warm median** |
|---:|---|---:|---:|---:|---:|---:|
| 128 | 35/35 | 0.99999999999967 | 2.98e-7 | 481 ms | 642 ms | **4.14 ms** |
| 512 | 35/35 | 0.99999999999887 | 2.98e-7 | 470 ms | 264 ms | **4.89 ms** |

The Mac h16c AOT twin also passed 70/70 (same numerics), but its timings were taken with another
lane's GPU evaluation running and are not reported. The w8 variant on Mac is gated on **CPU
only** (min cosine 0.999410, max |err| 5.67e-3, ranking exact); Mac GPU for w8 was not run.

The fixed grid computes every position, so pick the smallest grid that covers the text: S=128
for queries and short notes, S=512 for passages. **fp32 is the default.** w8 is a storage
option only — 22% smaller, not faster here — because the 180,000×384 fp32 vocabulary table is
276 MB of the bundle and palettization touches the 48 linear weights alone.

## Numerics gate

One gate at every stage, the oracle being official HF eager CPU fp32 (transformers 4.57.6):
per text cosine ≥ 0.999, max element error ≤ 0.02, L2-norm error ≤ 0.002; per query exact
top-1 over the 12 documents, retrieval-score error ≤ 0.01, and no inversion of any document
pair the oracle separates by ≥ 0.001; repeat drift ≤ 1e-6. A wrong-pairing control (every vector
matched to the wrong text) must FAIL.

- **Authoring** (`gate_granite_authoring.py`): the re-authored graph against every one of the 13
  saved hidden states, max |err| ≤ **1e-4** at fp32, both grids. Five mutations must trip it:
  all-global, all-local, ignore-padding and mean-pooling fail the embedding gate; a local radius of
  **63 instead of 64** passes the embedding gate (cos 0.99995) and fails only the layer gate —
  which is why the layer gate exists. Whole-model **fp16 fails** this layer gate on both grids.
- **Export**: the torch-exported, decomposed graph is gated before conversion, on both grids.
- **Runtime**: Mac CPU and GPU (JIT), Mac h16c AOT, iPhone h18p AOT — the tables above.
- **w8**: the same gate at prepared, finalized and decomposed stages, 48 `lut_to_dense` ops
  counted, palettes hashed; the iOS w8 export reuses the Mac palettes byte for byte.

[`models/granite-embedding-97m/gate-granite-embedding-97m.json`](https://github.com/john-rocky/coreai-model-zoo/blob/main/models/granite-embedding-97m/gate-granite-embedding-97m.json) in the zoo is the transcript: the eight runtime rows
(min cosine, max error, retrieval, timings, device/OS build), the tokenizer gate and the
authoring gate, each with the sha256 of the full record it summarizes.

## ⬇️ Bundle

This repo — one folder per variant, each self-contained: the bundle, `tokenizer/`, `reference.json` (the
35 HF fixtures with ids, masks and embeddings — the parity test) and `provenance/` (export
manifest with per-file sha256, the runtime gate record). `coreai-kit.json` at the root maps
platform → folder.

| Folder | Platform | Format | Bundle | Bytes |
|---|---|---|---|---:|
| `macos/fp32-s512/` **(default)** | macOS 27 | JIT `.aimodel` | `granite97m_fp32_s512_bound.aimodel` | 390,431,506 |
| `macos/fp32-s128/` | macOS 27 | JIT `.aimodel` | `granite97m_fp32_s128_bound.aimodel` | 389,989,146 |
| `ios/fp32-s512/` **(default)** | iOS 27, **h18p only** | AOT `.aimodelc` | `granite97m_fp32_s512_bound.h18p.aimodelc` | 390,308,788 |
| `ios/fp32-s128/` | iOS 27, h18p only | AOT `.aimodelc` | `granite97m_fp32_s128_bound.h18p.aimodelc` | 390,081,410 |
| `macos/w8-fp32table-s512/` | macOS 27 (CPU-gated) | JIT `.aimodel` | `granite97m_w8_fp32table_s512.aimodel` | 305,569,358 |
| `macos/w8-fp32table-s128/` | macOS 27 (CPU-gated) | JIT `.aimodel` | `granite97m_w8_fp32table_s128.aimodel` | 305,126,985 |
| `ios/w8-fp32table-s512/` | iOS 27, h18p only | AOT `.aimodelc` | `granite97m_w8_fp32table_s512_r02.h18p.aimodelc` | 305,479,184 |
| `ios/w8-fp32table-s128/` | iOS 27, h18p only | AOT `.aimodelc` | `granite97m_w8_fp32table_s128_r02.h18p.aimodelc` | 305,251,774 |

The `ios/` bundles are compiled for one device architecture (`h18p`, the iPhone 17 Pro) with
`xcrun coreai-build compile --platform iOS --min-deployment-version 27.0 --preferred-compute gpu
--architecture h18p` (coreai-build 3600.83.1). **Never load an iOS bundle on a Mac.** Other
phones need their own compile from the recipe; the source IR is reproducible, not shipped.

Convert yourself: [`conversion/granite_embedding/`](https://github.com/john-rocky/coreai-model-zoo/blob/main/conversion/granite_embedding/README.md)
— five staged scripts; [`recipe.toml`](https://github.com/john-rocky/coreai-model-zoo/blob/main/models/granite-embedding-97m/recipe.toml) names the commands.

## CoreAIKit (Swift)

**Not enrolled** in the kit catalog. The kit's `TextEmbedder` pads with 0, truncates after adding
the special tokens (losing SEP), applies its own BPE without `ignore_merges`, discovers a single
`.aimodel`, and has no grid / architecture selection — every one of those is wrong for this
model. Running it today means: the Swift tokenizer from this repo's `host/` folder, a fixed
grid, `AIModel` on the platform's folder. Enrolling it needs a `textEmbedding` driver that takes
the pad id, a SEP-preserving truncation, a per-platform variant path and an AOT-aware loader —
tracked as maintainer work, not a blocker on the bundle.

## The port in one lesson: gate the layers, not just the vector

ModernBERT's alternating local/global attention is the whole risk. The config says
`local_attention: 128`; the executed window is inclusive `|i − j| ≤ 64` — 129 keys — and a
window of 63 reproduces the final embedding to cos 0.99995 while every hidden state past layer 1
is wrong. Only a per-layer oracle catches it. Three more things the raw checkpoint settles that
the modeling file hides: layer 0 has no attention norm (adding one loads a missing weight),
the two RoPE thetas are per-layer-kind, and the CLS/L2 head needs an explicit `clamp_min`
epsilon because the converter's `F.normalize` decomposition drops it.

## License and limits

Apache-2.0 at the pinned upstream revision; this repo carries IBM's unmodified card as
`UPSTREAM_README.md` and a `LICENSE-NOTE.md` listing the changes (static graph, in-graph
pooling, optional w8 palettes, h18p compile). Not tested: other phones or OS builds, the Mac GPU
with w8, the Neural Engine, dynamic or batched shapes, S > 512, languages beyond the JA/EN
fixtures, retrieval quality on a benchmark, sustained thermals, true cache-cold load.