Card, coreai-kit.json manifest, source config.json

README.md

@@ -0,0 +1,186 @@
+---
+library_name: coreai
+license: apache-2.0
+base_model: ibm-granite/granite-embedding-97m-multilingual-r2
+tags:
+  - coreai
+  - sentence-similarity
+  - feature-extraction
+  - apple-silicon
+  - on-device
+  - modernbert
+language:
+  - multilingual
+  - ja
+  - en
+pipeline_tag: sentence-similarity
+---
+
+Core AI is Apple's on-device ML runtime in iOS 27 / macOS 27 and the successor to Core ML: PyTorch models are exported with Apple's `coreai-torch` (LLMs: `coreai.llm.export`) into `.aimodel` bundles that run on the GPU or the Neural Engine, e.g. Qwen3-8B 4-bit decodes at 94 tok/s on an M4 Max GPU, MLX 90 under the same protocol ([apple-silicon-llm-bench](https://github.com/john-rocky/apple-silicon-llm-bench), macOS 27 beta, 2026-06).
+
+<!-- gen-cards:devicemark begin (managed by scripts/gen-cards + tools/devicemark_row.py — edit cards.json, not this block) -->
+This model has no row on [DeviceMark](https://devicemark.github.io/), the on-device LLM leaderboard.
+<!-- gen-cards:devicemark end -->
+
+# Granite-Embedding-97M-Multilingual-R2 — Core AI export
+
+Zoo card, recipe and gate transcript: [coreai-model-zoo/models/granite-embedding-97m](https://github.com/john-rocky/coreai-model-zoo/blob/main/models/granite-embedding-97m/README.md).
+
+IBM's 97M-parameter **multilingual text embedder** — a ModernBERT encoder, 384-d CLS-pooled
+unit vectors, Japanese and English among its languages — as a static `.aimodel` for macOS 27
+and, ahead-of-time compiled, for the iPhone 17 Pro.
+[`ibm-granite/granite-embedding-97m-multilingual-r2`](https://huggingface.co/ibm-granite/granite-embedding-97m-multilingual-r2)
+(Apache-2.0, revision `835ad1408…`) is the **smallest embedder in this catalog** (390 MB fp32,
+against 1.2 GB for EmbeddingGemma-300m and 1.1 GB for Qwen3-Embedding-0.6B) and its **first
+encoder-architecture one** — every other embedder here is a causal decoder run as an encoder.
+Its retrieval quality relative to those three was **not** measured here: the fixture set below
+is a parity instrument (35 texts, 4 queries, 12 documents), not a benchmark.
+
+**This is an encoder, not a generator** — one forward over the right-padded grid returns one
+unit vector. No autoregressive loop, no KV cache, no LM head. It runs as a plain `.aimodel`
+through raw `AIModel.run` (like the vision encoders), not the pipelined generate engine.
+
+Architecture (`model_type: modernbert`): 12 layers, hidden 384, 12 heads × 32, GLU MLP 1536
+(SiLU), vocabulary 180,000, biasless everything (attention, MLP, LayerNorm ε 1e-5). Global
+attention at layers **0, 3, 6, 9** (RoPE θ 150,000); the other eight are **local**, a sliding
+window of inclusive radius 64 (129 keys per interior query, RoPE θ 160,000). Layer 0 has no
+attention pre-norm (the embedding LayerNorm serves). Pooling is CLS → L2 normalize, both in the
+graph.
+
+## Graph contract
+
+```
+input  "input_ids"       [1, S]    int32   right-padded to the grid S with 179935
+input  "attention_mask"  [1, S]    int32   1 over real tokens, 0 over padding
+output "embedding"       [1, 384]  fp32    CLS-pooled, L2-normalized
+S = 128 or 512 (export-time choice); batch = 1
+```
+
+**Host recipe** — the tokenizer is the whole contract, and the stock one is not enough:
+- **No prefix, no stripping, no normalization.** Query and document prompts are both empty in
+  the checkpoint. Raw whitespace is kept: sentence-transformers strips text before tokenizing,
+  the upstream README's `AutoTokenizer` path does not, and the two disagree on `"  東京駅から…\n"`.
+  The reference is the raw path.
+- Tokenize with the pinned `tokenizer.json`: regex `Split(Isolated)` → `ByteLevel` (no prefix
+  space) → byte BPE with **`ignore_merges = true`** (a whole pre-token that is in the vocabulary
+  wins; ` ક` is token 2999, not three). A BPE that ignores the flag tokenizes differently.
+- Truncate the **body to S−2**, then wrap: `[CLS 179934] body… [SEP 179938]`, right-pad with
+  **PAD 179935** and mask 0. Truncating after adding the specials loses SEP; padding with 0 is a
+  different token. Both are silent.
+- Similarity = dot product (unit vectors). Dimension truncation is not a property of this model.
+
+[`conversion/granite_embedding/_granite_tokenizer.py`](https://github.com/john-rocky/coreai-model-zoo/blob/main/conversion/granite_embedding/_granite_tokenizer.py) is that recipe with no HF import, and
+`host/GraniteTokenizer.swift` in this repo the same recipe in Foundation-only Swift; the gate
+holds both to `AutoTokenizer` exactly (ids and masks) over **681 texts × 2 grids = 1,362 cases**
+including every added token in five boundary contexts, and proves four mutations are caught
+(pad 0 / lose SEP / strip / `ignore_merges=false`).
+
+## Measured
+
+**iPhone 17 Pro** (iPhone18,1), iOS 27.0 build **24A437**, the compiled `h18p` bundles loaded by
+the native `AIModel` loader, GPU-preferred (MPSGraph/Metal plan). Every row: 35 HF texts, the
+gate below, 105 warm samples, thermal state fair before and after, caches retained (so "first"
+is process-first, not cache-cold). Peak footprint is the whole app process, tokenizer and file
+hashing included. Measured 2026-09-19.
+
+| Variant | S | Gate | Min cosine vs HF | Max \|err\| | Load | First after load | **Warm median** | Peak footprint |
+|---|---:|---|---:|---:|---:|---:|---:|---:|
+| fp32 | 128 | 35/35 | 0.999999999999407 | 1.97e-7 | 81 ms | 23.1 ms | **5.54 ms** | 640 MB |
+| fp32 | 512 | 35/35 | 0.999999999999486 | 2.38e-7 | 586 ms | 39.9 ms | **20.99 ms** | 640 MB |
+| w8 / fp32 table | 128 | 35/35 | 0.999410 | 5.67e-3 | 61 ms | 25.3 ms | 6.64 ms | 555 MB |
+| w8 / fp32 table | 512 | 35/35 | 0.999410 | 5.67e-3 | 447 ms | 138.7 ms | 23.07 ms | 553 MB |
+
+Each row matched 4/4 retrieval top-1s with 0 clear-pair flips and 0 repeat drift.
+
+**Mac** (M4 Max, Mac16,9), macOS 27.0 build 26A428, the JIT `.aimodel`, GPU-preferred, fp32.
+The driver refused to run while any foreign accelerator job was present; 105 warm samples.
+
+| S | Gate | Min cosine vs HF | Max \|err\| | Load | First after load | **Warm median** |
+|---:|---|---:|---:|---:|---:|---:|
+| 128 | 35/35 | 0.99999999999967 | 2.98e-7 | 481 ms | 642 ms | **4.14 ms** |
+| 512 | 35/35 | 0.99999999999887 | 2.98e-7 | 470 ms | 264 ms | **4.89 ms** |
+
+The Mac h16c AOT twin also passed 70/70 (same numerics), but its timings were taken with another
+lane's GPU evaluation running and are not reported. The w8 variant on Mac is gated on **CPU
+only** (min cosine 0.999410, max |err| 5.67e-3, ranking exact); Mac GPU for w8 was not run.
+
+The fixed grid computes every position, so pick the smallest grid that covers the text: S=128
+for queries and short notes, S=512 for passages. **fp32 is the default.** w8 is a storage
+option only — 22% smaller, not faster here — because the 180,000×384 fp32 vocabulary table is
+276 MB of the bundle and palettization touches the 48 linear weights alone.
+
+## Numerics gate
+
+One gate at every stage, the oracle being official HF eager CPU fp32 (transformers 4.57.6):
+per text cosine ≥ 0.999, max element error ≤ 0.02, L2-norm error ≤ 0.002; per query exact
+top-1 over the 12 documents, retrieval-score error ≤ 0.01, and no inversion of any document
+pair the oracle separates by ≥ 0.001; repeat drift ≤ 1e-6. A wrong-pairing control (every vector
+matched to the wrong text) must FAIL.
+
+- **Authoring** (`gate_granite_authoring.py`): the re-authored graph against every one of the 13
+  saved hidden states, max |err| ≤ **1e-4** at fp32, both grids. Five mutations must trip it:
+  all-global, all-local, ignore-padding and mean-pooling fail the embedding gate; a local radius of
+  **63 instead of 64** passes the embedding gate (cos 0.99995) and fails only the layer gate —
+  which is why the layer gate exists. Whole-model **fp16 fails** this layer gate on both grids.
+- **Export**: the torch-exported, decomposed graph is gated before conversion, on both grids.
+- **Runtime**: Mac CPU and GPU (JIT), Mac h16c AOT, iPhone h18p AOT — the tables above.
+- **w8**: the same gate at prepared, finalized and decomposed stages, 48 `lut_to_dense` ops
+  counted, palettes hashed; the iOS w8 export reuses the Mac palettes byte for byte.
+
+[`models/granite-embedding-97m/gate-granite-embedding-97m.json`](https://github.com/john-rocky/coreai-model-zoo/blob/main/models/granite-embedding-97m/gate-granite-embedding-97m.json) in the zoo is the transcript: the eight runtime rows
+(min cosine, max error, retrieval, timings, device/OS build), the tokenizer gate and the
+authoring gate, each with the sha256 of the full record it summarizes.
+
+## ⬇️ Bundle
+
+This repo — one folder per variant, each self-contained: the bundle, `tokenizer/`, `reference.json` (the
+35 HF fixtures with ids, masks and embeddings — the parity test) and `provenance/` (export
+manifest with per-file sha256, the runtime gate record). `coreai-kit.json` at the root maps
+platform → folder.
+
+| Folder | Platform | Format | Bundle | Bytes |
+|---|---|---|---|---:|
+| `macos/fp32-s512/` **(default)** | macOS 27 | JIT `.aimodel` | `granite97m_fp32_s512_bound.aimodel` | 390,431,506 |
+| `macos/fp32-s128/` | macOS 27 | JIT `.aimodel` | `granite97m_fp32_s128_bound.aimodel` | 389,989,146 |
+| `ios/fp32-s512/` **(default)** | iOS 27, **h18p only** | AOT `.aimodelc` | `granite97m_fp32_s512_bound.h18p.aimodelc` | 390,308,788 |
+| `ios/fp32-s128/` | iOS 27, h18p only | AOT `.aimodelc` | `granite97m_fp32_s128_bound.h18p.aimodelc` | 390,081,410 |
+| `macos/w8-fp32table-s512/` | macOS 27 (CPU-gated) | JIT `.aimodel` | `granite97m_w8_fp32table_s512.aimodel` | 305,569,358 |
+| `macos/w8-fp32table-s128/` | macOS 27 (CPU-gated) | JIT `.aimodel` | `granite97m_w8_fp32table_s128.aimodel` | 305,126,985 |
+| `ios/w8-fp32table-s512/` | iOS 27, h18p only | AOT `.aimodelc` | `granite97m_w8_fp32table_s512_r02.h18p.aimodelc` | 305,479,184 |
+| `ios/w8-fp32table-s128/` | iOS 27, h18p only | AOT `.aimodelc` | `granite97m_w8_fp32table_s128_r02.h18p.aimodelc` | 305,251,774 |
+
+The `ios/` bundles are compiled for one device architecture (`h18p`, the iPhone 17 Pro) with
+`xcrun coreai-build compile --platform iOS --min-deployment-version 27.0 --preferred-compute gpu
+--architecture h18p` (coreai-build 3600.83.1). **Never load an iOS bundle on a Mac.** Other
+phones need their own compile from the recipe; the source IR is reproducible, not shipped.
+
+Convert yourself: [`conversion/granite_embedding/`](https://github.com/john-rocky/coreai-model-zoo/blob/main/conversion/granite_embedding/README.md)
+— five staged scripts; [`recipe.toml`](https://github.com/john-rocky/coreai-model-zoo/blob/main/models/granite-embedding-97m/recipe.toml) names the commands.
+
+## CoreAIKit (Swift)
+
+**Not enrolled** in the kit catalog. The kit's `TextEmbedder` pads with 0, truncates after adding
+the special tokens (losing SEP), applies its own BPE without `ignore_merges`, discovers a single
+`.aimodel`, and has no grid / architecture selection — every one of those is wrong for this
+model. Running it today means: the Swift tokenizer from this repo's `host/` folder, a fixed
+grid, `AIModel` on the platform's folder. Enrolling it needs a `textEmbedding` driver that takes
+the pad id, a SEP-preserving truncation, a per-platform variant path and an AOT-aware loader —
+tracked as maintainer work, not a blocker on the bundle.
+
+## The port in one lesson: gate the layers, not just the vector
+
+ModernBERT's alternating local/global attention is the whole risk. The config says
+`local_attention: 128`; the executed window is inclusive `|i − j| ≤ 64` — 129 keys — and a
+window of 63 reproduces the final embedding to cos 0.99995 while every hidden state past layer 1
+is wrong. Only a per-layer oracle catches it. Three more things the raw checkpoint settles that
+the modeling file hides: layer 0 has no attention norm (adding one loads a missing weight),
+the two RoPE thetas are per-layer-kind, and the CLS/L2 head needs an explicit `clamp_min`
+epsilon because the converter's `F.normalize` decomposition drops it.
+
+## License and limits
+
+Apache-2.0 at the pinned upstream revision; this repo carries IBM's unmodified card as
+`UPSTREAM_README.md` and a `LICENSE-NOTE.md` listing the changes (static graph, in-graph
+pooling, optional w8 palettes, h18p compile). Not tested: other phones or OS builds, the Mac GPU
+with w8, the Neural Engine, dynamic or batched shapes, S > 512, languages beyond the JA/EN
+fixtures, retrieval quality on a benchmark, sustained thermals, true cache-cold load.

config.json

@@ -0,0 +1,45 @@
+{
+  "architectures": [
+    "ModernBertModel"
+  ],
+  "attention_bias": false,
+  "attention_dropout": 0.0,
+  "bos_token_id": 179934,
+  "classifier_activation": "silu",
+  "classifier_bias": false,
+  "classifier_dropout": 0.0,
+  "classifier_pooling": "cls",
+  "cls_token_id": 179934,
+  "decoder_bias": true,
+  "deterministic_flash_attn": false,
+  "dtype": "bfloat16",
+  "embedding_dropout": 0.0,
+  "eos_token_id": 179938,
+  "global_attn_every_n_layers": 3,
+  "global_rope_theta": 150000.0,
+  "gradient_checkpointing": false,
+  "hidden_activation": "silu",
+  "hidden_size": 384,
+  "initializer_cutoff_factor": 2.0,
+  "initializer_range": 0.02,
+  "intermediate_size": 1536,
+  "layer_norm_eps": 1e-05,
+  "local_attention": 128,
+  "local_rope_theta": 160000.0,
+  "max_position_embeddings": 32768,
+  "mlp_bias": false,
+  "mlp_dropout": 0.0,
+  "model_type": "modernbert",
+  "norm_bias": false,
+  "norm_eps": 1e-05,
+  "num_attention_heads": 12,
+  "num_hidden_layers": 12,
+  "pad_token_id": 179935,
+  "position_embedding_type": "absolute",
+  "repad_logits_with_grad": false,
+  "sep_token_id": 179938,
+  "sparse_pred_ignore_index": -100,
+  "sparse_prediction": false,
+  "transformers_version": "4.56.2",
+  "vocab_size": 180000
+}

coreai-kit.json

@@ -0,0 +1,66 @@
+{
+  "kind": "textEmbedding",
+  "model": "ibm-granite/granite-embedding-97m-multilingual-r2",
+  "revision": "835ad14087e140460703cf0fae09f97d469d65c2",
+  "variants": {
+    "macos": {
+      "path": "macos/fp32-s512",
+      "sizeMB": 396,
+      "format": "aimodel"
+    },
+    "macos-s128": {
+      "path": "macos/fp32-s128",
+      "sizeMB": 396,
+      "format": "aimodel"
+    },
+    "ios-gpu-h18p": {
+      "path": "ios/fp32-s512",
+      "sizeMB": 396,
+      "format": "aimodelc",
+      "architecture": "h18p",
+      "minOS": "27.0"
+    },
+    "ios-gpu-h18p-s128": {
+      "path": "ios/fp32-s128",
+      "sizeMB": 396,
+      "format": "aimodelc",
+      "architecture": "h18p",
+      "minOS": "27.0"
+    },
+    "macos-w8": {
+      "path": "macos/w8-fp32table-s512",
+      "sizeMB": 315,
+      "format": "aimodel"
+    },
+    "ios-gpu-h18p-w8": {
+      "path": "ios/w8-fp32table-s512",
+      "sizeMB": 315,
+      "format": "aimodelc",
+      "architecture": "h18p",
+      "minOS": "27.0"
+    }
+  },
+  "host": {
+    "tokenizer": "host/GraniteTokenizer.swift",
+    "pad_id": 179935,
+    "cls_id": 179934,
+    "sep_id": 179938,
+    "body_truncation": "S-2",
+    "prefix": "",
+    "strip": false,
+    "ignore_merges": true
+  },
+  "graph": {
+    "inputs": {
+      "input_ids": "[1,S] int32",
+      "attention_mask": "[1,S] int32"
+    },
+    "outputs": {
+      "embedding": "[1,384] fp32 L2-normalized"
+    },
+    "grids": [
+      128,
+      512
+    ]
+  }
+}