mlboydaisuke/Unlimited-OCR-CoreAI

Core AI is Apple's on-device ML runtime in iOS 27 / macOS 27 and the successor to Core ML: PyTorch models are exported with Apple's coreai-torch (LLMs: coreai.llm.export) into .aimodel bundles that run on the GPU or the Neural Engine, e.g. Qwen3-8B 4-bit decodes at 94 tok/s on an M4 Max GPU, MLX 90 under the same protocol (apple-silicon-llm-bench, macOS 27 beta 26A5353q, 2026-06-11).

This model has no row on DeviceMark, the on-device LLM leaderboard.

Unlimited-OCR → Core AI (on-device document OCR)

On-device document → structured-markdown OCR, end-to-end on Apple Core AI. A port of baidu/Unlimited-OCR (3B-A0.5B MoE, MIT): drop a document image, get back markdown — tables as HTML (<table><tr><td>…), formulas as LaTeX, reading order, and <|det|> layout boxes. Japanese + English + multilingual.

Runs on the stock coreai.runtime with no engine patch — the decoder is driven directly on inputs_embeds, so this is a pure-export port (not the static-input-buffer VLM path).

Use it

New to Core AI? Start with CoreAIKit 0.7.1. Follow its requirements and first-run steps for qwen3-0.6b, then open the same release's ChatDemo. The README records the tested OS/SDK and download size; model and device coverage is stated per example.

⚡ One line — run the kit's task op on this model (import CoreAIOps; no session, no model plumbing, downloads on first use):

let markdown = try await CoreAI.read(documentAt: url, options: .model("unlimited-ocr"))

Every op, one shape — Cookbook.

▶️ Run it (source) — the ReadDoc runner (GUI + CLI, one app for every document-OCR model in the catalog):

git clone --branch 0.7.1 --depth 1 https://github.com/john-rocky/coreai-kit
export DEVELOPER_DIR=/Applications/Xcode-27.0.0-RC.app/Contents/Developer
open -a /Applications/Xcode-27.0.0-RC.app coreai-kit/Examples/ReadDoc/ReadDoc.xcodeproj
# → Run, then pick "Unlimited-OCR" in the model picker

# agents / headless (macOS):
cd coreai-kit/Examples/ReadDoc
swift run -c release readdoc-cli --model unlimited-ocr --image sample.png

Use Xcode build 27A266a from the release's .xcode-pin; adjust the app path if your installation is named differently.

💻 Build with it — complete; the glue is kit API, copy-paste runs:

import CoreAIKit

let reader = try await KitDocReader(catalog: "unlimited-ocr")
let markdown = try await reader.read(imageAt: imageURL)
// markdown: the document as structured text — tables as <table>/<tr>/<td>,
// <|det|> layout boxes, reading order — fully on-device

The take-home is Examples/ReadDoc/Sources/QuickStart.swift — this exact code as one typed function, no UI; the CLI is an argument shell over it, and the GUI drives the same KitDocReader(catalog:) on the image you pick. One read(imageAt:) call per page; chunk a PDF into page images first. The output keeps the model's structural markup (tables as HTML, formulas as LaTeX, <|det|> boxes) — strip or render it as your app prefers.

Integration checklist

• SPM: https://github.com/john-rocky/coreai-kit (exact 0.7.1) → product CoreAIKit
• Info.plist: none needed
• Entitlements: none needed
• First run downloads the model — ~4,532 MB (Mac) — then it loads from the local cache (Application Support; progress via the downloadProgress callback)
• Measure in Release — Debug is ~3× slower on per-token host work

What's exciting (why you'd use it)

• Private OCR: invoices, receipts, contracts, papers, forms never leave the device.
• Structured, not just text: tables → HTML, equations → LaTeX, layout → boxes. RAG-ready ingestion.
• Flat latency: a static-shape decode graph (data-driven KV write + fixed-buffer R-SWA mask) keeps every tensor shape constant, so the runtime compiles once and decode stays flat at 12.7 ms/token (79 tok/s on M4 Max) — no growing-cache recompilation stalls.
• SOTA quality: the source model tops OmniDocBench v1.6 (93.92); this port is byte-faithful to the fp32 reference (decoder 0 flips at the sampled steps; vision encoder cos 1.000000).

Bundles

path	what	dtype	size
vision/unlimited_ocr_vision.aimodel	DeepEncoder (SAM-ViT + CLIP-ViT cascade) → 100 visual tokens	fp16	762 MB
decoder/unlimited_ocr_decoder.aimodel	DeepseekV2 R-SWA MoE decoder, functions prefill + decode sharing one weight set + KV state	sym8	3.2 GB
assets/embed_tokens.f16	token embedding table [129280,1280] (host row-gather)	fp16	316 MB
assets/{image_newline,view_seperator}.f16, assets/prompt_input_ids.i32, assets/recipe.json	arrangement constants + the assembly recipe	—	tiny
tokenizer/	fast tokenizer (tokenizer.json + configs)	—	—

Pipeline (Base mode, 640px)

image → preprocess (pad to 640², normalize mean=std=0.5)
      → vision .aimodel                         → visual tokens [1,100,1280]
      → arrange (10×10 + image_newline per row + view_seperator) → [111,1280]
      → scatter into embed_tokens(prompt_ids)   → prefix [1,115,1280]
      → decoder: prefill(prefix) + greedy decode (no_repeat_ngram=35) → tokens
      → detokenize (keep special tokens)        → markdown

The exact, verified recipe is in assets/recipe.json. Reference implementations (Python end-to-end

• a macOS app, CoreAIOCR, driving the stock runtime) are in the Core AI Model Zoo: conversion/unlimited_ocr/ and apps/CoreAIOCR/.

Notes

• Appropriate input: clean single-page documents (invoice / paper / report / table / formula), roughly square or portrait, with text still legible when fit to 640². Very dense small-text scans (newspaper) want the tiled crop_mode vision export (not included here; Base mode only).
• Prompt is fixed to document parsing (layout + structured extraction).
• License: MIT (inherited from baidu/Unlimited-OCR).

Community port — not affiliated with Apple or baidu.

---

More models in this format: Core AI Model Zoo — 75 models, each with the recipe that produced it.

Want a different model on-device? Open a request — free, open weights only; the export and its measured numbers get published publicly.