Document native OpenMed runtimes

README.md

@@ -1,5 +1,5 @@
 ---
-library_name: mlx
+library_name: openmed
 license: apache-2.0
 pipeline_tag: image-text-to-text
 base_model: CohereLabs/North-Micro-Vision-Instruct
@@ -17,10 +17,10 @@ language:
 - ar
 tags:
 - mlx
-- mlx-vlm
 - openmed
 - openmedkit
 - apple-silicon
+- ios
 - on-device
 - vision
 - multimodal
@@ -32,195 +32,202 @@ tags:
 
 # North Micro Vision Instruct — OpenMed MLX family
 
-On-device, native-resolution vision-language inference for Apple Silicon,
-prepared for OpenMed's privacy-first clinical document workflows and the
-future Cohere Compass runtime in [OpenMedKit](https://github.com/maziyarpanahi/openmed).
+Native OpenMed and OpenMedKit vision-language inference for Apple Silicon,
+including local clinical-document and chart workflows on Mac, iPhone, and iPad.
 
 These repositories contain MLX conversions of
 [`CohereLabs/North-Micro-Vision-Instruct`](https://huggingface.co/CohereLabs/North-Micro-Vision-Instruct),
-a compact 2.4B-parameter vision-language model released by Cohere under
-Apache 2.0. The same byte-identical README is used across the five precision
-variants; the repository name and `config.json` identify the precision.
+a compact 2.4B-parameter Cohere Compass vision-language model released under
+Apache 2.0. OpenMed owns the Python and Swift runtime paths described here; no
+`mlx-vlm` installation or model-repository Python code is required for use.
+
+The same byte-identical README is published across all five precision
+variants. The repository name, `config.json`, and `openmed-mlx.json` identify
+the precision.
 
 ## Choose a precision
 
 | Repository | Weight payload | Intended trade-off |
 | --- | ---: | --- |
-| [`OpenMed/North-Micro-Vision-Instruct-4bit-mlx`](https://huggingface.co/OpenMed/North-Micro-Vision-Instruct-4bit-mlx) | 2.02 GiB | Smallest affine variant; validate quality on your exact document set |
+| [`OpenMed/North-Micro-Vision-Instruct-4bit-mlx`](https://huggingface.co/OpenMed/North-Micro-Vision-Instruct-4bit-mlx) | 2.02 GiB | Smallest affine variant; validate quality on your document set |
 | [`OpenMed/North-Micro-Vision-Instruct-5bit-mlx`](https://huggingface.co/OpenMed/North-Micro-Vision-Instruct-5bit-mlx) | 2.25 GiB | Compact middle ground |
-| [`OpenMed/North-Micro-Vision-Instruct-6bit-mlx`](https://huggingface.co/OpenMed/North-Micro-Vision-Instruct-6bit-mlx) | 2.48 GiB | Recommended first on-device quality/size trial |
+| [`OpenMed/North-Micro-Vision-Instruct-6bit-mlx`](https://huggingface.co/OpenMed/North-Micro-Vision-Instruct-6bit-mlx) | 2.48 GiB | Recommended first quality/size trial |
 | [`OpenMed/North-Micro-Vision-Instruct-8bit-mlx`](https://huggingface.co/OpenMed/North-Micro-Vision-Instruct-8bit-mlx) | 2.93 GiB | Higher-fidelity quantized variant |
-| [`OpenMed/North-Micro-Vision-Instruct-bf16-mlx`](https://huggingface.co/OpenMed/North-Micro-Vision-Instruct-bf16-mlx) | 4.63 GiB | Full converted precision and quality reference |
+| [`OpenMed/North-Micro-Vision-Instruct-bf16-mlx`](https://huggingface.co/OpenMed/North-Micro-Vision-Instruct-bf16-mlx) | 4.63 GiB | Full converted-precision reference |
 
 The 4/5/6/8-bit repositories use 64-element affine weight groups. Their
 vision tower remains in source precision; eligible language-model layers are
-quantized. This matters for document and OCR work because visual feature
-extraction is not compressed just to reduce the decoder footprint.
+quantized. This preserves the source visual encoder for OCR and document work
+while reducing the decoder footprint.
 
-## Why this model is interesting for OpenMedKit
+## Python through OpenMed
 
-North Micro Vision accepts interleaved text and images, preserves native image
-aspect ratios, and was trained for OCR, document understanding, charts,
-grounding, captioning, multilingual prompts, and multi-image conversations.
-That combination maps naturally to an OpenMedKit document pipeline:
-
-1. Capture or import a page locally on iPhone, iPad, or Mac.
-2. Keep image normalization, OCR/VLM inference, PII handling, and structured
-   extraction on the user's device.
-3. Validate generated fields against the source page before they enter a
-   clinical record or workflow.
-4. Redact or pseudonymize with OpenMedKit policies before any explicitly
-   authorized export.
-5. Never auto-trigger diagnosis, treatment, disclosure, or another
-   consequential clinical decision from model output.
-
-OpenMedKit's design defaults remain important even when the model is local:
-no telemetry by default, no cloud fallback for PHI, no raw clinical text in
-logs or analytics, synthetic fixtures in committed tests, and human review for
-consequential use.
-
-## Runtime status: read this before using Swift
-
-The model weights use the standard MLX-VLM artifact layout and include the
-source tokenizer, chat template, native-resolution image processor metadata,
-and Cohere Compass configuration. They are validated today with the pinned
-Python MLX-VLM Compass runtime on Apple Silicon.
-
-The released OpenMedKit Swift package does **not yet contain a native Cohere
-Compass VLM implementation**. Do not assume that downloading one of these
-repositories through `OpenMedModelStore` is sufficient for iOS inference.
-These artifacts are deliberately packaged without conversion-time Python
-dependencies so a future OpenMedKit Compass loader can consume the same weight
-payloads, but native Swift/iOS support must still land and pass physical-device
-memory, thermal, image, and text parity gates.
-
-Until that runtime ships, use MLX-VLM on an Apple Silicon Mac for the model
-itself. OpenMedKit can still own capture, privacy policy, redaction, structured
-validation, and the surrounding app workflow. Do not send raw PHI from an iOS
-device to a Mac or service unless the user and deployment policy explicitly
-authorize that transfer.
-
-## Install the validated MLX runtime
-
-Cohere Compass support is pinned to the exact MLX-VLM port used for conversion
-and validation:
+Install an OpenMed revision that contains the native Compass runtime. Until the
+linked implementation PR is merged and released, install its tested branch:
 
 ```bash
 python -m pip install -U \
-  "git+https://github.com/Blaizzy/mlx-vlm.git@dd79a5d8caf3edafd6fa9e6326d7ce4977ddcbfc"
+  "openmed[mlx] @ git+https://github.com/maziyarpanahi/openmed.git@feature/cohere-compass-runtime"
 ```
 
-The validated environment uses MLX 0.32.0, MLX-LM 0.31.3, MLX-VLM 0.6.10
-from that commit, and Transformers 5.15.0. Once Cohere Compass support is in a
-released MLX-VLM package, a normal `pip install -U mlx-vlm` can replace the
-commit pin after you rerun your own parity tests.
-
-## Image + text example
+Image plus text:
 
 ```python
-from mlx_vlm import apply_chat_template, generate, load
+from openmed.mlx import OpenMedMLXVisionLanguageModel
 
-model_id = "OpenMed/North-Micro-Vision-Instruct-6bit-mlx"
-model, processor = load(model_id)
+model = OpenMedMLXVisionLanguageModel(
+    "OpenMed/North-Micro-Vision-Instruct-6bit-mlx"
+)
 
-prompt = apply_chat_template(
-    processor,
-    model.config,
+result = model.generate_with_metadata(
     "Read this synthetic document and list the visible medication and dose.",
-    num_images=1,
-)
-result = generate(
-    model,
-    processor,
-    prompt,
     image="synthetic-clinical-note.png",
     max_tokens=128,
-    temperature=0.0,
 )
+
 print(result.text)
+print(result.prompt_tokens, result.generation_tokens)
 ```
 
-Use synthetic or properly authorized content while prototyping. For a real
-clinical app, treat both the image and generated text as sensitive until the
-OpenMedKit privacy policy has been applied.
-
-## Text-only example
+Text-only generation uses the same loaded model:
 
 ```python
-from mlx_vlm import apply_chat_template, generate, load
-
-model, processor = load(
-    "OpenMed/North-Micro-Vision-Instruct-6bit-mlx"
-)
-prompt = apply_chat_template(
-    processor,
-    model.config,
-    "Explain in one sentence why local processing can improve document privacy.",
-    num_images=0,
+response = model.generate(
+    "Explain why local processing can improve clinical-document privacy.",
+    max_tokens=96,
 )
-result = generate(
-    model,
-    processor,
-    prompt,
-    max_tokens=80,
-    temperature=0.0,
-)
-print(result.text)
 ```
 
-System prompts are not recommended by the source model authors. Keep
-instructions concise and place the task in the user message.
+Pass a local artifact directory instead of the Hub repository ID for a fully
+offline deployment. OpenMed validates the Compass artifact, loads weights
+strictly, applies the bundled chat template, performs native-resolution image
+processing, and generates with MLX. It never enables remote model code.
 
-## OpenMedKit integration gates
+## Swift and iOS through OpenMedKit
 
-Before these weights are advertised as a native OpenMedKit/iOS runtime, the
-Swift implementation must demonstrate all of the following on a physical
-iPhone/iPad as well as Apple Silicon macOS:
+OpenMedKit implements Cohere Compass directly in Swift on MLX. Add the tested
+OpenMed branch until the implementation is merged and included in a tagged
+release:
 
-- strict weight loading for bf16 and every affine bit width;
-- tokenizer and chat-template parity with the validated Python path;
-- native-resolution image preprocessing and visual-token count parity;
-- coherent deterministic text-only generation;
-- correct OCR/document answers on synthetic fixtures;
-- bounded memory, launch latency, sustained thermals, and cancellation;
-- no network fallback, telemetry, or plaintext PHI logging;
-- provenance, model revision, and privacy-policy visibility in the app UI.
+```swift
+dependencies: [
+    .package(
+        url: "https://github.com/maziyarpanahi/openmed.git",
+        branch: "feature/cohere-compass-runtime"
+    ),
+]
+```
 
-That is the bar for “OpenMedKit-ready”; an MLX file extension alone is not.
+Load from Hugging Face and ask a question about a local image:
 
-## Validation performed before publication
+```swift
+import OpenMedKit
 
-Every OpenMed variant is loaded independently with strict tensor matching and
-must pass the same deterministic suite before any repository is made public:
+let model = try await OpenMedVisionLanguageModel.load(
+    modelID: "OpenMed/North-Micro-Vision-Instruct-6bit-mlx"
+)
 
-- coherent text-only privacy explanation;
-- exact fact extraction from a short synthetic text note;
-- exact extraction of name, synthetic record ID, medication, dose, frequency,
-  and allergy from a generated non-PHI clinical document image;
-- exact category/value extraction from a generated chart image;
-- model type, processor assets, quantization metadata, and weight-size checks;
-- bf16 SHA-256 payload parity with an independently published Cohere-linked
-  conversion made from the same pinned source and runtime port;
-- quantized payload-size parity plus an explicit nonzero token-embedding gate
-  (whole-file hashes are recorded but are not treated as a correctness proxy).
+let result = try await model.generate(
+    "List the visible medication and dose.",
+    imageURL: clinicalDocumentURL,
+    maxTokens: 128
+)
 
-The fixtures are synthetic and are not clinical-quality evidence. Passing
-these checks establishes conversion/runtime coherence, not safety or accuracy
-for a particular medical workflow. Evaluate the exact document types,
-languages, device targets, and failure modes your app will encounter.
+print(result.text)
+```
+
+For a bundled, app-managed, or air-gapped artifact:
+
+```swift
+let model = try await OpenMedVisionLanguageModel.load(
+    modelDirectory: localModelDirectory
+)
+```
 
-## Source model scope and limitations
+`OpenMedVisionLanguageGeneration` exposes decoded text, generated token IDs,
+prompt/generation token counts, and timing. Overloads support text-only input,
+`CIImage`, a local image URL, or multiple `UserInput.Image` values.
 
-The source checkpoint supports native-resolution and multiple images,
-multilingual conversations, visual question answering, grounding, OCR, and
-chart/document understanding. Its language backbone supports a long context,
-but Cohere reports multimodal training and validation up to 8K tokens; do not
-silently claim longer multimodal reliability.
+The initial Hub download is a network operation. Prompt and image inference is
+local after the artifact is cached, with no telemetry and no cloud inference
+fallback. For PHI workflows, pre-download or bundle the model before intake,
+keep raw documents out of logs and analytics, and apply OpenMedKit privacy
+policies before an explicitly authorized export.
+
+Implementation and review status: [OpenMed PR #2885](https://github.com/maziyarpanahi/openmed/pull/2885)
+
+## Why this model fits OpenMedKit
+
+North Micro Vision accepts interleaved text and images, preserves native image
+aspect ratios, and was trained for OCR, document understanding, charts,
+grounding, captioning, multilingual prompts, and multi-image conversations.
+That maps naturally to a privacy-first OpenMedKit pipeline:
+
+1. Capture or import a page locally on iPhone, iPad, or Mac.
+2. Run image normalization and VLM inference on the device.
+3. Validate names, identifiers, medications, measurements, and other generated
+   fields against the source page.
+4. Redact or pseudonymize with OpenMedKit policies before any authorized
+   disclosure.
+5. Preserve provenance and require human review before consequential use.
+
+Local inference reduces a network boundary; it does not make generated content
+automatically safe or correct. Never auto-trigger diagnosis, treatment,
+disclosure, or another consequential clinical action from model output.
+
+## OpenMed runtime validation
+
+Before the runtime and cards were published, every precision was loaded
+independently and run through the same synthetic suite in both native runtimes:
+
+- Python: `OpenMedMLXVisionLanguageModel`, strict safetensors loading;
+- Swift: `OpenMedVisionLanguageModel` in an Xcode Metal-backed test bundle;
+- coherent deterministic text-only privacy explanation;
+- exact `Tuesday` extraction from a short synthetic note;
+- correct name, synthetic record ID, medication, dose, frequency, and allergy
+  extraction from a generated clinical-document image;
+- exact `Screening, 42` extraction from a generated chart image;
+- tokenizer/chat-template and prompt-token-count parity;
+- native image resize, patch-grid, visual-token, and multimodal decode checks;
+- macOS execution for all five payloads and an iOS device-target build gate.
+
+The Python and Swift tests use the same prompts, image fixtures, token counts,
+and clinical fact acceptance criteria. Canonical fact and chart answers are
+also token-exact. Free-form sentences can choose equivalent near-tied tokens
+across MLX language bindings, so those are checked for coherence and required
+facts instead of brittle punctuation or wording.
+
+`openmed-runtime-validation.json` records the OpenMed runtime gates for this
+variant. `openmed-validation.json` preserves the original conversion and
+independent-reference certificate. Those fixtures are synthetic and are not
+clinical-quality evidence.
+
+## Artifact contract
+
+Each repository is data-only and includes:
+
+- `model.safetensors` plus its index;
+- `config.json` with `model_type: cohere_compass`;
+- tokenizer, chat-template, and generation configuration;
+- native-resolution image processor configuration;
+- `openmed-mlx.json` runtime/precision metadata;
+- OpenMed conversion and native-runtime validation reports.
+
+No conversion-time Python package or executable model code is stored in the
+repository. Both OpenMed runtimes consume the same artifact payload.
+
+## Scope and limitations
+
+The source model supports visual question answering, grounding, OCR, document
+and chart understanding, multilingual prompts, and multiple images. Cohere
+reports multimodal training and validation up to 8K tokens; do not silently
+claim longer multimodal reliability.
 
 The source model is not a reasoning model, has limited math and code ability,
-does not support tool calling or agentic workflows, and is intended as a
-compact foundation for prototyping and specialization. Native-resolution
-images can materially increase memory use and latency.
+does not support tool calling, and is intended as a compact foundation for
+prototyping and specialization. Native-resolution images can materially
+increase memory use, latency, and thermal pressure on mobile devices. Measure
+the exact precision, image sizes, and sustained workload on each target device.
 
 This model and OpenMedKit are not medical devices. Outputs can be incomplete,
 incorrect, or fabricated. A qualified human must verify consequential use.
@@ -229,11 +236,13 @@ incorrect, or fabricated. A qualified human must verify consequential use.
 
 - Source: [`CohereLabs/North-Micro-Vision-Instruct`](https://huggingface.co/CohereLabs/North-Micro-Vision-Instruct)
 - Pinned source revision: `373bda96ac70bf89f99f7048f420cf00dc07c149`
-- MLX-VLM Compass port: [`dd79a5d8caf3edafd6fa9e6326d7ce4977ddcbfc`](https://github.com/Blaizzy/mlx-vlm/commit/dd79a5d8caf3edafd6fa9e6326d7ce4977ddcbfc)
 - OpenMed / OpenMedKit: [github.com/maziyarpanahi/openmed](https://github.com/maziyarpanahi/openmed)
+- Independent conversion reference: the Cohere Compass port contributed to
+  MLX-VLM at revision `dd79a5d8caf3edafd6fa9e6326d7ce4977ddcbfc`
 
-Thank you to Cohere for releasing North Micro Vision and to Prince Canuma and
-the MLX-VLM contributors for the Apple Silicon runtime and Compass port.
+Thank you to Cohere for releasing North Micro Vision and to the MLX and
+MLX-VLM contributors whose prior Compass work provided a useful independent
+reference while OpenMed implemented and tested its own Python and Swift paths.
 
 The converted weights retain the source model's Apache 2.0 license. OpenMed's
 SDK source is separately licensed under Apache 2.0.

openmed-mlx.json

@@ -9,9 +9,9 @@
   },
   "runtime": {
     "local_only_recommended": true,
-    "mlx_vlm_revision": "dd79a5d8caf3edafd6fa9e6326d7ce4977ddcbfc",
-    "python": "mlx-vlm",
-    "swift": "OpenMedKit Cohere Compass runtime pending",
+    "python": "openmed.mlx.OpenMedMLXVisionLanguageModel",
+    "swift": "OpenMedKit.OpenMedVisionLanguageModel",
+    "openmed_implementation": "https://github.com/maziyarpanahi/openmed/pull/2885",
     "validated_multimodal_context": 8192
   },
   "source_model": "CohereLabs/North-Micro-Vision-Instruct",
@@ -24,13 +24,25 @@
       "image_clinical_document",
       "image_chart"
     ],
-    "report": "openmed-validation.json",
+    "report": "openmed-runtime-validation.json",
     "status": "passed",
     "strict_weight_load": true,
-    "text_and_image": true
+    "text_and_image": true,
+    "conversion_report": "openmed-validation.json",
+    "runtime_report": "openmed-runtime-validation.json",
+    "python": true,
+    "swift": true,
+    "ios_device_target_build": true,
+    "physical_ios_inference_run": false
   },
   "weights": {
     "format": "safetensors",
     "path": "model.safetensors"
+  },
+  "provenance": {
+    "independent_conversion_reference": {
+      "library": "mlx-vlm",
+      "revision": "dd79a5d8caf3edafd6fa9e6326d7ce4977ddcbfc"
+    }
   }
 }

openmed-runtime-validation.json

@@ -0,0 +1,83 @@
+{
+  "schema_version": 1,
+  "model_id": "OpenMed/North-Micro-Vision-Instruct-4bit-mlx",
+  "variant": "4bit",
+  "validated_at": "2026-08-14",
+  "passed": true,
+  "openmed": {
+    "implementation_pr": "https://github.com/maziyarpanahi/openmed/pull/2885",
+    "python_api": "openmed.mlx.OpenMedMLXVisionLanguageModel",
+    "swift_api": "OpenMedKit.OpenMedVisionLanguageModel"
+  },
+  "python": {
+    "passed": true,
+    "strict_weight_load": true,
+    "platform": "Apple M3 Ultra",
+    "runtime_versions": {
+      "mlx": "0.32.0",
+      "mlx-lm": "0.31.3",
+      "transformers": "5.15.0",
+      "huggingface-hub": "1.27.0",
+      "pillow": "12.3.0"
+    },
+    "test": "tests/integration/test_mlx_vlm_compass.py"
+  },
+  "swift": {
+    "passed": true,
+    "strict_weight_load": true,
+    "execution_platform": "Apple M3 Ultra, macOS 26.5.1, Xcode Metal test bundle",
+    "ios_device_target_build": true,
+    "physical_ios_inference_run": false,
+    "physical_ios_note": "No compatible physical iPhone was online for this publication run; all five payloads executed through the same Swift model and processor on Apple Silicon, and the generic iOS device target compiled successfully.",
+    "runtime_versions": {
+      "mlx-swift": "0.31.6",
+      "mlx-swift-lm": "42f08a872075fd07f9f1f40ec1a5e191e6aad86e",
+      "xcode": "26.6"
+    },
+    "test": "swift/OpenMedKit/Tests/OpenMedKitTests/OpenMedCompassTests.swift"
+  },
+  "cases": {
+    "text_privacy": {
+      "passed": true,
+      "acceptance": "one coherent sentence covering on-device locality, sensitive clinical data, and privacy risk reduction",
+      "prompt_tokens": 30
+    },
+    "text_fact_extraction": {
+      "passed": true,
+      "exact_response": "Tuesday",
+      "exact_generated_token_ids": [29445],
+      "prompt_tokens": 44
+    },
+    "image_clinical_document": {
+      "passed": true,
+      "required_facts": [
+        "Alex Rivera",
+        "SYN-2048",
+        "Metformin",
+        "500 mg",
+        "twice daily",
+        "Penicillin"
+      ],
+      "prompt_tokens": 1161,
+      "fixture": "synthetic_clinical_document.png"
+    },
+    "image_chart": {
+      "passed": true,
+      "exact_response": "Screening, 42",
+      "exact_generated_token_ids": [198759, 16, 225, 3304],
+      "prompt_tokens": 1053,
+      "fixture": "synthetic_clinic_chart.png"
+    }
+  },
+  "privacy": {
+    "fixtures_are_synthetic": true,
+    "remote_model_code": false,
+    "cloud_inference_fallback": false,
+    "telemetry": false
+  },
+  "reference": {
+    "conversion_report": "openmed-validation.json",
+    "source_model": "CohereLabs/North-Micro-Vision-Instruct",
+    "source_revision": "373bda96ac70bf89f99f7048f420cf00dc07c149"
+  }
+}