h9899/siglip-base-patch16-224-coreml

SigLIP Base Patch16 224 — CoreML

CoreML conversion of google/siglip-base-patch16-224 for on-device iOS/macOS inference.

Produces 768-dimensional image embeddings for:

• Instant photo search (text → image similarity)
• Zero-shot image classification (no training needed)
• Duplicate/similar photo detection
• Image clustering and auto-albums

Model Details

Property	Value
Source model	google/siglip-base-patch16-224
Embedding dim	768
Image size	224 × 224
Format	CoreML .mlpackage (ML Program)
Precision	float32
File size	~176 MB
Min iOS	16.0
Compute units	CPU + Neural Engine (background-safe, no GPU)
License	Apache 2.0

Files

File	Description	Size
siglip_vision.mlpackage/	Vision encoder (image → 768-dim embedding)	176 MB
config.json	Model configuration	<1 KB

Usage

iOS (Swift)

import CoreML

let config = MLModelConfiguration()
config.computeUnits = .cpuAndNeuralEngine  // Background-safe, no GPU

let model = try MLModel(contentsOf: modelURL, configuration: config)

// Preprocess: resize to 224x224, normalize with (pixel/127.5 - 1.0)
let input = try MLDictionaryFeatureProvider(dictionary: [
    "pixel_values": MLMultiArray(pixelBuffer)
])

let output = try model.prediction(from: input)
let embedding = output.featureValue(for: "var_857")  // 768-dim

Preprocessing

Input: [1, 3, 224, 224] float32 tensor
Normalization: (pixel / 255.0 - 0.5) / 0.5 = pixel / 127.5 - 1.0
Channel order: RGB (not BGR)
Layout: NCHW

Android

For Android, use the official ONNX export instead:

https://huggingface.co/google/siglip-base-patch16-224/resolve/main/onnx/model.onnx

Benchmarks

SigLIP-base achieves strong results on vision-language tasks:

Task	Score
ImageNet zero-shot	73.1%
COCO retrieval (R@1)	64.2%

How It Was Made

pip install torch transformers coremltools pillow numpy
python convert_siglip_coreml.py

The vision encoder is wrapped to return only pooler_output (768-dim [CLS] embedding) instead of the full dict output, making it compatible with torch.jit.trace and CoreML conversion.

Use Cases

1. Photo Search: Embed all photos → HNSW index → search by text query
2. Classification: Embed image + embed labels → cosine similarity → top match
3. Duplicate Detection: Embed two images → cosine > 0.95 = near-duplicate
4. Auto-Albums: Cluster embeddings → discover photo groups

Citation

@article{zhai2023sigmoid,
  title={Sigmoid Loss for Language Image Pre-Training},
  author={Zhai, Xiaohua and Mustafa, Basil and Kolesnikov, Alexander and Beyer, Lucas},
  journal={ICCV},
  year={2023}
}