Instructions to use mlboydaisuke/ColModernVBERT-CoreAI with libraries, inference providers, notebooks, and local apps. Follow these links to get started.
- Libraries
- ColPali
How to use mlboydaisuke/ColModernVBERT-CoreAI with ColPali:
# No code snippets available yet for this library. # To use this model, check the repository files and the library's documentation. # Want to help? PRs adding snippets are welcome at: # https://github.com/huggingface/huggingface.js
- Notebooks
- Google Colab
- Kaggle
gen-cards: regenerate Use-it block
Browse files
README.md
CHANGED
|
@@ -32,6 +32,50 @@ dense) and [Qwen3-Reranker](https://huggingface.co/mlboydaisuke/Qwen3-Reranker-0
|
|
| 32 |
(cross-encoder): **embed β rerank β visual-retrieval**, all on device.
|
| 33 |
|
| 34 |
<!-- gen-cards:use-it begin id=colmodernvbert (managed by scripts/gen-cards β edit cards.json / QuickStart.swift, not this block) -->
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 35 |
<!-- gen-cards:use-it end -->
|
| 36 |
|
| 37 |
## Two encoders (two graphs)
|
|
|
|
| 32 |
(cross-encoder): **embed β rerank β visual-retrieval**, all on device.
|
| 33 |
|
| 34 |
<!-- gen-cards:use-it begin id=colmodernvbert (managed by scripts/gen-cards β edit cards.json / QuickStart.swift, not this block) -->
|
| 35 |
+
## Use it
|
| 36 |
+
|
| 37 |
+
βΆοΈ **Run it (source)** β the [DocSearch runner](https://github.com/john-rocky/coreai-kit/tree/main/Examples/DocSearch)
|
| 38 |
+
(visual page search over bundled sample pages; the GUI (iPhone) adds tiled where-it-matched highlights):
|
| 39 |
+
|
| 40 |
+
```bash
|
| 41 |
+
git clone https://github.com/john-rocky/coreai-kit
|
| 42 |
+
open coreai-kit/Examples/DocSearch/DocSearch.xcodeproj
|
| 43 |
+
# β Run, then pick "ColModernVBERT" in the model picker
|
| 44 |
+
|
| 45 |
+
# agents / headless (macOS):
|
| 46 |
+
cd coreai-kit/Examples/DocSearch
|
| 47 |
+
swift run docsearch-cli --model colmodernvbert --query "monthly revenue trend"
|
| 48 |
+
```
|
| 49 |
+
|
| 50 |
+
π» **Build with it** β complete; the glue is kit API, copy-paste runs:
|
| 51 |
+
|
| 52 |
+
```swift
|
| 53 |
+
import CoreAIKitEmbeddings
|
| 54 |
+
|
| 55 |
+
let retriever = try await VisualDocumentRetriever(
|
| 56 |
+
catalog: "colmodernvbert")
|
| 57 |
+
var corpus: [VisualDocumentRetriever.PageEmbedding] = []
|
| 58 |
+
for url in pages {
|
| 59 |
+
corpus.append(try await retriever.encode(page: ImageFile.load(url).cgImage))
|
| 60 |
+
}
|
| 61 |
+
let hits = try await retriever.retrieve(query: query, over: corpus, topK: pages.count)
|
| 62 |
+
// hits: pages ranked by MaxSim, best match first β no OCR, pages are matched as pictures
|
| 63 |
+
```
|
| 64 |
+
|
| 65 |
+
The take-home is [`Examples/DocSearch/Sources/QuickStart.swift`](https://github.com/john-rocky/coreai-kit/blob/main/Examples/DocSearch/Sources/QuickStart.swift)
|
| 66 |
+
β this exact code as one typed function, no UI; the CLI is an argument shell over it, and
|
| 67 |
+
the GUI drives the same `VisualDocumentRetriever(catalog:)` with tiled per-page encoding.
|
| 68 |
+
Encode your corpus once and keep the `PageEmbedding`s β scoring a query is then host-side
|
| 69 |
+
MaxSim, no model call per page. `encodeTiled(page:)` localizes *where* a query matched.
|
| 70 |
+
|
| 71 |
+
**Integration checklist**
|
| 72 |
+
|
| 73 |
+
- SPM: `https://github.com/john-rocky/coreai-kit` β product **CoreAIKitEmbeddings**
|
| 74 |
+
- Info.plist: `NSPhotoLibraryUsageDescription` β only if you use PhotosPicker to import pages
|
| 75 |
+
- Entitlements: none needed
|
| 76 |
+
- First run downloads the model β 0.7 GB (Mac) / 0.7 GB (iPhone) β then it loads from the
|
| 77 |
+
local cache (Application Support; progress via the `downloadProgress` callback)
|
| 78 |
+
- Measure in Release β Debug is ~3Γ slower on per-token host work
|
| 79 |
<!-- gen-cards:use-it end -->
|
| 80 |
|
| 81 |
## Two encoders (two graphs)
|