gen-cards: regenerate Use-it block
Browse files
README.md
CHANGED
|
@@ -27,6 +27,54 @@ UI element**. Built on the **Qwen3-VL-4B** backbone, so it rides the Core AI zoo
|
|
| 27 |
Qwen3-VL pipeline. The zoo's **first GUI-grounding / computer-use model**.
|
| 28 |
|
| 29 |
<!-- gen-cards:use-it begin id=holo2-4b (managed by scripts/gen-cards β edit cards.json / QuickStart.swift, not this block) -->
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 30 |
<!-- gen-cards:use-it end -->
|
| 31 |
|
| 32 |
## Contents (`gpu-pipelined/`)
|
|
|
|
| 27 |
Qwen3-VL pipeline. The zoo's **first GUI-grounding / computer-use model**.
|
| 28 |
|
| 29 |
<!-- gen-cards:use-it begin id=holo2-4b (managed by scripts/gen-cards β edit cards.json / QuickStart.swift, not this block) -->
|
| 30 |
+
## Use it
|
| 31 |
+
|
| 32 |
+
βΆοΈ **Run it (source)** β the [VLChat runner](https://github.com/john-rocky/coreai-kit/tree/main/Examples/VLChat)
|
| 33 |
+
(GUI + CLI, one app for every vision-language model in the catalog):
|
| 34 |
+
|
| 35 |
+
```bash
|
| 36 |
+
git clone https://github.com/john-rocky/coreai-kit
|
| 37 |
+
open coreai-kit/Examples/VLChat/VLChat.xcodeproj
|
| 38 |
+
# β Run, then pick "Holo2 4B" in the model picker
|
| 39 |
+
|
| 40 |
+
# agents / headless (macOS):
|
| 41 |
+
cd coreai-kit/Examples/VLChat
|
| 42 |
+
swift run vlchat-cli --model holo2-4b --image screenshot.png --prompt "Localize an element on the GUI image according to my instructions and output a click position as Click(x, y) with x num pixels from the left edge and y num pixels from the top edge. Instruction: click the Submit button."
|
| 43 |
+
```
|
| 44 |
+
|
| 45 |
+
π» **Build with it** β complete; the glue is kit API, copy-paste runs:
|
| 46 |
+
|
| 47 |
+
```swift
|
| 48 |
+
import CoreAIKit
|
| 49 |
+
import FoundationModels
|
| 50 |
+
|
| 51 |
+
let vlm = try await KitVisionModel(catalog: "holo2-4b")
|
| 52 |
+
let session = LanguageModelSession(model: vlm)
|
| 53 |
+
let image = try ImageFile.load(imageURL) // any image file β CGImage + EXIF orientation
|
| 54 |
+
let reply = try await session.respond(to: Prompt {
|
| 55 |
+
prompt
|
| 56 |
+
Attachment(image.cgImage, orientation: image.orientation)
|
| 57 |
+
})
|
| 58 |
+
// reply.content: "Click(x, y)" in 0-1000-normalized coordinates for a grounding prompt,
|
| 59 |
+
// or a plain answer for a normal question - all generated on-device
|
| 60 |
+
```
|
| 61 |
+
|
| 62 |
+
The take-home is [`Examples/VLChat/Sources/QuickStart.swift`](https://github.com/john-rocky/coreai-kit/blob/main/Examples/VLChat/Sources/QuickStart.swift)
|
| 63 |
+
β this exact code as one typed function, no UI; the CLI is an argument shell over it, and
|
| 64 |
+
the GUI drives the same `KitVisionModel(catalog:)` behind a `LanguageModelSession`.
|
| 65 |
+
Holo2 is a GUI-grounding model: feed a screenshot and H Company's localization prompt
|
| 66 |
+
(see the card's grounding section) and it returns `Click(x, y)` in 0-1000-normalized
|
| 67 |
+
coordinates β multiply by `imageSize / 1000` for pixels. It also answers free-form
|
| 68 |
+
questions like its Qwen3-VL base.
|
| 69 |
+
|
| 70 |
+
**Integration checklist**
|
| 71 |
+
|
| 72 |
+
- SPM: `https://github.com/john-rocky/coreai-kit` β product **CoreAIKit**
|
| 73 |
+
- Info.plist: `NSPhotoLibraryUsageDescription` β only if you use PhotosPicker
|
| 74 |
+
- Entitlements (iOS): `com.apple.developer.kernel.increased-memory-limit`
|
| 75 |
+
- First run downloads the model β 5.5 GB (Mac) / 5.5 GB (iPhone) β then it loads from the
|
| 76 |
+
local cache (Application Support; progress via the `downloadProgress` callback)
|
| 77 |
+
- Measure in Release β Debug is ~3Γ slower on per-token host work
|
| 78 |
<!-- gen-cards:use-it end -->
|
| 79 |
|
| 80 |
## Contents (`gpu-pipelined/`)
|