mlboydaisuke commited on
Commit
100f230
Β·
verified Β·
1 Parent(s): 25c1f27

gen-cards: regenerate Use-it block

Browse files
Files changed (1) hide show
  1. README.md +48 -0
README.md CHANGED
@@ -27,6 +27,54 @@ UI element**. Built on the **Qwen3-VL-4B** backbone, so it rides the Core AI zoo
27
  Qwen3-VL pipeline. The zoo's **first GUI-grounding / computer-use model**.
28
 
29
  <!-- gen-cards:use-it begin id=holo2-4b (managed by scripts/gen-cards β€” edit cards.json / QuickStart.swift, not this block) -->
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
30
  <!-- gen-cards:use-it end -->
31
 
32
  ## Contents (`gpu-pipelined/`)
 
27
  Qwen3-VL pipeline. The zoo's **first GUI-grounding / computer-use model**.
28
 
29
  <!-- gen-cards:use-it begin id=holo2-4b (managed by scripts/gen-cards β€” edit cards.json / QuickStart.swift, not this block) -->
30
+ ## Use it
31
+
32
+ ▢️ **Run it (source)** β€” the [VLChat runner](https://github.com/john-rocky/coreai-kit/tree/main/Examples/VLChat)
33
+ (GUI + CLI, one app for every vision-language model in the catalog):
34
+
35
+ ```bash
36
+ git clone https://github.com/john-rocky/coreai-kit
37
+ open coreai-kit/Examples/VLChat/VLChat.xcodeproj
38
+ # β†’ Run, then pick "Holo2 4B" in the model picker
39
+
40
+ # agents / headless (macOS):
41
+ cd coreai-kit/Examples/VLChat
42
+ swift run vlchat-cli --model holo2-4b --image screenshot.png --prompt "Localize an element on the GUI image according to my instructions and output a click position as Click(x, y) with x num pixels from the left edge and y num pixels from the top edge. Instruction: click the Submit button."
43
+ ```
44
+
45
+ πŸ’» **Build with it** β€” complete; the glue is kit API, copy-paste runs:
46
+
47
+ ```swift
48
+ import CoreAIKit
49
+ import FoundationModels
50
+
51
+ let vlm = try await KitVisionModel(catalog: "holo2-4b")
52
+ let session = LanguageModelSession(model: vlm)
53
+ let image = try ImageFile.load(imageURL) // any image file β†’ CGImage + EXIF orientation
54
+ let reply = try await session.respond(to: Prompt {
55
+ prompt
56
+ Attachment(image.cgImage, orientation: image.orientation)
57
+ })
58
+ // reply.content: "Click(x, y)" in 0-1000-normalized coordinates for a grounding prompt,
59
+ // or a plain answer for a normal question - all generated on-device
60
+ ```
61
+
62
+ The take-home is [`Examples/VLChat/Sources/QuickStart.swift`](https://github.com/john-rocky/coreai-kit/blob/main/Examples/VLChat/Sources/QuickStart.swift)
63
+ β€” this exact code as one typed function, no UI; the CLI is an argument shell over it, and
64
+ the GUI drives the same `KitVisionModel(catalog:)` behind a `LanguageModelSession`.
65
+ Holo2 is a GUI-grounding model: feed a screenshot and H Company's localization prompt
66
+ (see the card's grounding section) and it returns `Click(x, y)` in 0-1000-normalized
67
+ coordinates β€” multiply by `imageSize / 1000` for pixels. It also answers free-form
68
+ questions like its Qwen3-VL base.
69
+
70
+ **Integration checklist**
71
+
72
+ - SPM: `https://github.com/john-rocky/coreai-kit` β†’ product **CoreAIKit**
73
+ - Info.plist: `NSPhotoLibraryUsageDescription` β€” only if you use PhotosPicker
74
+ - Entitlements (iOS): `com.apple.developer.kernel.increased-memory-limit`
75
+ - First run downloads the model β€” 5.5 GB (Mac) / 5.5 GB (iPhone) β€” then it loads from the
76
+ local cache (Application Support; progress via the `downloadProgress` callback)
77
+ - Measure in Release β€” Debug is ~3Γ— slower on per-token host work
78
  <!-- gen-cards:use-it end -->
79
 
80
  ## Contents (`gpu-pipelined/`)