deVision v0.5

Image + English questions → structured answers with calibrated probabilities. deVision pairs a SigLIP2 vision encoder with Laya's ModernBERT decision model. It scores the options of yes/no (noul) and multiple-choice (choice) questions instead of generating text, follows the Jev answer format, and runs without a GPU. Several questions about one image share a single image encoding.

Usage

pip install devision

import devision

model = devision.load("lukbit/devision")   # latest release; device defaults to auto
result = model.predict(
    image="photo.jpg",   # a path, an http(s) URL, a data URI, base64, bytes or a PIL image
    state="",            # text context; "" when there is none
    questions={
        "has_fork": {"type": "noul", "instructions": "Is there a fork in the image?"},
        "room": {"type": "choice", "instructions": "Which room is this?",
                 "criteria": {"kitchen": None, "bathroom": None, "bedroom": None}},
    },
)
print(result["answers"]["has_fork"]["noul"])        # probability of yes
print(result["answers"]["room"]["probabilities"])   # one probability per option, summing to 1

HTTP server with a browser demo at http://127.0.0.1:8000:

devision-serve --checkpoint lukbit/devision --port 8000
curl -s http://127.0.0.1:8000/v1/systemone -H 'Content-Type: application/json' \
  -d '{"image": "https://example.com/photo.jpg", "state": "", "questions": {"has_fork": {"type": "noul", "instructions": "Is there a fork in the image?"}}}'

Over HTTP, image is an http(s) URL, a data URI or base64; the server never reads its own files. Responses follow Jev; confidence for choice is (n * p_max - 1) / (n - 1). Invalid requests return 422 (devision.InvalidRequest in Python).

Good for: whether something is present and how many (ask counts as a choice), what kind of thing or scene, colours and materials, up/down and left/right in photos (as a choice). Not for: comparing two places on a diagram, chart or map; relative-position yes/no questions; small text; other languages or several images. Use the probabilities as thresholds and send uncertain cases to a person or a stronger model, after checking the thresholds on your own data.

Evaluation

Accuracy through decide with the fitted temperatures. Mismatched: the same questions with every picture swapped for an unrelated one. These are project evaluation subsets, not official leaderboard scores. Although test pictures are excluded from training, these sets have informed successive experiments and release decisions; they are not a fresh independent test. See the release limitations below.

Set	Questions	Accuracy	Mismatched	NLL	ECE
COCO object presence	1,120	0.952	0.505	0.189	0.070
VQAv2 multiple choice	1,453	0.905	0.373	0.251	0.026
COCO size	1,198	0.876	0.511	0.416	0.153
POPE, project filtered set	9,000	0.861	0.518	0.361	0.029
GQA val subset	986	0.845	0.558	0.346	0.029
COCO position	1,276	0.941	0.500	0.182	0.041
VQAv2 yes/no subset	1,000	0.721	0.501	0.567	0.062
COCO relative position	2,366	0.912	0.472	0.226	0.028
VSR, project held-out split	902	0.718	0.469	0.590	0.102
Visual7W, project held-out split	1,000	0.777	0.446	0.585	0.041
CountBench, 0–10 choices	491	0.515	0.106	1.261	0.079
COCO mirrored position	640	0.941	0.522	0.177	0.034
COCO mirrored relative position	1,786	0.923	0.490	0.210	0.020
What'sUp	820	0.518	0.233	1.174	0.096

Against Laya Vision 201M on the same questions (its published per-question predictions; difference in points, 95% interval from paired resampling by picture):

Set	Questions	Laya Vision	deVision	Difference
VQAv2 yes/no	4,887	0.717	0.749	+3.2 [+1.7, +4.7]
A-OKVQA	1,138	0.598	0.627	+2.9 [-0.3, +6.2]
ScienceQA with images	2,097	0.824	0.821	-0.3 [-2.3, +1.5]
ScienceQA natural science, needs the picture	323	0.700	0.601	-9.9 [-16.4, -3.4]

On the full POPE random / popular / adversarial sets (3,000 questions each) deVision scores 0.894 / 0.867 / 0.821, against Laya Vision's published 0.836 / 0.819 / 0.777 (aggregate scores only, not paired).

CPU latency: P50 301 ms, P95 321 ms (Linux x86_64, 4 threads, FP32, one question per request, warm-up excluded).

Model

Vision	SigLIP2-B/16, 256 × 256 letterboxed input, 64 visual tokens after a 2 × 2 merge and an MLP projector; training policy described below
Decision	Laya-initialised ModernBERT-large and Laya's decision head; each option is scored at its [MASK], then a temperature-scaled softmax
Size	About 518M parameters, FP32, 2.07 GB (LoRA merged)

This release continues v0.4 in one pass on one RTX 5070 (bf16), with SigLIP2's top four encoder layers and final LayerNorm fine-tuned at 5e-6 alongside the projector, decision head and ModernBERT LoRA (merged into the released weights). The input remains 256×256 with 64 visual tokens; the architecture is unchanged.

268,583 questions, one pass from v0.4, aimed at everyday uses v0.4 could not handle:

• phone screenshots (Android in the Wild: general and shopping tasks, with the screen state as the question) and web pages (Mind2Web: which marked element to act on), ScreenQA;
• first-person everyday scenes (EPIC-KITCHENS VISOR: what is being held, touched or done) and driving scenes (DriveLM, front camera);
• charts (ChartQA, distractors from the same chart), paper figures (ArxivQA), diagrams and science (AI2D, TQA, ScienceQA, IconQA, CLEVR);
• receipts and forms (CORD, FUNSD: which value a field holds), image–caption matching with hard negatives (COCO captions), meme moderation (Hateful Memes train), relative depth (COCO indoor photos labelled by a monocular depth model, kept only where the depth gap is large);
• replay of v0.4's natural-photo abilities (VQAv2, GQA, presence, size, position, relations, counting, Visual7W, VSR, A-OKVQA), whose targets were set to 0.5 × gold + 0.5 × v0.4's own answer so that the old abilities stay where they were.

Every picture of every evaluation set was excluded by id and image hash, including the pictures of the external benchmarks we report against (145 training rows were removed by this filter).

Temperatures were fitted at the end of training on a 1,362-question development set (charts, yes/no, presence, VQA and GQA choice), by question type and option-count bucket:

Bucket	Temperature
Choice, 2 options	1.1246
Choice, 3–5 options	1.1677
Noul (yes / no)	0.9890
Choice, any other count	1.1601

Against v0.4 on the same questions (95% intervals from resampling pictures; full table in accept_vs_v0.4.md):

set	v0.4	v0.5	difference
phone shopping screens (AitW, state questions)	21.4%	68.5%	+47.0 [+42.8, +51.2]
phone / web screens (AitW, Mind2Web)	38.6%	78.2%	+39.7 [+36.8, +42.7]
phone screen state	25.3%	52.4%	+27.1 [+24.0, +30.1]
first-person everyday scenes (EPIC-KITCHENS)	34.7%	71.7%	+37.0 [+35.4, +38.7]
charts (ChartQA)	30.7%	58.9%	+28.2 [+25.7, +30.9]
CV-Bench	55.1%	60.8%	+5.7 [+4.0, +7.3]
What'sUp	46.7%	51.8%	+5.1 [+2.7, +7.6]
POPE	86.5%	86.1%	−0.5 [−1.1, +0.1]

On held-out pictures of the new training sources: paper figures 32.0% → 51.5%, receipts / forms 39.3% → 53.4%, driving scenes 43.5% → 75.7%, caption matching 85.9% → 94.4%, memes 50.8% → 60.9%, relative depth 50.4% → 55.4% (all intervals above 0).

No other evaluation set (MMBench, MME, NaturalBench, RealWorldQA, CountBench, EgoThink, CoGenT, the laya-vision sets, our presence / position / relation / size / VSR / Visual7W / VQA tests) is significantly worse.

Limitations

• Scope. English, one image, yes/no and choice questions only; score is not implemented. At 256×256, small text (dense documents, infographics, desktop web pages) is hard to read.

• POPE. About 0.5 points below v0.4 on POPE (86.1% vs 86.5%, interval [−1.1, +0.1]) (object-presence hallucination); the owner accepted this small regression for the new abilities.

• Web pages. Our held-out phone and web screens improved strongly, but choosing the marked element to act on in dense desktop web pages stays near chance.

• Counting and science did not improve over v0.4 (CountBench 51.5% vs 50.1%, COCO counting 35.0% vs 34.5%, ScienceQA 82.1% vs 82.1%, none significant).

• Calibration. NLL is lower than or equal to v0.4's on 27 of 30 sets (strongly on the new uses), but the model is more over-confident on NaturalBench (NLL +0.029, significant); ECE is also higher on RealWorldQA, MMBench, NaturalBench and a few others (by 0.003–0.029). Refitting the temperatures on the development sets alone narrows but does not remove this.

• Labels. The relative-depth questions use a depth model's predictions, not measured depth; the memes source has its own research licence. Several sources are non-commercial; see the dataset list.

• Test-set reuse. Test sets informed several decisions in this cycle. This is one training seed.

• Release decision and original goals. The owner approved r24 as v0.5 on 2026-10-30 with the calibration trade-offs above. This release does not establish official ImageJevBench v0.3.0 admission, sub-1B leadership, or improvement across its five official domains. The five project-use evaluations above are a different protocol; no official rank is claimed.

Licence

Weights and code: Apache-2.0, like the three base models (Laya, SigLIP2, ModernBERT). The training datasets (listed in this card's metadata) have their own terms, some non-commercial (e.g. ScienceQA, CC BY-NC-SA 4.0); check them against your use.

Links

Code and training records (this release: tag model-v0.5) · evaluation details · provenance · Laya Vision