Semantic Layering Vision Benchmark — Results Report (Corrected)

Source: HuggingFace post by @EpsilonGreedyAI — June 2026

16 vision models. One mountain valley. Which actually sees it?

Test Image: Okanogan Valley, near Omak, WA — Northern Cascade Range foothills
Evaluator: Manual review + structured rubric
Framework: semantic-layering-prompt.md — single-pass, 14-section analysis
Confidence: Scores ±2 pts (single-image evaluation)

Quick Reference Card

Executive Summary

Sixteen vision-language models analyzed a single Okanogan Valley landscape across 10 depth-stratified reasoning tiers — from surface object detection through counterfactual viewpoint projection.

Top performers: Both Nex N2 Mini variants (94, 91) scored highest and were the only models to localize the region to the Okanogan/Cascade area. A surprise third-place entry: Qwable 9B Fable 5 (87), a 9B model that was the only non-Nex model to correctly identify the stone cairn fence posts and the only model to detect utility power lines crossing the valley. DeepSeek V4 (86) and Qwen 3.6 40B Deckard Heretic (86) tied for fourth with strong geological reasoning and regional awareness. Qwen 3.6 27B models (82–84) followed closely. The best previously-known sub-10B model was Qwen 3.5 9B at 77 — now surpassed by Qwable 9B at 87.

Critical failure modes: Four models placed the tree in front of the fence (occlusion reasoning failure). All three Gemma 4 variants misidentified stone fence posts as wood — a systematic material classification deficit. Model size correlated weakly with performance (r ≈ 0.4); vision-encoder architecture and reasoning depth were far stronger predictors.

Bottom line: For landscape-scale vision tasks requiring spatial reasoning, MoE models with agentic thinking (Nex, 35B total / 3B active) outperform dense vision-first models (Qwen3-VL 30B). Gemma 4 is unreliable for material classification at mid-distance.

Ranked Results

Model Metadata

Methodology

Test Design

Each model received the identical prompt from semantic-layering-prompt.md — a 14-section structured analysis request covering foreground, midground (fence, tree, secondary elements), background (left forest, mountains/sky, right terrain), depth ordering, occlusion reasoning, atmospheric inference, viewpoint/elevation, geological transition, and final synthesis. The prompt was appended to a single image upload with no system prompt modifications.

Scoring Framework

Each of the 10 tiers contributes points toward a 100-point composite:

Scoring Rules:

• T0: +1 per correct yes/no (5 positives: fence, mountains, deciduous tree, conifers, utility pole, clouds; 5 negatives: water, snow, road, building)
• T1: +2 per correct count within ±1 tolerance (posts: 5-7, deciduous trees: 1, utility poles: 1)
• T2: +2 per correct attribute (grass color, tree color, conifer color, post material, mountain color, grass texture). Partial credit (+1) for imprecise but not wrong answers.
• T3: +2.5 per correct spatial relation (fence/tree front-behind, right-of-tree, forest location, tree relative to fence+mountains)
• T4: +2.5 per correct depth judgment (ordering list, between-fence-and-mountains, pole front/behind)
• T5: +2 per correct material ID + reasoning (fence components, mountain texture, left-right surface contrast)
• T6: +2 per correct atmospheric inference (season, weather cloud/fog distinction, time of day, snow vs cloud)
• T7: +2.5 per occlusion judgment (above+through visibility, removal counterfactual, tree base occlusion)
• T8: +2.5 per geological insight (left-right transition description, biome/region ID, moisture gradient)
• T9: +3 per viewpoint judgment (elevation assessment, 50m forward projection, regional localization quality). Regional bonus: Okanogan/Cascades = +3, PNW = +2, Rockies/Sierra = +1, wrong continent = 0.

Composite Metrics

Per-Tier Heatmap

Scores normalized to tier maximum (darker = better [depending on device theme]).

 Model                                    T0  T1  T2  T3 T4  T5  T6  T7  T8 T9 │ Total
───────────────────────────────────────────────────────────────────────────────────────│───────
 1  Nex N2 Mini Apex MTP                  ██  ██  ██  ██  ██  ██  ██  ██  ██  ██ │  94
 2  Nex N2 Mini APEX I-Compact            ██  ██  ██  ██  ██  ██  ██  ██  ██  ██ │  91
 3  Qwable 9B Fable 5                     ██  █▓  ██  █▓  ██  ██  █▓  ██  █▓  ▒▓ │  87
 4  DeepSeek V4 (web)                     ██  ██  ██  ██  ██  ██  ██  ██  ██  █▓ │  86
 5  Qwen 3.6 40B Deckard Heretic          ██  ██  █▓  ██  █▓  ██  ▒▓  ██  ██  █▓ │  86
 6  Qwen 3.6 27B Abliterated              ██  ██  ██  ██  ██  ██  ██  ██  ██  █▓ │  84
 7  Qwen 3.6 27b Abl. i1                  ██  ██  ██  ██  ██  ██  ██  ██  █▓  █▓ │  82
 8  Unsloth Qwen 3.6 35B A3B              ██  ██  ██  ██  ██  ██  ██  ██  █▓  █▓ │  80
 9  Qwen 3.5 9B Unc. Aggressive           ██  ██  ██  ██  ██  ██  ██  █▓  █▓  █▓ │  77
10  Qwen 3 VL 8B Q4_K_M                   ██  ██  ██  █▓  █▓  ██  ██  █▓  █▓  █▓ │  67
11  Qwen3 VL 30B A3B Abliterated          ██  ░░  █▓  ░░  █▓  ██  ██  █▓  █▓  █▓ │  62
12  Qwen 3 VL 8B Q8_0                     ██  ██  ██  ░░  █▓  ██  ██  █▓  █▓  █▓ │  62
13  Gemma 4 E2B Uncensored                ██  █▓  █▓  ██  ██  ░░  ██  █▓  █▓  █▓ │  52
14  Qwythos 3.5 9B                        ██  █▓  █▓  ░░  █▓  █▓  ██  ░░  █▓  ░░ │  48
15  Gemma 4 12B it Q8_K_XL                ██  █▓  █▓  ██  ██  ░░  ██  █▓  █▓  ░░ │  45
16  Gemma 4 E4B Uncensored                ██  █▓  █▓  ░░  █▓  ░░  ██  ░░  █▓  ░░ │  40
───────────────────────────────────────────────────────────────────────────────────────│───────
█ = 90-100%   ▓ = 60-89%   ▒ = 30-59%   ░ = 0-29%

Correction note: Original heatmap contained a duplicate row for Qwen 3.6 40B Deckard Heretic (appeared at both position 4 and position 9). Removed duplicate — 16 rows → 15. Qwable 9B Fable 5 added from semantic-layering-responses4.md — 15 rows → 16.

Individual Model Scorecards

1. Nex N2 Mini Apex MTP — 94/100 (S-Tier)

Strengths: Pixel-level post counting in thinking trace. Regional localization to specific valley system. Thorough occlusion reasoning distinguishing above-fence from through-slat visibility. Self-correction on sun angle direction.

Weaknesses: Thinking trace shows mild over-analysis on post count (debated 7 vs 8 for ~20 lines) — thoroughness bordering on obsessive, but final answer was correct.

2. Nex N2 Mini Abliterated APEX I-Compact — 91/100 (S-Tier)

Strengths: Same regional knowledge as sibling model. Comprehensive thinking trace (~200 lines). Correct on all material, spatial, and depth judgments.

Weaknesses: Thinking trace slightly less pixel-precise on post count than Apex MTP. Minor hedging on tree species ("aspen/cottonwood/poplar" vs more confident single ID).

3. Qwable 9B Fable 5 (Empero-AI) — 87/100 (A-Tier)

Strengths: Best material classification of any model outside the Nex pair — correctly identified stone cairn fence posts where DeepSeek V4 (~236B), all Gemma 4 variants, and most Qwen models failed. Only model in the entire benchmark to detect utility power lines crossing the valley, a low-contrast thin linear element against complex background. Flawless depth ordering and occlusion reasoning. For a 9B model at Q4_K_M quantization, this is an exceptional result.

Weaknesses: Regional localization imprecise — lists 3 mountain ranges rather than committing. The model sees the scene accurately but lacks the geographic knowledge to place it. "What stands directly to the right of the prominent tree" answered generically rather than specifically identifying the conifer cluster and red/brown shrubs.

Architecture significance: This is a Qwen 3.5 derivative with a DPO/RLHF tune (Fable 5) optimized for visual instruction following. Its thinking trace shows methodical section-by-section reasoning without the obsessive over-analysis of the Nex models. It correctly resolved the stone-vs-wood material distinction efficiently — no 20-line post-count debate needed. Suggests the "Fable" tuning approach improves material classification fidelity in vision-language tasks.

4. DeepSeek V4 (chat.deepseek.com) — 86/100 (A-Tier)

Strengths: Most polished output. Produced the initial semantic layering framework from which the test prompt was derived. Strong occlusion and atmospheric reasoning.

Weaknesses: Hedged on tree species. Regional localization missed the Cascades entirely — placed it in Rockies. Additionally, failed to identify the ~5 smaller-diameter wooden vertical posts spaced between the stone pillars — the fence alternates stone columns with intermediate wooden posts supporting the horizontal rails. DeepSeek registered only one thin wooden pole (calling it a "utility pole") and otherwise collapsed these structural elements into the "slats" category. The Nex models, by contrast, distinguished them. Adjusted T5: 10→9 (−1).

5. Qwen 3.6 40B Deckard Heretic — 86/100 (A-Tier)

Strengths: Best regional awareness among Qwen models — explicitly lists "Cascade Range valleys" among candidates. Strong geological transition analysis. Detailed grass species speculation (rye grass). Stone posts, depth ordering, and occlusion reasoning all correct.

Weaknesses: Three errors: (1) Tree species: called it oak — a Western NA ecology gap. Cottonwood/aspen is correct for this region. (2) Time-of-day: 8-10 AM off by 3–5 hours. Fog pattern in mountains misled the model into assuming early morning despite bright midday lighting (actual: ~noon–1PM). (3) Regional: listed 3 ranges rather than committing — regional knowledge is present but imprecise.

6. Qwen 3.6 27B Abliterated — 84/100 (A-Tier)

Strengths: Identified "wattle" fence construction style. Accurate on all perceptual dimensions. Good atmospheric and occlusion reasoning.

Weaknesses: The European Alps guess is a significant regional error — the dry-stone + wattle fence style misled it to European pastoral landscapes. Correctly noted Rockies as alternative.

7. Qwen 3.6 27B Abliterated i1 — 82/100 (A-Tier)

Similar profile to sibling 27B model. Region: "Sierra Nevada or Rocky Mountain foothills." Solid perceptual accuracy, minor regional imprecision. (Abbreviated scorecard — full tier breakdown not provided in original.)

8. Unsloth Qwen 3.6 35B A3B MTP — 80/100 (A-Tier)

Strong perceptual accuracy. Region: "Pacific Northwest, British Columbia, or Rockies" — good range including PNW. Minor omissions in occlusion section. Solid all-around. (Abbreviated scorecard — full tier breakdown not provided in original.)

9. Qwen 3.5 9B Uncensored Aggressive — 77/100 (A-Tier)

Strong for a 9B model. Stone posts identified correctly. Region: "Northern Rocky Mountains (Montana/Idaho)." Minor omissions in occlusion reasoning and geological transition. (Abbreviated scorecard — full tier breakdown not provided in original.)

10. Qwen 3 VL 8B Q4_K_M — 67/100 (B-Tier)

Weaknesses: Quantization to Q4_K_M shows in reduced precision on spatial judgments. The model hedges and contradicts itself on tree-fence depth relationship.

11. Qwen3 VL 30B A3B Abliterated — 62/100 (B-Tier)

Critical errors:

• Tree position: IN FRONT of fence — fundamental occlusion failure (−4 on T3, −2.5 on T4)
• Post count: 4 — catastrophic undercount (−3 on T1)
• Forest type: "Deciduous trees mixed with coniferous" — reversed dominance (−2 on T2)

Analysis: Most disappointing result for a 30B-class model. The tree-in-front error suggests a failure to resolve occlusion cues — the model saw "bright orange tree" and "darker fence line below it" and defaulted to the tree being closer (bright things are typically nearer). It did not process that the fence slats cross in front of the tree trunk. The 4-post count suggests it saw only the most prominent stone pillars and missed the intermediate ones.

12. Qwen 3 VL 8B Q8_0 — 62/100 (B-Tier)

Same tree-in-front error as the 30B variant — suggests this is a Qwen3-VL architectural pattern, not a scale issue. Both the 8B and 30B Qwen3-VL models made this same error. The 27B text-first Qwen models did not.

13. Gemma 4 E2B Uncensored — 52/100 (C-Tier)

Critical failure: All Gemma 4 variants misidentify stone posts as wood, but the failure is a reasoning error, not a perceptual one. The vision encoder registers the stone texture (stacked, irregular, grey); the LLM's "fence post → wood" prior overrides the visual evidence. The E2B is the least severe case — it at least placed the tree behind the fence correctly and also identified the smaller intermediate wooden posts between the stone pillars ("additional smaller posts defining the gaps between the main sections"), a structural detail that even DeepSeek V4 missed.

14. Qwythos 3.5 9B — 48/100 (C-Tier)

Critical errors:

• Tree in front of fence
• Response is mostly a single-paragraph synthesis — misses most prompt sections
• No separate analysis of depth layers, occlusion, atmospheric inference
• Region: "Rocky Mountains or Sierra Nevada" — acceptable

Analysis: This model produced the shortest response. The thinking trace was more comprehensive than the output, suggesting the model can reason but either hit an output length constraint or was optimized for brevity. The output reads like a summary of the thinking, not a full response. Lowest completeness score.

15. Gemma 4 12B it Q8_K_XL — 45/100 (C-Tier)

Analysis: The "decorative brush bundles" / "stone crowns" description is the key diagnostic: the vision encoder registered the irregular stacked-stone shape and reported it; the LLM could not reconcile "stone texture on a fence post" with its "fence post = wood" prior, so it confabulated a compromise — wood posts with stone decorations. This is an LLM prior override, not a texture-to-material mapping failure at the encoder level. The encoder saw the difference; the LLM refused to believe it.

16. Gemma 4 E4B Uncensored — 40/100 (C-Tier)

Analysis: Worst overall performance. Combines both critical failure modes: tree-in-front (spatial — perceptual error) AND posts-as-wood (material — LLM prior override). The "uncensored" fine-tune may have degraded perceptual accuracy in exchange for reduced refusal rates. Notably, the E4B described the posts as "thick, irregular, naturally shaped" — words that describe stacked fieldstone as accurately as rough timber — showing the encoder transmitted the right texture signal; the LLM applied the wrong label.

Gap Analysis: Score Delta from #1

Where each model loses points relative to the leader. Positive deltas = points lost. Tiers listed in descending order of average loss across all models (hardest tiers first).

| # | Model | Total | T9 Reg | T5 Mat | T3 Spa | T4 Dep | T2 Attr | T7 Occl | T1 Cnt | T8 Geo | T6 Atm | T0 Obj |
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| 1 | Nex Apex MTP | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 |
| 2 | Nex APEX I-Comp | +3 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 |
| 3 | Qwable 9B Fable 5 | +7 | +5 | 0 | +2 | 0 | 0 | 0 | +1 | +3 | +2 | 0 |
| 4 | DeepSeek V4 | +8 | +7 | +1 | 0 | 0 | +2 | 0 | 0 | +2 | 0 | 0 |
| 5 | Qwen 40B Deckard | +8 | +5 | 0 | 0 | +1 | +4 | 0 | 0 | 0 | +4 | 0 |
| 6 | Qwen 27B Abl. | +10 | +9 | 0 | 0 | 0 | 0 | 0 | 0 | +3 | 0 | 0 |
| 7 | Qwen 27B i1 | +12 | +6 | 0 | 0 | 0 | 0 | 0 | 0 | +4 | 0 | 0 |
| 8 | Unsloth Qwen 35B | +14 | +6 | 0 | 0 | 0 | 0 | +2 | 0 | +4 | 0 | 0 |
| 9 | Qwen 9B Aggr. | +17 | +6 | 0 | 0 | 0 | 0 | +4 | 0 | +5 | 0 | 0 |
| 10 | Qwen VL 8B Q4 | +27 | +6 | 0 | +4 | +4 | 0 | +3 | 0 | +5 | 0 | 0 |
| 11 | Qwen3 VL 30B | +32 | +6 | 0 | +5 | +2.5 | +2 | +3 | +3 | +4 | 0 | 0 |
| 12 | Qwen VL 8B Q8 | +32 | +6 | 0 | +5 | +2.5 | 0 | +3 | 0 | +5 | 0 | 0 |
| 13 | Gemma 4 E2B | +42 | +6 | +7 | 0 | 0 | +6 | +3 | +2 | +5 | 0 | 0 |
| 14 | Qwythos 9B | +46 | +9 | +4 | +5 | +2.5 | +2 | +5 | +2 | +5 | 0 | 0 |
| 15 | Gemma 4 12B | +49 | +8 | +8 | 0 | 0 | +6 | +3 | +2 | +5 | 0 | 0 |
| 16 | Gemma 4 E4B | +54 | +9 | +8 | +5 | +2.5 | +6 | +5 | +2 | +4 | 0 | 0 |

Correction note: Original had "#8 Qwen 40B Deckard" — corrected to "#4".

Hardest tiers (avg loss): T9 Regional (+5.5) > T8 Geological (+3.1) > T2 Attribute (+2.1) > T5 Material (+1.9) > T3 Spatial (+1.6).

Easiest tiers (avg loss): T0 Object Presence (0.0) > T6 Atmospheric (+0.3) > T1 Counting (+0.7).

Interpretation: Every model passed object presence and atmospheric inference. The spread comes from regional knowledge, material classification, and spatial reasoning — exactly the tiers that distinguish scene-understanders from scene-describers.

Failure Mode Analysis

Failure Mode 1: Tree Position Reversal (4 models)

Affected: Qwen3 VL 30B, Qwen 3 VL 8B (both quants), Gemma 4 E4B, Qwythos 3.5 9B

Symptom: Model reports tree is in front of fence when it is actually behind.

Root Cause: Bright, salient object (orange tree) is interpreted as nearer than darker, less salient object (fence). Model fails to use occlusion cues — fence slats visibly cross in front of tree trunk.

Architecture Pattern: Affects Qwen3-VL family at multiple scales (8B and 30B) but NOT Qwen 3.6 text-first models or Nex series. Suggests vision-encoder-first architectures may be more susceptible than language-model-first architectures with vision bolted on.

Severity: Critical. Fundamental depth perception error.

Failure Mode 2: Stone Posts → Wood (3 models)

Affected: All Gemma 4 variants (E2B, E4B, 12B)

Symptom: Stone pillars described as wooden posts. E4B: "thick, irregular timber logs." 12B: "heavy squared timbers with decorative bundles."

Root cause — LLM prior override, not encoder blindness: The vision encoder does detect the stone texture. Evidence: the 12B describes "decorative stone crowns" on top of the posts and notes them as "irregular" — it registers the stacked-fieldstone visual signal. But the language model applies a strong "fence post → wood" prior that overrides the visual classification. The encoder says "stacked irregular grey material"; the LLM resolves that to "wood posts with stone decorations" (12B), "rough-hewn irregular wood" (E4B), or simply "wood" (E2B). This is the same class of error as DeepSeek's Rockies mislocalization: the LLM prior beats the visual evidence.

Architecture Pattern: Consistent across all three Gemma 4 sizes tested. This is not an encoder-level limitation — it is a reasoning failure where the language model's material priors dominate the vision encoder's output. The encoder can see the difference; the LLM won't believe it.

Severity: High. Material classification is a core vision task. More fixable than a perceptual deficit — a system prompt rule ("trust visual texture over material priors") would directly target this.

Failure Mode 3: Regional Mislocalization (3 models)

Affected: DeepSeek V4 (Rockies), Qwen 3.6 27B (Alps), Qwen 3.6 40B Deckard Heretic (Cascades candidate list but wrong tree)

Symptom: Model identifies correct biome but wrong continent or mountain range.

Root Cause: The dry-stone fence with stacked fieldstone posts is visually similar to Alpine/Tyrolean pastoral fencing. Models using visual fence-style cues for geography were misled to Europe. Models using vegetation cues (conifer + cottonwood/aspen) correctly identified Western North America.

Severity: Moderate. Geographical knowledge is secondary to perceptual accuracy for most use cases.

Failure Mode 4: Post Count Undercount (4 models)

Affected: Qwen3 VL 30B (4), all Gemma 4 variants (5-8 but wrong material)

Symptom: Qwen3 VL 30B counts 4 posts when 6 are visible (5 full + 1 partial top on left edge).

Root Cause: Model counts only the most visually prominent posts and misses partially occluded or edge-cropped ones. Genuine instance segmentation failure rather than counting error.

Severity: Moderate. Counting repeated elements is a known weakness of current VLMs.

Architecture Insights

Vision-encoder-first vs Language-model-first

Language-model-first and MoE architectures consistently outperformed vision-encoder-first architectures on spatial reasoning (tree position). The Nex models (35B total, 3B active) dominated through reasoning depth — their active-parameter count is low, but their 35B knowledge pool and extensive thinking traces compensated for what dense vision-first models missed.

Material classification presents a different failure pattern: Gemma 4's 100% error rate is an LLM prior override (the encoder sees stone; the LLM says wood), while Qwen3-VL's 0% error rate on materials but 66% error rate on spatial relations suggests vision-first architectures handle texture-to-material mapping well but fail on relational reasoning that requires integrating visual evidence with 3D mental models. The two failure types are architectural inverses of each other.

Thinking/Reasoning Depth

Strong correlation between thinking trace depth and overall score. Models that reasoned step-by-step about occlusion, material properties, and spatial relationships made fewer critical errors.

Quantization Impact (Qwen 3 VL 8B)

Minimal score difference (5 pts). Both quants gave post count 5 — within ±1 of the actual 6 (5 full + 1 partial). The Q4_K_M variant was slightly more cautious in spatial judgments (hedged with "same depth") which improved its score by avoiding the definitive wrong answer.

Key Findings

1. Regional localization is the best single discriminator of model quality. Only the top 2 models (both Nex variants) correctly identified the Okanogan/Cascade region. Every other model placed the scene somewhere between the Alps and the Rockies. This capability correlates strongly with overall score (r ≈ 0.85).

2. Occlusion reasoning separates scene-understanders from scene-describers. The tree-behind-fence question was the single most diagnostic item: models that got it right scored 80+; models that got it wrong scored ≤65. It tests whether the model builds a 3D mental model or describes a 2D image.

3. Gemma 4 has a systematic LLM prior override on material classification. All three variants registered the stone texture (the 12B even described "stone crowns") but the language model's "fence post → wood" prior overrode the visual evidence. This is a reasoning failure, not a perceptual one — the encoder sees the difference; the LLM overrides it. Contrast with Qwen3-VL's tree-in-front error (perceptual: brightness/salience dominates occlusion cues at the encoder level).

4. MoE + thinking depth outperforms dense vision-first architectures. The Nex models (35B total, 3B active) outscored both the dense Qwen3-VL 30B and the much larger DeepSeek V4 (~236B). Their MoE architecture provides a large knowledge pool at low inference cost, while extensive reasoning traces catch perceptual ambiguities.

5. Vision-first architectures struggle with depth from occlusion. Qwen3-VL models at both 8B and 30B scales made the same tree-in-front error, while Qwen 3.6 text-first models of comparable size did not. The vision encoder in vision-first architectures may dominate the final judgment, overriding contradictory signals from the language model.

6. Quantization has minimal impact on this task. The Q8_0 vs Q4_K_M comparison for Qwen 3 VL 8B showed only a 5-point difference, with the lower quant actually scoring slightly higher due to more cautious spatial hedging.

Recommendations

For Model Selection

• Best overall: Nex N2 Mini (either variant) for vision tasks requiring spatial reasoning and regional knowledge
• Best large model: DeepSeek V4 for polished, comprehensive scene analysis
• Best local model: Qwen 3.6 27B Abliterated for strong perceptual accuracy without cloud dependency
• Best regional awareness (Qwen family): Qwen 3.6 40B Deckard Heretic — explicitly lists Cascade Range valleys (but verify species and time-of-day on your own images)
• Avoid for landscape vision: Gemma 4 variants (LLM material priors override visual evidence — stone posts→wood); Qwen3-VL 8B/30B (spatial reasoning unreliable — tree-in-front perceptual error)

For Benchmark Improvement

• Add a material classification tier with explicit binary choices (wood/stone/metal) to isolate texture-to-material mapping from free-text description
• Add depth-from-occlusion forced-choice pairs (A in front of B or B in front of A?) to eliminate hedging
• Include a confidence calibration check — ask models to rate their confidence on spatial judgments
• Test with fence removed via inpainting as a control image to verify occlusion reasoning independently

For Further Investigation

• Why do Nex models have such strong regional knowledge? Training data source analysis needed.
• Is the Gemma 4 material deficit fixable via fine-tuning the prompt to distinguish posts vs columns, or is it a visual decoder deficit?
• Does the Qwen3-VL tree-position error persist across all images with foreground occlusion, or is it specific to this brightness/salience pattern?

Caveats & Limitations

1. Single-image evaluation. All scores derive from one image. Model rankings may shift with different scene types (urban, indoor, night, abstract). Treat scores as indicative, not definitive. A multi-image benchmark suite (10+ diverse scenes) would yield ±1–2 point confidence.

2. Manual scoring subjectivity. Tier scores were assigned by human review of free-text responses. While a structured rubric was applied consistently, different evaluators might vary ±2–3 points on borderline judgments. Automated LLM-as-judge re-scoring is planned for reproducibility.

3. Prompt sensitivity. All models received the identical prompt with no system-prompt modifications. Performance may vary with different prompting strategies (chain-of-thought, structured output formatting, role assignment).

4. Thinking mode variability. Some models were tested with thinking enabled, others without (per availability in Ollama). Direct comparison of "thinking vs. no-thinking" variants within the same model family was only possible for Qwen 3 VL 8B.

5. Quantization and hardware. Local models were run at various quantization levels on consumer GPU hardware. Cloud models (DeepSeek V4) ran at full precision. Quantization impact was minimal in this benchmark but may be larger for tasks requiring fine-grained texture discrimination.

6. Temporal snapshot. Model performance reflects the versions available in June 2026. Provider updates, fine-tune releases, and quantization improvements may change rankings. Re-benchmarking every 3–6 months recommended.

7. Geographic knowledge bias. The test image is from a specific North American region. Models with training data skewed toward Western North American landscapes had an inherent advantage on T9 (Regional).

8. No speed/cost data. Inference time and token efficiency were not tracked. Future runs should include timing instrumentation.

Appendices

A. Raw Response Data:

• semantic-layering-responses1.md — DeepSeek V4, Qwen3 VL 30B, Nex N2 Mini APEX I-Compact
• semantic-layering-responses2.md — Qwen 3.6 27B variants, Gemma 4 variants, Qwythos 9B, Qwen 40B
• semantic-layering-responses3.md — Nex N2 Mini Apex MTP, Qwen 3 VL 8B variants, Qwen 3.5 9B, Unsloth Qwen 35B, Gemma 4 12B
• semantic-layering-responses4.md — Qwable 9B Fable 5 (Empero-AI)

B. Test Prompt: semantic-layering-prompt.md

C. Scoring Rubric: semantic-layering-test.md

D. Test Image: IMG_0773.JPG — Okanogan Valley, near Omak, WA (Northern Cascade Range foothills). Autumn, ~noon–1PM.

E. Radar Chart: radar-comparison.png — multi-model radar chart showing normalized tier scores for the top 8 models.

Corrections Applied