Aikyam-Lab/gridvqa-models

GridVQA-X Models

This repository contains two paired reference models, M_pure and M_spur, built on identical transformer architectures (MDETR). These models, coupled with their corresponding datasets, together form a diagnostic framework to evaluate if Multimodal Explainable AI (MxAI) methods genuinely capture cross-modal synergy or simply report shallow feature correlations.

Model Descriptions

1. M_pure (The Faithful Spatial Reasoner)

• Training Distribution: Trained exclusively on the D_pure dataset.
• Behavioral Dynamics: Trained via explanation-guided dynamics using a two-phase optimization. Phase 1 forces explicit visual-textual token alignment via L1 and generalized IoU losses. Phase 2 handles Question Answering using class-frequency weighted cross-entropy to completely eliminate answer prior biases.
• Capabilities: Successfully internalizes true causal spatial-relational synergy, achieving robust accuracy across both clean and heavily distractor-crowded grids.

2. M_spur (The Shortcut / Bag-of-Words Model)

• Training Distribution: Trained exclusively on the D_spur dataset.
• Behavioral Dynamics: Structurally forced to rely on cross-modal shortcuts during training. It skips relational spatial geometry entirely and maps keywords directly to target visual volume.
• Capabilities: Achieves perfect accuracy (1.000) on its native spurious distribution, but fails catastrophically when evaluated on D_pure multi-hop queries.

Intended Diagnostic Use

These models are released explicitly to stress-test vision-language explainability algorithms (e.g., DIME, MultiSHAP, MultiViz, EMAP, InterSHAP):

• The Litmus Test: A faithful explainer must output completely different attribution heatmaps or synergy scalars for M_pure and M_spur on the same input question.
• The Reality Check: If your explainer highlights identical spatial regions for both models, it suffers from "model blindness" or is simply behaving as a superficial object detector.

Performance Benchmark Metrics

Evaluation Metric	M_pure on D_pure	M_spur on D_spur	M_spur on D_pure
Global Accuracy	>99%	100%	Catastrophic Failure (8%-14% on multi-hop)
Causal Pathway	True Spatial Relations	Bag-of-Words Shortcut	Unimodal Feature Collapse