File size: 9,378 Bytes
2a4ebec | 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105 106 107 108 109 110 111 112 113 114 115 116 117 118 119 120 121 122 123 124 125 126 127 128 129 130 131 132 133 134 135 136 137 138 139 140 141 142 143 144 145 146 147 148 149 150 151 152 153 154 155 156 157 158 159 160 161 162 163 164 165 166 167 168 169 170 171 | ---
library_name: transformers
pipeline_tag: image-text-to-text
base_model:
- Qwen/Qwen3-VL-2B-Instruct
- Qwen/Qwen3-VL-4B-Instruct
- Qwen/Qwen3-VL-8B-Instruct
tags:
- vision-language
- multimodal
- android
- gui
- software-testing
- rotation-bug
- qwen3-vl
- vllm
---
# RotVL
RotVL is a rotation-aware vision-language model fine-tuned for cross-orientation state-equivalence checking and for detecting, classifying, and localizing GUI defects caused by screen rotation in Android applications. It accompanies the paper *“RotDroid: Cross-Orientation State Equivalence Testing for Detecting GUI Rotation Bugs in Android Apps”*, accepted at the 37th IEEE International Symposium on Software Reliability Engineering (ISSRE 2026).
RotDroid supplies RotVL with two screenshots representing corresponding portrait and landscape states. RotVL determines whether a rotation-induced GUI defect is present and, when requested, returns the defective orientation, defect type, and bounding box.
## Released checkpoints
The three released checkpoints are:
| Checkpoint | Base model |
| ------------------------------------------------------ | -------------------- |
| [ImDim/RotVL-2B](https://huggingface.co/ImDim/RotVL-2B) | Qwen3-VL-2B-Instruct |
| [ImDim/RotVL-4B](https://huggingface.co/ImDim/RotVL-4B) | Qwen3-VL-4B-Instruct |
| [ImDim/RotVL-8B](https://huggingface.co/ImDim/RotVL-8B) | Qwen3-VL-8B-Instruct |
Each RotVL checkpoint is obtained by fine-tuning the corresponding Qwen3-VL-Instruct model on the [RotBench dataset](https://huggingface.co/datasets/ImDim/RotBench). It accepts two Android GUI screenshots and a text instruction, and produces JSON output for bug detection or bug classification/localization.
## Evaluation on RotBench
The models are evaluated on RotBench test splits for bug detection, bug classification, defective-orientation identification, and bounding-box localization. Accuracy (Acc.), precision (Prec.), recall (Rec.), and F1 are percentages. For coordinate localization, a lower center-point distance is better, and an area ratio closer to 1 is better.
### Bug detection (%)
| Model | Acc. | Prec. | Rec. | F1 |
| ------------------ | --------------: | --------------: | --------------: | --------------: |
| Qwen3-VL-2B | 53.24 | 62.82 | 53.24 | 42.48 |
| Qwen3-VL-4B | 61.47 | 61.54 | 61.47 | 61.41 |
| Qwen3-VL-8B | 66.47 | 67.65 | 66.47 | 65.90 |
| Qwen3-VL-32B | 58.82 | 59.73 | 58.82 | 57.84 |
| Qwen3-VL-235B | 60.00 | 60.72 | 60.00 | 59.32 |
| GPT-5.2 | 70.88 | 71.01 | 70.88 | 70.84 |
| RotVL-2B | 71.76 | 74.21 | 71.76 | 71.03 |
| RotVL-4B | 76.47 | 76.50 | 76.47 | 76.46 |
| **RotVL-8B** | **85.29** | **85.37** | **85.29** | **85.29** |
### Bug classification (%)
| Model | Acc. | Prec. | Rec. | F1 |
| ------------------ | --------------: | --------------: | --------------: | --------------: |
| Qwen3-VL-2B | 20.59 | 8.76 | 20.59 | 9.56 |
| Qwen3-VL-4B | 21.47 | 20.54 | 21.47 | 13.03 |
| Qwen3-VL-8B | 30.59 | 52.49 | 30.59 | 26.02 |
| Qwen3-VL-32B | 30.00 | 65.02 | 30.00 | 25.89 |
| Qwen3-VL-235B | 37.06 | 46.05 | 37.06 | 33.77 |
| GPT-5.2 | 49.71 | 65.70 | 49.71 | 49.95 |
| RotVL-2B | 48.53 | 59.51 | 48.53 | 42.97 |
| RotVL-4B | 44.41 | 65.37 | 44.41 | 39.54 |
| **RotVL-8B** | **62.65** | **66.71** | **62.65** | **62.68** |
### Bug localization: orientation (%)
| Model | Acc. | Prec. | Rec. | F1 |
| ------------------ | --------------: | --------------: | --------------: | --------------: |
| Qwen3-VL-2B | 54.12 | 54.41 | 54.12 | 53.34 |
| Qwen3-VL-4B | 52.06 | 75.53 | 52.06 | 37.75 |
| Qwen3-VL-8B | 54.41 | 69.03 | 54.41 | 43.58 |
| Qwen3-VL-32B | 53.82 | 75.99 | 53.82 | 41.31 |
| Qwen3-VL-235B | 56.18 | 74.48 | 56.18 | 46.10 |
| GPT-5.2 | 68.82 | 70.61 | 68.82 | 68.13 |
| RotVL-2B | 57.35 | 68.95 | 57.35 | 49.56 |
| RotVL-4B | 71.47 | 77.38 | 71.47 | 69.84 |
| **RotVL-8B** | **80.29** | **82.36** | **80.29** | **79.97** |
### Bug localization: coordinates
| Model | Center-point distance (px) | Area ratio |
| ------------------ | -------------------------: | -------------: |
| Qwen3-VL-2B | 466 | 73.29 |
| Qwen3-VL-4B | 499 | 53.72 |
| Qwen3-VL-8B | 435 | 38.45 |
| Qwen3-VL-32B | 478 | 30.07 |
| Qwen3-VL-235B | 459 | 45.06 |
| GPT-5.2 | 276 | 9.76 |
| RotVL-2B | 237 | 4.87 |
| RotVL-4B | 263 | 16.62 |
| **RotVL-8B** | **198** | **3.17** |
RotVL-8B achieves the best result in every evaluated dimension. Its improvements over the strongest baseline are statistically significant for bug detection, bug classification, bug localization error (`p < 0.001`).
## Evaluation on natural bugs
Bug detection generalization is also evaluated using 100 runtime cross-orientation screenshot pairs sampled from 44 real-world applications with developer-confirmed rotation bugs.
| Model | Accuracy | Precision | Recall | F1 |
| ------------------ | --------------: | --------------: | --------------: | --------------: |
| Qwen3-VL-8B | 63.00 | 70.22 | 63.00 | 66.41 |
| GPT-5.2 | 65.00 | 59.21 | 65.00 | 61.97 |
| **RotVL-8B** | **70.00** | **74.81** | **70.00** | **72.32** |
RotVL-8B performs best on this natural bug set, indicating that its gains are not limited to the synthetic bug patterns in RotBench.
## Download
Choose one of `2B`, `4B`, or `8B` and download the corresponding checkpoint with the Hugging Face CLI. The following example downloads RotVL-8B:
```bash
MODEL_SIZE=8B
hf download "ImDim/RotVL-${MODEL_SIZE}" \
--local-dir "RotVL-${MODEL_SIZE}"
```
## Serve with vLLM
The tested model-server setup uses Linux, NVIDIA V100 GPUs, CUDA 12.2 or later, and vLLM 0.11.1. Start the selected checkpoint as an OpenAI-compatible service:
```bash
MODEL_SIZE=8B
CUDA_VISIBLE_DEVICES=0 VLLM_ALLOW_LONG_MAX_MODEL_LEN=1 \
vllm serve "RotVL-${MODEL_SIZE}" \
--served-model-name "RotVL-${MODEL_SIZE}" \
--limit-mm-per-prompt '{"image": 2}' \
--host 0.0.0.0 \
--port 8000 \
--dtype float16 \
--api-key xxx \
--max-model-len 32768 \
--tensor-parallel-size 1
```
The value passed to `--served-model-name` must match the model name used by the client.
## Output formats
### Bug detection
RotVL is prompted to return a strict JSON object:
```json
{"bug": true}
```
### Bug classification and localization
For a defective pair, RotVL is prompted to return:
```json
{
"type": "layout-overlap",
"image": "portrait",
"bbox_2d": [100, 200, 400, 500]
}
```
- `type` is one of `layout-overlap`, `layout-clip`, `layout-miss`, `direction-mismatch`, and `state-loseinput`.
- `image` is `portrait` or `landscape`.
- `bbox_2d` is `[x_min, y_min, x_max, y_max]` in the defective image's pixel coordinates.
## Intended uses
RotVL is intended for:
- use as the visual defect detector in RotDroid;
- research on Android GUI rotation-bug detection;
- evaluation on RotBench.
## Use with RotDroid
For integration details and instructions on running RotDroid, see the [RotDroid source repository](https://github.com/ImDiM/RotDroid).
## Related resources
- RotDroid source code: https://github.com/ImDiM/RotDroid
- RotBench dataset: https://huggingface.co/datasets/ImDim/RotBench
- Full artifact: https://doi.org/10.5281/zenodo.21897206
## Citation
```bibtex
@inproceedings{qin2026rotdroid,
title = {RotDroid: Cross-Orientation State Equivalence Testing for Detecting GUI Rotation Bugs in Android Apps},
author = {Qin, Mengdi and Jiang, Bo},
booktitle = {Proceedings of the 37th IEEE International Symposium on Software Reliability Engineering (ISSRE)},
year = {2026}
}
```
|