README.md

---
license: apache-2.0
pipeline_tag: image-text-to-text
---
<div align="center">

<h1>🚀 VisCo: Leveraging Large Language Models as Intrinsic Encoders for Visual Token Compression</h1>

<h3>ACM Multimedia 2026</h3>

<p align="center">
  <a href="https://github.com/Zyvpeng">Yupeng Zheng</a><sup>*</sup>,
  <a href="https://github.com/Jacky-hate">Kai Zou</a><sup>*</sup>,
  <a href="https://scholar.google.com/citations?user=kReWULQAAAAJ&hl=en&oi=ao">Bin Liu</a><sup>†</sup>,
  <a href="https://scholar.google.com/citations?user=7620QAMAAAAJ&hl=en&oi=ao">Nenghai Yu</a><sup>†</sup>
</p>

<p align="center">
  <sup>*</sup>Equal contribution &nbsp;&nbsp;
  <sup>†</sup>Corresponding author
</p>

<p align="center">
  <a href="https://arxiv.org/abs/2607.12756">
    <img src="https://img.shields.io/badge/arXiv-2607.12756-b31b1b.svg">
  </a>
  <!-- <a href="https://your-project-page.github.io">
    <img src="https://img.shields.io/badge/Project-Page-blue">
  </a>-->
  <a href="https://huggingface.co/zpatrick/VisCo">
    <img src="https://img.shields.io/badge/🤗%20Model-HuggingFace-yellow"> 
  </a>
  <a href="https://github.com/Zyvpeng/VisCo/blob/main/LICENSE">
    <img src="https://img.shields.io/badge/License-Apache--2.0-green">
  </a>
  <a href="https://github.com/Zyvpeng/VisCo">
    <img src="https://img.shields.io/github/stars/Zyvpeng/VisCo?style=social">
  </a>
</p>

<p align="center">
  <strong>
    Compress hundreds of visual tokens into only a few memory tokens while preserving multimodal understanding.
  </strong>
</p>

</div>

## 🔥 Highlights

- **Intrinsic self-compression.** VisCo reuses the pretrained VLM itself as a parameter-sharing autoencoder instead of introducing a separate compression network.
- **Lightweight adaptation.** Only memory tokens and encoder-side LoRA adapters are trained; the shared pretrained backbone is kept intact for decoding.
- **Hierarchical KV passing.** Layer-wise key-value states associated with memory tokens are transferred from the encoder directly into the decoder KV cache.
- **Extreme compression.** VisCo retains 85.3% of LLaVA-1.5-7B performance with one visual token and 91.4% of Qwen2-VL-2B performance with 18 tokens.
- **Broad evaluation.** Experiments cover LLaVA-1.5-7B, Qwen2-VL-2B, and Qwen2-VL-7B on six multimodal benchmarks.


## 📊 Main Results

### LLaVA-1.5-7B

Average is the mean percentage of performance retained relative to the original model.

| Method | Tokens | GQA | MMB | MMB-CN | MME | POPE | MMVet | Average |
|:--|--:|--:|--:|--:|--:|--:|--:|--:|
| LLaVA-1.5-7B | 576 | 62.0 | 64.3 | 58.3 | 1510.7 | 85.9 | 31.1 | 100.0% |
| FastV | 32 | 41.5 | 37.8 | 33.2 | 884.6 | 32.5 | 20.7 | 57.6% |
| SparseVLM | 32 | 48.3 | 51.4 | 40.6 | 1046.7 | 67.9 | 18.6 | 72.6% |
| PruMerge+ | 32 | 51.1 | 56.8 | 47.0 | 940.8 | 70.9 | 21.4 | 77.5% |
| DivPrune | 32 | 54.9 | 57.6 | 49.1 | **1284.9** | 81.5 | 26.3 | 87.8% |
| VisPruner | 32 | 52.2 | 58.4 | 52.7 | 1271.0 | 72.7 | **28.8** | 87.8% |
| **VisCo** | **32** | **58.5** | **62.3** | **57.2** | 1152.9 | **81.9** | 27.9 | **91.8%** |
| PruMerge+ | 1 | 26.4 | 13.7 | 13.7 | 568.9 | 40.4 | 12.5 | 35.4% |
| VisPruner | 1 | 41.8 | 22.4 | 25.8 | 764.4 | 49.0 | 12.0 | 48.8% |
| **VisCo** | **1** | **58.2** | **59.0** | **52.4** | **1191.2** | **78.2** | **20.7** | **85.3%** |

### Qwen2-VL

| Base model | Method | Tokens | GQA | MMB | MMB-CN | MME | POPE | MMVet | Average |
|:--|:--|--:|--:|--:|--:|--:|--:|--:|--:|
| Qwen2-VL-2B | Origin | 144 | 59.8 | 67.3 | 61.8 | 1465.8 | 81.5 | 42.5 | 100.0% |
| Qwen2-VL-2B | VisionZip | 36 | 53.2 | 64.3 | 56.7 | 1372.4 | 75.3 | 35.5 | 91.0% |
| Qwen2-VL-2B | **VisCo** | **36** | **61.1** | 64.0 | **58.9** | 1368.3 | **82.9** | **35.6** | **95.2%** |
| Qwen2-VL-2B | VisionZip | 18 | 48.5 | 55.2 | 41.6 | 1300.9 | 62.7 | 25.8 | 76.1% |
| Qwen2-VL-2B | **VisCo** | **18** | **59.7** | **62.9** | **57.8** | **1358.0** | **82.4** | **28.9** | **91.4%** |
| Qwen2-VL-7B | Origin | 144 | 64.8 | 76.1 | 71.6 | 1664.8 | 82.6 | 56.1 | 100.0% |
| Qwen2-VL-7B | VisionZip | 36 | 58.3 | 72.4 | 63.4 | 1530.0 | 75.3 | 39.1 | 87.7% |
| Qwen2-VL-7B | **VisCo** | **36** | **62.6** | **74.0** | **70.0** | **1551.0** | **83.3** | **40.4** | **92.9%** |
| Qwen2-VL-7B | VisionZip | 18 | 55.3 | 66.4 | 58.3 | 1443.8 | 71.7 | 33.9 | 81.3% |
| Qwen2-VL-7B | **VisCo** | **18** | **61.8** | **70.0** | **68.8** | **1496.5** | **81.7** | **39.2** | **90.4%** |

## 🛠️ Installation

```bash
git clone https://github.com/Zyvpeng/VisCo.git
cd VisCo

conda create -n visco python=3.10 -y
conda activate visco
pip install --upgrade pip
pip install torch transformers peft datasets safetensors qwen-vl-utils
pip install flash-attn --no-build-isolation
```

The current implementation additionally uses `torch`, `transformers`, `peft`, `datasets`, `safetensors`, `qwen-vl-utils`, and FlashAttention for LLaVA.

## ⚡ Quick Start

The case examples read `/home/ypzheng/VLM_ICAE/data/case/data.json`. Each item must provide `image` and `question` fields. Run commands from the repository parent so that `VisCo` is importable.

### Qwen2-VL case inference

```bash
cd /home/ypzheng
conda activate mmicae2

python -m VisCo.inference.case_qwen2 \
  --model_name_or_path /home/ypzheng/Qwen2-VL-2B-Instruct \
  --output_dir /path/to/qwen2_visco_checkpoint.pt \
  --fixed_mem_size 18 \
  --train false \
  --per_device_train_batch_size 1
```

### LLaVA-1.5 case inference

```bash
cd /home/ypzheng
conda activate mmicae2

python -m VisCo.inference.case_llava15 \
  --model_name_or_path /home/ypzheng/llava15 \
  --output_dir /path/to/llava_visco_checkpoint.pt \
  --fixed_mem_size 32 \
  --train false \
  --per_device_train_batch_size 1
```

`--output_dir` is used by the inference scripts as the checkpoint file path and is loaded with `torch.load`.

## 🚂 Training

Both training scripts fine-tune for one epoch on LLaVA-665K-style JSON data. Update the dataset path near the top of the selected script before launching. Qwen2-VL trains LoRA on `q_proj` and `v_proj`; LLaVA-1.5 trains LoRA on the attention `q/k/v/o` projections. Memory-token embeddings are learned jointly, while decoding reuses the shared backbone without the encoder LoRA adapters.

### Qwen2-VL-2B

```bash
cd /home/ypzheng
torchrun --nproc_per_node=1 -m VisCo.train.train_qwen2 \
  --model_name_or_path /home/ypzheng/Qwen2-VL-2B-Instruct \
  --output_dir /path/to/output/qwen2_visco \
  --fixed_mem_size 18
```

### LLaVA-1.5-7B

```bash
cd /home/ypzheng
torchrun --nproc_per_node=8 -m VisCo.train.train_llava15 \
  --model_name_or_path /home/ypzheng/llava15 \
  --output_dir /path/to/output/llava_visco \
  --fixed_mem_size 32
```

Training uses the tokenization/mapping functions and data collators in `utils/preprocess.py`, then delegates optimization and checkpointing to `train_model` in the same module.

## 🧪 Evaluation

The released evaluation entry points cover MME for both backbones. They load the checkpoint passed through `--output_dir`, preprocess the benchmark with the matching backbone-specific mapping function, and report MME accuracy and accuracy+ scores.

```bash
# Qwen2-VL
python -m VisCo.inference.mme_qwen2 \
  --model_name_or_path /home/ypzheng/Qwen2-VL-2B-Instruct \
  --output_dir /path/to/qwen2_visco_checkpoint.pt \
  --fixed_mem_size 18 \
  --train false

# LLaVA-1.5
python -m VisCo.inference.mme_llava15 \
  --model_name_or_path /home/ypzheng/llava15 \
  --output_dir /path/to/llava_visco_checkpoint.pt \
  --fixed_mem_size 32 \
  --train false
```

The paper evaluates GQA, MMBench, MMBench-CN, MME, POPE, and MMVet. Additional benchmark entry points will be added as they are cleaned for release.


## 📌 Citation

If you find this project useful, please cite:

```bibtex
@inproceedings{zheng2026visco,
  title     = {VisCo: Leveraging Large Language Models as Intrinsic Encoders for Visual Token Compression},
  author    = {Zheng, Yupeng and Zou, Kai and Liu, Bin and Yu, Nenghai},
  booktitle = {Proceedings of the 34th ACM International Conference on Multimedia},
  year      = {2026}
}
```

## 🙏 Acknowledgements

This project builds on [LLaVA](https://github.com/haotian-liu/LLaVA), [Qwen2-VL](https://github.com/QwenLM/Qwen2-VL), [Hugging Face Transformers](https://github.com/huggingface/transformers).