--- license: apache-2.0 pipeline_tag: image-text-to-text ---

🚀 VisCo: Leveraging Large Language Models as Intrinsic Encoders for Visual Token Compression

ACM Multimedia 2026

Yupeng Zheng^*, Kai Zou^*, Bin Liu^†, Nenghai Yu^†

^*Equal contribution ^†Corresponding author

Compress hundreds of visual tokens into only a few memory tokens while preserving multimodal understanding.

## 🔥 Highlights - **Intrinsic self-compression.** VisCo reuses the pretrained VLM itself as a parameter-sharing autoencoder instead of introducing a separate compression network. - **Lightweight adaptation.** Only memory tokens and encoder-side LoRA adapters are trained; the shared pretrained backbone is kept intact for decoding. - **Hierarchical KV passing.** Layer-wise key-value states associated with memory tokens are transferred from the encoder directly into the decoder KV cache. - **Extreme compression.** VisCo retains 85.3% of LLaVA-1.5-7B performance with one visual token and 91.4% of Qwen2-VL-2B performance with 18 tokens. - **Broad evaluation.** Experiments cover LLaVA-1.5-7B, Qwen2-VL-2B, and Qwen2-VL-7B on six multimodal benchmarks. ## 📊 Main Results ### LLaVA-1.5-7B Average is the mean percentage of performance retained relative to the original model. | Method | Tokens | GQA | MMB | MMB-CN | MME | POPE | MMVet | Average | |:--|--:|--:|--:|--:|--:|--:|--:|--:| | LLaVA-1.5-7B | 576 | 62.0 | 64.3 | 58.3 | 1510.7 | 85.9 | 31.1 | 100.0% | | FastV | 32 | 41.5 | 37.8 | 33.2 | 884.6 | 32.5 | 20.7 | 57.6% | | SparseVLM | 32 | 48.3 | 51.4 | 40.6 | 1046.7 | 67.9 | 18.6 | 72.6% | | PruMerge+ | 32 | 51.1 | 56.8 | 47.0 | 940.8 | 70.9 | 21.4 | 77.5% | | DivPrune | 32 | 54.9 | 57.6 | 49.1 | **1284.9** | 81.5 | 26.3 | 87.8% | | VisPruner | 32 | 52.2 | 58.4 | 52.7 | 1271.0 | 72.7 | **28.8** | 87.8% | | **VisCo** | **32** | **58.5** | **62.3** | **57.2** | 1152.9 | **81.9** | 27.9 | **91.8%** | | PruMerge+ | 1 | 26.4 | 13.7 | 13.7 | 568.9 | 40.4 | 12.5 | 35.4% | | VisPruner | 1 | 41.8 | 22.4 | 25.8 | 764.4 | 49.0 | 12.0 | 48.8% | | **VisCo** | **1** | **58.2** | **59.0** | **52.4** | **1191.2** | **78.2** | **20.7** | **85.3%** | ### Qwen2-VL | Base model | Method | Tokens | GQA | MMB | MMB-CN | MME | POPE | MMVet | Average | |:--|:--|--:|--:|--:|--:|--:|--:|--:|--:| | Qwen2-VL-2B | Origin | 144 | 59.8 | 67.3 | 61.8 | 1465.8 | 81.5 | 42.5 | 100.0% | | Qwen2-VL-2B | VisionZip | 36 | 53.2 | 64.3 | 56.7 | 1372.4 | 75.3 | 35.5 | 91.0% | | Qwen2-VL-2B | **VisCo** | **36** | **61.1** | 64.0 | **58.9** | 1368.3 | **82.9** | **35.6** | **95.2%** | | Qwen2-VL-2B | VisionZip | 18 | 48.5 | 55.2 | 41.6 | 1300.9 | 62.7 | 25.8 | 76.1% | | Qwen2-VL-2B | **VisCo** | **18** | **59.7** | **62.9** | **57.8** | **1358.0** | **82.4** | **28.9** | **91.4%** | | Qwen2-VL-7B | Origin | 144 | 64.8 | 76.1 | 71.6 | 1664.8 | 82.6 | 56.1 | 100.0% | | Qwen2-VL-7B | VisionZip | 36 | 58.3 | 72.4 | 63.4 | 1530.0 | 75.3 | 39.1 | 87.7% | | Qwen2-VL-7B | **VisCo** | **36** | **62.6** | **74.0** | **70.0** | **1551.0** | **83.3** | **40.4** | **92.9%** | | Qwen2-VL-7B | VisionZip | 18 | 55.3 | 66.4 | 58.3 | 1443.8 | 71.7 | 33.9 | 81.3% | | Qwen2-VL-7B | **VisCo** | **18** | **61.8** | **70.0** | **68.8** | **1496.5** | **81.7** | **39.2** | **90.4%** | ## 🛠️ Installation ```bash git clone https://github.com/Zyvpeng/VisCo.git cd VisCo conda create -n visco python=3.10 -y conda activate visco pip install --upgrade pip pip install torch transformers peft datasets safetensors qwen-vl-utils pip install flash-attn --no-build-isolation ``` The current implementation additionally uses `torch`, `transformers`, `peft`, `datasets`, `safetensors`, `qwen-vl-utils`, and FlashAttention for LLaVA. ## ⚡ Quick Start The case examples read `/home/ypzheng/VLM_ICAE/data/case/data.json`. Each item must provide `image` and `question` fields. Run commands from the repository parent so that `VisCo` is importable. ### Qwen2-VL case inference ```bash cd /home/ypzheng conda activate mmicae2 python -m VisCo.inference.case_qwen2 \ --model_name_or_path /home/ypzheng/Qwen2-VL-2B-Instruct \ --output_dir /path/to/qwen2_visco_checkpoint.pt \ --fixed_mem_size 18 \ --train false \ --per_device_train_batch_size 1 ``` ### LLaVA-1.5 case inference ```bash cd /home/ypzheng conda activate mmicae2 python -m VisCo.inference.case_llava15 \ --model_name_or_path /home/ypzheng/llava15 \ --output_dir /path/to/llava_visco_checkpoint.pt \ --fixed_mem_size 32 \ --train false \ --per_device_train_batch_size 1 ``` `--output_dir` is used by the inference scripts as the checkpoint file path and is loaded with `torch.load`. ## 🚂 Training Both training scripts fine-tune for one epoch on LLaVA-665K-style JSON data. Update the dataset path near the top of the selected script before launching. Qwen2-VL trains LoRA on `q_proj` and `v_proj`; LLaVA-1.5 trains LoRA on the attention `q/k/v/o` projections. Memory-token embeddings are learned jointly, while decoding reuses the shared backbone without the encoder LoRA adapters. ### Qwen2-VL-2B ```bash cd /home/ypzheng torchrun --nproc_per_node=1 -m VisCo.train.train_qwen2 \ --model_name_or_path /home/ypzheng/Qwen2-VL-2B-Instruct \ --output_dir /path/to/output/qwen2_visco \ --fixed_mem_size 18 ``` ### LLaVA-1.5-7B ```bash cd /home/ypzheng torchrun --nproc_per_node=8 -m VisCo.train.train_llava15 \ --model_name_or_path /home/ypzheng/llava15 \ --output_dir /path/to/output/llava_visco \ --fixed_mem_size 32 ``` Training uses the tokenization/mapping functions and data collators in `utils/preprocess.py`, then delegates optimization and checkpointing to `train_model` in the same module. ## 🧪 Evaluation The released evaluation entry points cover MME for both backbones. They load the checkpoint passed through `--output_dir`, preprocess the benchmark with the matching backbone-specific mapping function, and report MME accuracy and accuracy+ scores. ```bash # Qwen2-VL python -m VisCo.inference.mme_qwen2 \ --model_name_or_path /home/ypzheng/Qwen2-VL-2B-Instruct \ --output_dir /path/to/qwen2_visco_checkpoint.pt \ --fixed_mem_size 18 \ --train false # LLaVA-1.5 python -m VisCo.inference.mme_llava15 \ --model_name_or_path /home/ypzheng/llava15 \ --output_dir /path/to/llava_visco_checkpoint.pt \ --fixed_mem_size 32 \ --train false ``` The paper evaluates GQA, MMBench, MMBench-CN, MME, POPE, and MMVet. Additional benchmark entry points will be added as they are cleaned for release. ## 📌 Citation If you find this project useful, please cite: ```bibtex @inproceedings{zheng2026visco, title = {VisCo: Leveraging Large Language Models as Intrinsic Encoders for Visual Token Compression}, author = {Zheng, Yupeng and Zou, Kai and Liu, Bin and Yu, Nenghai}, booktitle = {Proceedings of the 34th ACM International Conference on Multimedia}, year = {2026} } ``` ## 🙏 Acknowledgements This project builds on [LLaVA](https://github.com/haotian-liu/LLaVA), [Qwen2-VL](https://github.com/QwenLM/Qwen2-VL), [Hugging Face Transformers](https://github.com/huggingface/transformers).