README.md

---
library_name: transformers
pipeline_tag: text-to-image
license: other
---

# InstanceControl: Sa2va-Instance-4B (Stage 1)

This repository contains the `Sa2va-Instance-4B` checkpoint, which serves as **Stage 1** (Instance Parsing Model) for **InstanceControl**, presented in the paper [InstanceControl: Controllable Complex Image Generation without Instance Labeling](https://huggingface.co/papers/2606.31924).

* **Project Page:** [InstanceControl Homepage](https://instancecontrol.github.io/InstanceControl/)
* **GitHub Repository:** [InstanceControl GitHub](https://github.com/liuxiaoyu1104/InstanceControl)
* **Paper:** [arXiv:2606.31924](https://huggingface.co/papers/2606.31924)

## Model Description

InstanceControl is a multi-instance controllable generation method that eliminates the need for manual instance labeling. It uses a Vision-Language Model (VLM)—specifically this `Sa2va-Instance-4B` model—to automatically parse instance descriptions from text prompts and predict instance masks based on visual conditions (such as Canny edges, depth, or HED).

## Usage

For detailed instructions on setup, environment installation, and running the inference pipeline, please refer to the [official GitHub repository](https://github.com/liuxiaoyu1104/InstanceControl).

### Predict Instance Masks (Stage 1)

You can run the model to predict instance masks using the following command:

```bash
python stage1_Sa2VA/projects/llava_sam2/evaluation/gcg_eval_our_folders.py \
  --model_path /path/to/Sa2va-Instance-4B \
  --image_dir ./example/canny \
  --json_dir ./example/json \
  --save_dir ./results/json_pred_canny
```

## Citation

If you find this project useful, please cite the authors' work:

```bibtex
@article{liu2026instancecontrol,
  title={InstanceControl: Controllable Complex Image Generation without Instance Labeling},
  author={Xiaoyu Liu and Huan Wang and Fan Li and Zhixin Wang and Jiaqi Xu and Ming Liu and Wangmeng Zuo},
  journal={arXiv preprint arXiv:2606.31924},
  year={2026}
}
```