🚀 VisCo: Leveraging Large Language Models as Intrinsic Encoders for Visual Token Compression

ACM Multimedia 2026

Yupeng Zheng^*, Kai Zou^*, Bin Liu^†, Nenghai Yu^†

^*Equal contribution    ^†Corresponding author

Compress hundreds of visual tokens into only a few memory tokens while preserving multimodal understanding.

🔥 Highlights

• Intrinsic self-compression. VisCo reuses the pretrained VLM itself as a parameter-sharing autoencoder instead of introducing a separate compression network.
• Lightweight adaptation. Only memory tokens and encoder-side LoRA adapters are trained; the shared pretrained backbone is kept intact for decoding.
• Hierarchical KV passing. Layer-wise key-value states associated with memory tokens are transferred from the encoder directly into the decoder KV cache.
• Extreme compression. VisCo retains 85.3% of LLaVA-1.5-7B performance with one visual token and 91.4% of Qwen2-VL-2B performance with 18 tokens.
• Broad evaluation. Experiments cover LLaVA-1.5-7B, Qwen2-VL-2B, and Qwen2-VL-7B on six multimodal benchmarks.

📊 Main Results

LLaVA-1.5-7B

Average is the mean percentage of performance retained relative to the original model.

Method	Tokens	GQA	MMB	MMB-CN	MME	POPE	MMVet	Average
LLaVA-1.5-7B	576	62.0	64.3	58.3	1510.7	85.9	31.1	100.0%
FastV	32	41.5	37.8	33.2	884.6	32.5	20.7	57.6%
SparseVLM	32	48.3	51.4	40.6	1046.7	67.9	18.6	72.6%
PruMerge+	32	51.1	56.8	47.0	940.8	70.9	21.4	77.5%
DivPrune	32	54.9	57.6	49.1	1284.9	81.5	26.3	87.8%
VisPruner	32	52.2	58.4	52.7	1271.0	72.7	28.8	87.8%
VisCo	32	58.5	62.3	57.2	1152.9	81.9	27.9	91.8%
PruMerge+	1	26.4	13.7	13.7	568.9	40.4	12.5	35.4%
VisPruner	1	41.8	22.4	25.8	764.4	49.0	12.0	48.8%
VisCo	1	58.2	59.0	52.4	1191.2	78.2	20.7	85.3%

Qwen2-VL

Base model	Method	Tokens	GQA	MMB	MMB-CN	MME	POPE	MMVet	Average
Qwen2-VL-2B	Origin	144	59.8	67.3	61.8	1465.8	81.5	42.5	100.0%
Qwen2-VL-2B	VisionZip	36	53.2	64.3	56.7	1372.4	75.3	35.5	91.0%
Qwen2-VL-2B	VisCo	36	61.1	64.0	58.9	1368.3	82.9	35.6	95.2%
Qwen2-VL-2B	VisionZip	18	48.5	55.2	41.6	1300.9	62.7	25.8	76.1%
Qwen2-VL-2B	VisCo	18	59.7	62.9	57.8	1358.0	82.4	28.9	91.4%
Qwen2-VL-7B	Origin	144	64.8	76.1	71.6	1664.8	82.6	56.1	100.0%
Qwen2-VL-7B	VisionZip	36	58.3	72.4	63.4	1530.0	75.3	39.1	87.7%
Qwen2-VL-7B	VisCo	36	62.6	74.0	70.0	1551.0	83.3	40.4	92.9%
Qwen2-VL-7B	VisionZip	18	55.3	66.4	58.3	1443.8	71.7	33.9	81.3%
Qwen2-VL-7B	VisCo	18	61.8	70.0	68.8	1496.5	81.7	39.2	90.4%

🛠️ Installation

git clone https://github.com/Zyvpeng/VisCo.git
cd VisCo

conda create -n visco python=3.10 -y
conda activate visco
pip install --upgrade pip
pip install torch transformers peft datasets safetensors qwen-vl-utils
pip install flash-attn --no-build-isolation

The current implementation additionally uses torch, transformers, peft, datasets, safetensors, qwen-vl-utils, and FlashAttention for LLaVA.

⚡ Quick Start

The case examples read /home/ypzheng/VLM_ICAE/data/case/data.json. Each item must provide image and question fields. Run commands from the repository parent so that VisCo is importable.

Qwen2-VL case inference

cd /home/ypzheng
conda activate mmicae2

python -m VisCo.inference.case_qwen2 \
  --model_name_or_path /home/ypzheng/Qwen2-VL-2B-Instruct \
  --output_dir /path/to/qwen2_visco_checkpoint.pt \
  --fixed_mem_size 18 \
  --train false \
  --per_device_train_batch_size 1

LLaVA-1.5 case inference

cd /home/ypzheng
conda activate mmicae2

python -m VisCo.inference.case_llava15 \
  --model_name_or_path /home/ypzheng/llava15 \
  --output_dir /path/to/llava_visco_checkpoint.pt \
  --fixed_mem_size 32 \
  --train false \
  --per_device_train_batch_size 1

--output_dir is used by the inference scripts as the checkpoint file path and is loaded with torch.load.

🚂 Training

Both training scripts fine-tune for one epoch on LLaVA-665K-style JSON data. Update the dataset path near the top of the selected script before launching. Qwen2-VL trains LoRA on q_proj and v_proj; LLaVA-1.5 trains LoRA on the attention q/k/v/o projections. Memory-token embeddings are learned jointly, while decoding reuses the shared backbone without the encoder LoRA adapters.

Qwen2-VL-2B

cd /home/ypzheng
torchrun --nproc_per_node=1 -m VisCo.train.train_qwen2 \
  --model_name_or_path /home/ypzheng/Qwen2-VL-2B-Instruct \
  --output_dir /path/to/output/qwen2_visco \
  --fixed_mem_size 18

LLaVA-1.5-7B

cd /home/ypzheng
torchrun --nproc_per_node=8 -m VisCo.train.train_llava15 \
  --model_name_or_path /home/ypzheng/llava15 \
  --output_dir /path/to/output/llava_visco \
  --fixed_mem_size 32

Training uses the tokenization/mapping functions and data collators in utils/preprocess.py, then delegates optimization and checkpointing to train_model in the same module.

🧪 Evaluation

The released evaluation entry points cover MME for both backbones. They load the checkpoint passed through --output_dir, preprocess the benchmark with the matching backbone-specific mapping function, and report MME accuracy and accuracy+ scores.

# Qwen2-VL
python -m VisCo.inference.mme_qwen2 \
  --model_name_or_path /home/ypzheng/Qwen2-VL-2B-Instruct \
  --output_dir /path/to/qwen2_visco_checkpoint.pt \
  --fixed_mem_size 18 \
  --train false

# LLaVA-1.5
python -m VisCo.inference.mme_llava15 \
  --model_name_or_path /home/ypzheng/llava15 \
  --output_dir /path/to/llava_visco_checkpoint.pt \
  --fixed_mem_size 32 \
  --train false

The paper evaluates GQA, MMBench, MMBench-CN, MME, POPE, and MMVet. Additional benchmark entry points will be added as they are cleaned for release.

📌 Citation

If you find this project useful, please cite:

@inproceedings{zheng2026visco,
  title     = {VisCo: Leveraging Large Language Models as Intrinsic Encoders for Visual Token Compression},
  author    = {Zheng, Yupeng and Zou, Kai and Liu, Bin and Yu, Nenghai},
  booktitle = {Proceedings of the 34th ACM International Conference on Multimedia},
  year      = {2026}
}

🙏 Acknowledgements

This project builds on LLaVA, Qwen2-VL, Hugging Face Transformers.