SparseVLM: Visual Token Sparsification for Efficient Vision-Language Model Inference

Yuan Zhang^1,3* , Chun-Kai Fan^1*, Junpeng Ma^2*, Wenzhao Zheng^3✉️, Tao Huang⁴, Kuan Cheng¹,

Denis Gudovskiy⁵, Tomoyuki Okuno⁵, Yohei Nakata⁵, Kurt Keutzer³, Shanghang Zhang^1✉️

¹School of Computer Science, Peking University, ²Fudan University,

³UC Berkeley, ⁴The University of Sydney, ⁵Panasonic Holdings Corporation

📜 News

🔥 [2025/03/06] We released SparseVLM v1.5! Higher Accuracy, Flexible Pruning Manner, and Compatibility with FlashAttention 2!

🔥 [2024/10/15] We released SparseVLM and its Project Page! The Code is now open-source!

✒️ Contents

• News
• Contents
• Overview
• Preparation
• Usage
• Citation
• Acknowledgment

👀 Overview

In vision-language models (VLMs), visual tokens usually consume a significant amount of computational overhead, despite their sparser information density compared to text tokens. To address this, existing methods extract more compact image representations by modifying the image encoder or projector. While some recent works further sparsify vision tokens during the decoding, they still ignore the guidance from the language tokens, which contradicts the multimodality paradigm. We argue that visual tokens should be sparsified adaptively based on the question prompt, as the model might focus on different parts (e.g., foreground or background) when dealing with various questions, as shown in Figure below. Unlike previous methods with text-agnostic visual sparsification (c) e.g., recent FastV, our SparseVLM (b) is guided by question prompts to select relevant visual patches.

👨‍💻 Preparation

1. Clone this repository and navigate to SparseVLMs folder

git clone https://github.com/Gumpest/SparseVLMs.git
cd SparseVLMs

2. Install necessary package

conda create -n SparseVLMs python=3.10 -y
conda activate SparseVLMs
pip install -e .
pip install transformers==4.37.0
pip install flash_attn==2.3.3

3. Download Multimodal Benchmark

Please follow the detailed instruction in LLaVA-Evaluation.

🎯 Usage

Specifically, --retained_tokens in script indicates the number of tokens to be retained after the SparseVLM algorithm. It supports three numbers of tokens, including 192, 128, and 64. If a specific number of tokens is required, please make modifications in ./llava/model/language_model/score.py

1. Example for evaluating MME results (default 192 tokens):

CUDA_VISIBLE_DEVICES=0 bash scripts/v1_5/eval/mme.sh

2. Example for evaluating POPE results (default 192 tokens):

CUDA_VISIBLE_DEVICES=0 bash scripts/v1_5/eval/pope.sh

3. Example for evaluating ScienceQA results (default 192 tokens):

CUDA_VISIBLE_DEVICES=0 bash scripts/v1_5/eval/sqa.sh

4. Example for evaluating TextVQA results (default 192 tokens):

CUDA_VISIBLE_DEVICES=0 bash scripts/v1_5/eval/textvqa.sh

5. Example for evaluating MMBench results (default 192 tokens):

CUDA_VISIBLE_DEVICES=0 bash scripts/v1_5/eval/mmbench.sh

License

This project is released under the Apache 2.0 license.

Citation

If you use SparseVLM in your research, please cite our work by using the following BibTeX entry:

@article{zhang2024sparsevlm,
  title={SparseVLM: Visual Token Sparsification for Efficient Vision-Language Model Inference},
  author={Zhang, Yuan and Fan, Chun-Kai and Ma, Junpeng and Zheng, Wenzhao and Huang, Tao and Cheng, Kuan and Gudovskiy, Denis and Okuno, Tomoyuki and Nakata, Yohei and Keutzer, Kurt and others},
  journal={arXiv preprint arXiv:2410.04417},
  year={2024}
}

Acknowledgment

We extend our gratitude to the open-source efforts of TCFormer, LLaVA, MiniGemini and VideoLLaVA.