FVG-PT: Adaptive Foreground View-Guided Prompt Tuning for Vision-Language Models
Abstract
Foreground attention shifts during CLIP-based prompt tuning are addressed through an adaptive module that enhances foreground view quality and mitigates generalization degradation.
CLIP-based prompt tuning enables pretrained Vision-Language Models (VLMs) to efficiently adapt to downstream tasks. Although existing studies have made significant progress, they pay limited attention to changes in the internal attention representations of VLMs during the tuning process. In this paper, we attribute the failure modes of prompt tuning predictions to shifts in foreground attention of the visual encoder, and propose Foreground View-Guided Prompt Tuning (FVG-PT), an adaptive plug-and-play foreground attention guidance module, to alleviate the shifts. Concretely, FVG-PT introduces a learnable Foreground Reliability Gate to automatically enhance the foreground view quality, applies a Foreground Distillation Compensation module to guide visual attention toward the foreground, and further introduces a Prior Calibration module to mitigate generalization degradation caused by excessive focus on the foreground. Experiments on multiple backbone models and datasets show the effectiveness and compatibility of FVG-PT. Codes are available at: https://github.com/JREion/FVG-PT
Community
For CLIP-based prompt tuning, incorporating external knowledge to enhance model capability has become a common strategy. However, existing methods typically rely on LLMs to introduce knowledge from the text modality, while paying limited attention to the visual modality. In contrast, this paper aims to introduce visual foreground supervision to control the incorrect shift of visual attention during the prompt tuning process.
Regarding implementation, the authors also release both visual foreground datasets and general pipelines, with the expectation of inspiring subsequent research in this direction.
This is an automated message from the Librarian Bot. I found the following papers similar to this paper.
The following papers were recommended by the Semantic Scholar API
- MMLoP: Multi-Modal Low-Rank Prompting for Efficient Vision-Language Adaptation (2026)
- PAND: Prompt-Aware Neighborhood Distillation for Lightweight Fine-Grained Visual Classification (2026)
- DeAR: Fine-Grained VLM Adaptation by Decomposing Attention Head Roles (2026)
- Seeing Clearly, Reasoning Confidently: Plug-and-Play Remedies for Vision Language Model Blindness (2026)
- Fine-tuning Pre-trained Vision-Language Models in a Human-Annotation-Free Manner (2026)
- CLIPoint3D: Language-Grounded Few-Shot Unsupervised 3D Point Cloud Domain Adaptation (2026)
- VisualAD: Language-Free Zero-Shot Anomaly Detection via Vision Transformer (2026)
Please give a thumbs up to this comment if you found it helpful!
If you want recommendations for any Paper on Hugging Face checkout this Space
You can directly ask Librarian Bot for paper recommendations by tagging it in a comment: @librarian-bot recommend
Models citing this paper 1
Datasets citing this paper 1
Spaces citing this paper 0
No Space linking this paper
Collections including this paper 0
No Collection including this paper