A Region-Prompted Adapter Tuning for Visual Abductive Reasoning
Authors:
Hao Zhang,
Yeo Keat Ee,
Basura Fernando
Abstract:
Visual Abductive Reasoning is an emerging vision-language (VL) topic where the model needs to retrieve/generate a likely textual hypothesis from a visual input (image or its part) using backward reasoning based on commonsense. Unlike in conventional VL retrieval or captioning tasks, where entities of texts appear in the image, in abductive inferences, the relevant facts about inferences are not re…
▽ More
Visual Abductive Reasoning is an emerging vision-language (VL) topic where the model needs to retrieve/generate a likely textual hypothesis from a visual input (image or its part) using backward reasoning based on commonsense. Unlike in conventional VL retrieval or captioning tasks, where entities of texts appear in the image, in abductive inferences, the relevant facts about inferences are not readily apparent in the input images. Besides, these inferences are causally linked to specific regional visual cues and would change as cues change. Existing works highlight cues utilizing a specific prompt (e.g., colorful prompt). Then, a full fine-tuning of a VL foundation model is launched to tweak its function from perception to deduction. However, the colorful prompt uniformly patchify ``regional hints'' and ``global context'' at the same granularity level and may lose fine-grained visual details crucial for VAR. Meanwhile, full fine-tuning of VLF on limited data would easily be overfitted.
To tackle this, we propose a simple yet effective Region-Prompted Adapter (RPA), a hybrid parameter-efficient fine-tuning method that leverages the strengths of detailed cues and efficient training for the VAR task. RPA~consists of two novel modules: Regional Prompt Generator (RPG) and Adapter$^\textbf{+}$. The prior encodes ``regional visual hints'' and ``global contexts'' into visual prompts separately at fine and coarse-grained levels. The latter extends the vanilla adapters with a new Map Adapter, which modifies the attention map using a trainable low-dim query/key projection. Additionally, we propose a new Dual-Contrastive Loss to regress the visual feature toward features of factual description and plausible hypothesis. Experiments on the Sherlock demonstrate that RPA outperforms previous SOTAs, achieving the 1st rank on leaderboards (Comparison to Human Accuracy: RPA~31.74 vs CPT-CLIP 29.58).
△ Less
Submitted 7 January, 2024; v1 submitted 18 March, 2023;
originally announced March 2023.
