STLLaVA-Med: Self-Training Large Language and Vision Assistant for Medical

Abstract

Large Vision-Language Models (LVLMs) have shown significant potential in assisting medical diagnosis by leveraging extensive biomedical datasets. However, the advancement of medical image understanding and reasoning critically depends on building high-quality visual instruction data, which is costly and labor-intensive to obtain, particularly in the medical domain. To mitigate this data-starving issue, we introduce Self-Training Large Language and Vision Assistant for Medical (STLLaVA-Med). The proposed method is designed to train a policy model (an LVLM) capable of auto-generating medical visual instruction data to improve data efficiency, guided through Direct Preference Optimization (DPO). Specifically, a more powerful and larger LVLM (e.g., GPT-4o) is involved as a biomedical expert to oversee the DPO fine-tuning process on the auto-generated data, encouraging the policy model to align efficiently with human preferences. We validate the efficacy and data efficiency of STLLaVA-Med across three major medical Visual Question Answering (VQA) benchmarks, demonstrating competitive zero-shot performance with the utilization of only 9% of the medical data. Our implementation is available at https://github.com/heliossun/STLLaVA-Med.

1 Introduction

Large Vision-Language Models (LVLMs) have demonstrated impressive performance across a wide range of medical challenges Li et al. (2023); Moor et al. (2023); Hu et al. (2024) by fine-tuning through biomedical visual instruction data. Similar to general LVLMs Liu et al. (2023a); Chen et al. (2023), existing methods tailored for biomedical tasks primarily focus on collecting high-quality medical data to enhance task generalization and visual understanding. However, collecting medical data necessitates specialized expertise from physicians and raises privacy concerns, making the process both time-consuming and costly. To address this data-starving issue, recent studies Li et al. (2023) have explored leveraging larger models/APIs (e.g., GPT-4 Achiam et al. (2023)) to generate medical data. Nevertheless, this kind of method does not fully resolve the high API costs Deng et al. (2024) associated with building instructional data and still requires large-scale pre-training data to align medical images and text (see Fig. 1 left).

Figure 1: Left: Comparison of total medical data usage between LLaVA-Med (530K) and STLLaVA-Med (50k). Right: Comparison results on three medical VQA datasets. STLLaVA-Med reports better/comparable performance, using much less medical training data.

To bridge the gap in medical data acquisition, we propose Self-Training Large Language and Vision Assistant for Medical (STLLaVA-Med), a new training pipeline that enables LVLMs to automatically generate medical instruction data governed by Direct Preference Optimization (DPO) Rafailov et al. (2023). Different from previous self-training approaches Wang et al. (2023); Deng et al. (2024), which generate answers for fixed/pre-defined questions (e.g., summarization and report), this work automatically generates open-ended questions and answers them, to enhance the diversity of self-training data and further improve medical image reasoning.

Moreover, achieving precise control of the generated model response is also challenging due to its unsupervised nature Rafailov et al. (2023); Zhao et al. (2023); Azar et al. (2023); Mehta et al. (2023). Existing methods for gaining such steerability, such as reinforcement learning Ouyang et al. (2022) and DPO Rafailov et al. (2023) from human feedback, mainly rely on collecting human labels to evaluate the relative quality of model generations and fine-tune the unsupervised LVLM to align with human preferences, which still burdens data collection in biomedical domains. To this end, the proposed STLLaVA-Med implements DPO by leveraging a larger LVLM with better general medical knowledge to supervise the policy model.

Overall, the proposed STLLaVA-Med realizes self-training in two stages – 1) reasoning and learning to question and 2) preference revelation. In Stage 1, to enhance the model’s reasoning and questioning skills, we incorporate questions within the visual instructional data as an additional learning objective following Sun et al. (2024). After the first-stage training, STLLaVA-Med is able to automatically generate question-answer pairs. In Stage 2, we leverage GPT-4o OpenAI (2024) as a medical expert to further supervise fine-tuning STLLaVA-Med through DPO, ensuring it adheres to our designed preferences (e.g., detail, relevance, and accuracy) on the auto-generated data. We summarize the contributions of this work as follows:

• We propose a novel self-training approach for LVLMs that enhances medical reasoning skills with less medical data. Our approach improves the data efficiency of training LVLMs for specific domains.

• The proposed STLLaVA-Med enables the automatic construction of medical instructional data, supervised by a stronger and heavy LVLM (i.e., GPT-4o) and governed through DPO, which allows our LVLM to adhere to preferences in a self-training way.

• Experiments on three major medical VQA benchmarks demonstrate that our method achieves highly competitive zero-shot performance compared to existing methods yet utilizing only 9% of the medical data.

2 STLLaVA-Med

In this section, we introduce STLLaVA-Med (see Fig. 2) given by our proposed two-stage self-training algorithm, which is designed to enhance the data efficiency when training an LVLM for medical tasks. Specifically, we optimize the LVLM – a policy model – in two stages sequentially. The policy model $\pi_{\theta}$ parameterized by $\theta$ first learns to automatically generate question-answer pairs for self-training, then utilizes DPO to precisely control the prediction behavior. Please refer to Appendix B.1 for model architecture details.

Stage1: Reasoning and learning to question.
The main part of self-training is automatic question generation and answering. Specifically, we follow Sun et al. (2024) by adding a special token <vusr> and set the question-tokens to learnable, to jointly fine-tune $\pi_{\theta}$ for reasoning and questioning on visual instructional data $\mathcal{D}_{ft}$.

Figure 2: Model architecture of STLLaVA-Med and self-training pipeline. Left: stage 1 aiming to optimize the model $\pi_{\theta}$ improving medical image reasoning and learning to question. Right: in stage 2, we first prompt $\pi_{\theta}$ to auto-generate preference data under the guidance of GPT-4o, then supervise $\pi_{\theta}$ for DPO fine-tuning.

Given the visual instruction data $\mathcal{D}_{ft}$ = $\{(X_{v},X_{c})\}$, where the conversation $X_{c}=\{X_{q}^{(j)},X_{a}^{(j)}\}^{M}_{j=1}$ consists of $M$ QA pairs, the text $X_{c}$, and the image $X_{v}$ are encoded to sequential embeddings as $H_{c}=\{H_{q}^{(j)},H_{a}^{(j)}\}^{M}_{j=1}$ and $H_{v}$ by word embedding and vision encoder. We minimize the negative log-likelihood loss for the vq: visual questioning and qa: answering as the following:

$$\mathcal{L}_{vq}=\sum_{v,c\in\mathcal{D}_{ft}}-\mathrm{log}\pi_{\theta}(H_{q}^{(j+1)}\mid H_{v},H_{c}^{(1:j)}),$$

$${\mathcal{L}_{qa}=\sum_{v,c\in\mathcal{D}_{ft}}-\mathrm{log}\pi_{\theta}(H_{a}^{(j+1)}\mid H_{v},H_{c}^{(1:j)},H_{q}^{(j+1)})},$$

where $j\in\left\{1,\cdots,M\right\}$ indicates the index of question or answer within the conversational data $H_{c}=\{H_{q}^{(j)},H_{a}^{(j)}\}^{M}_{j=1}$. To mitigate the heavy computational overhead, we fine-tune LoRA Hu et al. (2022) in both the vision encoder and LLM. Thus, the learnable parameters $\theta$ of the policy model $\pi_{\theta}$ during fine-tuning represent a combination of all the parameters of LLM-LoRA, ViT-LoRA, and the vision-to-language projector. In addition, we skip the vision-language alignment on medical image-text pairs for data efficiency by loading the pre-trained weights Sun et al. (2024) on general-purpose data. Eventually, our model can raise and answer questions for a medical image, enabling automatic QA generation in the following stage.

Stage2: Preference revelation.
We apply DPO Rafailov et al. (2023) to fine-tune the unsupervised LVLM to align with pre-defined preferences. Unlike previous works Rafailov et al. (2023); Zhao et al. (2023), we employ $\pi_{\theta}$ to automatically generate a preference dataset $D_{pref}=\{(X_{v},X_{q},X_{a_{w}},X_{a_{l}})\}$. Specifically, we prompt $\pi_{\theta}$ to generate an image-related question $X_{q}$ and two different answers $X_{a}$, which are labeled as win $X_{a_{w}}$ and loss $X_{a_{l}}$ answers by GPT-4o. For all the experiments, we first map $X_{q}$, $X_{a_{w}}$, $X_{a_{l}}$, and $X_{v}$ to $H_{q}$, $H_{a_{w}}$, $H_{a_{l}}$, and $H_{v}$ using the same word embedding and vision encoder in stage 1, and fine-tune $\pi_{\theta}$ through DPO by minimizing the following negative log-likelihood loss:

$$\mathcal{L}_{DPO}(\pi_{\theta};\pi_{ref})=-\mathbb{E}_{v,q,a_{w},a_{l}\in\mathcal{D}_{pref}}[\mathrm{log}\sigma(\beta\mathrm{log}$$ (1)

$$\frac{\pi_{\theta}(H_{a_{w}}\mid H_{v},H_{q})}{\pi_{ref}(H_{a_{w}}\mid H_{v},H_{q})})-\beta\mathrm{log}\frac{\pi_{\theta}(H_{a_{l}}\mid H_{v},H_{q})}{\pi_{ref}(H_{a_{l}}\mid H_{v},H_{q})})],$$

where $\beta$ is a parameter controlling the deviation from the base reference policy $\pi_{ref}$, which prevents the policy model from deviating too far from the distribution of correct generation, as well as maintaining the generation diversity and preventing mode-collapse to single high-reward answers Rafailov et al. (2023). Notably, $\pi_{\theta}$ and $\pi_{ref}$ are initialized from the same weights at beginning, and only $\pi_{\theta}$ is optimized during training. In this way, we fit an implicit reward to precisely control the model generation by the pre-defined preference such as accuracy and detail.

3 Self-training Datasets

Medical LVLMs Li et al. (2023); Zhang et al. (2023); Moor et al. (2023) generally adopt pre-training on massive medical data, to realize medical image-text alignment. However, the proposed STLLaVA-Med does not involve such a medical corpus pre-training, providing new insights into data efficiency. To fine-tune the LVLM for medical tasks, we utilize a filtered open-source medical instructional dataset Med-60k-IM Li et al. (2023) as $\mathcal{D}_{ft}$ due to image unavailability. Table 2 provides the medical data statistics for training LLaVA-Med and STLLaVA-Med. We show how to employ the policy model to auto-generate a preference dataset for DPO fine-tuning in the following process:
Auto-generated Questions. We randomly sample 10k medical images from Med-60k-IM datasets and prompt $\pi_{\theta}$ to generate questions.
GPT-4o guided preference data collection. We prompt $\pi_{\theta}$ to predict two answers to each generated question. Specifically, to ensure the difference between answers, we set the temperature scaling to 1.2, $TopK=100$, and $TopP=0.95$, encouraging the model to generate more diverse and non-repetitive output. In previous research Rafailov et al. (2023), the preference data were annotated by human annotators. In contrast, this work utilizes GPT-4o as a simulated expert since we observe its excellent biomedical performance Yue et al. (2023) and the best downstream task performance in Table 1. We prompt GPT-4o (see Appendix B.2 for prompt design) with all the information to label the answers with win or loss, treated as $X_{a_{w}}$ and $X_{a_{l}}$ within $\mathcal{D}_{pref}$.

Table 1: Comparison with other methods on three benchmarks. Open questions are evaluated by Recall and F1 score, and closed questions are evaluated by accuracy. All models are using 7B LLM. STLLaVA-Med w/o DPO is the ablated version of our final model. Notably, LLaVA-Med was trained on the original Med-60k-IM Li et al. (2023), which has 20k more samples than the Med-IM we used in this work due to image unavailability.

Dataset	Method	VQA-RAD			SLAKE			PVQA
		Recall	F1 Score	Closed	Recall	F1 Score	Closed	Recall	F1 Score	Closed
	GPT-4o OpenAI (2024)	51.60	9.23	63.97	59.06	8.90	71.63	24.14	3.29	75.97
w/o Med-IM	LLaVA-v1.5 Liu et al. (2023a)	23.63	9.53	50.74	35.23	8.84	52.16	11.85	2.73	52.76
	SQ-LLaVA Sun et al. (2024)	23.91	6.29	52.57	40.04	9.65	57.45	11.24	2.63	53.73
Med-IM	LLaVA-Med Li et al. (2023)	32.68	8.65	59.56	40.84	8.21	46.88	12.03	2.47	55.23
	STLLaVA-Med w/o DPO	33.81	10.37	59.16	40.13	10.97	55.53	10.38	2.68	52.05
	STLLaVA-Med	37.12	10.83	60.35	46.69	11.46	57.69	11.92	2.72	52.90

4 Experiments

4.1 Datasets and Metrics

Datasets. We conduct experiments on the widely-used medical VQA benchmark dataset VQA-RAD Lau et al. (2018), SLAKE Liu et al. (2021), and PVQA He et al. (2020). Specifically, VQA-RAD contains QA pairs generated by clinicians, where the images are evenly distributed over the head, chest, and abdomen. Questions are categorized into 11 categories: abnormality, attribute, modality, organ system, color, counting, etc. SLAKE is a Semantically-Labeled Knowledge-Enhanced dataset for medical VQA. The original dataset contains Chinese and English QA, but we only consider the English subset in our study. Besides, SLAKE includes richer modalities and covers more human body parts than the currently available dataset. PathVQA is a dataset of pathology images. Each image has questions about multiple aspects, such as location, shape, color, appearance, etc. Overall, the medical questions are categorized into two types: open-ended questions such as why, what, how, where, etc., and closed-ended questions with one-word answers (yes/no).

Evaluation Metrics. We report accuracy for closed questions. For open-ended questions, we compute recall as the proportion of correctly predicted words out of the reference sentence, and F1 score as a balance metric between recall and precision.

Table 2: Statistics of medical training data.

Method	#Images	#QA-Pairs
LLaVA-Medpt	467710	467710
LLaVA-Medft	56708	164231
ours	37452	108545

4.2 Overall Performance

The evaluation results in Table 1 are divided into three sections based on rows. We compile all the experiments locally with a single run. The first row indicates the upper bound of zero-shot medical performance; the next two rows and the last three rows reflect the performance of LVLMs trained without and with medical data. As shown in Table 1, even without pre-training on medical data, STLLaVA-Med achieves competitive performance with LLaVA-Med only after fine-tuned instructional data Med-IM. After DPO training, we observed a performance improvement over open-ended questions, demonstrating the effectiveness of supervised preference optimization. Also, in Fig. 3, the answer of STLLaVA-Med has higher relevance, more detail, and accuracy than STLLaVA-Med w/o DPO, proving that self-training on the auto-generated preference dataset controls the model following pre-defined preference. See Appendix C.2 for more results. Moreover, we have several observations based on the results in Table 1.

1) Medical image-text alignment is unnecessary for LVLM-Med. As we observed, fine-tuning a pre-trained general-purpose LVLM (e.g., LLavA-V1.5, SQ-LLaVA) on a small set of medical instruction data achieves the same performance as fully training on medical data (LLaVA-Med). This suggests that general-purpose LVLM with strong vision-language alignment can be easily adapted to medical tasks after light-weight fine-tuning.

2) High-quality medical instruction data can further improve STLLaVA-Med. We believe training the model on better instructional data can enrich the auto-generated data’s diversity, complexity, and professionalism.

Figure 3: Qualitative evaluation of methods w and w/o preference revelation.

5 Conclusion

In this work, we have proposed a self-training vision-language assistant for medical (STLLaVA-Med), a novel approach designed to enhance the data efficiency of training LVLMs for medical tasks. The proposed two-stage self-training pipeline enables STLLaVA-Med to automatically construct data. By leveraging the supervision of GPT-4o, we have optimized the model to align with pre-defined preferences at a reduced cost and with minimal human effort, addressing the challenges associated with medical data acquisition. Experimental results on three benchmarks demonstrate that STLLaVA-Med achieves exceptional medical reasoning capabilities using medical data at a minimum level. We aspire for our work to inspire future research aimed at enhancing the efficiency of training LVLMs in broad medical domains.

6 Limitations

Although the proposed approach improves medical reasoning ability, the performance of self-training is highly dependent on the quality and relevance of the auto-generated medical instructional data. It indicates that we still need the instructional data in self-training stage1 covers wider range of medical tasks and professional expertise, which may still be difficult to collect for some diseases or some types of medical images.

7 Ethics Statements

We conduct experiments and analysis on public datasets, PMC-15M, VQA-RAD, SLAKE, and PVQA, where all medical images and texts were de-identified, ensuring the privacy and confidentiality of patients. While our method reduces the need for extensive labeled datasets, its outputs are still machine-generated, requiring critical human oversight, particularly when used in clinical decision-making.

Appendix A Related Work

A.1 Large Vision-Language Model

As the field of Large Language Models (LLMs) and instruction tuning undergoes rapid advancements, the academic research community is increasingly focusing on integrating visual information into LLM frameworks to enhance visual instruction tuning. This emerging research area has seen the development of various methodologies, building on the foundational work of CLIP Radford et al. [2021] and diverse LLM architectures such as Vicuna Zheng et al. [2023], Llama2 Touvron et al. [2023], and Qwen-VL Bai et al. [2023]. Notably, LLaVA Liu et al. [2023b] pioneered the integration of an LLM with a CLIP vision encoder to create a vision-language model, demonstrating significant capabilities in image-text dialogue tasks through strategies of pre-training alignment and targeted instruction tuning. Subsequent research has focused on refining visual instruction tuning by improving the quality and diversity of datasets used during the pre-training and fine-tuning phases. Building upon these advancements, recent studies, including LLaVA-v1.5 Liu et al. [2023a] and ShareGPT4V Chen et al. [2023], have achieved notable success in general vision-language comprehension, showcasing their ability to handle complex question-answering tasks. This progression underscores the importance of sophisticated data handling and model-tuning strategies in develo** effective vision-language models.

Figure 4: Qualitative evaluation of methods w and w/o preference revelation.

Figure 5: Prompt for GPT-4o to grade the preference of two answers generated by STLLaVA-Med from stage 1. The answer with the higher score will be designated as the winning response, while the other will be classified as rejected.

Table 3: Comparison of fine-tuning performance on three benchmarks. Open questions are evaluated by Recall and F1 score, and closed questions are evaluated by accuracy. All models are using 7B LLM. STLLaVA-Med w/o DPO is the ablated version of our final model.

Dataset	Method	VQA-RAD			SLAKE			PVQA
		Recall	F1 Score	Closed	Recall	F1 Score	Closed	Recall	F1 Score	Closed
Med-IM+ VQA-RAD+ SLAKE+PVQA	LLaVA-v1.5 Liu et al. [2023a]	43.44	36.41	70.59	52.76	46.98	64.18	35.91	35.47	91.15
	STLLaVA-Med w/o DPO	52.07	45.38	75.74	56.10	50.77	67.31	38.05	37.76	92.13
	STLLaVA-Med	52.60	45.92	76.10	57.37	50.84	67.31	38.30	38.00	92.13

Alignment fine-tuning. Following supervised fine-tuning (SFT), alignment fine-tuning has emerged as a key method to further enhance the performance of Large Language Models (LLMs) by aligning them with human preferences Ouyang et al. [2022]. Initial approaches utilized on-policy reinforcement learning (RL) methods, such as proximal policy optimization (PPO) Schulman et al. [2017], to train a reward model based on preference data Bai et al. [2022]. The introduction of direct policy optimization (DPO) Rafailov et al. [2023], Dubois et al. [2023], Azar et al. [2023], Mehta et al. [2023] has marked a significant shift towards direct learning from human preferences, bypassing the need for an explicit reward model. Another effective strategy is iterative preference fine-tuning, which repeatedly optimizes the model on newly generated preference pairs in successive iterations, thereby improving performance. Despite extensive research on alignment fine-tuning for LLMs, the application of these techniques to Large Vision-Language Models (LVLMs) has been comparatively limited. Early attempts Zhou et al. [2024], Deng et al. [2024] have focused on constructing preference datasets using human-labeled data or GPT-4 generations, followed by fine-tuning with a DPO loss.

Appendix B Approach

B.1 Model Architecture

The proposed STLLaVA-Med model consists of three main components: 1) A pre-trained vision encoder CLIP-ViT Radford et al. [2021] that extracts a sequence embedding of image tokens for an input image; 2) A trainable projection block with two linear layers to map the enhanced image tokens to the language domain tokens, handling the dimension misalignment between the vision and language domain; and 3) Our LLM backbone implemented by the pre-trained Vicuna Zheng et al. [2023] to predict the next token upon the previous embedding sequence.

B.2 Preference Data Generation

In previous research Rafailov et al. [2023], preference data were annotated by human annotators. In contrast, this work employs GPT-4o as a simulated expert to classify the answers generated by STLLaVA-Med. As shown in Fig. 5, we provide the detailed prompt design for GPT-4o to label the answers with either win or loss. Due to its multi-modal understanding capabilities, GPT-4o can directly take images as input. In Fig. 6, we provide qualitative results about the preference data generated by STLLaVA-Med and guided by GPT-4o.

Table 4: Medical data statistics of training.

Method	#Images	#QA-Pairs
Med-IM	56708	164231
VQA-RAD	313	3064
SLAKE	546	11934
PVQA	37452	26034

Figure 6: Preference data visualization. The win and loss answer were classified by GPT-4o.

Appendix C Experiments

C.1 Implementation Details

Our proposed self-training pipeline involves two stages. In stage 1, we fine-tune the policy model $\pi_{\theta}$ on instructional data with global batch size as $128$. During training, we insert LoRAHu et al. [2022] with $rank=128$ and $\alpha=256$ into the language model (LLM-LoRA) and LoRA with $rank=32$ and $\alpha=64$ into the vision encoder (ViT-LoRA). We optimize the model using AdamW Loshchilov and Hutter [2019] optimizer for one epoch by setting the learning rate to $2\times 10^{-4}$ for LoRA, and $5\times 10^{-5}$ for the other layers.
In stage 2, we fine-tune $\pi_{\theta}$ on the auto-generated preference dataset. Similar to stage 1, we utilize LoRA for light-weight training. We optimize the model using AdamW Loshchilov and Hutter [2019] optimizer for one epoch by setting the learning rate to $5\times 10^{-6}$ for LoRA, and $2\times 10^{-5}$ for the other layers. We train the model on 4 A100s for 10 hours.

C.2 Additional Results

In addition to evaluating zero-shot performance, we conducted experiments involving fine-tuning the model on downstream tasks. To maintain task generalizability and domain specificity, we compiled a new medical instructional dataset, combining Med-IM Li et al. [2023] with QA pairs from the training sets of VQA-RAD, SLAKE, and PVQA. Table 4 details the number of medical images and QA pairs within each dataset. After fine-tuning the models on this visual instruction dataset, we observed a clear improvement in downstream task performance, as shown in Table 3. The performance gap between the baseline model LLaVA-v1.5 and the proposed STLLaVA-Med demonstrates the effectiveness of the self-training pipeline. Additionally, the improvement between STLLaVA-Med without DPO and STLLaVA-Med illustrates the effectiveness of preference alignment within the self-training pipeline. However, we found this improvement is not as significant as the improvement over zero-shot scenario. One explanation is the inconsistency between our designed preference and the ground truth preference. For VQA-RAD, SLAKE, and PVQA, the ground truth are short phrases, but the preference we are trying to optimize is detailed and relevance. This gives us an insight that the human expert should be involved in future medical tasks evaluation.

In Fig. 4, we provide more qualitative results of medical VQA. As can be seen, STLLaVA-Med follows human preference by generating more detailed and accurate answer than the model without DPO fine-tuning.