MIA-Bench: Towards Better Instruction Following Evaluation of Multimodal LLMs

Abstract

We introduce MIA-Bench, a new benchmark designed to evaluate multimodal large language models (MLLMs) on their ability to strictly adhere to complex instructions. Our benchmark comprises a diverse set of 400 image-prompt pairs, each crafted to challenge the models’ compliance with layered instructions in generating accurate responses that satisfy specific requested patterns. Evaluation results from a wide array of state-of-the-art MLLMs reveal significant variations in performance, highlighting areas for improvement in instruction fidelity. Additionally, we create extra training data and explore supervised fine-tuning to enhance the models’ ability to strictly follow instructions without compromising performance on other tasks. We hope this benchmark not only serves as a tool for measuring MLLM adherence to instructions, but also guides future developments in MLLM training methods.

Figure 1: An example from MIA-Bench, featuring an image and a complex instruction to test models’ compliance with layered instructions that are compositional in nature. Responses from GPT-4v [1] and InternVL-v1.5 [2] are evaluated using GPT-4o as the judge.

1 Introduction

Figure 2: Comparison of various multimodal LLM benchmarks. (Left) Fixed-form visual question answering, often features short answers or multi-choice formats, such as MMMU [3]. This format is popular due to its ease of evaluation. (Middle) Open-ended responses, such as LLaVA-Bench (in the Wild) [4]. (Right) The proposed MIA-Bench, which also uses open-ended responses but focuses on evaluating precise adherence to complex instructions within the prompt.

The rapid advancement of Multimodal Large Language Models (MLLMs) [1, 4, 5, 6, 7, 8, 9, 10, 11] has been a defining feature of recent AI research, showcasing increased model capabilities to comprehend and respond to visual inputs, often termed as multimodal “instruction following”.

To measure the progress of instruction following, many multimodal benchmarks have been developed, which can be roughly divided into two broad categories: ($i$) fixed-form visual question answering (VQA), often with short answers or using a multi-choice QA format; and ($ii$) free-form conversations with open-ended responses. Many current benchmarks have adopted the first format, including VQAv2 [12], TextVQA [13], ScienceQA [14], MME [15], MMBench [16], SEED-Bench [17], MathVista [18], and MMMU [3]. These benchmarks are popular due to their ease of use in evaluating metrics and presenting model comparisons.

However, as visual assistant models, the ability to engage users in free-form conversations is also crucial. Benchmarks in this format include LLaVA-Bench [4], MM-Vet [19], VisIT-Bench [20], InfiMM-Eval [21], and the most recent Vibe-Eval [22] and LLaVA-Bench-Wilder [23]. Typically, the free-form model responses are evaluated using external models as the judge. These benchmarks are closer to daily-life visual chat scenarios; however, the type of “instruction following” examined in these benchmarks usually gauges a model’s ability to perform tasks in a broad, often loosely defined manner. Yet, the precise adherence to complex instructions within prompts – a critical aspect for evaluating LLMs [24, 25, 26] – remains less explored in the context of multimodal LLMs.

To this end, we introduce MIA-Bench,a new benchmark specifically designed for evaluating strict “instruction adherence”. Our instruction adherence metric measures the precision with which MLLMs can execute layered and compositional instructions. This involves not only recognizing the content of the instructions, but also meticulously executing the detailed demands without deviation (e.g., answering in a given number of sentences, including specific elements, etc.). By establishing this stricter criterion, our benchmark aims to push the boundaries of model precision and reliability in practical applications, ensuring that outputs not only align with the general intent of the instructions, but also match the exact specifications provided. An example from MIA-Bench is provided in Figure 1, and its comparison with previous MLLM benchmarks is illustrated in Figure 2.

MIA-Bench consists of 400 meticulously created image-prompt pairs, and encompasses diverse image contents including animals, food, landmarks, sport, art, landscape, text, etc. to cover a broad spectrum of real-world scenarios. In constructing this benchmark, we sought not only to evaluate the current capabilities of state-of-the-art MLLMs, but also to push the boundaries of what these models can achieve when rigorously tested against structured and layered instructions. The final prompts are of various complexity levels, and compositional in nature, with five base instruction categories, which are tailored to probe the models’ linguistic dexterity, grammatical accuracy, and descriptive fidelity. For example, the prompt in Figure 1 is composed of five base categories, including description, mention, grammar, length limit, and genre.

We evaluate a wide array of MLLMs on the proposed benchmark, ranging from closed-source models (e.g., GPT-4o [27], Gemini Pro [10], Claude-3 [28], Reka [29]) to open-source ones (e.g., LLaVA-NeXT [30], Intern-VL-Chat-1.5 [2], CogVLM2 [8], Phi-3-Vision [31]). Our investigations reveal notable variations in model performance, highlighting great opportunities for improvement.

To address these challenges, we further propose to generate training data tailored for supervised fine-tuning (SFT), where we aim to refine the models’ abilities to process and comply with multifaceted instructions. Results from our SFT experiments indicate a promising enhancement in the models’ performance to strictly adhere to instructions, without hurting performance on other benchmarks.

Our contributions are summarized as follows. ($i$) We construct MIA-Bench, a new benchmark to comprehensively evaluate MLLMs on their capability to strictly adhere to instructions. ($ii$) We provide a detailed analysis of popular MLLMs, and suggest training methods for enhanced instruction following. For this purpose, we created training data and conducted experiments for additional supervised fine-tuning. MIA-Bench will be open-sourced, and we hope this benchmark can serve as a useful resource to stimulate further research on multimodal instruction adherence.

2 MIA-Bench

Figure 3: Examples from MIA-Bench, with detailed information on the instruction composition, base instruction weight and type.

MIA-Bench consists of 400 image-prompt pairs, with examples shown in Figure 3. The images are collected from diverse sources, including COCO 2017 validation set [32], SBU [33], TextVQA [34], and Flickr. Images in the Flickr subset are photos of a variety of themes, including animals, art, architectures, text, food, math, etc. Images from the other three sources are randomly sampled from each corresponding source. Figure 4 shows the top 15 image content categories and the distribution of the 8 sub-instruction categories in MIA-Bench. The image content is labeled using GPT-4v. For each image, we manually write diverse and challenging instructions that contain multiple sub-instructions.

Figure 4: The top-15 image content categories and the distribution of the 8 sub-instruction categories in MIA-Bench.

When constructing the instructions, we follow three principles, detailed below.

• Correctness. The instruction needs to be answerable by humans. For example, asking about objects that do not exist in the image makes the prompt unanswerable.

• No answer leakage. The instruction should not contain the answer to itself. ‘What color is the green object?’ is an example of answer leakage.

• Image-dependent. MMStar [35] pointed out that on some multimodal benchmarks, MLLMs can generate correct answers without accessing images half of the time. Multi-modal capabilities are necessary to correctly answer MIA-Bench prompts.

2.1 Instruction Categories

In this paper, we use instruction to refer to the entire textual input, which in MIA-Bench can generally be viewed as a composition of multiple individual requests or constraints. We refer to these individual components as sub-instruction. Instructions in MIA-Bench are of diverse complexity, and sub-instructions contained are of multiple categories, summarized in Figure 4.

The sub-instructions in MIA-Bench can be categorized into description, length limit, mention, genre, grammar, math, perspective, and OCR, detailed below.

• ‘description’ refers to describing a certain part of the image, with the exception of text-rich parts of the image, which falls under the ‘OCR’;

• ‘length limit’ refers to the limitation of response length (e.g., in exactly two sentences, using exactly 60 words);

• ‘mention’ refers to mentioning or not mentioning certain objects or entities (e.g., highlighting two similarities and one difference, comparing and contrasting the condition of the buildings with the activity on the street);

• ‘genre’ refers to requests for a specific written form (e.g., write a poem, write a narrative, with at least one pun included, all while weaving in a subtle theme of change);

• ‘grammar’ refers to grammatical requirements (e.g., use present tense, use capitalized letters, use integers);

• ‘math’ refers to requirements to come up with a solution to math problems, or to identify errors in solutions to math problems, or to generate a valid math problem given table, charts, etc.;

• ‘perspective’ refers to requirements specifying the viewpoint of an object or person in the image. This requires MLLMs to correctly identify what can or cannot be seen from the specified position, and understand the spatial relationship of objects in its surrounding with itself (e.g., imagine you are the lady in the image, describe what you can see without turning your head around);

• ‘OCR’ refers to requirements related to understanding OCR information in text-rich images such as menus, tickets, bills, etc. For example, given a photo of a ticket, the sub-instruction asking about the price printed on the ticket falls into this category.

Figure 5: The most frequently used verbs and co-occurring nouns in MIA-Bench.

Figure 5 shows the most frequently used verbs and co-occurring nouns in MIA-Bench. To guarantee the diversity of prompts, when writing the instructions, we contribute instructions of various levels of complexity: basic, intermediate, advanced, creative, and complex. The basic category is the simplest; the instructions normally only contain one or two sub-instructions, such as “What is the color of the cat?”, or “Describe the sofa in two words.”. The intermediate category consists of instructions that contain three or more sub-instructions, but are in general easy for MLLMs to follow. The advanced category contains instructions that are challenging and contain three or more sub-instructions. The creative category contains instructions that instruct MLLMs to generate creative pieces of text, such as poems. The complex category is a combination of the previous two categories; the instructions in this category are the most complicated as they usually contain multiple challenging sub-instructions. While we found these categories useful to elicit a diverse instruction set, we also found that practical examples were often difficult to categorize objectively. As a result, we only used these categories for data collection, but are not reporting per-category results.

2.2 Response Evaluation Method

We adopt GPT-4o [27] to score MLLMs’ responses on each instruction and return a total score using the following prompt:

Each response is graded by first assessing how well it follows each sub-instruction, then computing the total score. Figure 6 shows an example of how responses from different MLLMs are evaluated and scored. Each sub-instruction in an instruction is assigned a maximum score ranging from 1 to 10; sum of the weight of all sub-instructions in an instruction is 10. For the example in Figure 6, there are 4 sub-instructions (denoted from S1 to S4); the first is worth 4 points and the rest is worth 2 points each. The response from GPT-4o partially follows the first sub-instruction which requires the response to be from the perspective of the dog, as the dog should not be able to see the car behind the man without turning around. The dog should be able to see the guitar, thus GPT-4o gets 2 points out of 4 for the first sub-instruction. It successfully follows the other 3 sub-instructions, achieving full score for them. Thus, the final score GPT-4o reaches is 8 out of 10. We always assign larger weight (6 if there are two sub-instructions, 4 if there are three or more sub-instructions) to the sub-instruction in the description category unless this category is absent in some cases, as usually a major part of the response is addressing this sub-instruction. For each MLLM, we compute the average score it gets on all 400 responses, and represent the ratio of the average score divided by 10 using percentage. We also compute the average score for each instruction category.

Figure 6: An example with responses from four MLLMs and their evaluation scores.

3 Experiments

In this section, we first present results of different MLLMs on MIA-Bench in Section 3.1, with additional supervised fine-tuning exploration in Section 3.2.

3.1 Benchmark Results

Model	Meta-Avg	Description	Len-Limit	Genre	Grammar	Mention	Math	Perspective	OCR
Open Source
Fuyu-8b [36]	24.52	52.06	24.52	17.06	17.18	36.43	22.62	66.67	33.09
Kosmos-2 [37]	26.06	50.95	38.52	11.55	19.78	28.70	17.26	50.83	41.88
InstructBLIP-13b [5]	38.16	50.54	39.57	29.34	38.43	42.28	12.50	50.00	30.42
Sphinx [9]	50.99	75.33	53.51	60.45	48.28	57.75	47.41	70.00	61.04
Idefics-2-8b [38]	51.42	59.37	62.73	48.07	64.09	46.20	46.51	48.33	61.97
Yi-VL-34b [39]	53.90	74.89	52.05	59.09	55.91	57.25	54.17	41.85	70.09
mPLUG-Owl2 [40]	57.86	75.01	65.25	63.39	60.26	57.70	57.22	65.00	62.08
CogVLM-Chat [8]	58.95	60.42	57.86	67.94	60.55	62.92	36.67	60.83	61.87
ShareGPT4V [41]	59.41	81.08	63.49	63.88	58.46	62.49	52.98	82.50	72.29
DeepSeek-VL-7b-chat [42]	60.96	86.31	63.26	72.11	54.79	63.75	67.39	74.17	77.85
LLaVA-1.5-7b [4]	62.18	78.00	68.60	63.95	64.18	65.89	47.31	86.67	60.75
LLaVA-NeXT-7b-vicuna [30]	62.27	79.21	68.01	65.63	60.95	63.33	46.67	90.00	65.54
Qwen-VL-Chat [7]	63.09	80.51	74.22	66.95	63.11	63.01	45.00	75.83	66.01
LLaVA-1.5-13b [4]	63.55	80.98	70.15	64.54	59.30	67.42	45.11	69.17	76.28
XComposer2-7b [43]	67.71	83.47	76.16	73.66	67.69	67.01	48.61	77.50	68.06
LLaVA-NeXT-13b-vicuna [30]	69.16	86.75	69.88	82.07	64.77	74.99	48.56	77.50	75.83
CogVLM2 [8]	73.43	87.60	74.52	83.47	71.97	77.01	71.53	90.83	87.16
InternVL-Chat-v1.5 [2]	75.42	89.13	78.21	79.92	78.16	77.54	76.11	87.50	80.92
LLaVA-NeXT-34b [30]	75.61	88.02	83.50	86.58	71.57	75.83	68.06	87.50	80.26
Phi-3-vision [31]	76.02	84.90	84.46	86.52	67.93	74.70	78.16	74.17	83.96
MiniCPM-Llama3-v2.5 [44]	76.27	84.12	79.44	80.33	81.25	76.99	64.08	81.67	76.59
LLaVA-NeXT-110b [30]	79.84	86.99	84.86	82.49	79.04	80.10	71.94	80.83	75.45
Proprietary
Gemini-Pro [10]	70.63	82.77	72.83	78.76	76.91	71.67	81.45	89.29	84.11
Reka [29]	76.95	91.05	79.91	85.16	78.98	82.08	82.53	77.50	81.08
Claude-3-Haiku [28]	78.25	86.86	77.53	90.27	73.41	82.62	82.22	57.50	86.49
Claude-3-Sonnet [28]	79.44	88.06	82.71	90.54	79.60	82.05	82.22	76.67	84.43
Claude-3-Opus [28]	84.50	90.50	86.03	91.19	83.82	85.49	85.92	65.00	86.84
GPT-4v [1]	86.11	90.03	87.61	94.59	80.12	89.37	85.63	59.17	85.26
GPT-4o [27]	88.58	90.82	92.73	94.29	85.70	90.66	87.07	92.50	86.54

Table 1: Evaluation results of a wide array of MLLMs on MIA-Bench.

In total, we have evaluated 29 popular MLLMs on MIA-Bench. Results are reported in Table 1. Observations are summarized as follows.

• Overall, the best performance was achieved by GPT-4o [27], with a score 88.58, showcasing its superiority across different categories of instruction adherence.

• The ability to describe content accurately was best exhibited by Reka [29]. Other models like Claude-3-Opus [28], GPT-4v [1] and GPT-4o also achieved scores higher than 90. This suggests that these models are good at generating coherent and contextually appropriate text.

• In the genre category, the highest proficiency was shown by GPT-4v and GPT-4o with scores above 94, suggesting an exceptional grasp of language nuances. Among open-source models, Phi-3-Vision [31] and LLaVA-NeXT-34b [30] show strong performance with scores of 86.52 and 86.58, respectively. The lowest score on this metric was by Kosmos-2 [37], with a mere 11.55, pointing to difficulties in understanding or generating linguistically complex sentences.

• GPT-4o excelled in grammar with a score of 85.70, which indicates superior ability in syntax correctness and sentence structuring that matches specific requirements in the instruction. Among the open-source models, MiniCPM-Llama3-V-2.5 [44] is notable with a score of 81.25. Contrastingly, Fuyu-8b [36] scored the lowest with 17.18, reflecting major challenges in grammar adherence.

• GPT-4o also showed the best performance with a score of 92.73 in respecting prescribed length limits, which is crucial for tasks requiring concise and precise answers. Among open-source models, LLaVA-NeXT-110b [30] stands out with a score of 84.86.

• Results from LLaVA series also suggest a strong correlation between LLM size and MIA-Bench performance across metrics.

Model	MME	MMMU	MMB	MMVet	HallB	Math	Meta	MIA	MIA
						Vista	Ranking		Ranking
GPT-4v [1]	1926.6	56.8	77/74.4	67.6	46.5	49.9	2	86.11	1
Gemini-Pro-1.0 [10]	1933.4	47.9	73.6/74.3	64.3	45.2	45.2	4	70.63	5
Claude-3-Opus [28]	1586.8	59.4	63.3/59.2	58.1	37.8	50.5	5	84.50	2
InternVL-Chat-V1-5 [2]	2187.8	45.2	82.2/82	62.8	49.3	53.5	1	75.42	4
LLaVA-NeXT-34b [23]	2028	51.1	81.1/79	48.9	47.6	47.7	3	75.61	3

Table 2: Meta ranking of five state-of-the-art MLLMs on existing multimodal benchmarks compared with their ranking on MIA-Bench.

Correlation with other benchmarks. In Table 2, we compare the ranking of 5 state-of-the-art MLLMs on MIA-Bench as well as their meta ranking on MME [15], MMMU [3] , MMBench [45], MMVet [19], HallusionBench [46], and MathVista [18] (meta ranking is computed by averaging rankings across these benchmarks). Our findings reveal a discrepancy between the two sets of rankings. Notably, InternVL-Chat-V1.5 [2], which holds the highest meta-ranking among the five MLLMs on the other benchmarks, ranks the lowest on MIA-Bench. Conversely, Claude-3-Opus, which has the lowest meta-ranking, secures the second position on the MIA-Bench. This indicates that excelling in tasks evaluated by existing benchmarks does not necessarily translate to superior instruction adherence capability assessed by MIA-Bench.

Correlation with LLM backbone performance. To determine if the performance on MIA-Bench is attributable solely to the underlying LLMs, we also evaluate several MLLMs on IFEval [26], a benchmark that assesses the instruction adherence capability of LLMs, and compare their ranking with that on MIA-Bench. This comparison is shown in Appendix, which shows that the instruction adherence capabilities of MLLMs do not consistently align with their LLMs’ adherence capability.

Other external models as the judge. Since the evaluation uses GPT-4o as the judge, it is natural to conjecture that GPT-4o may favorably score its own responses. To alleviate this concern, we use Claude-3, a strong performer in Table 1, to evaluate responses from GPT-4o and itself, and compare their scores with each other. The prompt used to grade responses is the same as the one used in GPT-4o grading. We find that even using Claude-3 Opus to score its own and GPT-4o’s responses, GPT-4o still achieves a superior score. When scored by Claude-3-Opus, GPT-4o achieves 89.84 score in contrast to Claude-3-Opus’ 85.89. Based on this observation, we use GPT-4o for evaluation by default, and observe that results from multiple runs may have around 1% variation.

3.2 Supervised Fine-Tuning (SFT)

The performance of small-scale models such as LLaVA-NeXT-13b is less ideal on MIA-Bench. In this section, we study the use of supervised fine-tuning to enhance model performance.

Additional SFT data construction. First, we randomly sample 1000 images from COCO 2017 training set, and use GPT-4v to generate five instructions for each image, using the prompt below.

Model	Total Score	Description	Length Limit	Genres	Grammar	Mention	Math	Perspective	OCR
LLaVA-NeXT-13b [30]	69.16	86.75	69.88	82.07	64.77	74.99	48.56	77.50	75.83
LLaVA-NeXT-13b*	78.85	86.90	86.80	88.02	71.34	81.01	60.87	84.17	72.65

Table 3: Detailed results on MIA-Bench before and after (denoted by *) supervised fine-tuning on additional constructed diverse instruction-tuning data. We re-ran the baseline.

Model	MMBench	TextVQA	VQA2	LLaVA-itw	POPE	VizWiz	MathVista	MIA-Bench
LLaVA-NeXT-13b [30]	70.6	64.26	82.80	85.8	87.7	60.41	33.0	69.16
LLaVA-NeXT-13b*	68.6	63.20	82.58	83.4	86.9	59.72	32.0	78.85

Table 4: Results on MIA-Bench and other major multimodal benchmarks before and after (denoted by *) supervised fine-tuning on additional diverse instruction-tuning data. We re-ran the baseline.

We then manually process the generated instructions. The cleaned data for SFT consists of 5000 image-prompt pairs.

Then, we use GPT-4v to generate responses to the constructed prompts. To evaluate the quality of these responses, we sampled 100 responses and manually checked if they adhere to the instructions. We find that 90% of the responses successfully followed all instructions in the prompt, serving as a proper ground-truth response for model training. Examples of this additional training data is provided in the Appendix.

Results. Using LLaVA-NeXT-13b as the backbone, we train the model for 1 epoch on the constructed SFT data. Results on MIA-Bench and other benchmarks are summarized in Table 4, with detailed results on MIA-Bench reported in Table 3. The performance on MIA-Bench has been significantly improved by around 10 points, at the cost of minor regressions across other benchmarks. Examples are shown in Figure 7 to compare responses from LLaVA-NeXT-13b before and after SFT.

Figure 7: Examples of LLaVA-NeXT-13b responses before and after supervised fine-tuning on additional diverse instruction-tuning data.

4 Related Work

Multimodal LLMs and Benchmarks. Multimodal Large Language Models (MLLMs) have recently emerged as a significant research focus. LLaVA [4] and MiniGPT-4 [47] pioneered visual instruction tuning, and the past year has witnessed a boom of open-source MLLMs based on this concept. Prominent examples include InstructBLIP [5], mPLUG-Owl(-2/Doc) [40, 48, 49], Qwen-VL [7], CogVLM [8], SPHINX(-X) [9, 50], InternLM-XComposer2-VL [51], InternVL(-1.5) [52, 2], VILA [53], MM1 [11], Mini-Gemini [54], Idefics2 [55], Phi-3-vision [31], to name a few. There is also a rich body of literature on enabling MLLMs for referring and grounding [56, 57, 58, 59, 60, 61, 62, 63], image generation and editing [64, 65, 66], etc.

Various benchmarks have been proposed to evaluate the performance of MLLMs across different dimensions. Benchmarks like VQAv2 [12], TextVQA [34], ScienceQA [14], MME [15], MMbench [45], SEED-Bench [17], MathVista [18], and MMMU [3] aim to assess comprehensive multimodal understanding abilities. Additionally, there are benchmarks that specifically study model hallucination, including POPE [67], MHalDetect [68], GAVIE [69], HallusionBench [46], and MAD-Bench [70]. Many of these benchmarks have gained popularity within the community due to their use of multiple-choice evaluations. However, they do not accurately reflect the common use cases for MLLMs, where user interactions are typically open-ended. To address this, benchmarks like LLaVA-Bench [4], MM-Vet [19], and Vibe-Eval [22] have been proposed. Our MIA-Bench also falls into this category; however, we focus on studying the exact instruction adherence of MLLMs, a metric that previous benchmarks have only loosely measured.

Instruction Following Benchmarks for LLMs. Several benchmarks have been proposed to measure the instruction adherence ability of LLMs. Instruction-Following Eval (IFEval) [25] is a benchmark for assessing LLMs’ adherence ability to the given instructions. Its approach emphasizes verifiable instructions, which enhance objectivity and reproducibility in evaluations. IFEval creates 541 prompts spanning 25 instruction types, revealing a significant performance gap in instruction adherence ability between GPT-4 [1] and PaLM-2 [71]. This demonstrates the benchmark’s ability to effectively differentiate between models in adherence ability. On the other hand, InfoBench [26] introduces a new metric called Decomposed Requirements Following Ratio (DRFR) for assessing the instruction-adherence capabilities of LLMs. DRFR dissects complex instructions into simpler sub-instructions, allowing for a granular evaluation of compliance with various task aspects. InfoBench contains 500 diverse instructions consisting of 2,250 decomposed questions in multiple constraint categories. The evaluation of advanced LLMs using this framework highlights their strengths and areas for improvement, especially in complex instruction adherence scenarios. Compared with these previous work, we are the first known effort that specifically focuses on benchmarking the instruction adherence ability of multimodal LLMs.

5 Conclusion

This paper introduces MIA-Bench, a benchmark designed to evaluate the ability of MLLMs to strictly adhere to complex instructions within prompts. Through the analysis of 400 image-prompt pairs from diverse sources, our findings highlight variability in model performance and much room for improvement, underscoring a critical need for enhanced training methods to improve instruction compliance. We further explored supervised fine-tuning (SFT) using LLaVA-NeXT as the backbone, which yielded promising results. Going forward, future research can expand on both SFT and alignment methods such as RLHF [72, 73] and DPO [74, 75], enhancing MLLMs to achieve higher accuracy and reliability in practical applications across diverse instructional contexts.

Limitation

In designing the instructions for our benchmark, we incorporated a wide range of categories to enhance the diversity of sub-instructions. Nonetheless, the real world presents an infinite variety of instructions, many of which may pose significant challenges for MLLMs.

Appendix A Appendix

A.1 Comparison between MIA-Bench and IFEval

Figure 8: Comparison between MIA-Bench and IFEval.

A.2 Examples of diverse instruction-tuning data used for SFT

Figure 9: Examples of diverse instruction-tuning data used for SFT.

A.3 Examples of how MLLMs respond to instructions in MIA-Bench.

Figure 10: An example of how MLLMs respond to instructions in MIA-Bench.

Figure 11: Another example of how MLLMs respond to instructions in MIA-Bench.

Figure 12: The third example of how MLLMs respond to instructions in MIA-Bench.