InstructDoc: A Dataset for Zero-Shot Generalization of Visual Document Understanding with Instructions

Figure 2 shows the source datasets in InstructDoc. We collected 30 publicly available datasets and 12 tasks in VDU areas from DUE (Borchmann et al. 2021) as well as through manual searches. Following the task clusters defined in previous works (Wei et al. 2021; Dai et al. 2023), we divided the QA datasets that require different reasoning abilities into different tasks. As a result, we divided the collected datasets into the following tasks:

• •

  Key Information Extraction (KIE) assigns each word a semantic entity label from predefined categories (Šimsa et al. 2023; Jaume, Ekenel, and Thiran 2019; Sun et al. 2021; Park et al. 2019; Huang et al. 2019).

• •

  Single-page QA is a task of QA on single-page documents and focuses on document layout and textual content understanding (Tanaka, Nishida, and Yoshida 2021; Chen et al. 2021; Mishra et al. 2019; Mathew, Karatzas, and Jawahar 2021; Tüselmann et al. 2022).

• •

  Single-page QA w/ Discrete Reasoning requires various arithmetic abilities, including addition, sorting, or counting (Zhu et al. 2022).

• •

  Single-page QA w/ Visual Reasoning requires a set of abilities, including object (e.g., icon) recognition, commonsense understanding, and relation extraction on single-page documents (Lu et al. 2021; Kembhavi et al. 2016; Lu et al. 2022; Kembhavi et al. 2016).

• •

  Single-page QA w/ Discrete & Visual Reasoning requires both discrete and visual reasoning (Mathew et al. 2022; Masry et al. 2022) on single-page documents.

• •

  Multi-page QA w/ Multi-hop & Discrete & Visual Reasoning requires understanding the content relationship via multi-hop reasoning as well as discrete/visual reasoning on multi-page documents (Tanaka et al. 2023; Landeghem et al. 2023).

• •

  Document NLI is a task of natural language inference that predicts the entailment relationship between two sentences in a document (Borchmann et al. 2021)

• •

  Dialogue involves a human-agent interaction on the basis of document images (Zhang et al. 2023).

• •

  Captioning involves producing descriptions of documents (Hsu, Giles, and Huang 2021; Wang et al. 2021).

• •

  Classification involves classifying a document from a set of candidate labels (Harley, Ufkes, and Derpanis 2015).

• •

  Document Layout Analysis (DLA) determines a document’s components with bounding boxes (Li et al. 2020; Pfitzmann et al. 2022)

• •

  Image-Text Matching (ITM) requires the model to determine whether a given OCR text and image match.

We found that two KIE datasets (FUNSD and CORD) are challenging because they contain abbreviated queries that are difficult for humans to comprehend. To bridge the gap between humans and machines, we replace these queries with complete and more easily understandable phrases (e.g., menu.vatyn $\to$ menu_whether_price_tax_included).

For each dataset, we manually crafted five to ten distinct instruction templates in a unified format. For QA tasks, the answers have diverse styles in the original datasets; for example, DocVQA’s answer is extractive, which requires the model to extract a contiguous span of words from the document, but VisualMRC’s answer is generative, which is not limited to the word spans. Hence, an instruction that sufficiently describes an arbitrary VDU task should include intent and answer style or only intent. Specifically, as shown in Figure 1, intent describes how the task can be performed and answer style describes how the model generates the output. If each dataset provides query and options, we fill it in annotated instruction templates.

We split InstructDoc into 23 held-in and seven held-out datasets. For the held-out evaluation, we aim to understand how instruction tuning on the held-in datasets improves the zero-shot generalization performance on unseen datasets, including (i) Test${}_{\text{Cross-Dataset}}$: FUNSD and CORD datasets, (ii) Test${}_{\text{Cross-Task}}$: ChartQA, InfoVQA, and TabFact datasets, and (iii) Test${}_{\text{Cross-Domain}}$: DUDE and SlideVQA datasets. All other datasets were held-in ones to train our model. Note that the held-out datasets were carefully selected in order to avoid data contamination.

To encode a document image, we use a pre-trained CLIP (Radford et al. 2021) vision encoder to extract its visual features $\mathbf{z}^{\text{vis}}$. Additionally, we process the document image using an OCR engine and apply a sub-word tokenizer to obtain $M$ word tokens $\{s_{i}\}_{i=1}^{M}$ and their corresponding bounding boxes $\{(x_{i}^{1},y_{i}^{1},x_{i}^{2},y_{i}^{2})\}_{i=1}^{M}$, where ($x^{1}$, $y^{1}$) and ($x^{2}$, $y^{2}$) represent the coordinates of the top-left and bottom-right corners, respectively. To learn the visual layout of the image, we construct a spatially aware OCR representation $\mathbf{z}_{i}^{\text{ocr}}=\mathbf{z}_{i}^{\text{word}}+\mathbf{z}_{i}^{\text{bbox}}$ with learnable embedding layers $\mathbf{W}^{\{s,x,y,h,w\}}$, where OCR text features are calculated as $\mathbf{z}^{\text{word}}_{i}=\mathbf{W}^{s}(s_{i})$ and spatial features are calculated as $\mathbf{z}^{\text{bbox}}_{i}=\mathbf{W}^{x}(x^{1}_{i},x^{2}_{i})+\mathbf{W}^{y}(y^{1}_{i},y^{2}_{i})+\mathbf{W}^{h}(y^{2}_{i}-y^{1}_{i})+\mathbf{W}^{w}(x^{2}_{i}-x^{1}_{i})$. Similarly, we encode an instruction by $\mathbf{W}^{s}$ and obtain its features $\mathbf{z}^{\text{ins}}$.

We introduce Document-former, which is a trainable module to bridge the gap between an image encoder and an LLM, enabling extraction of document content/layout that LLMs can understand. The architecture of Document-former is a stack of Transformer blocks with cross-attention layers. To map document features into the LLM’s space, we use a set of $m$ learnable tokens $\mathbf{z}^{\text{token}}\in\mathbb{R}^{d}$, where $d$ is the dimension of the hidden size. These tokens $\mathbf{z}^{\text{token}}$ interact with $\mathbf{z}^{\text{vis}}$ through cross-attention layers and with the input sequence, composed of $\mathbf{z}^{\text{ins}}$ and $\mathbf{z}^{\text{ocr}}$, through self-attention layers. As a result, we obtain $\mathbf{z}^{\text{doc}}$ and transform it via a projection feed-forward network (FFN) layer to $\mathbf{h}^{\text{doc}}\in\mathbb{R}^{m\times d^{\text{LLM}}}$, which have the same dimension $d^{\text{LLM}}$ as the LLM’s input embedding.

The LLM receives the document embeddings $\mathbf{h}^{\text{doc}}$, the instruction, and OCR tokens as input and outputs the answer $\mathbf{A}$, token by token. The parameters of the LLM are initialized from an instruction-tuned FlanT5.

To achieve task-agnostic learning, we formulate the process of learning all held-in tasks in a unified sequence-to-sequence abstraction through instructions. To train the LLM efficiently, we update only the parameters of the Document-former (including $\mathbf{W}^{\{s,x,y,h,w\}}$) and the projection FFN layer, while kee** other parameters frozen during training. We optimize the model by minimizing the negative log-likelihood between the ground-truth and predictions.

We also support performing reasoning across multiple document pages. As shown in Figure 3b, each image is processed individually by the image encoder and Document-former, and their resulting document embeddings are mean-pooled together before being fed into the LLM. The OCR input to the LLM consists of concatenated tokens extracted from each page.

We compared InstructDr with seven state-of-the-art (SOTA) MLLMs, including LLaVA (Liu et al. 2023a), MiniGPT-4 (Zhu et al. 2023) and mPLUG-Owl (Ye et al. 2023b), which align CLIP visual encoder with Vicuna (Chiang et al. 2023) trained on a dialogue generated by GPT-4 (OpenAI 2023); BLIP-2 (Li et al. 2023), which connects a FlanT5 with a vision encoder; InstructBLIP (Dai et al. 2023), which fine-tunes BLIP-2 with instructions on scene images; and LLMDoc (Ye et al. 2023a) and LLaVAR (Zhang et al. 2023), which fine-tune mPULG-Owl/LLaVA on the DocOwl/LLaVAR datasets. Additionally, we used Supervised SOTA models (Appalaraju et al. 2023; Chen et al. 2023; Huang et al. 2022; Landeghem et al. 2023) on each dataset and two text-based LLMs, ChatGPT (gpt-3.5-turbo-0613) and GPT-4. To control the answer’s length, we added control phrases (e.g., use 1 to 3 words to answer) to the selected instructions.

We followed the evaluation protocol of each dataset, we used ANLS (Biten et al. 2019) for InfoVQA, DUDE, Text-VQA and ST-VQA, EM for SlideVQA, Relaxed Accuracy (RAcc.) for ChartQA, entity F1 (eF1) for FUNSD and CORD, Accuracy (Acc.) for TabFact, and ROUGE-L for VisualMRC as evaluation metrics. Additionally, we used F1 as the optional metrics.

Following (Wei et al. 2021), we balanced the training instances of different tasks by sampling a maximum of 5k instances for each held-in dataset while kee** all evaluation instances. We used the AdamW (Loshchilov and Hutter 2017) with a weight decay of 0.05. We applied a linear warmup during the initial 1,000 steps and used a cosine learning rate decay with a minimum learning rate of 0. We set the number of learnable tokens $m$ to $32$. All images of the model input were resized to $224$. We trained on eight A100 (40G) GPUs for three epochs and completed the training within two hours. If each dataset does not provide OCR, we extracted it via the Google Vision API.

Table 2 shows that our model achieved the highest performance on all datasets compared with other MLLMs. InstructDr consistently outperformed its original backbone, BLIP-2, by a significant margin, indicating that instruction tuning on InstructDoc effectively enhances performance on unseen VDU datasets, tasks, and domains. In contrast, InstructBLIP, which is instruction-tuned BLIP-2 trained on scene images, performed worse than BLIP-2. This is because that InstructBLIP does not assume that the images might contain text during instruction tuning. BLIP-2 fine-tuned on InstructDoc falls short of achieving the same level of performance compared with InstructDr, indicating that InstructDr is better suited for comprehending diverse real-world documents. This conclusion is further supported by the results presented in Table 4, where ablations of Document-former, spatial information, and strategy of gathering multi-page features have a significant negative impact on performance.

As shown in Table 3, our model outperformed ChatGPT on all datasets. Additionally, InstructDr achieved competitive results with supervised SOTA models and GPT-4 on the DUDE and SlideVQA datasets that require multiple reasoning skills (e.g., discrete, visual, and multi-hop reasoning). This indicates that our model can effectively learn diverse skills through instruction tuning with InstructDoc.

As shown in Table 4, removing instructions (i.e., only query and options as the model input) significantly decreased zero-shot performance during training or/and test time, indicating the effectiveness of incorporating instructions. This result was observed on the high-quality instruction-tuning datasets (Wei et al. 2021; Xu, Shen, and Huang 2023). Moreover, our instruction annotations, including query rephrasing and answer styles, helped to improve the zero-shot performance.

Figure 4 shows the performance variance when the models were given five different instructions; InstructDr exhibited the smallest performance variance and outperformed the other models. This indicates InstructDoc empowers the model with the ability to deal with a variety of instructions. Our results also suggest that using multiple instructions per dataset is important for achieving decent performance.

As shown in Figure 5, as the number of task clusters increases, we can observe an improvement in models’ zero-shot performance.

We further fine-tuned InstructDr (only Document-former module) on a specific dataset to investigate the knowledge and transferability of our instruction-tuned model weights. Table 5 shows the fine-tuning performance on held-in (VisualMRC) and held-out (DUDE, SlideVQA) tasks. InstructDr achieved state-of-the-art finetuning performance on VisualMRC, DUDE, and SlideVQA using a unified model. Compared with BLIP-2, InstructDr exhibited superior fine-tuning performance on both held-in/out datasets, validating InstructDr as a better weight initialization model for task-specific fine-tuning.

Table 6 shows the zero-shot performance of scene-text VQA (Singh et al. 2019; Biten et al. 2019) on scene images, which are the unseen image types in InstructDoc but were used for training our base model, BLIP-2. Note that ST-VQA’s images include the part of COCO (Lin et al. 2014) that InstructBLIP was trained on. This result indicates that InstructDr can effectively learn visual reasoning skills without forgetting the abilities of the original backbone.

Figure 6 visualizes output examples, where the left/center/right examples require table/visual/hand-written text understanding skills. ChatGPT gave incorrect answers because it can only consider text information. Moreover, while BLIP-2 could not follow instructions (e.g., use 5 to 10 words) and extract items from structured text, InstructDr accomplished diverse VDU tasks with instructions. As shown in the right example, all models affected OCR quality, causing incorrect answers.