Web2Code: A Large-scale Webpage-to-Code Dataset and Evaluation Framework for Multimodal LLMs

Abstract

Multimodal large language models (MLLMs) have shown impressive success across modalities such as image, video, and audio in a variety of understanding and generation tasks. However, current MLLMs are surprisingly poor at understanding webpage screenshots and generating their corresponding HTML code. To address this problem, we propose Web2Code, a benchmark consisting of a new large-scale webpage-to-code dataset for instruction tuning and an evaluation framework for the webpage understanding and HTML code translation abilities of MLLMs. For dataset construction, we leverage pretrained LLMs to enhance existing webpage-to-code datasets as well as generate a diverse pool of new webpages rendered into images. Specifically, the inputs are webpage images and instructions, while the responses are the webpage’s HTML code. We further include diverse natural language QA pairs about the webpage content in the responses to enable a more comprehensive understanding of the web content. To evaluate model performance in these tasks, we develop an evaluation framework for testing MLLMs’ abilities in webpage understanding and web-to-code generation. Extensive experiments show that our proposed dataset is beneficial not only to our proposed tasks but also in the general visual domain, while previous datasets result in worse performance. We hope our work will contribute to the development of general MLLMs suitable for web-based content generation and task automation. Our data and code are available at https://github.com/MBZUAI-LLM/web2code.

Figure 1: Our motivation for constructing the Web2Code dataset stems from the limitations of previous models, such as LLaVA [33], which are trained on general datasets and struggle to generate high-quality webpages, as in the second row. Our dataset aims to significantly enhance the quality of webpage generation as in third row while maintaining a strong level of general multimodal ability.

1 Introduction

Multimodal large language models (MLLMs) have achieved explosive growth in the past few years. Leveraging the rich commonsense knowledge in large language models (LLMs), MLLMs are remarkably successful at processing and reasoning about various modalities such as image [2, 35], video [60, 53], and audio [40] in a broad range of tasks such as recognition [52], reasoning [59], and question-answering [39], all using language as the intermediate representation. However, existing MLLMs are surprisingly poor at understanding webpage screenshots and generating the HTML code to express their latent states. For instance, given the instruction “Parse the HTML code for this webpage”, the well-known LLaVA-1.5 [33] generates generic, pale code that fails to preserve most of the original webpage’s features (see Figure 1), which hampers its utility in applications such as UI prototy**, automation agents, and accessibility (e.g., noting available buttons and options given webpage screenshot).

The essential ingredients behind the progress in MLLMs are arguably large-scale instruction datasets [9, 63] and evaluation benchmarks [16, 58] – the former for aligning multimodal inputs with the massive knowledge in LLMs [27, 35], and the latter for standardized comparison which facilitates model development. However, existing instruction datasets and benchmarks typically focus on general settings (e.g., visual QA and reasoning) and pay insufficient attention to webpage understanding and webpage-to-code generation, which requires a unique combination of capabilities such as optical character recognition (OCR), spatial reasoning, and long-text generation, among others. While previous work has developed datasets for these tasks [4, 22], they lack instruction information and are unsuitable for integration with general-purpose MLLMs. On the other hand, popular benchmarks [36, 25] evaluate some of the required capabilities in isolation, but not fully in combination for visual parsing and reasoning over webpages.

To fill this gap, we propose a new instruction tuning dataset and evaluation suite named Web2Code. Web2Code contains a total of 1179.7k webpage based instruction-response pairs. The responses consist of not only the HTML code, but also the structured questions and answers about the webpage, which assist a model in better understanding its information. For dataset collection, we use GPT-3.5 and GPT-4 to clean existing data (e.g. WebSRC [10]) as well as to generate completely new webpages in HTML code. To evaluate the MLLM’s success at webpage understanding and HTML parsing, we propose the Webpage Understanding Benchmark (WUB) and Webpage Code Generation Benchmark (WCGB), two tasks that test the model’s abilities to answer questions about a webpage and generate its HTML code, respectively. For the latter task, we find that traditional text similarity metrics are insufficient for evaluating the fidelity of the generated code, and instead propose to render the output HTML back to a webpage screenshot, and use GPT-4V [42] to evaluate the quality of the resulting webpage [61].

To demonstrate the utility of our dataset, we train LLaVA-style MLLMs with our dataset included in the instruction finetuning stage. Quantitative results show that finetuning on our dataset not only clearly improves the image-to-HTML-code translation ability of the MLLM, but also leads to slight improvements in the model’s perception and reasoning abilities in general benchmarks. In contrast, including previous datasets without our data conversion leads to degraded general performance, suggesting our dataset is suitable for inclusion in MLLM instruction-tuning for additional capabilities without impacting existing ones.

2 Related Work

MLLM Dataset. At present, there is a substantial amount of large-scale visual instruction data, primarily generated using GPT. SVIT [63] and LRV-Instruction [32] are both generated by GPT4 based on manual prompts to adjust the instruction data, including rich high-quality dialogue question and answer, complex reasoning question and answer, reference question and answer, and image detail description task datasets; similarly, ShareGPT4V [9], LLaVAR [62], LVIS-Instruct4V [55] use GPT-4V [42] to generate millions of high-quality image-text pairs, aiming to enhance the perception, reasoning and planning capabilities of MLLM. Commonly used image data sources include LAION [48], CC [49], SBU [46], COCO [31], VG [21], VQAv2 [18].

MLLM. Instruction Tuning MLLM has made great progress in recent years. The structure of MLLM usually contains a visual encoder, vision-language map** module and LLM. LLaVA-v1.5 [33] only uses MLP as the vision-language map** module and the successful application of instruction tuning on MLLM has inspired people. The community has explored various feasible structures, which can be divided into attention structures BLIP2 [27], InstructBLIP [12], Qwen-VL [3], ShareGPT4V [9] and non-attention structures LLaVA [33], Shikra [8] according to the vision-language map** module. At the same time, various open source and more powerful LLMs, such as Vicuna1.5 [11], InternLM2 [5] also help MLLM achieve richer and more extensive instruction following capabilities. Qwen-VL [3], OtterHD [24], mPLUG-Owl [56], InternLM-XComposer2-4KHD [13] increase the resolution of images, while LLaVA-NeXT [34], Mini-Gemini [28], MM1 [41] split the input image into several image crops. In addition, BRAVE [19], MoVA [64], DeepSeek-VL [38], OmniFusion [17] apply supplementary vision encoders to obtain abundant visual features, e.g. DINOv2 [43], SAM [20]. Furthermore, more computer vision models are utilized for different tasks, which include image segmentation, detection and OCR, in MOAI [23], CuMo [26] and SpatialVLM [7]. Subsequently, MoE [30] was applied to MLLM to expand the scale of training data at the same computing scale.

Code Study. There are various code studies related to LLM. Sarker et al. [47] focused on generating code functions through formula hints, trying to improve the robustness of the syntax, and systematically tested the robustness of the syntax. From the perspective of security. Finkman et al. [15] said that these code assistance tools may inadvertently disclose the developer’s proprietary code to the code assistant service provider in the process of hel** development, thus they proposed a complementary method to reduce the risk of code leakage while providing effective advice to developers. In addition to code leakage, the code generated by LLM has also caused concerns in industries and other fields. To address this issue, Ye et al. [57] proposed a new zero-shot synthetic code detector based on the similarity between code and its rewritten variants. In the evaluation work on code generation, Du et al. [14] proposed a new computational efficiency benchmark Mercury and a new metric Beyond for the efficiency evaluation of code. They experimentally showed that direct preference optimization can be used as a robust baseline for improving computational efficiency compared to supervised fine-tuning, which paves a promising path for future exploration of efficient code generation.

3 Dataset Construction

Overview. Our Web2Code instruction tuning dataset construction and instruction generation process involves four key components: (1) Creation of new webpage image-code pair data: We generated high-quality HTML webpage-code pairs following the CodeAlpaca prompt [6] using GPT-3.5 and convert them into instruction-following data. (2) Refinement of existing webpage code generation data: We transform existing datasets including WebSight [22] and Pix2Code [4] into an instruction-following data format similar to LLaVA data [33], so they can be used as instruction-following data to train MLLMs. (3) Creation of a new text question-answer pair data: We generated a new question-answer pair dataset utilizing our new GPT-3.5 generated data from (1) for webpage understanding. (4) Refinement of existing webpage understanding data: We refine the WebSRC [10] question-answer data to improve its quality using the GPT-4. Each component is elaborated in detail as follows:

Figure 2: Qualitative example of generated question-answer pair dataset. Questions cover diverse aspects of the web page understanding.

Figure 3: WebSRC data refinement for improved Quality. Left: Before refinement; Right: After refinement, the quality has been improved and duplications have been excluded.

DWCG: Creation of new webpage image-code pair data for code generation. To augment our dataset with high-quality data, we employed GPT-3.5 to generate 60K HTML pages following the guidelines and prompts in CodeAlpaca [6]. Using Selenium WebDriver, we then created web image screenshots from the generated HTML code. These web image-code pairs were subsequently converted into an instruction-following data format similar to the LLaVA data format [33], enabling their use in training Multimodal Large Language Models (MLLMs). The example of the instruction is shown in Figure 16. The generation of instruction was done in two stages using prompts fed to GPT-4: (a) During stage 1, the prompt shown in Figure 12 resulted in the creation of generic instructions. (b) This was followed by augmenting the instruction from (a) with the GPT generated instructions using the prompt shown in Figure 13 to include stylistic information. This allowed the model to learn two styles: Modern and Bootstrap style as shown in Figure 21 and Figure 22, respectively.

DWCG${}_{\texttt{R}}$: Refinement of existing webpage code generation data. To enhance the capability of our model in the task of HTML code generation, we leverage the Pix2code [4] and WebSight [22] datasets. To mitigate the detrimental impact on model performance from random letters in Pix2Code data, we replace these random letters with meaningful text using GPT-4, thereby refining the webpages into diverse webpages encompassing product landing pages, personal portfolios, blogs, and other categories. We then visually rendered each sample by taking screenshots of the browser view of each webpage. Further, we convert all these data into LLaVA instruction following data format using the same strategy as used for DWCG. We note that DWCG and WebSight webpages follow Modern style while Pix2Code follow Bootstrap style.

DWU: Creation of a new question-answer pair data for webpage understanding. For the purpose of fine-tuning our models through an instruction-following manner, we utilized the capabilities of GPT-4 to generate webpage code-based question-answer pairs. We generated 10 question-answer pairs using GPT-4 for a subset of 24.35K webpage data, resulting in a total of 243.5K question-answer data points. This includes, a set of 230K question-answer pairs for GPT-3.5 based webpages, a set of 13.5K newly generated question answer pairs for refined Pix2Code images. These pairs were meticulously crafted to align with our image-based evaluation criteria, ensuring that each question probes specific aspects of the visual and content quality reflected in the generated web images. This strategy enhances the model’s performance by integrating a nuanced understanding of the evaluation parameters into its learning process. Figure  2 shows a qualitative example from our generated question-answer pairs. The prompt used for question-answer generation is shown in Figure 11.

DWU${}_{\texttt{R}}$: Refinement of existing webpage understanding data. To increase our instruction-following dataset with high-quality instruction-following examples for webpages, we integrated the WebSRC dataset into our training regime. Prior to inclusion, we meticulously filtered the existing question-and-answer pairs from the WebSRC dataset to ensure relevance and quality. This involves duplication removal and quality optimization as shown in Figure 3. Specifically, we found that WebSRC data contains several questions related to the same answer. To this end, we first removed those duplicates and then employed GPT-4 to assess and enhance the quality of answers. This process not only refined the dataset into 51.5K high-quality instruction data but also ensured that the model’s training was influenced by high-fidelity, instructionally sound data, thereby improving its ability to follow complex web-based instructions.

3.1 Statistics and Analysis

Figure 5 shows the word cloud of the answer set of our question-answer dataset. The word cloud highlights the most frequently occurring terms, with "section," "color", "button", and "website" being the most prominent, indicating a strong emphasis on structural and design elements in the data. This reflects the detailed focus on the layout and visual aspects of the dataset.

Figure 5 illustrates the distribution of the most common HTML tags in our GPT-3.5 generated HTML data. The distribution shows a high frequency of essential structural tags such as <div>, <p>, <meta>, <img>, and <a>, indicating that the generated pages include a diverse range of elements necessary for rich and varied web content. The significant presence of <h2>, <input>, <html>, <head>, and <body> tags further reinforces the completeness and structural integrity of the generated HTML documents.

To estimate the difficulty levels of our HTML-based webpage dataset, we provide several quantitative measures and compare them with recent and similar existing datasets, namely WebSight [22], Design2Code [50], and Pix2Code [4] (See Table 1).

Design2Code is primarily used for testing and has a small size of 484 examples, limiting its versatility and robustness. In contrast, our dataset, intended for both training and testing, is significantly larger (884.7K examples) and more complex, making it more suitable for develo** robust models. Overall, our benchmark examples are more challenging and cover a broader spectrum of complexities compared to prior efforts such as WebSight.

Dataset	WebSight [22]	Design2Code [50]	Pix2Code [4]	DWCG (ours)	DWCG${}_{\texttt{R}}$ (ours)
Instruction	-	-	-	✓	✓
Source	Synthetic	Real-World	Synthetic	Synthetic	Synthetic
Size	823K	484	1.7K	60K	824.7K
Avg Length (tokens)	647$\pm$216	31216$\pm$23902	658.7$\pm$98.0	471.8$\pm$162.3	652.85$\pm$157.0
Avg Tag Count	19$\pm$8	158$\pm$100	51.6$\pm$8.0	28.1$\pm$10.6	35.3$\pm$9.0
Avg DOM Depth	5$\pm$1	13$\pm$5	8.0$\pm$0.0	5.3$\pm$1.0	6.5$\pm$1.0
Avg Unique Tags	10$\pm$3	22$\pm$6	17.0$\pm$0.0	13.6$\pm$2.7	13.5$\pm$2.5

Table 1: Comparison of dataset statistics among webpage code generation datasets: WebSight, Design2Code, Pix2Code, our DWCG, and our DWCG${}_{\texttt{R}}$. DWCG is a newly generated GPT-3.5-based dataset, while DWCG${}_{\texttt{R}}$ is the refined dataset that utilizes WebSight and Pix2Code datasets.

3.2 Distribution

Dataset	DWU	DWU${}_{\texttt{R}}$
Instruction	✓	✓
Size	243.5K	51.5K

Table 2: Distribution of DWU and DWU${}_{\texttt{R}}$ datasets. Both datasets include high-quality question-answer pairs for webpage understanding.

Our instruction-following dataset contains 1,179.7K instruction data points. This includes 884.7K website image-code pairs and 295K question-answer pairs.

The 295K question-answer pairs consist of 243.5K GPT-4 based question-answer pairs (DWU Data) and 51.5K question-answer pairs from WebSRC image-based data, as shown in Table 2. Our evaluation dataset comprises 1,198 webpage screenshot images, sourced from diverse origins, including WebSight, Pix2Code, GPT-3.5 based data, and manual processes, ensuring a broad representation of web content. Additionally, we utilize 5,990 "yes" / "no" question-answer pairs generated from the GPT-4 Vision API for our Webpage Understanding Benchmark, as shown in Section 6.

Figure 4: Word Cloud for the answer set of the GPT4 based DWU dataset.

Figure 5: The distribution of most common 20 tags in GPT-3.5 based HTML data.

4 A New Evaluation Framework for Webpage

Our proposed evaluation framework includes two schemes: (1) Webpage Understanding Benchmark (WUB): An offline evaluation using "yes" / "no" questions. (2) Webpage Code Generation Benchmark (WCGB): An online evaluation (using GPT-4 Vision) based on image similarity.

4.1 Evaluation Metric for HTML Code Generation

In the realm of assessing code quality, particularly in terms of final visual appeal and overall functionality, existing methods that rely on code similarity metrics fall short. These traditional approaches often lack the precision and reliability needed for nuanced evaluations of code effectiveness. To address these shortcomings, we have developed a novel approach: regenerating the webpage using the model’s predicted HTML code and capturing screenshots of these generated webpages. This process, automated using the Selenium WebDriver extension in Python, shifts the focus from the less reliable code similarity assessments to a more accurate and visually oriented method. By comparing images of the generated webpages, we can more effectively evaluate the aesthetic and functional aspects of the code, offering a more comprehensive understanding of its quality.

Figure 6: Evaluation benchmark for webpage generation and webpage understanding. Left: WCGB utilizes GPT4 Vision based online evaluation for image level comparison; Right: WUB employs an offline evaluation based on question-answer pairs.

We propose two benchmarks for assessing webpage understanding and code generation capabilities.

WUB: This benchmark comprises 5,990 high-quality question-answer pairs generated from GPT-4 Vision API (See prompt 15), based on 1,198 webpage screenshot images, where each answer is either "yes" or "no". These images are sourced from diverse data origins, including WebSight, Pix2Code, GPT-3.5, and manual processes, ensuring a broad representation of web content. Figure 10 shows a qualitative sample data we used for WUB. We test these pairs on various multimodal image understanding models by comparing the predicted answers to the ground truth, with the final accuracy score serving as the evaluation metric as depicted on the right side of Figure 6. Qualitative data examples in our WUB benchmark are shown in Figure 10.

WCGB: Utilizing the same images as the WUB, this benchmark evaluates a multimodal model tasked with generating HTML code from webpage images based on specific instructions. Unlike traditional code-level evaluations, this benchmark assesses the generated webpage’s fidelity at the image level. We convert the predicted HTML codes back into images using Selenium WebDriver to allow a direct visual comparison with the ground truth images. The evaluation, depicted on the left side of Figure 6, considers 10 different aspects, which are further categorized into four evaluation matrices using the GPT-4 Vision API. This image-level evaluation provides a more accurate measure of the model’s code generation capabilities, acknowledging that identical webpages can be constructed from varying codes. The prompt used for evaluation is shown in Figure 14. This framework consists of 10 distinct criteria, which we group into four categories, each encompassing specific criteria that are scored on a 0-10 scale, as shown in Section D of Appendix.

4.2 Quantitative Evaluation for HTML Code Generation of MLLMs

We have evaluated the trained models using various data configurations and backbones on our WUB and WCGB benchmarks. The performance of the models on the code generation benchmark is presented in Table 3, while the results for webpage understanding are shown in Table 4.

LLM Backbone	DWCG	DWU	DWCG${}_{\texttt{R}}$	DWU${}_{\texttt{R}}$	VSA $\uparrow$	CAD $\uparrow$	TCC $\uparrow$	UII $\uparrow$	Overall $\uparrow$
CystalCoder-7B [37]	-	-	-	-	3.832	3.678	3.411	3.992	3.728
	✓	-	-	-	7.812	7.899	8.138	8.112	7.990
	✓	✓	-	-	8.010	8.102	8.266	8.124	8.126
	✓	✓	✓	✓	8.023	8.071	8.321	8.334	8.187
CrystalChat-7B [37]	-	-	-	-	4.714	4.572	4.865	5.147	4.825
	✓	✓	-	-	7.900	8.001	8.204	8.215	8.080
	✓	✓	✓	✓	8.384	8.287	8.417	8.488	8.394
Vicuna1.5-7B [11]	-	-	-	-	3.042	3.250	3.333	3.167	3.198
	✓	✓	✓	✓	7.876	7.687	7.267	7.563	7.598
LLaMA3-8B [1]	✓	✓	✓	✓	8.522	8.564	8.421	8.611	8.530

Table 3: The performance of different LLM backbones under various data configurations on our Webpage Code Generation Benchmark (WCGB). "VSA" denotes Visual Structure and Alignment, "CAD" represents Color and Aesthetic Design, "TCC" represents Textual and Content Consistency, and "UII" denotes User Interface and Interactivity.

LLM Backbone	DWCG	DWU	DWCG${}_{\texttt{R}}$	DWU${}_{\texttt{R}}$	Accuracy (%)
CrystalCoder-7B [37]	-	-	-	-	73.54
	✓	-	-	-	71.81
	✓	✓	-	-	73.74
	✓	✓	✓	✓	73.61
CrystalChat-7B [37]	-	-	-	-	73.94
	✓	✓	-	-	73.48
	✓	✓	✓	✓	74.14
Vicuna1.5-7B [11]	-	-	-	-	71.12
	✓	✓	✓	✓	71.23
LLaMA3-8B [1]	✓	✓	✓	✓	74.84

Table 4: The accuracy of webpage understanding under various data configurations and LLM backbones. All models are instruction-tuned and evaluated on our WUB benchmark. We note that the general domain data (i.e., LLaVA) is included in all data configuration as default.

To be specific, our dataset components have an orthogonal contribution to the overall improvements on both the WUB and the WCGB benchmarks. Table 3 demonstrates improvements in webpage code generation quality when adding DWU, DWCG + DWU, and DWCG${}_{\texttt{R}}$ + DWU${}_{\texttt{R}}$. For example, the results based on instruction-tuned CrystalChat show step-wise improvements on the WCGB benchmark when adding DWCG + DWU, and DWCG${}_{\texttt{R}}$ + DWU${}_{\texttt{R}}$ (4.825$\rightarrow$8.080$\rightarrow$8.530 on the overall metric). Interestingly, the instruction-tuned CrystalChat-7B with the general domain data only shows poor WCGB performances while it shows on par WUB performance compared to the use of further webpage datasets. Similar trends are also found in the Vicuna1.5-7B backbone, while adding the proposed dataset shows significant improvements, achieving an overall score of 7.598.

Similarly, Table 4 further demonstrates the effectiveness of the proposed dataset on the webpage comprehension capabilities. For example, the four rows of the instruction-tuned CrystalCoder-7B show that not only using DWCG with DWU can prevent degradation in language understanding, but also the full components even enhance the webpage comprehension capabilities. Overall, we found the proposed dataset can enhance both webpage understanding capability and webpage code generation abilities under various LLM backbones, and LLaMA3-8B archives the best performance among all on both webpage code generation and webpage understanding.

4.3 Visualizations for Qualitative Evaluation

As shown in Figure 7, we compare the results between the original image which is the real-world webpage sample, the rendered image generated by using LLM backbones of Vicuna1.5-7B and CrystalChat-7B, respectively. CrystalChat-7B is a code-enhanced LLM and our visualization demonstrates that it achieves the better quality of generation than Vicuna1.5-7B even the performance is slightly worse on general multimodal domain, as presented in Table 6. Moreover, as in Figure 9, our rendered webpage from the model trained on our web dataset closely resembles the original image, indicating the positive impact of the web2code dataset. We further visualize our generation in Figure 9 when the input is a hand-drawn webpage to examine the adaptation ability of our model.

(a) Original

(b) Our Vicuna1.5-7B

(c) Our CrystalChat-7B

Figure 7: Visualization comparison using different backbones. Using the code-enhanced LLM backbone CrystalChat-7B achieves better quality of generation than Vicuna1.5-7B.

(a) Original

(b) GT-Code to Image

(c) Ours

Figure 8: Visualization comparison between ground-truth code generated image and our result. The style and layout of the generated webpage image are similar to the ground-truth image.

(a) Hand drawn webpage and our generation

(b) Hand drawn webpage and our generation

Figure 9: Visualization of our CrystalChat-7B generation when the input is a hand-drawn webpage.

5 General Evaluation of MLLMs Using Web2Code

Setup and Overview. Our model training framework mainly follows the design of LLaVA-1.5 [33] where we leverage the capabilities of both a pre-trained visual encoder, an LLM and a projector to connect visual features into the word embedding space. The model consists of (1) a pre-trained CLIP ViT-L/14 [44] visual encoder with a resolution of 336$\times$336 and a patch size of 14, which has good feature representation already aligned with the text embedding space. (2) As for the LLM backbones, we leverage CrystalChat [37] as the base model and compare it with other latest LLM backbones like Vicuna1.5 [11], LLaMA2 [54], LLaMA3 [1] and CrystalCoder [37]. Training details and hyperparameters are presented in the Appendix A.

General Evaluation Metrics for MLLMs. MME [16] serves as an extensive evaluative benchmark, aiming to assess the perceptual and cognitive capability of MLLMs within 14 sub-tasks. Additionally, we also evaluate the performance of our models on text-oriented visual question-answering tasks employing a diverse set of benchmark datasets including ScienceQA [39] and TextVQA [51]. Furthermore, We assess our models’ ability toward anti-hallucination through POPE [29].

LLM Backbone	DWU	DWCG	DWU${}_{\texttt{R}}$	DWCG${}_{\texttt{R}}$	MME-P	MME-C	POPE	SciQA	TextVQA
CrystalChat-7B [37]	-	-	-	-	1456.53	308.21	86.86	67.77	57.84
	✓	-	-	-	1438.51	292.14	87.10	68.27	58.15
	✓	✓	-	-	1478.82	297.14	86.13	67.92	57.41
	✓	✓	✓	✓	1449.54	279.64	86.53	68.32	57.86

Table 5: Component analysis on CrystalChat-7B backbone under various data configurations. We note that the general domain data (i.e., LLaVA) is included in all data configuration as default.

Effects of Web2Code on General Domain. Here, we first perform instruction tuning using Web2Code on various LLM backbones and then we evaluate those MLLMs on the general domain of visual language understanding. Throughout extensive experiments under various data configurations, we observed that the proposed dataset Web2Code can be incorporated with the conventional visual language instruction tuning dataset of LLaVA [33] without harming performances on the general domain. Table 5 summarizes the results. Specifically, both proposed Web Understanding data (DWU or DWU${}_{\texttt{R}}$) and Web Code Generation data (DWCG or DWCG${}_{\texttt{R}}$) do not hurt or even can be beneficial to the visual language understanding. For example, we observed that adding DWU to CrystalChat achieves comparable or even better performances on POPE (86.86$\rightarrow$87.10), SciQA (67.77$\rightarrow$68.27), and TextVQA (57.84$\rightarrow$58.15). Somewhat surprisingly, we further found that adding DWCG can even improve visual language understanding. For example, the second and third rows of CrystalChat show +40.31 and +5.00 points higher improvements in MME-P and MME-C benchmarks, respectively. Moreover, adding refined data DWU${}_{\texttt{R}}$ and DWCG${}_{\texttt{R}}$ are still effective in the visual language domain, by achieving comparable (or even better) performances on overall benchmarks. For example, the last row indicates that adding DWU${}_{\texttt{R}}$ and DWCG${}_{\texttt{R}}$ preserves comparable performances on overall benchmarks and even achieves +0.4 higher points on the SciQA benchmark.

6 Conclusion

We have presented Web2Code, a benchmark that consists of a high-quality, large-scale webpage-to-code instruction tuning dataset containing 1179.7k entries and an evaluation suite for the webpage understanding and webpage-to-HTML translation abilities of MLLMs. Through extensive experiments, we have demonstrated that our proposed dataset is clearly effective at enhancing these abilities of MLLMs as well as general visual proficiency, while existing datasets lead to inferior performance. We hope our work will attract the community’s attention and facilitate progress toward foundation models serving as virtual assistants for content generation and task automation.

Limitations and Ethics Statement. The Web2Code project provides a comprehensive dataset and evaluation framework for fine-grained multimodal large language models. This can significantly enhance the capabilities of LLMs in understanding and generating web code from instructions, leading to advancements in web development automation, improved coding assistance tools and platforms. By enabling more accurate and context-aware code generation, it can boost productivity for developers and make coding more accessible to beginners. However, the primary limitations of the Web2Code include the potential for biased dataset that may not cover all possible HTML coding scenarios, potentially leading to gaps in model performance, some webpages that include humans may be privacy sensitive. Ensuring high-quality annotations and comprehensive coverage of all possible HTML and code structures is challenging. Also, handling complex, real-world HTML and code scenarios might still be beyond the current capabilities of the models trained on this dataset. Moreover, the proposed evaluation framework may not capture all aspects of the code generation quality, such as code efficiency, readability, or adherence to best practices.

Appendix

Appendix A Training Details and Hyperparameters

We follow the instruction-tuning protocol of LLaVA-1.5 [33]. In the pretraining step, we employ the caption data to optimize the projector, while kee** the vision encoder and LLM frozen. Meanwhile, we optimize the projector and LLM in the instruction tuning step. During the pretraining phase, we utilize a batch size of 256, while for the instruction tuning phase, we employ a batch size of 128. The learning rate is set at $1e^{-3}$ during pretraining and adjusted to $2e^{-5}$ for instruction tuning, with both phases incorporating a cosine decay schedule. We also apply a learning rate warmup with a decay factor of 0.03, and no weight decay is used. Both pretraining and instruction tuning are conducted for one epoch each, consistently using the AdamW optimizer.

Appendix B More Effects of Web2Code on General Domain

Here, we first perform instruction tuning using Web2Code on various LLM backbones and then we evaluate those MLLMs on the general domain of visual language understanding.

Comparison on different LLM backbones. We compare the general domain abilities of various LLM backbones under the same data configuration of LLaVA + DWU + DWCG; Table 6 summarizes the results of instruction-tuned MLLMs. Specifically, we found that instruction-tuned CrystalChat-7B, Vicuna1.5-7B, and LLaMA2-7B show superior performances in the general domain compared to CrystalCoder and CodeLlama. For example, CrystalChat shows +132.89 points higher than CodeLlama in MME-P (i.e. perception domain). Somewhat surprisingly, instruction-tuned CrystalChat showed the strongest performance on TextVQA, which requires visual reasoning based on text in images.

LLM Backbone	MME-P	MME-C	POPE	SciQA	TextVQA
CodeLlama-7B [45]	1345.93	258.92	85.28	61.87	55.23
CrystalCoder-7B [37]	1351.22	274.64	86.05	61.63	50.11
CrystalChat-7B [37]	1478.82	297.14	86.14	67.92	57.41
Vicuna1.5-7B [11]	1488.26	268.21	87.05	69.31	56.40
LLaMA2-7B [54]	1448.58	304.29	86.87	67.87	55.62

Table 6: Comparison of different LLM backbones on visual language understanding benchmarks. All models are instruction-tuned on the general domain data (i.e. LLaVA) with DWU and DWCG.

Appendix C Qualitative Data Examples in WUB Benchmark

The qualitative data examples in our WUB benchmark are shown in Figure 10. It covers different aspects of webpage understanding based on "yes" / "no" question-answer pairs.

Figure 10: Qualitative data examples in our WUB benchmark. It covers different aspects of webpage understanding based on "yes" / "no" question-answer pairs.

Appendix D WCGB Criteria

Our proposed WCGB framework consists of 10 distinct criteria, which we group into four categories, each encompassing specific criteria that are scored on a 0-10 scale:

1. Visual Structure and Alignment

   • Layout Consistency: Measures the arrangement of structural webpage elements like headers, footers, and sidebars.

   • Element Alignment: Assesses the alignment of images, buttons, and text boxes.

   • Proportional Accuracy: Checks for consistency in sizes and aspect ratios of visual elements.

   • Visual Harmony: Examines the overall balance and harmony in design.

2. Color and Aesthetic Design

   • Color Scheme and Aesthetic Match: Focuses on the similarity in color schemes, including hues and saturation.

   • Aesthetic Resemblance: Looks at the overall aesthetic appeal and style (modern, minimalistic, traditional, etc.).

3. Textual and Content Consistency

   • Font Characteristics and Consistency: Assesses uniformity in font type, size, style, and weight.

   • Textual Content Match: Evaluates the match in words and sentences.

   • Numeric and Special Character Accuracy: Checks for consistency in numbers, dates, and special characters.

4. User Interface and Interactivity

   • User Interface Consistency: Assesses the similarity in design language and appearance of UI elements like menus, buttons, and forms.

Appendix E Prompt Templates

E.1 Prompt Used to Question and Answer Generation for DWU Data

Figure 11: Prompt used to generate Question Answer pairs using GPT4 for DWU data.

E.2 Prompts Used For Instruction generation of DWCG and DWCG${}_{\texttt{R}}$

Figure 12: Prompt used to generate instructions for DWCG using GPT4, feeding input as Seed instructions and output as GPT generated instructions shown in Figure 16.

Figure 13: Prompt used to generate webpage style instruction for DWCG using GPT4, feeding input as Seed instructions and output as GPT generated webpage style instructions shown in Figure 16.

E.3 Prompt Used For GPT4-Vision Evaluation in WCGB benchmark

Figure 14: The prompt utilized for evaluation in WCGB with the employment of GPT4-Vision.

E.4 Prompt Used For QA Generation for WUB benchmark

Figure 15: The prompt employed to generate "yes" / "no" Question Answer pairs for the WUB benchmark through the utilization of GPT4-Vision.

Appendix F Data samples

Figure 16: Examples of webpage to code generation instruction tuning data. These web image-code pairs were converted into an instruction-following data format close to the LLaVA data format.

Figure 17: Examples of webpage to code generation instruction tuning data in DWCG. These web image-code pairs were converted into an instruction-following data format close to the LLaVA data format.

Figure 18: Examples of webpage to code generation instruction tuning data in DWCG${}_{\texttt{R}}$. The refined instruction tuning dataset for webpage code generation, by utilizing GPT-4.

Figure 19: Examples of webpage understanding data in DWU. For a webpage, there is a set of diverse question-answer pairs about web understanding.

Figure 20: Examples of webpage understanding data in DWU${}_{\texttt{R}}$. The refined instruction tuning dataset for web understanding tasks.

Figure 21: Examples of Modern styles webpages.

Figure 22: Examples of Bootstrap styles webpages.