Update README.md

README.md

@@ -22,20 +22,20 @@ and Evaluation Framework for Multimodal LLMs](https://arxiv.org/pdf/2406.20098).
 ## Web2Code Dataset
 Our Web2Code instruction tuning dataset construction and instruction generation process
 involves four key components: 
-1.  Creation of new webpage image-code pair data: We generated
-high-quality HTML webpage-code pairs following the CodeAlpaca prompt [6] using GPT-3.5 and
-convert them into instruction-following data. (2) Refinement of existing webpage code generation
-data: We transform existing datasets including WebSight [ 22 ] and Pix2Code [ 4] into an instruction-
-following data format similar to LLaVA data [33 ], so they can be used as instruction-following data
-to train MLLMs. (3) Creation of a new text question-answer pair data: We generated a new question-
-answer pair dataset utilizing our new GPT-3.5 generated data from (1) for webpage understanding.
-(4) Refinement of existing webpage understanding data: We refine the WebSRC [ 10] question-answer
-data to improve its quality using the GPT-4. 
+1. Creation of new webpage image-code pair data **(DWCG)**: We generated high-quality HTML webpage-code pairs following the CodeAlpaca prompt using GPT-3.5 and convert them into instruction-following data.
+2. Refinement of existing webpage code generation data **(DWCG<sub>R</sub>)**: We transform existing datasets including WebSight and Pix2Code into an instruction- following data format similar to LLaVA data, so they can be used as instruction-following data to train MLLMs.
+3. Creation of a new text question-answer pair data **(DWU)**: We generated a new question-answer pair dataset utilizing our new GPT-3.5 generated data for webpage understanding.
+4. Refinement of existing webpage understanding data **(DWU<sub>R</sub>)** : We refine the WebSRC question-answer data to improve its quality using the GPT-4. 
+
+The Web2Code instruction tuning dataset was released in [Web2Code: A Large-scale Webpage-to-Code Dataset and Evaluation Framework for Multimodal LLMs](https://huggingface.co/datasets/MBZUAI/Web2Code).
 
 ## Evaluations
 
-## Webpage Code Generation Benchmark
+## Webpage Code Generation Benchmark (WCGB)
+
+Utilizing the same images as the WUB, this benchmark evaluates a multimodal model tasked with generating HTML code from webpage images based on specific instructions. Unlike traditionalcode-level evaluations, this benchmark assesses the generated webpage’s fidelity at the image level. We convert the predicted HTML codes back into images using Selenium WebDriver to allow a direct visual comparison with the ground truth images. The evaluation, depicted on the left side of Figure 6, considers 10 different aspects, which are further categorized into four evaluation matrices using the GPT-4 Vision API. 
 
+### Results 
 | LLM Backbone           | DWCG | DWU | DWCG<sub>R</sub> | DWU<sub>R</sub> | VSA ↑  | CAD ↑  | TCC ↑  | UII ↑  | Overall ↑ |
 |------------------------|------|-----|------------------|------------------|--------|--------|--------|--------|------------|
 | **CrystalChat-7B**     |      |     |                  |                  | 4.714  | 4.572  | 4.865  | 5.147  | 4.825      |
@@ -46,11 +46,11 @@ data to improve its quality using the GPT-4.
 |                        | ✓    |     |                  |                  | 6.871  | 6.660  | 6.589  | 6.897  | 6.754      |
 |                        |      | ✓   |                  |                  | 3.898  | 3.489  | 3.340  | 3.651  | 3.595      |
 |                        | ✓    | ✓   | ✓                | ✓                | **7.876**  | **7.687**  | **7.267**  | **7.563**  | **7.598**      |
-
+| **Llama3-8B**          | ✓    | ✓   | ✓                | ✓                | **8.522**  | **8.564**  | **8.421**  | **8.611**  | **8.530**      |
 **Table 1:** The performance of different LLM backbones under various data configurations on our Webpage Code Generation Benchmark (WCGB). "VSA" denotes Visual Structure and Alignment, "CAD" represents Color and Aesthetic Design, "TCC" represents Textual and Content Consistency, and "UII" denotes User Interface and Interactivity.
 
-## Webpage Understanding Accuracy
-
+## Webpage Understanding Benchmark (WUB)
+### Results
 | LLM Backbone           | DWCG | DWU | DWCG<sub>R</sub> | DWU<sub>R</sub> | Accuracy (%) |
 |------------------------|------|-----|------------------|------------------|--------------|
 | **CrystalChat-7B**     |      |     |                  |                  | 73.94        |
@@ -60,6 +60,7 @@ data to improve its quality using the GPT-4.
 |                        | ✓    |     |                  |                  | 68.11        |
 |                        |      | ✓   |                  |                  | 70.82        |
 |                        | ✓    | ✓   | ✓                | ✓                | **71.23**    |
+| **Llama3-8B**          | ✓    | ✓   | ✓                | ✓                | **74.84**    |
 
 **Table 2:** The accuracy of webpage understanding under various data configurations and LLM backbones.
   All models are instruction-tuned and evaluated on our WUB benchmark. We note that the general domain data (i.e., LLaVA) is included in all data configuration as default.
@@ -75,7 +76,7 @@ data to improve its quality using the GPT-4.
 
 
 
-## Evaluation
+## General Evaluations
 
 General Evaluation Metrics for MLLMs. MME serves as an extensive evaluative benchmark,
 aiming to assess perceptual and cognitive capability of MLLMs within 14 sub-tasks. Additionally, we also evaluate the performance of our models on text-oriented visual question answering tasks employing a diverse set of benchmark datasets including ScienceQA and TextVQA. Furthermore, we assess our models’ ability toward anti-hallucination through POPE. 
@@ -88,14 +89,11 @@ aiming to assess perceptual and cognitive capability of MLLMs within 14 sub-task
 
 **Table 3:** Comparison of different LLM backbones on visual language understanding benchmarks. All models are instruction-tuned on the general domain data (i.e. LLaVA)*
 
-
-
 ## Data and Training Details
 
 ### Pretrain Data
 LLaVA Visual Instruct Pretrain LCS-558K is a filtered subset of the LAION, CC, and SBU datasets, featuring a more balanced distribution of concept coverage. The file includes multimodal synthesized conversations generated from image-caption pairs by incorporating randomly selected instructions such as "Describe this image." It is used for pretraining in LLaVA, with the raw CC-3M caption serving as the default answer.
 
-TO-DO: Add new short caption data
 ### Finetune Data
 The finetuning data contains the following:
 
@@ -120,19 +118,6 @@ The dataset chosen was created by LLaVA with academic-task-oriented VQA data mix
 
 **Table 4:** Instruction-following Data Mixture of LLaVA-1.5.*
 
-#### Web2Code Data
-
-The Web2Code instruction tuning dataset was released in [Web2Code: A Large-scale Webpage-to-Code Dataset
-and Evaluation Framework for Multimodal LLMs](TODO: Add link). The dataset construction and instruction generation process involves four key components:
-
-DWCG: We created new webpage image-code pair data DWCG by generating high-quality HTML webpage-code pairs following the CodeAlpaca prompt using GPT-3.5 and converting them into instruction-following data.
-
-DWCG<sub>R</sub>: We refined existing webpage code generation data by transforming existing datasets, including WebSight and Pix2Code, into an instruction-following data format similar to LLaVA data, so they can be used as instruction-following data to train MLLMs.
-
-DWU: We created new text question-answer pair data by generating a new question-answer pair dataset utilizing our new GPT-3.5 generated data for webpage understanding.
-
-DWU<sub>R</sub>: We refined the WebSRC question-answer data to improve its quality using GPT-4.
-
 ### Code Datasets
 
 | Dataset | DWCG (ours) | DWCG<sub>R</sub> (ours) |