A Suite of Generative Tasks for Multi-Level Multimodal Webpage Understanding (2023)

Abstract: Webpages have been a rich, scalable resource for vision-language and language only tasks. Yet only pieces of webpages are kept in existing datasets: image-caption pairs, long text articles, or raw HTML, never all in one place. Webpage tasks have resultingly received little attention and structured image-text data left underused. To study multimodal webpage understanding, we introduce the Wikipedia Webpage suite (WikiWeb2M) containing 2M pages with all of the associated image, text, and structure data. We verify its utility on three generative tasks: page description generation, section summarization, and contextual image captioning. We design a novel attention mechanism Prefix Global, which selects the most relevant image and text content as global tokens to attend to the rest of the webpage for context. By using page structure to separate such tokens, it performs better than full attention with lower computational complexity. Extensive experiments show that the new data in WikiWeb2M improves task performance compared to prior work.

## introduction webpages are a source of multimodal, structured content which have been used for both pretraining and finetuning purposes. large scale noisy text or multimodal datasets scraped from the web have been used to pretrain large language or contrastive models @xcite @xcite . downstream tasks built from webpages have included instruction following, image captioning, news captioning, image-sentence retrieval, and image-article retrieval @xcite @xcite @xcite @xcite . figure 1 : tasks we study with wikiweb2m. our dataset provides a unified webpage sample that contains all text, image, and structure, enabling new tasks like page description generation. for image captioning and section summarization, remaining page text and images provide useful context, aiding task performance. yet limited prior work has studied tasks to evaluate multimodal webpage understanding itself. many classification and generation problems can be studied with webpages: taxonomic webpage classification, webpage retrieval, web image captioning, and page summarization. however, to date there is no open source, multimodal dataset that retains all webpage content. e.g., the wikipedia image text (wit) dataset @xcite does not retain html structure and misses out on text sections. thus, we propose the new wikipedia webpage (wikiweb2m) dataset of over 2m pages, which unifies webpage content to include all text, images, and their location (e.g., section index) in a single sample. figure 1 shows an example of our wikiweb2m benchmark suite; we design a set of tasks that require webpage understanding at varying degrees of dataset # webpage sections # images structural heading text image both total unique total wit (en) ---199,872 2,847,929 3,047,801 3,660,211 4,955,835 @xcite @xcite @xcite @xcite @xcite @xcite @xcite wikiweb2m's tasks will allow for general study of multimodal content understanding with manyto-many text and image relationships and can also specifically improve interaction with web content. for example, a webpage description may provide a user who is blind more agency by allowing them to preview content before listening to the entire body of image and text with a screen reader @xcite . in addition to contextual captioning and section summarization aiding assistive technology, these tasks can be used for modern content generation, as there is growing interest in providing multimodal web snippets @xcite . the study of webpages in a multimodal context has even been motivated from a sociological and anthropological perspective @xcite . while we curate a new dataset with wikipedia, we note it is just one of many domains that could be used to study multimodal webpage understanding. instructional websites, news articles, recipes, blogs, and more have bodies of text and images interleaved by layout or html structure. we utilize the t5 @xcite ) framework to address the wikiweb2m tasks. one challenge in modeling webpage tasks is the length of the input data (i.e., a long sequence results from flattening webpage text and images). while the full attention originally used in t5 is performant, it results in a quadratic computational complexity with respect to the input sequence length. thus, we define a new mixture of local-global attention, prefix global, which uses our structured webpage data to select the most salient text and images as global tokens in the prefix of our input sequence. prefix global is ultimately more efficient, meaning longer input sequences can be used to reach better task performance. our results can be beneficial to the many structured image-text domains outside of webpages such as mobile apps, figures, posters, infographics, and documents. we include ablations across multiple axes: the pretrained checkpoint we initialize from, the input sequence length, the feature inputs, and the attention mechanism. we importantly find that images improve performance for all tasks, while prior work on contextual image captioning claimed otherwise @xcite . we are also able to improve task performance now that we have access to the entire page's content. still, there is plenty of room to improve upon our benchmark suite. we summarize our contributions below: ## the wikiweb2m dataset we create the wikipedia webpage (wikiweb2m) dataset to have an all-in-one multimodal webpage dataset where all text, image, and structure content is retained. wikiweb2m is built by starting with the wikipedia image text (wit; @xcite english pages foot_0 . we re-scrape webpage samples and keep all text, image, and structure available, providing more contextual data which can be used to model existing tasks like contextual image captioning, as well as enable new webpage understanding tasks like page description generation. we start with wit urls to create a high quality multimodal webpage dataset that has already gone through extensive content and topic filtering. each webpage sample includes the page url, page title, section titles, section text, images and their captions, and indices for each section, their parent section, their children sections, and more. this differs from wit, which defined individual samples as image-caption pairs with metadata (e.g., originating section title). appendix a.3 includes a comparison of fields available in wikiweb2m versus wit. in we provide section counts by type: structural, heading, text only, image only, and both text and image. structural and heading sections do not contain immediate text. the former has subsections. for heading sections, a section title was available, while the content linked to a different article, was empty, or only had tables. we only retain sections if they are content sections (e.g., not the "see also" section). a significant 6.8m text sections are in wikiweb2m, none of which were available in wit. for image quality control, we keep jpeg and png image types 3 . we make a random 90/5/5 split and ## prefix global attention when structured image-text data is available, we need not treat all images and text equally. with webpages, it may be more sensible to isolate certain parts as more important. e.g., in contextual image captioning, the model should focus on the target image and section it came from, while using the rest of the page as additional context. we can now isolate these inputs with the wikiweb2m dataset because we have structural metadata signaling where each image and text element are located, as opposed to a bag of images and a single long body of text. i.e., the new structure available in our dataset can both serve as new inputs to the model and enable new attention mechanisms. we thus propose prefix global, a local-global attention, to capitalize on this intuition. a mixture of local and global attention weights provides the means to designate certain inputs as "global" tokens which can specially attend to the rest of the input sequence, while others only have local attention to a radius of r tokens to the left and right. not only is it desirable to prioritize more salient image and text content from the input data, but it can also reduce the computational complexity of the attention mechanism. while full attention is performant by allowing all input tokens to attend to each other, it results in a quadratic computational complexity (o(l 2 ) for a sequence of length l). figure 2 illustrates our prefix global and prior work's transient global attention schemes, where in each the ith row represents what the ith token can attend to. @xcite introduced longt5 as an adaptation of the t5 model with transient global (tglobal) attention to balance the efficiency of local attention, which allows for much longer input sequences to be held in memory, with the higher performance of full attention. tglobal resulted in similar or better performance than full attention with much longer sequence lengths, while having a complexity of o(l(r + k)) for a variety of text summarization tasks (where @xmath0. in addition to the local attention of tglobal (see the blue attention diagonal in figure 2 (left)), "transient" global tokens are defined on the fly per layer. tglobal defines k globals as the average of every 16 input tokens, which are additionally attended to by all other inputs. as a result, tglobal has more global tokens as the sequence length increases. in contrast, shown in figure 2 (right), prefix global uses a constant number of global tokens. specifically, it takes a prefix of the input sequence. this is inspired by the leading sentence bias @xcite @xcite , which shows that earlier content in a body of text is often of greater importance. we define different prefixes for each task in section 4. while we use section structure to define our prefixes, prefix global can use structure from other sources: html/the document object model, rendered webpage regions, pdf document layouts, or simply knowing a priori what task inputs are most salient. prefix global has a computational complexity of o((lk) • r + k • l) for k global tokens, similar to local-global attention schemes etc @xcite , longformer @xcite , and big-bird @xcite . however, prefix global does not require any special pretraining and instead finetunes directly from full attention checkpoints (t5 in our case). this is distinct from longt5, which also required pretraining with tglobal attention to be effective. thus, as we show in section 5 with prefix global's higher performance, it is both a more flexible and performant attention. we also are the first to demonstrate using a local-global attention with multimodal inputs, and further show prefix global's ability to be performant in multimodal finetuning from a text-only checkpoint. ## experiments we now detail the model variants used for experiments, parameter settings for reproducing our set up, the metrics used for evaluation, and key ablations we perform. ## results we now detail experimental results, first evaluating performance and efficiency of each attention type at different sequence lengths. then, we report input feature, model size, and annotation ablations. ## related work webpage tasks have been studied with text only html for web element classification, html description generation, and web navigation. @xcite proposed finetuning large language models for these tasks. reinforcement learning methods have also trained agents to perform language commands in handcrafted web environments @xcite @xcite . wikipedia has previously been used to develop downstream tasks. for example, wit @xcite released image-caption pairs from wikipedia, in addition to some contextual section text. while wit does not contain all of the page content, @xcite studied contextual image captioning with the available annotations. this is a webpage task and not strictly an imagetext problem, as additional section text is included to aid in wikipedia image captioning, where captions often contain finer-grained, knowledge based information. atomic also studied ways to improve multimodal retrieval with wikipedia by defining more realistic evaluation sets @xcite . @xcite proposed cm3, a transformer with a causally masked pretraining objective. while cm3 relied on pretraining data from the web containing the images and html of a webpage, this dataset was not open sourced. their results illustrated that rich html data could be used to learn representations for tasks such as image generation, image in-filling, and entity disambiguation and linking. this demonstrates that webpage data can generalize to non-webpage tasks, but leaves webpage specific problems unexplored. to our knowledge there is no open source multimodal webpage data that captures all modalities. c4 was recently extended to a multimodal version, mmc4 @xcite . however, mmc4 does not retain structure, and instead uses clip scores to noisily match images to chunks of text that it could be aligned with. @xcite has not yet been used for pretraining or downstream applications. in mobile apps, the closest domain to webpages, there are two open source datasets that contain all modalities (text, image, and structure): rico @xcite and motif @xcite . ## conclusion in this paper we study three generative tasks for multimodal webpage understanding: page description generation, section summarization, and contextual image captioning. to do so, we present the wikiweb2m dataset, which retains all of the text, images, and structure from more than 2m pages. we propose a new attention, prefix global, which outperforms full attention by allowing the most salient text and images to specially attend to all inputs. extensive ablations on attention mechanism, sequence length, model size and checkpoint, input features and section type reveal the most impactful factors on our benchmark suite and verify using wikiweb2m to study webpage understanding.