Table and Image Generation for Investigating Knowledge of Entities in Pre-trained Vision and Language Models (2023)

Abstract: In this paper, we propose a table and image generation task to verify how the knowledge about entities acquired from natural language is retained in Vision & Language (V & L) models. This task consists of two parts: the first is to generate a table containing knowledge about an entity and its related image, and the second is to generate an image from an entity with a caption and a table containing related knowledge of the entity. In both tasks, the model must know the entities used to perform the generation properly. We created the Wikipedia Table and Image Generation (WikiTIG) dataset from about 200,000 infoboxes in English Wikipedia articles to perform the proposed tasks. We evaluated the performance on the tasks with respect to the above research question using the V & L model OFA, which has achieved state-of-the-art results in multiple tasks. Experimental results show that OFA forgets part of its entity knowledge by pre-training as a complement to improve the performance of image related tasks.

## introduction vision & language (v&l), which is the fusion of vision and language tasks, has achieved great success in tasks such as caption generation from images @xcite and image generation from texts @xcite . this progress has been driven by pre-trained v&l models that are trained on large-scale v&l datasets @xcite . to generate appropriate captions and images for input, pre-trained v&l models need to have prior knowledge of the features of the objects they are generating @xcite . these models retain knowledge about entities in particular by inheriting parameters from pre-trained language models used in natural language processing to indirectly utilize data resources such as wikipedia. in this way, v&l models @xcite @xcite @xcite title image caption table table generation title caption (table) image generation title (image) et al., 2022; saharia et al., 2022) map the inherited textual knowledge into visual representations through additional training on v&l datasets. this learning process raises a number of questions, such as whether the knowledge about entities acquired from natural language is adequately retained in the pre-trained v&l model, or whether it is enhanced by combining it with image features. these are important in understanding the limits of what can be generated by the pre-trained v&l model. to answer these questions, we propose a task of generating tables and images of infoboxes in english wikipedia. figure 1 shows an example of the target infobox, in which either tables or images are generated by the proposed task. in both cases, the model must know the entities to generate them properly. we collected about 200,000 infoboxes to construct the wikipedia table and image generation (wikitig) dataset necessary to perform the proposed task. in addition, we used ofa @xcite , a pre-trained v&l model that has achieved state-of-the-art performance in various v&l tasks. our evaluation of the table generation revealed that part of the knowledge in the v&l model acquired from natural language is lost when the v&l model is pre-trained. we also found that additional knowledge for entities was acquired by supplementing image information, which was not possible solely from textual data. in image generation, we found that ofa can generate more accurate images by using the knowledge expressed in the table. we also found that the models trained only on natural language can infer table knowledge, which increases the diversity of generated images. our code and dataset will be released at https://github.com/kamigaito/wikitig . ## vision & language models many pre-trained v&l models have achieved stateof-the-art performance on various tasks by inheriting the weights of the conventional pre-trained models for natural language and images @xcite @xcite @xcite before learning v&l datasets. our study examines how the knowledge represented in the pre-trained model for natural language is transformed through such a learning process. we select ofa, which has achieved state-of-the-art performance in multiple v&l tasks, as our target model. figure 2 shows the network structure of ofa and its relation to each dataset 2 . ofa uses vqgan @xcite on the decoder to transform images into discrete sequences so that the same transformer @xcite is used for image and natural language generation. because ofa inherits 2 appendix a describes the data for the pre-training. ## table and image generation in this section, we describe two tasks for verifying knowledge behavior in the v&l model: in the following subsections, we explain the details of each task. ## dataset creation we created the wikipedia table and image generation (wikitig) dataset by extracting infoboxes from the html dump data of the english wikipedia foot_4 . to ensure consistency in the format of infoboxes, we limited the extraction target to those containing a title in the first row and an image in the second row, as shown in figure 1 . in order to use only entities with sufficient information, we targeted entities for which the table was not empty. in addition, to ensure reliable correspondence, only rows one column wide, which often describe groups, and rows two columns wide, which often consist of a header and its value, were targeted for extraction. the target images are limited to those in jpeg, png, and gif formats. since some captions do not include a title, we used a hyphen to join the title at the beginning of the caption in such cases. ## related work following the advancements in v&l models @xcite , there have been various studies that investigate v&l models. @xcite conducted a comprehensive analysis of v&l models including the difference between model structures. through their analysis, they revealed the importance of text information in v&l tasks over image information. several studies focused on the performance differences between v&l models and text-only models. @xcite investigated the improvement of linguistic representations by pre-training v&l models on physicalqa (piqa) @xcite and the probing framework of @xcite . they concluded that the benefit of pretrained v&l models for text-only tasks is marginal. @xcite ; hagström and johansson (2022) compared the performance of v&l models and text-only models on the text-only benchmark, glue @xcite and determined that the text-only model achieved higher scores than the v&l models. however, even though various kinds of v&l models @xcite @xcite @xcite inherit language-related knowledge from pre-trained language-only models, how the knowledge is inherited has yet to be investigated. our work clarifies this by using our created dataset, wikipedia table and image generation (wikitig). ## conclusion this paper investigates how knowledge about entities are preserved in a pre-trained v&l model which is originally transferred from a pre-trained natural language model. we analyzed a pre-trained v&l model by creating the wikipedia table and image generation (wikitig) dataset for generating images and tables of the infoboxes in wikipedia. wikitig consists of 200,000 infoboxes and their corresponding images from english wikipedia. experimental results on a pre-trained v&l model ofa @xcite showed that the model forgot part of the knowledge about entities during pre-training, and the image information did not fully compensate for the forgotten knowledge.