V is T o T : Vision-Augmented Table-to-Text Generation (2022)

Abstract: Table-to-text generation has been widely studied in the Natural Language Processing community in the recent years. We give a new perspective to this problem by incorporating signals from both tables as well as associated images to generate relevant text. While tables contain a structured list of facts, images are a rich source of unstructured visual information. For example, in the tourism domain, images can be used to infer knowledge such as the type of landmark (e.g., church), its architecture (e.g., Ancient Roman), and composition (e.g., white marble). Therefore, in this paper, we introduce the novel task of Vision-augmented Table-To-Text Generation (VisToT, defined as follows: given a table and an associated image, produce a descriptive sentence conditioned on the multimodal input. For the task, we present a novel multimodal table-to-text dataset, WikiLandmarks, covering 73,084 unique world landmarks. Further, we also present a competitive architecture, namely, VT3 that generates accurate sentences conditioned on the image and table pairs. Through extensive analyses and experiments, we show that visual cues from images are helpful in (i) inferring missing information from incomplete or sparse tables, and (ii) strengthening the importance of useful information from noisy tables for natural language generation. We make the code and data publicly available.

## introduction structured data-to-text generation is a well-studied problem that demands a model to produce meaningful and factually accurate sentences based on its comprehension of the source context that are usually in the form of fact graphs, tables, or hierarchical spreadsheets @xcite . many datasets and several competitive 1 https://vl2g.github.io/projects/vistot conventional transformer-based table -to-text: "lough leane is a 4700 acre estate in killarney, county kerry, ireland." vision-augmented table -to-text (this work): "lough leane is a large lake in killarney, county kerry, ireland." ground truth: "lough leane is the largest of the three lakes of killarney, in county kerry." -to-text. in this novel proposed task, the goal is to explore the utility of visual cues besides the information in the table for natural language generation. note the presence of large lake in the text generated by our proposed model; on the other hand, existing table-to-text models wrongly generate estate in the output text. specifically, for table-to-text, several benchmark datasets have been proposed including wik-ibio @xcite , webnlg @xcite , e2e @xcite , dart @xcite , @xcite , and wikipedia person and animal dataset @xcite . these datasets vary in terms of their lexical richness, syntactic variation, and semantic and linguistic adequacy. further, there are several recent methods utilizing neural encoder-decoder frameworks to generate text descriptions from tables @xcite @xcite . despite this progress, the current table-to-text literature has underexplored the challenge of including a multimodality requirement in this task. while tables provide a series of facts at the conceptual level, images provide unstructured visuallevel signals. therefore, the information in the two modalities can be complementary and may lead dataset domain #data-has text pairs images? weathergov @xcite weather 22.1k ✗ wikibio @xcite biography 728.3k ✗ rotowire @xcite baseketball 4.9k ✗ webnlg @xcite selected dbpedia categories 25.3k ✗ e2e @xcite restaurants 50.6k ✗ logicnlg @xcite open domain 37.0k ✗ totto @xcite open domain 136.2k ✗ wikilandmarks (ours) world 73.2k ✓ landmarks to significant improvements in generated text, especially for domains such as tourism, dining, and products, where images contain rich information. images could also act as a significant source of additional information when tables are sparse or noisy. hence, in this paper, we propose an extension to the typical table-to-text task, where not just tables but also associated images are needed to generate meaningful and complete sentences. we refer to this novel task as vision-augmented table -to-text generation or vistot. figure 1 shows an example for this task. note the presence of "large lake" in the text generated by our proposed model; on the contrary, existing table-to-text models wrongly generate "estate" in the output text. unlike table-to-text datasets that only contain (table, text) pairs, vistot requires a dataset that maps (table, image) pair to a text. hence, we contribute a new dataset -wikilandmarks, with infoboxes (tables) corresponding to 73,084 unique landmarks, ∼10 images on average per landmark, and a descriptive text with ∼35 tokens on average. further, to perform vistot on wikiland-marks, we first experiment with standard previously proposed methods of encoding table and images. next, to handle the joint tables and image information more accurately, we propose a novel model, namely visual-tabular data-to-text transformer or vt3. for encoding image information, we experiment with faster rcnn @xcite , vit @xcite , clip-vit @xcite and swin @xcite . for encoding tables, we represent key-value pairs in a sequence. both the table and visual encodings are fed as input to our model initialized from the pretrained bart @xcite to generate text. to adapt the proposed model for our task, we present three novel pretraining strategies, namely image- ## vt3: visual-tabular data-to-text transformer to address vistot, we present a novel trainable neural architecture namely visual-tabular datato-text transformer (or vt3 in short). vt3 is a bart-based encoder-decoder transformer model. the overall architecture of vt3 and our proposed novel pretraining tasks are illustrated in figure 4 . next, we describe the architectural details of vt3 followed by proposed pretraining strategies. ## conclusion we have highlighted the possible benefits of leveraging multimodal data for better sentence generation by introducing a novel task, viz. vistot and a large-scale accompanying dataset namely wiki-landmarks. we presented a competitive model -vt3 that is trained using three novel task-specific pertaining strategies, namely, image-table matching, masked value modeling, and image captioning. vt3 shows impressive performance gain over those models which only rely on tabular data for table -to-text generation. in general, inspired by our proposed novel task and dataset, we look forward to exciting future research on bridging multimodal cues for more precise nlg.