Unified Language Representation for Question Answering over Text, Tables, and Images (2023)

Abstract: When trying to answer complex questions, people often rely on multiple sources of information, such as visual, textual, and tabular data. Previous approaches to this problem have focused on designing input features or model structure in the multi-modal space, which is inflexible for cross-modal reasoning or data-efficient training. In this paper, we call for an alternative paradigm, which transforms the images and tables into unified language representations, so that we can simplify the task into a simpler textual QA problem that can be solved using three steps: retrieval, ranking, and generation, all within a language space. This idea takes advantage of the power of pre-trained language models and is implemented in a framework called Solar. Our experimental results show that Solar outperforms all existing methods by 10.6-32.3 pts on two datasets, MultimodalQA and MMCoQA, across ten different metrics. Additionally, Solar achieves the best performance on the WebQA leaderboard.

## introduction information overload is a major problem in today's society due to the vast amount of information available. question answering (qa) systems can help alleviate this problem by providing users with concise and accurate answers foot_1 @xcite . however, traditional qa models are limited to text or structured data and are unable to utilize the vast amount of multimodal knowledge available on the internet @xcite , much of which is in non-text formats, such as images @xcite , data tables @xcite , etc. for example, on the wikipedia page for the united states, a user's questions about the us census, the color of the statue of liberty, and the us capital city could be answered using data tables, images, and paragraphs on the page, respectively. to address this challenge, one strategy is to train separate qa models for each modality and use a classifier to determine which modality to obtain information from @xcite . while this method is straightforward, it lacks the ability of cross-modal reasoning as models of different modalities are integrated without interaction. however, both human thought and the current benchmarks require reasoning across modalities @xcite . an alternative approach is to train a multi-modal model that can accept input from multiple sources simultaneously and produce the answer directly @xcite . this method allows for reasoning between modalities, but it can be challenging to train due to the diversified inputs (e.g. text, image, and table can be combined in seven modal combinations). for example, murag @xcite requires 3 million image-text pairs for pre-training to achieve decent results. in this paper, we introduce solar, a unified language representation for multi-modal qa. unlike existing qa methods performing in the multimodal space, solar transforms images and tables into natural languages at the input stage, simplifying the multi-modal qa task into a simpler textual qa problem. this transformation has at least three advantages as follows: (1) it helps to overcome the diversified modal combinations and enable crossmodal reasoning by putting together the textual representation of relevant clues belonging to multiple modalities; (2) language models have learned an enormous amount of world knowledge and patterns by reading billion-level text corpora @xcite , compared with the much smaller multi-modal pre-training datasets used in murag. the memorized information can serve as reusable background knowledge and basic skills for com- race track santa derby santa park kentucky derby churchill downs question: what animal races in the race of warrior at the churchill downs track? a group of people riding on the backs of horses; kentucky derby racing brown horse; white large stadium; riding white jockey row one's race is santa derby, the track is santa park. row one's race is kentucky derby, the track is churchill downs. title: kentucky derby warrior was an australian bred thoroughbred racehorse that won the 1869 melbourne cup at odds of 20/1. text image table global local unified language space for multi-modal qa ……. retrieve rank what animal races …at the churchill downs track?... row one's race is kentucky derby, the track is churchill downs... kentucky derby… racing brown horse generate horse plex qa @xcite ; ( 3 ) language has a higher information density compared to images @xcite and tables, the transformation from low-density to high-density can be done losslessly for tables and with minimal loss for images using global and local textualization strategies. then we retrieve relevant clues in the unified language space for each question, rank clues to get the most related top-n ones, and input them into the decoding model to generate the desired answer. though simple, our experiments show that solar achieves state-of-the-art performance on three different multi-modal qa datasets. on multi-modalqa and mmcoqa, we outperform sophisticated baselines by 10.6-32.3 pts across ten metrics. without the need of pre-training on external vision-language data or additional tricks, solar also achieves the first position on the official webqa leaderboard at the time of writing foot_2 . at the same time, solar has reduced 99.7% of storage space. ## method in this section, we will introduce solar in detail. as shown in figure 1 , it consists of two parts: a unified language representation and a unified qa model. the former is responsible for mapping images, tables and texts in different modal spaces into a unified language space. the latter generates an answer through three steps of retrieval, ranking and decoding in the language space. we discuss the related work in appendix a. ## conclusion in this paper, we approach the multi-modal qa task from a new perspective, where all the information sources are aligned into the language space to take advantage of the power of pre-trained language models and facilitate cross-modal reasoning. our experiments show the promise of this approach, as it outperforms baselines by a large margin and achieves the top position at public leaderboard with a single model. we hope this work can lead to a discussion: can language become a unified interface for people and models to understand the world?