Multimodal Chart Retrieval: A Comparison of Text, Table and Image Based Approaches (2024)

Abstract: We investigate multimodal chart retrieval, addressing the challenge of retrieving image-based charts using textual queries. We compare four approaches: (a) OCR with text retrieval, (b) chart derendering (DePlot) followed by table retrieval, (c) a direct image understanding model (PaLI-3), and (d) a combined PaLI-3 + DePlot approach. As the table retrieval component we introduce Tab-GTR, a text retrieval model augmented with table structure embeddings, achieving state-of-the-art results on the NQ-Tables benchmark with 48.88% R@1. On in-distribution data, the DePlot-based method (b) outperforms PaLI-3 (c), while being significantly more efficient (300M vs 3B trainable parameters). However, DePlot struggles with complex charts, indicating a need for improvements in chart derendering - specifically in terms of chart data diversity and the richness of text/table representations. We found no clear winner between methods (b) and (c) in general, with the best performance achieved by the combined approach (d), and further show that it benefits the most from multi-task training.

## introduction multimodal retrieval is the task of retrieving a relevant piece of information from a multimodal dataset, given a query. this task has been extensively studied in the context of text and image retrieval @xcite ) or text and table retrieval @xcite , but has received relatively little attention in the context of visually grounded images such as charts and scientific figures. charts are an important source of information within scientific and technical domains. they are often used to summarizing multifaceted data, conveying communicate insights @xcite as well as facilitating the interpretation of intricate domains encompassing financial data analysis, news reporting, and scientific disciplines @xcite . in this context, chart retrieval assumes a pivotal role as a potent tool addressing diverse real-world applications. within the scholarly research domain @xcite , chart retrieval can enable the exploration and analysis of scientific literature that incorporates visual elements. furthermore, in the realm of fact verification @xcite , chart retrieval offers distinct advantages, par-ticularly in domains such as financial and climate data, where visual representations provide valuable insights and patterns. to the best of our knowledge, this work is the first to investigate multimodal retrieval on chart images, addressing the limited research in this domain. we begin by establishing a powerful table retrieval model that serves as a backbone for subsequent experiments, starting from the assumption that images of charts are a visual representation of an underlying table. to this end, we propose extending text retrieval models with row and column embeddings modeling the table structure, borrowing the main ideas from @xcite ; @xcite . our proposed model, tab-gtr, achieves state-of-the-art results on the nq-tables dataset @xcite , resulting in an improvement of 4.4 absolute points in r@1. for chart retrieval, we compare four approaches, leveraging existing findings in the literature, as also graphically summarized in figure 1: (a) @xmath0. an ocr model, namely tesseract @xcite , converts the chart image into a textual representation. the text is then processed by a text retrieval model, that is tab-gtr. a chart de-rendering model, namely de-plot @xcite , converts the chart image into a table representation. the table is then processed by a table retrieval model, that is tab-gtr. we make use of the output of deplot @xcite as an additional table input to pali-3 along the image data. the model was additionally augmented with table structure embeddings @xcite . we evaluate the four approaches on a dataset of charts. due to the lack of available chart retrieval data, we adapt the chartqa, scicap, and chart2text datasets for retrieval. our extensive experimentation shows that a chart derendering pipeline coupled with a table retrieval model outperforms the vlm setup, when applied in-distribution data (e.g. chartqa). however, deplot fails to generalize to more complicated charts (e.g. sci-cap), where it falls behind an ocr baseline. we conclude analyzing the shortcomings of the chart derendering model suggesting that future work in this area should focus on developing more robust chart derendering pipelines that are able to handle a wider range of chart types and annotations. if realized these improvements can enable (a) more efficient resource utilization, as deplot + tab-gtr pipeline is significantly more efficient, with 300m trainable parameters compared to 3b of the pali-3 encoder; (b) flexible applications of 0-shot chart derendering with large language prompting/retriever models, as done in @xcite . ## related work text / table retrieval. text retrieval has been extensively studied in the literature @xcite . in this work, we build upon existing work and repurpose a generalizable text retriever model to work on table inputs, following the same ideas of @xcite and @xcite . by building on top of a pre-trained text retrieval model @xcite we achieve better performance than @xcite and @xcite , without the need for hard-negative mining or more complex tri-encoder setup. although the task of table retrieval is not new @xcite , to the best of our knowledge, there is no method that adapts the methodology for the task of chart retrieval. ## problem setup we consider multimodal retrieval problems where a textual query is used to retrieve a document that can be a table, an image (specifically of a chart) or a combination of both. ## table retrieval with tab-gtr we present tab-gtr, a multimodal extension of the gtr @xcite model that handles both text and tabular data. we extend the t5 encoder architecture following the approach of @xcite by adding twodimensional positional embeddings that encode the table structure. the overview of the model architecture is shown in figure 2 . given an input text t and input table with n columns and m rows and text c i,j in cell at column 1 ≤ @xmath1. for each token we add two additional discrete features text_col and text_row: columns and rows are embedded into feature vectors and the embeddings added to the token embeddings before being fed to the transformer encoder. this provides the network with absolute positional embeddings of the table row and column corresponding to the tokens. we also use relative positional attention bias inherited from the t5 architecture, which is based on the linearized token sequence and not aware of the table structure. ## conclusions in this paper, we tackled the problem of chart retrieval, which, to the best of our knowledge, has not been explored before, at least in the context of text query to chart retrieval. from the assumption that chart images are visual representations of an underlying table, we established a sota table retrieval backbone, tab-gtr, combining the findings of @xcite ; @xcite ; @xcite . we found that when a good (e.g. human-annotated) table representation is available the tab-gtr model outperforms other chart retrieval methods. in the situation that only image data is available, our experiments on 5 datasets show that: we conclude that the approaches provide complementary benefits: a vlm can be extended with deplotter input to achieve both higher performance on chart data well represented by the deplotter and better flexibility.