Integrating Table Representations into Large Language Models for Improved Scholarly Document Comprehension (2024)

Abstract: We address the challenge of interpreting and reasoning over scientific tables with Large Language Models (LLMs), a crucial aspect of scholarly documents. Despite significant progress in natural language processing, the integration of tabular data into scientific LLMs remains limited. We propose an innovative approach leveraging intermediate task pre-training on table question-answering datasets, followed by model adaptation to comprehend tables in computer science literature. Our findings reveal that incorporating table understanding substantially improves the performance of LLMs on scientific literature understanding tasks, which we showcase in peer-review score prediction. This improvement underscores the importance of utilizing tabular data in the training of scientific language models. The code and models are publicly available at [this link](https://github.com/buseskorkmaz/Integrating-Table-Representations-into-LLMs).

## introduction large language models (llms) have experienced significant advancements in recent years and have been adapted to numerous natural language understanding and generation tasks @xcite . particularly in the scientific community, they have received increasing attention with their applications to scientific literature understanding tasks such as citation prediction @xcite , paper classification @xcite , scientific literature search @xcite and paper recommendation @xcite to accelerate scientific discovery. in addition, domain-specialized research assistant language models have been developed @xcite @xcite . although these specialized models on scientific texts demonstrate success in the scientific literature understanding benchmarks such as maple @xcite and scirepeval @xcite , the benchmarks and the corpora used in the training of these scientific language models predominantly focus on textual data. a critical component -and the focus of this study -often remains overlooked, which is tables. tables encapsulate key findings, offering a condensed view of the research outcomes. in this work, contrary to the existing approaches, we hypothesize that understanding tables can significantly enhance the performance of llms on scientific literature tasks by providing a more holistic understanding of research papers. we first tackle the challenge of interpreting tables and reasoning over them to answer questions requiring arithmetic operations and choosing information from specific cells through intermediate task pre-training. then, we adapt our trained model to comprehend scientific tables in published computer science papers. this training process is designed to enable the models to reason with scientific table data. the scientific tables dataset we use is fundamentally different from the datasets used in intermediate task pre-training for table question-answering, incorporating more extensive summaries of scientific tables. finally, we demonstrate that utilizing table representations extracted from fine-tuned llms with our approach improves the prediction of peer-review scores. overall, we develop a pipeline that allows llms to incorporate scientific knowledge from tables. the main contributions of this work are: (i) we apply an intermediate task pre-training approach that allows llms to understand tables, (ii) we do a detailed comparison of scientific table understanding by different models with different sizes, architectures, and under various settings, and (iii) we show how learning to represent scientific tables improves the understanding of scholarly documents, using the peer-review score prediction as a case study. ## conclusions scientific language model development and document comprehension have accelerated progress in recent years parallel to advancements in large language models. however, their ability to effectively understand and reason over tabular data in scientific literature has remained under-explored. in this work, we addressed this issue by proposing an approach that combines intermediate task pre-training on table question-answering datasets with model adaptation to comprehend tables in computer science literature. our experiments demonstrated that by understanding tables better, llms can achieve higher performance in scientific literature understanding tasks. we validated this claim through a case study on peer-review score prediction, where our best-performing model, pre-trained on wikitq and sqa datasets and fine-tuned on the scigen dataset, outperformed other settings in terms of mean squared error and f1-score. these results emphasize the importance of integrating tabular data into the training process of scientific language models. moreover, our qualitative analysis showed that the proposed approach generates more informative and contextually relevant summaries of scientific tables compared to generalist table models and models without intermediate task pre-training or fine-tuning. this finding suggests that our method can enhance the comprehension of scientific literature by providing more accurate and descriptive table representations. future research directions could include extending our approach to other scientific domains, exploring the integration of table representations with other elements of scientific papers (e.g., figures and equations), and developing more sophisticated table encoding techniques. additionally, incorporating larger and more diverse datasets for pre-training and fine-tuning could further improve the performance of llms on scientific literature tasks. figure 4 : the original caption of the table is "randomized beam search. all listed models use gradient weight qmml and tokens to represent execution history.". taken from @xcite . figure 5 : the original caption of the table is "test errors (%) and computational time of the proposed method (adaptivenet) and the bayesian optimization (bo) with different budgets in the experiment of adaptation of stochastic network. the mean values over 30 trials are reported in the proposed method, and the value in parentheses denotes the standard deviation. for the bayesian optimization, the result of a single run is reported.". taken from @xcite . figure 6 : the original caption of the table is "the performances of the im2latex-100k bi-lstm model. we discover that the look-ahead improves the model from the greedy search method-noted that la is more directly comparable to the greedy search because of their same beam size. we also show the scores of the beam search for the reference". taken from @xcite .