MATSA : Multi-Agent Table Structure Attribution (2024)

Abstract: Large Language Models (LLMs) have significantly advanced QA tasks through in-context learning but often suffer from hallucinations. Attributing supporting evidence grounded in source documents has been explored for unstructured text in the past. However, tabular data present unique challenges for attribution due to ambiguities (e.g., abbreviations, domain-specific terms), complex header hierarchies, and the difficulty in interpreting individual table cells without row and column context. We introduce a new task, Fine-grained Structured Table Attribution (FAST-Tab), to generate row and column-level attributions supporting LLM-generated answers. We present MATSA, a novel LLM-based Multi-Agent system capable of post-hoc Table Structure Attribution to help users visually interpret factual claims derived from tables. MATSA augments tabular entities with descriptive context about structure, metadata, and numerical trends to semantically retrieve relevant rows and columns corresponding to facts in an answer. Additionally, we propose TabCite, a diverse benchmark designed to evaluate the FAST-Tab task on tables with complex layouts sourced from Wikipedia and business PDF documents. Extensive experiments demonstrate that MATSA significantly outperforms SOTA baselines on TabCite, achieving an 8-13% improvement in F1 score. Qualitative user studies show that MATSA helps increase user trust in Generative AI by providing enhanced explainability for LLM-assisted table QA and enables professionals to be more productive by saving time on fact-checking LLM-generated answers.

## introduction recent advances in llms have enhanced questionanswering capabilities @xcite , but they are prone to hallucination, 1 demo video: https://youtu.be/ufunwvzfn18 the company recorded a revenue of 201,351 mil in the year 2017 and 199,014 mil in the year 2016. cumulative total revenue for 2016-17 was 409,365 mil. total per mile revenue was11.99 and 11.55 cents in 2017 and 2016, respectively t multi-agent table structure attribution ## experiments we evaluate the matsa on our proposed tabcite benchmark. tables in this benchmark are derived from wikipedia pages and sec filings, which are paired with questions, free-form answers, and ground truth row/column attributions. table image / pdf upload question input answer input start attribution workflow reset app row attributions column attributions attribution explanation totto includes tables with extreme size variations, merged cells, and complex column hierarchies, representative of real-world distributions. (2) fetaqa (nan et al., 2021) (free-form table question answering) is a dataset consisting of table qa pairs from wikipedia that mimic humanlike multi-hop qa reasoning over evidence table cells to generate long-form coherent answers. while tables in fetaqa lack complex header hierarchies, the dataset is designed to require retrieving and reasoning over evidence cells from multiple rows for answer grounding. (3) aitqa (katsis et al., 2022) (airline industry table qa) is a domain-specific dataset of tables gathered from us sec 10-k annual reports of publicly traded airline companies that requires reasoning with complex column and row header hierarchies containing domain-specific vocabulary. table distribution is similar to that found in scientific and business documents. answers in aitqa are provided as singular table entities, which we converted into complete statements using gpt-4. we extracted the rows and columns corresponding to the supporting cells in above-listed datasets to get the set of ground truth row/column attributions baseline: we evaluate the effectiveness of matsa with recent baselines: (1) few-shot in-context learning (gao et al., 2023b) prompts llms with few-shot examples to generate answers with inline citations; (2) post-hoc retrieval (gao et al., 2023b) using a dense retriever to retrieve top-k rows/columns for answer attribution. evaluation metrics: as predictions output by matsa are not ranked, we evaluate the attribution quality using precision, recall, and f1 score. given a table with total d rows (or columns) , d ′ retrieved rows (or columns), and d ground truth rows (or columns) , we evaluate: (1) citation recall ( n 1 d ′ ∩ d d ) to determine if the model captures all supporting rows/columns, and (2) citation precision ( n demo ui: we used gradio for the demo ui hosted locally or on the aws cloud platform. ## matsa demo app figure 3 shows the matsa demo app. the app was built using gradio foot_3 and uses openai gpt-4o and gpt-4v (vision) models. the interface includes an upload panel for table images and questions, option to type in the answer statement or let the llm generate the answer based on table context. matsa helps users visualize the cited rows and columns in different colors. the users also have the ability to read the llm generated explanation for the row/column attributions, and can reset the interface to restart. ## results main results: ## user evaluation we conducted a user evaluation to assess the attribution accuracy of matsa and perceived usefulness of having fine-grained attribution on tables. recruitment & methodology: sixteen participants were recruited via prolific foot_4 . our evaluation dataset was comprised of 100 long-form table qa pairs randomly sampled from our proposed tabcite corpus. participants were asked to review the fine-grained attribution produced by matsa shown as highlights obtained for the table qa . participants were asked to rate (1) the usefulness of the attribution in helping them verify the accuracy of the answers, (2) the accuracy of the attribution, and (3) list any improvements on the attribution table qa pair from tabcite benchmark where question/answer are unclear as reported by evaluation participants. or feedback. figure 4 shows our hosted interface that was used for user study with participants recruited on prolific. they were presented with the question-answer and the related table, with and without attribution highlights. participants rated the attribution accuracy and usefulness in helping verify answer citations. usefulness & accuracy: overall, the participants had a positive feedback for the fine-grained table attributions produced by matsa. figure 5 shows the ratings for usefulness and accuracy. the majority of users found the attributions extremely useful (224/335, 66.86%) and very useful (61/335, 18.5%) for verifying the accuracy of table qa. participants found the attributions to be completely accurate (276/335, 82.38%) and somewhat accurate requiring minor corrections (34/335, 10.15%). through qualitative feedback, participants described the attributions as easy to understand, helpful in reducing reading time of the tables ("i could sift through the table quickly") and making verification easier ("...can help me to locate the answer quickly."). user feedback: the participants also provided feedback for cases where attribution could be improved. in some cases participants reported additional row/columns could be included in the attribution to make them more helpful (19/100). in other cases, some unnecessary row/columns could be removed (15/100). additionally, in our evaluation dataset a small portion of the qa pairs were found to have either an inaccurate answer or the question was unclear (figure 6 ), which in turn impacted participant ratings of the usefulness and accuracy. qualitative examples: figure 7 shows an example table qa pair from the tabcite benchmark where attribution is accurate as reported by evaluation participants. figure 6 shows an example table qa pair from the tabcite benchmark where question/answer are unclear as reported by evaluation participants. we found that a small portion of table qa pair from tabcite benchmark where attribution is accurate as reported by evaluation participants. ## target audience matsa is targeted to help students, professionals, and other users of llm-based chat systems interacting with pdfs or text document. some of the common use cases that we envision for this system are: (1) enable users to fact check llm-generated answers grounded in tabular data, (2) post-hoc text attribution for financial documents, product manuals, wikipedia-style web pages, (3) generate annotation data for instruction-tuning llm models to retrogressively generate inline citations with text. system license: the matsa system is a proprietary system developed for research experimentation and development. at this stage, we do not plan to publicly open-source it for any commercial or non-commercial purposes. ## conclusion we introduce fast-tab, a novel task for finegrained table structure attribution to provide citations from table rows and columns to support factual claims in llm-generated answers to tabular questions. we present the tabcite benchmark, which includes table qa and row/column attributions from wikipedia and business pdf documents with complex layouts. our multi-agent llm framework, matsa, converts tables into html, augments raw table data with descriptive context, and retrieves semantically relevant rows/columns that support atomic facts in the answers. future work may extend these methods to low-resource domains and other semi-structured documents, such as charts, info graphics, and diagrams.