T able L lama: Towards Open Large Generalist Models for Tables (2024)

Abstract: Semi-structured tables are ubiquitous. There has been a variety of tasks that aim to automatically interpret, augment, and query tables. Current methods often require pretraining on tables or special model architecture design, are restricted to specific table types, or have simplifying assumptions about tables and tasks. This paper makes the first step towards developing open-source large language models (LLMs) as generalists for a diversity of table-based tasks. Towards that end, we construct TableInstruct, a new dataset with a variety of realistic tables and tasks, for instruction tuning and evaluating LLMs. We further develop the first open-source generalist model for tables, TableLlama, by fine-tuning Llama 2 (7B) with LongLoRA to address the long context challenge. We experiment under both in-domain setting and out-of-domain setting. On 7 out of 8 in-domain tasks, TableLlama achieves comparable or better performance than the SOTA for each task, despite the latter often has task-specific design. On 6 out-of-domain datasets, it achieves 5-44 absolute point gains compared with the base model, showing that training on TableInstruct enhances the model’s generalizability. We open-source our dataset and trained model to boost future work on developing open generalist models for tables.

## introduction semi-structured tables are prevalent data structures to store and present information in almost every domain, ranging from scientific research, business reports, and healthcare records to financial statements. a variety of table-based tasks have been proposed, such as entity linking @xcite , schema augmentation @xcite , and table-based question answering @xcite @xcite , which 1 code, model and data are available at: https:// osu-nlp-group.github.io/tablellama/ . most existing methods for table-based tasks have at least one of the following limitations: (1) require table pretraining @xcite @xcite and/or special model architecture design for tables @xcite @xcite , (2) only support limited, specific types of tables and tasks @xcite , (3) make strong simplifying assumptions (see the "in-domain" part of section 2.1) about tables and tasks @xcite . on the other hand, language models like t5 @xcite have been shown to excel in grounding language to structured knowledge @xcite . in addition, instruction tuning @xcite @xcite appears as an important technique that can guide llms to follow instructions to complete a variety of tasks. under this background, we seek to answer the following question: can we build a generalist model to handle a variety of table-based tasks using llms and instruction tuning? some exemplar tasks are shown in figure 1 . such a generalist model shall meet the following desiderata: first, it should not only work well on diverse tablebased tasks, but also generalize to unseen tasks. since new table data and tasks can be constructed dynamically as new information arrives, it is hard to collect training data that covers all tasks and all tables, which requires a model to be inherently generalizable to tasks and datasets it has never seen before. second, it should work on real-world tables and realistic tasks. the model should not make strong assumptions to only handle simplified synthetic tables and tasks, but must embrace practical challenges such as handling complex numerical reasoning on large hierarchical spreadsheets as well table types wikipedia tables spreadsheets tablellama in-domain evaluation tasks out-of-domain evaluation tasks hybrid table passage qa table fact verification table grounded dialogue generation highlighted cells description fine-tuning table fact verification hierarchical table qa column type annotation evaluate as a large number of candidates for classification and ranking tasks. in pursuing this goal, we realize there lacks a comprehensive collection of realistic tables and tasks that can support the development and evaluation of generalist models. therefore, we construct tableinstruct, by meticulously selecting representative table-based tasks from widely used datasets, unifying the format for all tasks and manually annotating instructions. tableinstruct shown in tableinstruct boasts a collection of 14 datasets of 11 tasks in total, with both in-domain and outof-domain evaluation settings. our training data includes 8 tasks, which are curated from 1.24m tables containing 2.6m instances spanning from table interpretation, table augmentation, table-based qa, and table-based fact verification. we choose 8 datasets for these 8 tasks for in-domain evaluation and leave the other 6 datasets for 4 tasks for out-ofdomain evaluation. the in-domain training tasks can enable the model to learn more fundamental table understanding abilities such as table interpretation and table augmentation, while we choose tasks that require more high-level reasoning abilities such as table qa and cell description to test the model's generalization ability. this extensive range of tables and diverse tasks not only provide valuable resources for table modeling, but also foster a more comprehensive evaluation of generalist models. (2) the use of real-world tables and realistic tasks. tableinstruct uses authentic real-world instead of overly simplified synthetic task data com-pared with existing work @xcite . we incorporate a large number of wikipedia tables and spreadsheets from statistical scientific reports with varied length of contents, realistic and complex semantic types from freebase @xcite for column type annotation and relation extraction, and a large referent entity corpus with rich metadata from wikidata @xcite for entity linking. in addition, we include complicated numerical reasoning tasks with hierarchical table structure and existing manually annotated table qa and fact verification tasks. by doing so, we aim to equip models with the capability to cope with realistic and complex table-based tasks. tableinstruct requires models to accommodate long inputs ( in summary, our main contributions are: tablellama, an open-source llm-based generalist model fine-tuned on tableinstruct. experiments show that compared with the sota on each task that often has special pre-training or model architecture design for tables, tablellama can achieve similar or even better performance on almost all of the in-domain tasks. for out-ofdomain tasks, compared with the base model, tablellama can achieve 5-44 absolute point gains on 6 datasets, and compared with gpt-4, tablellama has less gap or even better zeroshot performance on 4 out of 6 datasets, which demonstrate that tableinstruct can substantially enhance model generalizability. 2 tableinstruct benchmark unlike existing datasets predominantly designed for training task-specific table models, our objective is to bridge the gap between multiple complex task-specific models and one simple generalist model that can deal with all the table-based tasks without extra model-design efforts. to achieve this, our approach for constructing tableinstruct adheres to the following principles. first, instead of collecting multiple datasets from highly homogeneous tasks, we try to diversify the tasks and table types. we pick representative table-based tasks that necessitate different abilities of models, such as table interpretation, table augmentation, table qa and table fact verification from wikipedia tables and spreadsheets in statistical scientific reports. second, we select realistic tasks and construct highquality instruction data in a unified fashion without simplifying assumptions (see "in-domain" part of 2.1). tableinstruct will support powerful modeling and realistic evaluation approaches, ensuring a valuable and practical dataset for research. ## experimental setup model construction. although a few existing llms @xcite can handle longer than 4k contexts, their training time is quadratically increasing with context length, which becomes very costly for us to further finetune them on tableinstruct due to our large data scale. as longlora @xcite has been shown as an effective and efficient technique to train long-context llms with shift short attention, we adopt it as our backbone model. shift short attention splits context length into several groups and conducts attention in each group individually. the tokens are shifted by half group size in half attention heads to ensure the information flow between neighboring groups. for example, longlora can use shift short attention with group size 2048 to approximate total 8196 context length training, which leads to less computation cost with similar performance compared to fine-tuning with vanilla attention. we fine-tune longlora on tableinstruct to get our generalist model tablellama. existing sota models. in our evaluation settings, we have 9 out of 14 sota models utilize table pretraining and/or have special model architecture design for tables. the detailed description for each sota model is in appendix a. evaluation metrics. we follow the above baselines to use their evaluation metrics. for column type annotation, relation extraction and kvret, we use micro f1. for entity linking, tabfact, feverous, hybridqa, wikisql and wikitq, we use accuracy. for row population and schema augmentation, we use map. for hitab, we use execution accuracy @xcite . for fetaqa and totto, we use bleu @xcite . training and inference details. we choose lon-glora 7b @xcite , fully fine-tuning version with 8k context length limit as our base model. the fully fine-tuning version replaces the vanilla attention in llama 2 with shift short attention. we fine-tune the model with huggingface transformers library @xcite . we merge all eight datasets and repeat three smaller datasets (i.e., fetaqa, hitab and tabfact) for six times and randomly shuffle them as our final training data. we use a learning rate of 2e-5 and set the batch size at 3. we streamingly train the model on 48 a100 80gb gpus and use a cosine scheduler with a 3% warm-up period for 2 epochs. to efficiently train the model, we employ deepspeed training with zero-2 stage @xcite . for both training and inference, we set the input length as 8192. for inference on tablellama, as different tasks have different lengths of the ground truth, we use 64 as the output length for column type annotation, relation extraction, entity linking, hitab, tabfact, feverous, hybridqa, wikisql and wikitq, 128 for schema augmentation, fetaqa, kvret and totto, and 512 for row population. for column type annotation and entity linking, we uniformly sample a subset from the original test data as our test set due to the large test size. for row population, we filter out the examples with more than 500 candidate entities from the original test set and randomly sample a subset as our test set. for all the downsampled test set, we reproduce the sota results using the sota model. for closed-source llms, we use the gpt-4-1106preview version for gpt-4, which is the latest version that supports 128k context and reports the best performance. for gpt-3.5, we use the gpt-3.5turbo-1106 version, which supports 16k context. ## related work table representation learning. given the vast amount of knowledge stored in tables, various table-based tasks have been proposed @xcite , such as column type annotation @xcite , row population @xcite , table qa @xcite @xcite , etc. in order to handle the semi-structured tables, existing work puts their efforts into designing special model architectures, such as turl with structure-aware attention @xcite , tuta with tree-based attention @xcite and tabert with vertical self-attention mechanism @xcite ; or designing special encodings such as table position encoding @xcite , and numerical encoding @xcite to better encode the table structure and infuse more information to the neural architecture. in addition, some work focuses on table pretraining @xcite @xcite to encode knowledge in large-scale tables. however, although such existing works have shown promising progress, they are still data-specific and downstream task-specific, which requires special design tailored for tables and table-based tasks. our work proposes tableinstruct to unify different table-based tasks and develops a one-for-all llm tablellama to reduce those extra efforts during modeling. this high-level insight is similar to unifiedskg @xcite , which unifies a diverse set of structured knowledge grounding tasks into a text-to-text format. however, unified-skg deals with different knowledge sources such as databases, knowledge graphs and web tables and does not explore instruction tuning, while we focus on a wide range of realistic tasks based on real-world tables via instruction tuning. in addition, a concurrent work @xcite synthesizes diverse table-related tasks and finetunes closesource llms such as gpt-3.5 via instruction tuning. compared to theirs, we collect more realistic and complex task data such as hitab as well as classification and ranking tasks with candidates from freebase and wikidata and develop open-source llms for table-based tasks. we believe both our constructed high-quality table instruction tuning dataset and the trained model can be valuable resources for facilitating this line of research. ## conclusion this paper makes the first step towards developing open-source large generalist models for a diversity of table-based tasks. towards that end, we construct tableinstruct and develop the first opensource generalist model for tables, tablellama. we evaluate both in-domain and out-of-domain settings and the experiments show that tablellama has gained strong table understanding ability and generalization ability.