Equipping Language Models with Tool Use Capability for Tabular Data Analysis in Finance (2024)

Abstract: Large language models (LLMs) have exhibited an array of reasoning capabilities but face challenges like error propagation and hallucination, particularly in specialised areas like finance, where data is heterogeneous, and precision is paramount. We explore the potential of language model augmentation with external tools to mitigate these limitations and offload certain reasoning steps to external tools that are more suited for the task, instead of solely depending on the LLM’s inherent abilities. More concretely, using financial domain question answering datasets, we apply supervised finetuning on a LLAMA-2 13B CHAT model to act both as a task router and task solver. The task router dynamically directs a question to either be answered internally by the LLM or externally via the right tool from the tool set. Our tool-equipped SFT model, RAVEN, demonstrates an improvement of 35.2% and 5.06% over the base model and SFT-only baselines, respectively, and is highly competitive with strong GPT-3.5 results. To the best of our knowledge, our work is the first that investigates tool augmentation of language models for the finance domain.

## introduction augmenting large language models (llms) with tools has emerged as a promising approach to further complement llms' capabilities with specialised mechanisms, leading to improved accuracy and reliability @xcite . this approach offloads tasks, fully or partially, to a deterministic offline tool such as a python interpreter @xcite , calculator @xcite , knowledge base @xcite , or online apis of models and services @xcite @xcite . this paradigm holds particular appeal in fields demanding precision, such as finance @xcite and healthcare (luo et al., 2022; singhal 1 code, model, and data: https://raven-lm.github.io @xcite . specifically, the specialised terminology within the finance domain and the diverse range of data sources, encompassing both structured and unstructured data, along with the complex numerical reasoning requirements across such heterogeneous sources, render it an ideal candidate for potential improvements through tool augmentation. nevertheless, there has been limited research dedicated to this specialised domain. a satisfying review of existing works on tool augmentation of llms is beyond the scope of this work; however, this space can be divided into two primary directions: (1) approaches that require an llm at the center and uses few-shot incontext learning to either provide tool and api documentations, or demonstrations that involve tool use @xcite @xcite , and (2) approaches that build fine-tuned smaller lms under a static tool use protocol @xcite , or through expensive annotations collected from commercial llms @xcite . in this work, our primary focus lies in demonstrating the potential of tool augmentation within the finance domain. acknowledging the utmost significance of privacy concerns within the financial sector, we have chosen to adopt a fully offline approach, equipping a language model with diverse tool utilisation mechanisms. more concretely, we employ parameter efficient fine-tuning (peft) @xcite to equip a llama 2 13b chat @xcite with tool use capabilities. our approach differs from previous research in two significant ways. first, we do not rely on costly annotations of training examples produced by commercial language models. second, we enhance existing question-answering training datasets by incorporating instructions and merge data representing various tasks. this approach instructs the model to adapt dynamically and determine the most appropriate mechanism (either internal or tool-based) to address each specific query. how much was the included change in fair value of the company's servicing asset included in its servicing fees? greensky, inc. notes to consolidated financial statements (continued) (united states dollars in thousands, except per share... {"header": ["", "", "year ended december 31,", ""], "rows": [["", "2019", "2018", "2017"], ["merchant fees", "@xmath0297,776", "@xmath1 in millions)", "", "", ""], "rows": [["for the year ended december 31:", "2018", "2017", "yr.-to-yr. percent/ margin change"], ... which team started 2010? {"header": ["season", "series", "team name", "races", "poles","wins", "points", "finalplacing"], "rows": [[ "2002", "formularenault 2.0 brazil", ... determine the sentiment of the following. metsa-botnia will finance the payment of dividends , the repayment of capital and the repurchase of its own shares with the funds deriving from its ... using the language model the task router infers the optimal task format to use conditioned on the given prompt. the task solver re-formats the instruction according to the selected template by the task router and sends it to the language model again. the pipeline will branch between serving the response directly or calling a tool api to perform an intermediate evaluation before serving the final output. our model, raven, achieves significant improvements in reasoning over structured data. for example, compared to the base model we demonstrate a lift in exact match accuracy of 63.8% (21.68% → 85.52%) on the wiki-sql @xcite . despite being much smaller in size, raven also outperforms gpt-3.5 on all benchmarks with an absolute average accuracy lift of 9.2%. ## raven we use the llama 2 13b chat @xcite model as the backbone and fine-tune it using low rank adaptation (lora) @xcite . in this section we provide training details of raven. the overall architecture of raven is shown in figure 1 . ## experiments we compare with the base llama 2 13b chat with and without sft 6 . we also report gpt-3.5 foot_5 (5-shot), gpt-3.5 (chain-of-thought @xcite ) and gpt-3.5 (5-shot + tools). the sft model trained with tool use is denoted as raven. when tool use fails due to ill-formed arguments we have a fallback mechanism to produce the answer by the sft model, denoted as backoff. for training details and hardware, see appendix b. we evaluate the models using exact match. the task router has determined the correct type 100% of the time, except for tat-qa where the accuracy was 90.62%. ## conclusion in this paper we have demonstrated the feasibility of equipping a llama 2 13b chat model with tool use capabilities via fine-tuning a mere 0.2% of its parameters on a relatively small and diverse dataset. the augmentation with tools remarkably elevated the performance of the base model by an average of 35.2% across 4 datasets, surpassing even a significantly larger gpt-3.5 model by 9.2%. additionally, through a comparative analysis of question answering datasets we demonstrate the effectiveness of augmenting language models with external tools, showing significant improvements in accuracy when addressing multi-hop questions with tools.