D ata T ales: A Benchmark for Real-World Intelligent Data Narration (2024)

Abstract: We introduce DataTales, a novel benchmark designed to assess the proficiency of language models in data narration, a task crucial for transforming complex tabular data into accessible narratives. Existing benchmarks often fall short in capturing the requisite analytical complexity for practical applications. DataTales addresses this gap by offering 4.9k financial reports paired with corresponding market data, showcasing the demand for models to create clear narratives and analyze large datasets while understanding specialized terminology in the field. Our findings highlights the significant challenge that language models face in achieving the necessary precision and analytical depth for proficient data narration, suggesting promising avenues for future model development and evaluation methodologies.

## introduction data narration, the process of transforming intricate data into compelling narratives @xcite , plays a critical role in shaping business decision-making. by distilling vast amounts of information into digestible narratives, it empowers executives with clear and actionable insights @xcite . moreover, it fosters accessibility to valuable information, reaching a wider audience. however, traditional manual approaches are burdened by both time constraints and the potential for inaccuracies. consequently, there has been a longstanding anticipation for models capable of autonomously extracting meaningful insights from data @xcite . the rise of large language models (llms), such as gpt-3 @xcite and llama @xcite , signifies a beacon of hope within the field. these models demonstrate extraordinary ## the datatales benchmark we outline our data collection procedure employed for compiling datatales. subsequently, we conduct a comprehensive analysis to underscore its unique contributions. ## experimental setup we define the task of financial data narration as follows: given market movement data {t i,j |@xmath0, @xmath1, where t i,j is the row of entity i on date j, a data narration model m generates a report y narrating the market data: models. we explore both open-access models, specifically llama2-7b-chat and llama2-13b-chat @xcite , alongside closeaccess models like gpt-3.5-turbo and @xcite . these models are renowned for their robust capabilities, particularly in terms of zero-shot generalization on new tasks. 7 evaluation setup. we evaluate real-world scenarios with no training data and examine the model's learning ability from examples. we assess both zero-shot and fine-tuned scenarios, splitting data based on time (first 80% for training, remaining 20% for validation and testing). we fine-tune with adamw and a linear scheduler (learning rate: 1e-4, batch size: 16). we load models in 8-bit mode and fine-tune for 5 epochs on datatales using human report: "the dow jones industrial average fell by 1.2%, and the msci world index slid by 0.6% for the day." "the dow jones industrial average fell by" "1.2%" tabular data generation constraint: numerical value ? exact match? figure 5 : illustration of the factuality evaluation process. we provide the model with the prefix of human reports and assess whether its predicted numerical values align with those provided by experts, thus evaluating the accuracy of the content. lora due to resource constraints, and perform greedy decoding in inference. evaluations are conducted in two tabular data settings: (1) same-day data and ( 2 ) historical data spanning one week, to discern the influence of historical data on performance. tabular data is linearized row-by-row for model input. although few-shot models have shown better performance @xcite , input limitations of models like llama-2 posed challenges in incorporating complete data-report pairs for in-context learning without compromising content. future studies should consider integrating few-shot learning by examining models with extended context windows or refining methods for condensing data. metrics. we evaluate generated text quality based on factual accuracy, insightfulness and language style. (1) factuality is evaluated using a method inspired by mcqa tasks @xcite . we first use a named entity recognition model to identify numerical values in the generated text. as shown in figure 5 , the model then predicts numeric tokens given contextual information, and these predictions are compared with the original report to assess accuracy. a detailed explanation of this process is provided in appendix c. (2) insightfulness is evaluated through human assessments of freely generated model narrations. two finance-background evaluators score them based on impact (breadth of the claim) and significance (magnitude of changes described) on a 1-5 scale, inspired by @xcite , based on sentence samples that were generated on the same tabular data. the insightfulness score is the average of the two. (3) style is assessed using bleu @xcite to compare n-grams with human-provided narratives. ## experimental result and discussion ## conclusion & future work we identify a crucial gap in financial data narration: the lack of benchmarks that offer deep insights, essential for real-world applications. to address this, we introduce datatales, a novel dataset covers complex analytical operations, and enhanced by extensive data for more impactful and significant insights and domain-specific languages for higher proficiency. the complexity of datatales poses significant challenges to state-of-the-art models, in terms of both their poor accuracy and the lack of insight on their generated text. this complexity arises from the requirements of performing complex analytical operations, incorporating large input data, and accessing relevant knowledge. moving forward, we propose three focused areas to advance financial data narration. firstly, refining the analysis by using methods like datashot @xcite and table2analysis @xcite to recommend insights with specific metrics as an intermediate step prior text narrative generation to enhance analytical quality. secondely, integrating visuals into narratives, with methods like foresight @xcite , voder @xcite , and calliope @xcite , to improve storytelling. lastly, developing new automated evaluation focused on accuracy and insight quality using table fact-checking models @xcite @xcite and insight evaluation frameworks @xcite ) respectively.