When Reasoning Meets Information Aggregation: A Case Study with Sports Narratives (2024)

Abstract: Reasoning is most powerful when an LLM accurately aggregates relevant information. We examine the critical role of information aggregation in reasoning by requiring the LLM to analyze sports narratives. To succeed at this task, an LLM must infer points from actions, identify related entities, attribute points accurately to players and teams, and compile key statistics to draw conclusions. We conduct comprehensive experiments with real NBA basketball data and present SportsGen, a new method to synthesize game narratives. By synthesizing data, we can rigorously evaluate LLMs’ reasoning capabilities under complex scenarios with varying narrative lengths and density of information. Our findings show that most models, including GPT-4o, often fail to accurately aggregate basketball scores due to frequent scoring patterns. Open-source models like Llama-3 further suffer from significant score hallucinations. Finally, the effectiveness of reasoning is influenced by narrative complexity, information density, and domain-specific terms, highlighting the challenges in analytical reasoning tasks.

## introduction accurately aggregating information is essential for reasoning @xcite @xcite . this is especially true when reasoning over longitudinal data @xcite . for example, when assessing patient outcomes over years to determine the effectiveness of new medications, models must accurately track changes to provide insights into trends, patterns, and potential causal relationships. while llms hold significant potential in reasoning, understanding how they aggregate information to support reasoning is key to ensuring their robustness in decision-making. sports data have recently emerged as a popular testing ground for llms @xcite @xcite . unlike health data, sports narratives are readily accessible from diverse sources, e.g., espn, cbs sports, bbc sport, and sports illustrated. 2 these narratives provide detailed event descriptions over extended game periods, making them an excellent case study to explore how llms reason with longitudinal data. figure 1 presents a snippet from an nba basketball game. the play-by-plays capture key actions of the game, such as passes, shots, and fouls, along with timestamps and team-player affiliations. our research involves using llms to track scoring actions within these detailed game descriptions and to link entities for comprehensive game analysis. these games include a mix of expected and unexpected events, presenting unique challenges for reasoning tasks. llms's reasoning abilities, including multi-hop, deductive, inductive, and abductive reasoning, have garnered increasing attention @xcite @xcite @xcite @xcite . for example, @xcite introduce a dataset designed to evaluate language models on multi-step reasoning with 1,000word murder mysteries and object placement. @xcite explore latent multi-hop reasoning and found evidence of its use in fact composition. @xcite study the reversal curse that llms trained on 'a is b' fail to learn that 'b is a.' distinguishing from previous studies, our approach introduces a quantitative aspect to reasoning with game narratives, where the llm must summarize key statistics to derive conclusions. we focus on analytical reasoning in this work, which involves identifying relationships between elements, drawing inferences and synthesizing infor-analyze the team-player affiliations and play-by-play descriptions below to determine the total points scored by each team (player). mation @xcite . notably, we evaluate if llms perform better in a more focused and manageable context when using a divide-and-conquer strategy. sports narratives, for instance, can be segmented along multiple dimensions. we explore division methods that simplify multi-hop reasoning or facilitate information aggregation, thereby easing the load on llms. we further introduce a new discounted cumulative accuracy (dca) metric. dca differs from traditional accuracy by allowing a small margin of error. this helps us identify scenarios where llms significantly hallucinate, rather than grounding predictions on the narratives. our experiments reveal multiple limitations of state-ofthe-art llms in using divide-and-conquer strategies for analytical reasoning. with data synthesis, we further stress test llms' reasoning abilities in complex and novel scenarios. llms are required to handle a range of narratives that differ in length, complexity, and information density, and may also involve a shift from natural language to symbolic reasoning. our approach, sportsgen, enhances the control over narrative complexity when compared to human-written game narratives and few-shot prompting. it contributes to ongoing research in data synthesis @xcite @xcite by creating synthetic sports narratives that help evaluate the analytical capabilities of llms. our research contributions are summarized as follows. by exploring how llms aggregate information, we gain insights into their potential in longitudinal studies, such as patient health management, where the llm must identify and interpret recurring patterns to make critical decisions. ## related work success in reasoning hinges on effective content selection. recent studies have placed a growing emphasis on llm's reasoning capabilities, exploring various types such as deductive, inductive, abductive, and multi-hop reasoning @xcite @xcite @xcite . efforts to enhance llms' reasoning abilities include prompting, supervised fine-tuning, and adjustments to decoding @xcite @xcite @xcite @xcite . in our paper, we propose a quantitative approach to reasoning by requiring the llm to identify and consolidate relevant facts. similar to how frequency indicates salient content in multidocument summarization @xcite @xcite @xcite , we conjecture that how llms aggregate information is critical in addressing complex reasoning challenges. numerical reasoning is applied in financial question answering and solving mathematical word problems, and numerous datasets have been de-veloped for these purposes @xcite @xcite @xcite @xcite . our study explores a new angle-investigating llms' ability to analytically solve problems using divide-and-conquer strategies @xcite . sports data are complex and multifaceted. our focus is on pinpointing specific divide-and-conquer strategies where llms are particularly effective. sports data plays a pivotal role in various language tasks, such as real-time summarization of games, data-to-text generation, and analysis of commentator bias @xcite @xcite @xcite @xcite . it also influences the analysis of domains like game reviews and gameplay logs, enhancing our understanding of gameplay commentary dynamics @xcite @xcite . building on research by @xcite , our study explores how well llms can track scoring actions across lengthy narratives and accurately link entities for in-depth game analysis. with sportsgen for data synthesis @xcite @xcite , we take a step further by creating diverse narratives that vary in style, complexity, and level of detail to better assess llm effectiveness. ## analytical reasoning with divide-and-conquer strategies our research uses analytical reasoning to accurately calculate team points from sports narratives. we specifically focus on espn's nba game play-byplay descriptions (figure 1 ). on average, an nba game consists of 466 plays and 6,229 tokens, with the longest narrative containing up to 7,322 tokens. to succeed at this task, an llm must understand the basic rules of the game, distinguishing between scoring plays, such as three-pointers and field goals, and non-scoring actions like passing. further, the model must correctly attribute each scoring action to the appropriate player and team, which requires nuanced reasoning. finally, the llm must aggregate all the scored points to determine the final scores for the teams. this process tests the llm in three critical areas: knowledge acquisition, point referencing, and effective score aggregation. errors in any of these areas can lead to incorrect results. player-centric division. we are particularly interested in exploring whether llms can perform analytical reasoning in a more focused and manageable context using a divide-and-conquer strategy. for instance, sports narratives can be segmented along multiple dimensions. in our player-centric division, the llm assesses player performance based on play-by-play descriptions. here, the llm receives a list of players, without their team affiliations, and is tasked with tracking and summarizing each player's performance throughout the game. the final results are presented in json format. following this, we link the individual players' scores with their respective teams to calculate the overall team scores. this player-centric approach employs llms to track individual player performances, allowing for a direct calculation of team scores from player data. it lessens the burden on llms to use multi-hop reasoning for inferring team scores from player scores. ## creating data with sportsgen we introduce sportsgen, a new method that synthesizes sports narratives by modeling game dynamics. sportsgen offers a platform to assess llms' reasoning capabilities in novel scenarios that may not exist in real data. since synthesized narratives have not been encountered by llms during pretraining, they also serve as a valuable benchmark for future llm assessments. previously, data synthesis has been utilized in fine-tuning, distilling, and evaluating llms for tasks such as hallucination detection, topic classification, sarcasm and humor detection @xcite @xcite @xcite . our work expands on these efforts by focusing on the specific challenges in generating game narratives. 3 setting up team profiles. we create profiles for two basketball teams, team t 1 and team t 2 , for a specific game. each team has five players, filling the roles of point guard (pg), shooting guard (sg), small forward (sf), power forward (pf), and center (c). each team has an efficiency score e from 0 to 100, which reflects the points they score per 100 possessions. 4 we start with real nba teams for these profiles, and later users can select either real or synthetic players to create fantasy teams. turn-by-turn game generation. our sports-gen renders realistic sports narratives using a turnby-turn approach, which mirrors the back-and-forth dynamics seen in live sports games. in each turn, a team controls the ball and performs a series of plays 5 such as passes, shots, and fouls, which are key actions during the game. after each turn, possession generally shifts to the opposing team. each quarter consists of alternating turns between the two teams, with each turn being given an allotted time frame. for example, a turn might unfold as follows: "lakers take possession, make a quick pass, and take a shot that misses," all within about 15 seconds. each basketball game lasts 48 minutes. our approach continues to add turns until nearing the end of a quarter's time limit. if the ending turn exceeds this limit, it will be truncated. ## experiments sports data are accessible from a variety of sources such as espn, bbc sport, cbs sports, and sports illustrated. this work utilizes the basketball playby-plays provided by @xcite consisting of 28,492 nba games from 2002 to 2023, gathered from espn. due to resource constraints, we subsampled the original dataset to d h , corresponding to the number of human-written narrative quarters. further, our proposed method, sportsgen, generates synthesized game narratives with varied scoring to non-scoring (s:ns) ratios including 1:2, 1:3, 1:4, and 1:5. d s represents the number of game quarters synthesized by sportsgen. accuracy. our goal is to enable llms' to track and predict the total points scored by each team at the end of a game quarter. we choose quarter-level evaluation because current llms do not perform well at tracking points for full games. the accuracy metric strictly evaluates whether the model's predictions matches with the actual scores for each quarter and team. here, each team is considered as a separate data point. for human-written narratives, our dataset includes detailed box scores sourced from espn. for our sportsgen generated narratives, ground-truth scores are obtained during the generation process. discounted cumulative accuracy (dca). "exact match" accuracy is not an ideal metric for track- we use p t to denote the proportion of instances where the prediction error is exactly t points from the true value (eq.( 2 )). let n represent the total number of instances evaluated. for each instance, |s ns * n | measures the absolute difference between the predicted and actual values. if this error surpasses t, the instance does not contribute to the score. higher accuracy and dca scores indicate better-performing systems. in the sections below, we evaluate leading llms on their analytical reasoning with game narratives using various divide-and-conquer strategies. we also compare human-written sports narratives against our sportsgen narratives to assess llm effectiveness. ## conclusion we investigate llms' analytical reasoning abilities, using divide-and-conquer strategies to determine where they perform best in analyzing sports narratives. with sportsgen, we further generate a variety of narratives that vary in style, complexity, and detail to better assess llm performance. our research not only paves the way for future llm assessments but also sets an understanding of how llms may handle complex, longitudinal data.