S ports M etrics: Blending Text and Numerical Data to Understand Information Fusion in LLM s (2024)

Abstract: Large language models hold significant potential for integrating various data types, such as text documents and database records, for advanced analytics. However, blending text and numerical data presents substantial challenges. LLMs need to process and cross-reference entities and numbers, handle data inconsistencies and redundancies, and develop planning capabilities such as building a working memory for managing complex data queries. In this paper, we introduce four novel tasks centered around sports data analytics to evaluate the numerical reasoning and information fusion capabilities of LLMs. These tasks involve providing LLMs with detailed, play-by-play sports game descriptions, then challenging them with adversarial scenarios such as new game rules, longer durations, scrambled narratives, and analyzing key statistics in game summaries. We conduct extensive experiments on NBA and NFL games to assess the performance of LLMs on these tasks. Our benchmark, SportsMetrics, introduces a new mechanism for assessing LLMs’ numerical reasoning and fusion skills.

## introduction large language models (llms) are more powerful than ever. openai's gpt-4 turbo (2023) features a 128k context window, allowing it to process over 300 pages of text in a single prompt. claude v2.1 (2023) steps it up with a 200k token window, equivalent to roughly 150,000 words or more than 500 pages. mistral ai (2023) has created a sparse mixture of experts model capable of processing up to 32k tokens. the developments suggest language models can now engage with vast amounts of text content and data, opening doors to exciting new applications in various domains. one of the most promising uses of llms is in handling a combination of unstructured texts and structured data. for example, determining if a patient can be discharged from the hospital may involve reviewing doctor notes, radiology and pathology reports, lab results, and other records that blend ## related work there is a growing need for a benchmark to evaluate llms' information fusion capabilities, which offers clear, quantitative scores for comparing various llms. for example, chatbot arena @xcite utilizes elo ratings @xcite , mt-bench comprises of 80 multi-turn questions, and mmlu focuses on a model's multitask accuracy across 57 tasks @xcite . multi-document summarization offers a promising benchmark @xcite @xcite . however, developing a summary scoring system poses challenges due to variables such as summary length, content coverage, and faithfulness @xcite @xcite @xcite @xcite . sports data, which combines static knowledge with player dynamics, presents an untapped opportunity for benchmarking llms. combining information from a blend of textual and numerical records poses a significant challenge. in traditional multi-document summarization, the system creates a concise summary from a set of topically related documents. giorgi et al. @xcite show that this task remains difficult in an "open-domain" setting, where the document set is generated by a retriever and may include irrelevant information. with the growing popularity of the nba game score, developed by former espn writer john hollinger, provide a rough measure of a player's productivity in a basketball game. source: https://www.nbastuffer.com/analytics101/game-score in 1979, the ncaa created the passing efficiency formula with specific scaling factors to ensure an average passer would have a rating of exactly 100 for yards-per-attempt and completion percentage. source: https://stassen.com/football/pass-eff/ passing efficiency (pe) figure 3 : we adopt the nba's game score, originally designed for player evaluation, to measure a team's overall efficiency. for american football, we apply ncaa's passing efficiency formula. sports data has been utilized in various natural language tasks, including data-to-text generation for sports games @xcite @xcite @xcite , real-time game summarization from live commentaries @xcite ; and other aspects such as sports commentator bias @xcite . beyond sports, there's significant interest in annotating and analyzing large-scale game-related corpora, such as reviews and gameplay logs, and summarizing gameplay commentaries @xcite @xcite . we anticipate that insights from our sportsmetrics benchmark will benefit these areas, enhancing our understanding of game narratives and player dynamics. ## the sportsmetrics benchmark we collect nba and nfl play-by-play data from espn.com. the descriptions capture the essence of each game. they are typically written by espn's sports journalists, who are experts in their respective sports. we reached out to espn as necessary to ensure adherence to their data policies. in figure 1 , we use "time" to indicate the exact moment of each action on the game clock, while "play" details the actions occurring at those times. scoring actions, which change the game's score, are identified but not disclosed to llms during our experiments, as are team points. additionally, we collect data on players' team affiliations and the game's box scores for our analysis. our task requires llms to track key stats across thousands of play-by-play records, which is a nontrivial effort. an ideal llm needs to associate each action with the right player and their team in order to calculate team-level statistics. it must also monitor multiple key statistics simultaneously, such as field goals, free throws, rebounds, assists, blocks, steals, personal fouls, and turnovers in a basketball game. we believe an llm's ability to summarize key details and fill in the missing statistics in game summaries demonstrates its capabilities in data fusion and numerical reasoning. we need a comprehensive scoring metric to evaluate llms' ability to monitor key game statistics. simply reporting individual metrics such as team points, rebounds, assists, and blocks for each team is inefficient and does not provide a holistic view of game analysis. to address this, we employ expertdeveloped team statistics formulas, as illustrated in figure 3 . we adopt the nba's "game score" by john hollinger, originally for player evaluation, to measure a team's overall effectiveness in basketball. it considers both positive (points, rebounds, you are a helpful assistant tasked with analyzing sports games. you have been given play-by-play breakdowns of a basketball game between two teams. the "time" column shows the exact time on the game clock when each play took place. the game clock counts down, so this column displays times in a descending order. the "play" column describes the action that happened at the respective times. it provides details of specific plays, movements, and outcomes on the court. team players are listed in two rows, each row representing one of the two basketball teams involved in the game. your task is to fill in the missing key statistics from a basketball game recap. each missing statistic is marked with '___'. game recap: ```jalen duren had ___ points and ___ rebounds as the detroit pistons overcame a careerhigh ___ points from zach lavine to beat the chicago bulls ___-___ on saturday night.``f irst, create an internal memory as a json object. initially, this json object only has placeholders for team points, like this: initial memory: {"chicago bulls": {"points": null}, "detroit pistons": {"points": null}} next, add the necessary key game or player statistics to the working memory to complete the missing information. these statistics might include categories such as 'field goals made', 'field goals attempted', 'free throws made', 'free throws attempted', 'rebounds', 'assists', 'blocks', 'steals', 'points' and others. this json object will later be populated with relevant data that will be used to fill in the blanks. working memory example: {"jalen duren": {"points": null, "rebounds": null}, "zach lavine": {"points": null}, "chicago bulls": {"points": null}, "detroit pistons": {"points": null}} now, you will be given a new game recap. your goal is to create a working memory as a json object, which can be used to fill in the missing key statistics in the game recap. game recap: ```{game-recap}```c an this working memory, represented as a json object, effectively complete the missing key statistics for the given game recap? if yes, just respond with the working memory. if not, improve the memory structure. do not take into account or modify the null value. game recap: ```franz wagner scored 24 of his ___ points in the second half, paolo banchero added ___ points, and the orlando magic overcame nikola jokic's triple-double wednesday night to record their fifth straight victory, ___-___ over the denver nuggets.``ì nitially, you are given a json object where all values are set to null. based on the provided play-by-play breakdown and teamplayer data, you will update these key statistics in json format. working memory example: {"jalen duren": {"points": 23, "rebounds": 15}, "zach lavine": {"points": 51}, "chicago bulls": {"points": 102}, "detroit pistons": {"points": 102}} game recap: ```jalen duren had ___ points and ___ rebounds as the detroit pistons overcame a career-high ___ points from zach lavine to beat the chicago bulls ___-___ on saturday night.``ò utput: ```jalen duren had 23 points and 15 rebounds as the detroit pistons overcame a career-high 51 points from zach lavine to beat the chicago bulls 118-102 on saturday night.``ǹ ow, you will be given a new game recap and its working memory, represented as a json object. working memory: {working-memory} game recap: ```{game-recap}``` output: initially, the llm creates an internal json object as its memory. it then enriches this memory by adding necessary game or player statistics, where all values are set to null, and further reflects on whether this memory is sufficient to accomplish the task. lastly, the llm uses detailed play-by-play and team-player data to update the json object's values; it finally utilizes this updated memory to fill in the blanks in the game summary. assists) and negative (missed shots, turnovers) factors. for american football, we apply ncaa's "passing efficiency" formula, as the nfl rule is more complex. in the following sections, we evaluate llms under different adversarial scenarios to assess their robustness. ## experiments we evaluate various llms in our sportsmetrics benchmark. these models are listed in llms' ability to integrate information is tested under three adversarial scenarios: (a) 'newrule,' which assigns every scoring action just one point, regardless of the move, (b) 'swap' which randomly selects two players from each team to swap their affiliations in the team-player table, (c) 'shuffle,' which duplicates any non-scoring action with a 20% chance (@xmath0.2) before shuffling the play-byplays. we assess llms' performance in these scenarios and report the deviation of predicted team points from actual scores as ∆newrule, ∆swap and ∆shuffle. in in figure 6 , we organize games based on the length (number of tokens) of their play-by-play descriptions, with the x-axis showing the games and the y-axis the deviation scores from various llms, where lower scores indicate better performance. we perform a regression analysis to demonstrate each llm's trend in handling games of increasing length. gpt-3.5-turbo-1106 and gemini-pro stand out, maintaining nearly flat curves, which corresponds with their superior performance as shown in we note that basketball teams typically score between 100 to 120 points. our findings show that the smallest prediction gap for ∆points is 9.45, while the largest can exceed 100. this indicates the difficulty in accurately tracking key game statistics over long contexts, as standard llms can produce predictions significantly off from actual scores due to hallucinations. among the three adversarial scenarios, the new rule is relatively simpler as it requires llms to assign one point to every scoring action, focusing on counting these actions instead of distinguishing between types (3-pointers vs. free throws) and adding them up for a team's score. in this scenario, llama-2-13b-chat scores lower than all other llms. in particularly, passing yards are vital in the nfl games, often leading to scoring opportunities like touchdowns and field goals. on average, nfl teams average 200 to 250 passing yards per game. we find that the top model, gpt-4-1106-preview, exhibits a 34.77-yard discrepancy in passing yards prediction, while the open-source llama-2-13b-chat lags significantly in comparison. this highlights the difficulty of tracking passing yards, a task even more challenging than summarizing basketball points, with most models struggling to accurately aggregate such data. 0 2 4 6 8 10 12 14 16 accuracy (%) 5.22 7.51 7.94 7.94 9.12 11.24 16.72 gpt-3.5-turbo-0613 gpt-3.5-turbo-1106 llama-2-13b-chat mistral-7b-instruct gemini-pro gpt-4-1106-preview claude-2.1 figure 8: accuracy of various llms in filling missing key statistics from basketball game recaps. claude-2.1 shows strong performance, while mistral-7b-instruct achieves the highest accuracy among standard llms. our results suggest that the difference in performance between gpt-3.5-turbo-1106 and gpt-4 across basketball and football games stems from the scoring frequency in each sport. basketball's frequent scoring presents a challenge for gpt-4-1106-preview to track all actions, while football, with less frequent scoring, is somewhat easier for the model to track. gpt-4-1106-preview is optimized for handling extremely long contexts and it is less accurate in tracking frequent scoring. this distinct characteristic accounts for the varied performance of both models. in figure 7 , we test llms' robustness against adversarial conditions. in the left subfigure, we vary the difficulty of identifying scoring events by either dropping or duplicating non-scoring events. e.g., at probability @xmath1.5, we eliminate any non-scoring event with a 50% chance; at @xmath2.2, we duplicate any non-scoring event with a 20% chance, before shuffling the entire game description. the y-axis measures the deviation from the actual box score, with smaller values indicating better model performance. we observe that gpt-3.5-turbo-1106 and gemini-pro perform the best, whose curves are quite flat, indicating their robustness to a varying level of noise in the play-by-plays. overall, llms perform well when non-scoring events are removed, yet their performance drops as more non-scoring events are added, akin to searching for a needle in a larger haystack. further, we randomly swapped n players' affiliations in the team-player table and replaced n players' names with science fiction characters, all without changing the play-by-play texts. our findings are shown in the middle and right subfigures. we find that claude-2.1, gemini-pro, and gpt-3.5-turbo-1106 are the top performers. interestingly, renaming players significantly decreases all models' performance. this suggests llms may use familiar basketball player names from their pretraining to guess team scores, rather than analyzing the actual play-by-plays. gpt-4-1106-preview is the least adaptable to these adversarial conditions among the long-context llms. we also observe a notable performance disparity exists between opensource and proprietary llms. we assess the accuracy of various llms in completing missing key statistics from basketball game recaps. the types of missing data include a player's total points, team scores, assists, rebounds, and other stats. an llm must understand the recap's context to precisely estimate the missing statistic. to do this, llms create a json object as its working memory. they then calculate the needed statistics using play-by-play and team-player data and use this memory object to fill in the blanks. figure 8 presents the results of this task. claude-2.1 shows strong performance, while mistral-7b-instruct achieves the highest accuracy among standard llms. this task requires that llms possess strong instruction-following capabilities to build an effective working memory. figure 5 provides sample working memories from various llms. although complex structures are possible, they increase the risk of errors when populating values. models such as gpt-4-1106-preview and llama-2-13b-chat face difficulties in creating a working memory. they hallucinate field values or fail to accurately fill fields with aggregated values from play-by-play data. by contrast, claude-2.1's memory structure is the best in terms of efficiency, focusing on essential game statistics. our task crucially evaluates llms' memory management skills when handling complex data queries. ## conclusion we introduce sportsmetrics, a novel benchmark designed to evaluate llms in sports data analytics. it assess llms' numerical reasoning and fusion abilities through challenges such as new game rules, lengthy descriptions, scrambled narratives and key stats analysis in game summaries. sportsmetrics highlights llms' potential in fields such as multiplayer gaming and collaborative workspaces.