Input Matters: Evaluating Input Structure’s Impact on LLM Summaries of Sports Play-by-Play (2025)

Abstract: A major concern when deploying LLMs in accuracy-critical domains such as sports reporting is that the generated text may not faithfully reflect the input data. We quantify how input structure affects hallucinations and other factual errors in LLM-generated summaries of NBA play-by-play data, across three formats: row-structured, JSON and unstructured. We manually annotated 3,312 factual errors across 180 game summaries produced by two models, Llama-3.1-70B and Qwen2.5-72B. Input structure has a strong effect: JSON input reduces error rates by 69% for Llama and 65% for Qwen compared to unstructured input, while row-structured input reduces errors by 54% for Llama and 51% for Qwen. A two-way repeated-measures ANOVA shows that input structure accounts for over 80% of the variance in error rates, with Tukey HSD post hoc tests confirming statistically significant differences between all input formats.

## introduction despite recent advancements, large language models (llms) are known to produce factually incorrect output @xcite . these errors occur frequently enough to make llm-based nlg applications unacceptable in accuracy-critical domains, unless deployed in a human-in-the-loop setting where mistakes can be corrected. sports reporting is one such domain, and in this paper, we investigate the factual accuracy of llms for generating nba game summaries. this is a challenging text generation task because play-by-play data is highly detailed, encoding entities (teams, players), events/actions (e.g., jump shot, layup, free throw, offensive rebound), relationships (e.g., player -attempts -shot; player -commits -foul), and attributes (e.g., scores, time, period, distance). we hypothesise that structuring the complex play-by-play data is important for producing fac-{ " time ": { " period ": " ot1 " , " clock ": " 2:13.0 " }, " team ": " los ␣ angeles ␣ lakers " , " play_details ": { " description ": " lebron ␣ james ␣ makes ␣ 3-pt ␣ jump ␣ shot ␣ from ␣ 25 ␣ ft " , " event ": { " type ": " shot " , " action ": " jump ␣ shot " , " outcome ": " made " , " distance ": " 25 ␣ ft " , " points ": 3 }, " players ": { " primary_player ": " lebron ␣ james " } }, " score ": { " points_scored ": " lal :␣3" , " cumulative ": { " lal ": 127 , " atl ": 125 } } } tually accurate summaries, whereas unstructured input (garbage in) yields more factual errors (garbage out). to empirically study the impact of input structuring on model output, we evaluate three input formats: unstructured (natural play-byplay descriptions; see recent work by @xcite shows that different tables define structure and formatting in dis- to investigate this, we evaluate two open-source llms, llama-3.1-70b @xcite and qwen2.5-72b @xcite . we use a manual annotation protocol adapted from @xcite and @xcite to assess how input structure affects factual errors in generated summaries. the key findings from our study are as follows: • a two-way repeated-measures anova shows that structured inputs significantly reduce error rates compared with unstructured text, with json providing the strongest improvements (@xmath0.05). prior work on data-to-text generation for basketball @xcite @xcite has used hierarchical encoder models and related architectures to generate summaries from box scores (aggregated player and team statistics), yet consistent factual errors have been reported @xcite . the advent of long-context llms (up to 128k tokens) @xcite has enabled studies on richer inputs; for example, @xcite use nba play-byplay data to compute player and team points. their work focuses on reasoning and aggregation over play-by-play data, supporting the value of using a single rich dataset for controlled analysis. in contrast, we generate paragraph-style summaries from complete play-by-play data and compare three input structures: unstructured input with 13,000 tokens, row-structured input with 28,000 tokens, and json input with 70,000 to 80,000 tokens, producing summaries of 450 to 500 words. @xcite demonstrated that automatic metrics are insufficient for measuring hallucinations. building on @xcite and @xcite , we adapted their manual error annotation protocols to study factual errors at a granular level, making minor task-specific adjustments for the nba play-by-play data to evaluate the errors in our generated summaries. ## results we manually annotated 3,312 errors across 180 play-by-play summaries generated for 30 nba games (2 models × 3 input structures). llama-3.1-70b averaged 408 words per summary, whereas qwen2.5-72b averaged 505 words. to compare factual accuracy across both models and all three input structures, we report normalised error rates, i.e., errors per 100 words (see ## discussion fig. 5 presents partial-game summaries with error annotations to demonstrate a few examples of factual errors produced by our models for three different inputs (taken from the first overtime of the game shown in ## conclusion factual inaccuracies remain a major challenge for deploying llms in real-world applications. our study shows that structured inputs, especially hierarchical json, substantially reduce errors in nba play-by-play summaries: error rates dropped by up to 69% for llama and 65% for qwen compared to unstructured text. a two-way repeated-measures anova confirmed that input structure explained over 80% of the variance in error rates, with significant differences across all formats. as future work, we will develop an llm-as-judge framework to complement costly manual evaluation by automatically assessing atomic factual claims against playby-play input logs. we will also extend this work to ice hockey play-by-play data to identify general data-structuring features that minimise factual inaccuracies across multiple sports datasets.