SOCCER : An Information-Sparse Discourse State Tracking Collection in the Sports Commentary Domain (2021)

Abstract: In the pursuit of natural language understanding, there has been a long standing interest in tracking state changes throughout narratives. Impressive progress has been made in modeling the state of transaction-centric dialogues and procedural texts. However, this problem has been less intensively studied in the realm of general discourse where ground truth descriptions of states may be loosely defined and state changes are less densely distributed over utterances. This paper proposes to turn to simplified, fully observable systems that show some of these properties: Sports events. We curated 2,263 soccer matches including time-stamped natural language commentary accompanied by discrete events such as a team scoring goals, switching players or being penalized with cards. We propose a new task formulation where, given paragraphs of commentary of a game at different timestamps, the system is asked to recognize the occurrence of in-game events. This domain allows for rich descriptions of state while avoiding the complexities of many other real-world settings. As an initial point of performance measurement, we include two baseline methods from the perspectives of sentence classification with temporal dependence and current state-of-the-art generative model, respectively, and demonstrate that even sophisticated existing methods struggle on the state tracking task when the definition of state broadens or non-event chatter becomes prevalent.

## introduction state tracking, the task of maintaining explicit representations of user requests and agent responses, has long been a key component of dialogue systems @xcite @xcite . the same challenge arises during reading comprehension of procedural texts (recipes, how-to guides, etc.) where systems focus on predicting changes of object attributes at the entity-level (a car window may transition from foggy to clear) @xcite . however, both of these state tracking variants rely on transaction-based or turn-based data such as transactional dialogues or procedure descriptions that are information-dense. few works have studied state tracking tasks where state changes occur infrequently while a large proportion of messages are "chatter". as an alternative to altogether unrestricted state tracking-a task that is daunting due to the complexity of even describing ground-truth states in a discrete manner-we resort to a simpler and more self-contained setting: sports competitions. given the stream of natural language utterances with which a commentator describes the events in a real-world setting (here a sports competition), an ideal natural language understanding system would maintain and reason over a coherent and accurate representation of the match based on how the commentator described it. this representation can, in turn, be used for downstream tasks such as inference or language generation. sports matches provide an ideal test bed for state tracking due to their self-contained, fully observable nature and their inherent interpretability in the form of the temporal evolution of scores. however, existing sports-related commentary collections such as described by @xcite and @xcite do not provide such within-match temporal information. to this end, we collect temporally-aligned commentaries and live scores of soccer matches along with other meta information from the website goal.com and compile the dataset soccer. to the best of our knowledge, soccer is the first temporally-aligned collection of sports match commentary and state. it contains over 2,200 matches from tournaments such as the uefa champions league or the uk premier league between 2016 and 2020. across these matches, there are over 135,000 individual comments and approximately 31,000 events. a simplified example is shown in figure 1 . to demonstrate the potential of state tracking for open-domain discourse, we use the proposed dataset to investigate to what degree state-of-theart systems are able to track the progression of events described in the commentary. this overview includes two model classes: classification models that treat match events as different class labels, and generative language models such as gpt-2 @xcite that model context and events in a causal manner. our experiments show that both methods do not perform well on soccer and only slightly outperform distributional heuristics, leaving considerable room for improvement. the novel contributions of this paper are threefold: (1) we propose a new task of tracking event occurrences via state changes, (2) we create soccer, a general discourse state tracking dataset that contains temporally-aligned human-composed commentary and in-game events, serving as the training and evaluation dataset for this task, and (3) we provide two intuitive baselines demonstrating the difficulty of this task and presenting exciting opportunities for future research. ## dataset construction we collect time-stamped commentary with key events of 2,263 soccer matches in total. the matches stem from four major soccer tournaments including the uefa champions league, uefa europa league, premier league and series a between 2016 and 2020. soccer consists of over 135,000 time-stamped pieces of commentary and 31,000 within-match events. this section describes our data collection and preparation process in detail. ## state definition and task proposal for each match m in the dataset m , there is a set of timestamps t @xmath0. as input, we are given a stream of commentaries c @xmath1. the output will be a set of general match states s @xmath2. s t contains a set of events e (t) i,j , where i represents the event types (@xmath3, assist, yellow card, red card, switch}) and j denotes the event actor (@xmath4, guest}). given the sparse distribution of s t , we propose two alternative variants of the variable to assess the difficulty of state tracking at different granularity levels of state resolution. team level. in this simplest notion of state, events are tracked at the team level. in other words, e (t) i,@xmath5, no}. consider the event of the home team scoring a goal e (t) goal, home at time t as an example: given the commentary c t and other related meta-information, a model is tasked with determining the value of e (t) goal, home to be yes if the home team indeed scored a goal in a given minute, or no otherwise. player level. at this significantly increased level of resolution, all events are additionally associated with their player agents @xmath6, where p denotes the collection of players. concretely, the variable e (t) i,j is mapped to either the related players' names p or a none answer to each event at time t. to facilitate this form of state, match metainformation includes lineups that associate present players with teams. ## analysis and baseline experiments in the following, we provide descriptive statistics of the soccer dataset and include two model baselines for recognizing match events resulting in changes of states. ## results this section reports the results on the test set of soccer. as a naïve distributional baseline, we compute the ratio of negative cases in the test set to be 0.9766. in ## conclusion in this paper, we introduce soccer, the first discourse state tracking collection in the sports commentary domain. we propose two different levels of state granularity and provide two performance benchmarks for models ranging from gru @xcite for embedding temporal dependency to gpt-2 @xcite for causal language modeling. the dataset shows a much lower information density than many existing resources on state tracking, making it considerably more challenging. we believe that, in conjunction with the wide vocabulary of player-level notions of state, this property makes soccer an exciting resource on which our community can advance discourse state tracking to a broader range of settings than have been studied previously.