Neural Network-Based Generation of Sport Summaries: A Preliminary Study (2021)

Abstract: This paper presents a global summarization method for live sport commentaries for which we have a human-written summary available. This method is based on a neural generative summarizer. The amount of data available for training is limited compared to corpora commonly used by neural summarizers. We propose to help the summarizer to learn from a limited amount of data by limiting the entropy of the input texts. This step is performed by a classification into categories derived by a detailed analysis of the human-written summaries. We show that the filtering helps the summarization system to overcome the lack of resources. However, several improving points have emerged from this preliminary study, that we discuss and plan to implement in future work.

## introduction in this paper, we propose a new approach for sport commentary summarization. this approach is still an ongoing work, and the results presented here are preliminary. sport commentaries represent an interesting resource, as the live commentaries we work on are associated with a summary written by an expert : the commentator himself. indeed, the commentator writes a summary at the end of every game. automatically generating a game summary would release the commentator from a part of his heavy workload and thus would free up his time for more complex and rewarding tasks, such as game in-depth analysis. summarizing live sport events commentaries is a challenging task. first of all, they are live written; new commentaries can conflict with or complete former ones. for example, if a soccer player scores two goals, the live commentaries about each goal are relevant information. however, extracting the live commentaries about each goal will not be suffi-cient in order to generate a good summary. it would indeed lead to producing redundant summaries. moreover, if you consider a game as an event, it is composed of several subevents. some of them are deemed important enough to be commented. however, most of those commented subevents are not important enough to appear in a summary. so, live commentaries are mostly made up of noise : in a soccer game, there will be more shots than goals, even more substitutions than goals, which are the most important information of a game. this noise has to be filtered. therefore, studies have to be carried out about the relevance of an information in the game summarization context. the style of human-written summaries differs from the one of live commentaries. for all these reasons, statistical extractive summarization models are not relevant for this kind of data. extractive models consist in fact in extracting relevant sentences from source corpora and put them together in order to build a summary. the difference in style between summaries and commentaries and the noise in commentaries are a substantial hindrance to building live commentaries summaries with extractive models. as for neuronal abstractive models, fast-growing these past few years, they need huge training corpora to be efficient : several hundreds of thousands of documents associated to their summary. however, we can only assemble a corpus that covers five years of a national championship -approximately 1700 game commentaries associated with their summary. moreover, neural abstractive summarizers are mostly designed for news summarization. news summarization fits well neural summarizers as neural models can only take a limited number of words for input. journalists use an inverted pyramid structure, so the most important information is packed in the first paragraph. taking the first n words as input ensures that a neural model will work only with important information. live commentaries do not have such a hierarchical structure, and commonly used statistical indicators do not seem to be useful. therefore, selecting the input of a neural model is a challenge in our application. in order to overcome these obstacles, we need to train our own abstractive summarization model. we have to reduce the data noise in order to allow the abstractive model to converge quickly with a limited amount of data. we hypothesize that the human summaries variability is low enough to make it possible for the model to learn despite having a small corpus as input. we propose a global summarization method that aims to lower the input data entropy in order to enhance the automatic summaries quality. this method relies on a an information selection prior to learning in order to shorten the input texts and thus get rid of data that are useless to the summary generation. the paper is structured as follows: in a first part, we present the related work. in a second part, we introduce our corpus and its features. then we describe our method, followed by the experiments and the results. we end by discussing the results and exposing our perspectives. ## related work the automatic generation of sports commentary summaries is, to our knowledge, a subject that is very little discussed in the literature. we can mainly cite the work of @xcite which, from game commentaries available online, generates an extractive summary. the method consists of three main steps: a first step of modeling sentences according to surface clues defined empirically as the sum of the tf-idf sentence words, the presence or absence of important words in the sentence such as "red card", "goal"... then, from a set of reference sentences, a learning step predicts the sentences rouge score according to these surface clues. finally, a last ranking step allows the sentences with the best scores to be incorporated into the summary. according to these authors, however, the approach suffers from several limitations. since the process is sentencecentered, the summaries generated have a lot of redundancy between sentences. furthermore, learning tends to penalize short sentences that are sometimes wrongly considered less informative because their direct contribution to the rouge score is lower. on this same task, we can also cite the work of @xcite whose particularity is to propose a system of generative summary based on the definition of a specialized ontology for soccer games. thus, from the data extracted from commentaries in an ontology, handwritten rules are triggered in order to rephrase the information and generate a summary. the main limitations of the approach are the need for an exhaustive ontology population (players, teams,...) as well as a generation of stereotypical summaries because they are built from the same rules. on the same issue but from a very different angle, the work of @xcite starts from the comments of twitter users during games and produces subjective summaries. for each official commentary related to an event during the game, the supporters' comments on twitter are analyzed on a 4 minutes window, the goal being to extract the most representative tweet of this event from the subjective point of view of the supporters of each team. for that purpose, the tweets are first distributed between the two teams. a user is defined as a supporter of a team if in his comments the team is overrepresented compared to other teams. in a second step, for each team, the most important topics are detected using a variant of the tf-idf. finally, for each team, the most representative tweet of the subjects found is selected, without further processing. more recently, the work of @xcite ) presents a model able to produce nba games summaries. based only on game statistics, it can generate a summary composed of two parts: a game overall summary and a player centered summary. this model uses latest deep learning techniques with a wasserstein generative adversarial networks (wgan) proposed by @xcite . however, despite the model used, the method only generates stereotyped summaries filling a fixed template. for the overall summary, the template is defined as follows: except for the learned phrase, all elements in [] are directly assigned from game statistics. the learned model will only affect the learned phrase used to characterize with words to which degree team a wins or loses against team b. to avoid the pitfall of generating stereotypical summaries, given the difference in style between the summaries and the live commentaries, the high noise level in the commentaries and the low volume of data, we choose to stand out from previous methods by taking advantage of recent advances in neuronal generation. to do so, we approach the problem from the perspective of neuronal generation preceded by a noise reduction step in the source texts to allow the generative model to converge quickly. ## live commentaries corpus we extracted all the available ligue 1 soccer games commentaries and their associated humanwritten summary from l'equipe website ( http: //www.lequipe.fr ). the archived games with live commentaries cover a five seasons period from 2014 to 2019. game commentaries are 8000 words long on average and 80 interventions of the commentator. live commentaries about the game are interspersed with facts about the game players: information about a recent trade, an ongoing goals streak... manual summaries are 55 words long on average (cf. figure 1 ). therefore, live commentaries are significantly longer than 400 words which is the length commonly used by most neuronal summarization models in order to reduce their complexity @xcite . the complexity of automatic summarization based on such live commentaries is thus far too important, especially given the small amount of documents we can use to learn. figure 1 shows the three last minutes of paris vs lille commentaries. it displays the noise in these documents. one can especially notice a poll posted by the commentator between 90+1' and 90+2' minutes, and three commentaries that one can consider as noise for the purpose of generating a short summary: extra time announcement and three missed actions. one can also notice the difference in style between the summary and the commentaries. the specificities of these documents force us to rethink the automatic summarization process by first filtering only relevant information, then generating a summary in order to mimic manual summaries style. ## our model automatically summarizing soccer game commentaries presents a major difficulty: standard frequency-based techniques for evaluating the importance of a word or sentence are ineffective due to the source documents specificity: the most important information about a game is often the rarest one -result, expulsions, goals. this, combined with the style of the summary which is radically different from the commentaries, led us to abandon the extractive methods for an abstract method. however, the small number of documents that can be used for learning is very restrictive foot_0 . we hypothesize that, given the small average size of summaries and their low linguistic variability, the decoder part of an encoder/decoder model can learn how to generate a summary using style elements of manual summaries. on the other hand, the relatively large size of the commentaries makes the encoding task more complicated, if not impossible given the small size of the learning corpus. therefore, we choose to reduce the entropy of the source texts in order to allow an encoder/decoder to learn how to generate summaries with a restricted corpus size. the simplest idea is to keep in the input commentaries only those that are deemed relevant for the development of a summary, before learning an automatic summary model based on these filtered commentaries. we detail both steps here. ## experiments to test our approach, we compare two automatic summary models: an extractive method, textrank @xcite as well as a generative method, pointer-generator on the sample of the last 167 league 1 games that were not used during any learning step. experimental setup : both methods are tested with and without prior filtering of the sentences judged relevant, according to the method presented in §4.1.2, in order to validate the hypothesis that reducing entropy in source texts has a positive effect on model convergence and on the quality of the summaries produced. the workflows for generative methods with and without pre-filtering are shown in figures 4 and 5 . figure 5: pipeline architecture with classification the language model uses a word embedding layer of dimension 64. the recurrent cells of the encoder and decoder are of dimension 64. the size of the output layer is the size of the vocabulary, which is 4480. the classification model uses the same word embedding layer that it retrieves from the language model after training and two layers of lstm (bi-lstm) each of size 16, the output layer is of size 2 (0 or 1 for important and unimportant). the pointer-generator model uses an encoder and a decoder with bi-lstms of dimension 128. the vocabulary size of the model is 50000. during training, the model takes texts truncated to 400 words and produces summaries of no more than 100 words, which is much more than the number of words used in human summaries. in order to reduce the size of the problem and speed up training, we have a batch size of 4. error backpropagation is done with the adam optimizer with a learning rate of 0.15. the models were learned over 30000 iterations (80 epochs). it takes fewer iterations than the @xcite model to get results because our training set is much smaller. evaluation metric: summaries are evaluated with the commonly used rouge package @xcite . the rouge-n score is a metric that compares the n-grams in common between the reference summary and the summaries to be evaluated. we took as a reference summary the summaries written by the commentator at the end of the game. we use the specific configuration that showed the best correlation with human evaluations in @xcite (a rouge-2 precision score). baselines: we compare the generative method to two extractive methods: one without, and one with filtering commentaries. we use textrank @xcite , a method comparable to @xcite but designed for mono-document summarization. it is a graph-based method that considers summarization as the extraction of the most central sentences in a graph. the implementation used is the one of @xcite , freely available online 2 . even if textrank method was designed in 2004, it was shown in @xcite that it still compares to more recent methods when there is no correlation between sentence position and centrality. ## results the results are presented in the we also find that extractive models are better in recall but less accurate than generative models. the extractive model used here indeed maximizes the number of words in the abstracts, unlike the generative model. as a result, extractive summaries are composed of 69 words on average compared to 44 words for generative summaries, so they can carry more information. it is also noted that while the generative summaries present relevant information, it is often in a home draw for nantes that sinks unsurprinsigly in score, lille are splitting for the relegation zone. fc nantes loses for brave losc players, but finds itself the brace of rankings. figure 6 : example of a summary generated by our generative system (we try to retranscript syntactic errors in our translation in italics) extractive summaries contain a great deal of redundancy and irrelevant information. using surface clues other than purely frequency clues, such as @xcite , would surely partly solve this problem. figure 8 presents an extractive summary for the same game as the summaries of figures 6 and 7 . ## discussion our model offers relevant information, but with an approximate linguistic quality. we assume that le fc nantes et lille se séparent sur un match nul après une fin de match haletante. malgré un sala décisif (doublé), les canaris pourront nourrir des regrets après avoir mené deux fois au score. nantes reprend sa 5e place devant montpellier tandis que lille sort de la zone rouge. fc nantes and losc split in a draw after a thrilling end of game. despite a decisive sala (brace), the canaris will have regrets after having led two times. nantes takes back its 5th place ahead of montpellier, while lille moves out of the relagation zone. sala signe un doublé ! thomasson insiste à gauche dans la surface mais se heurte à un lillois. lucas lima hérite du ballon et propose un centre du pied gauche. totalement libre aux 6 mètres, sala place sa tête et trompe maignan sur la droite. sala attaquant c l'equipe grâce à son buteur sala (9e but en l1 cette saison), le fc nantes mène très logiquement à la pause tant les canaris ont été solides et efficaces. but de sala ! il centre pour thomasson qui prolonge vers sala. sala attaquant sala scores a brace! thomasson keeps pushing on the left of the box but gets blocked by a losc player. lucas lima gets the ball and crosses with his left foot. free at the 6-yard line, sala heads the ball and beats maignan on his left side. sala foward. c l'equipe thanks to his scorer sala (9th goal in l1 this season), the fc nantes leads unsurprisingly after the first half as they were solid and efficient. sala scores! he crosses for thomasson who passes towards sala. sala forward figure 8: example of an extractive summary (manual translation in italics) as we assumed, the extractive model results in very noisy juxtapositions of commentaries that contain a lot of irrelevant information. the results are therefore far away from human summaries. this is shown on the one hand by reading the summaries produced and on the other hand by the evaluation of the rouge score. commentary filtering prior to learning does improve the quality of the generated summaries. learning is simplified by reducing noise and the size of the input data. we did not perform a manual evaluation of the summaries, because the work is still in its early stages and the time needed for a manual evaluation, e.g. pyramid @xcite is better spent in late stages. however, we analyzed the automatic summaries produced by our method, and we made the following observations: this can be due to several causes: first, even if we filtered the commentaries, they are still longer on average than the 400 words commonly used as input of neural summarizers (and used by our summarizer as well). given the small amount of data available for training and the fact that information is cut from the input texts, it can explain that major information is missing. second, the filtered commentaries are still noisy. instead of using filtering techniques, information extraction techniques could be used to fill predefined templates for the 17 important information categories we defined in §4.1.1. this would lead to more concise input texts, focusing on the core of each relevant information only. third, some information cannot be found in the commentaries. we think of championship ranks, ongoing streaks, which are rarely raised in pregame commentaries, or the overall technical quality of a game, which can be derived from game statistics (percentage of completed passes). however, these statistics can be extracted and given as input of the pointer-generator decoder. this way, the pointer-generator would have access to the information needed to generate sentences conveying what our analysis of the data considered as major information. ## conclusion in this paper we presented a model that allows the generation of abstractive summaries of specialized documents with limited data in french language. our goal was to show that for summary generation and in specific contexts, abstract models could converge more quickly by reducing the entropy of the input data. our preliminary results show that after having filtered the input texts and even with a small amount of data, the neural summarizer reaches a much higher precision, and also a better linguistic quality. we found that much of the information needed to manually generate summaries is not present in the live commentaries. indeed, many important facts: absence of a player, efficiency, domination of a team, balanced game, are often only deductible from non textual data. systematically providing this input data to a generative system can help it to improve summary generation. in this way, we plan to add to the text sequences the relevant statistics for the generation of summaries. we also plan to provide more focused and concise texts as input to a neural generative summarizer in order to improve its summaries even with a limited amount of data. we could also improve a language model by using extra texts about soccer games, and thus improve the linguistic quality of the generated summaries.