L ive QA : A Question Answering Dataset over Sports Live (2020)

Abstract: In this paper, we introduce LiveQA, a new question answering dataset constructed from play-by-play live broadcast. It contains 117k multiple-choice questions written by human commentators for over 1,670 NBA games, which are collected from the Chinese Hupu1 website. Derived from the characteristics of sports games, LiveQA can potentially test the reasoning ability across timeline-based live broadcasts, which is challenging compared to the existing datasets. In LiveQA, the questions require understanding the timeline, tracking events or doing mathematical computations. Our preliminary experiments show that the dataset introduces a challenging problem for question answering models, and a strong baseline model only achieves the accuracy of 53.1% and cannot beat the dominant option rule. We release the code and data of this paper for future research.

## introduction the research of question answering (qa), where a system needs to understand a piece of reading material and answer corresponding questions, has drawn considerable attention in recent years. while various qa datasets have been constructed to study how a qa system can understand a specific passage, the common sense knowledge and so on @xcite @xcite , most questions in these datasets could be given their answers by extracting from a few relevant sentences so that the model only needs to find a small set of supporting evidences,whose temporal ordering does not effect the final answer. in other words, these questions are raised only considering a fixed document. however, in the real-life question answering, a question could have its timelines. to infer the answer, a good model needs to understand series of timeline information. for example, the question "how many points did lebron james have?" would have different answers based on the time when the question was asked during a basketball game, and the answer would continuously change during the game. the other question "which team would first earn 10 points?" would require a system to track down information of scoring points along the timeline until one team achieves 10 points. according to the analysis above, we consider the timeline-based question answering problem as a gap which has not been covered by existing datasets. thus, in this work we hope to construct a dataset where passages and questions both have timelines and question respondents are required to judge what information should be gathered for the questions involved in a timeline. such a timeline inferenceinvolved qa dataset introduces a new research line of reading comprehension, that evaluates the ability of understanding temporal information of a qa model. additionally, the real-world questions are often involved in some math calculation, such as addition, subtraction and counting. to answer the questions correctly, one not only needs to locate some specific sentences, but also do calculation or comparison on the extracted evidence. for example, "how many points did the winner team win?" needs one system to perform subtraction on the final score to get the correct answer. to these ends, we construct a qa dataset liveqa based on a hupu-live-broadcasting-dataset, which is a set of chinese live-broadcasting passages of nba. hupu is a sports news website that has livebroadcasting for basketball games. in the hupu-live-broadcasting, the host of one sport game describes the details of the game vividly with emotion and different sentence structures, and presents many gamerelated quizzes during the game. we collect the description texts and their quizzes into liveqa. answering the quizzes requires one model to correctly understand the timeline information of the context: some quizzes ask about information of one-whole quarter of the game or which player reaches a certain score earlier. thus, the model needs to fully understand the temporal information of the live-broadcasting and then performs inference based on the temporal information. figure 1 shows four question examples in the liveqa dataset. answering the first two questions requires an addition math operation, and the 3 rd and 4 th questions need comparison operation. meanwhile, we can see that all these questions are time-dependent and require temporal inference. in summarize, the main characteristics of our liveqa dataset include the following two aspects. firstly, the questions are time-awared. the model needs temporal inference to obtain the final answer. secondly, in our dataset, reading comprehension is not limited to extracting a few specific text spans from the document, but is involved with math calculation. these characteristics make liveqa challenging for previous qa systems to answer its questions. in this paper, we present an analysis of the resulting dataset to show how these characteristics appear in the data. we also show how questions are involved with temporal inference, and these questions also require mathematical inference. to demonstrate how these characteristics affect the performance of the qa model, we design a pipeline method, which first tries to find supporting sentences and then uses a strong baseline multi-hop inference model named gated-attention reader, to judge the baseline performance on liveqa. our experimental results show that such strong baseline model only slightly exceeds random choice, which achieve 53.1% and cannot beat the dominant option rule. the analysis and experimental results show how this dataset can effectively examine how a qa system can perform multi-hop temporal and mathematical inference, which is not covered by previous studies. the following of this paper is organized as follows: in section 2, we give a brief introduction of current qa research lines and research on live text processing. in section 3, we describe how we constructed the dataset. in section 4, we give statistics of the dataset and analyse the timelineness and mathematical inference in the data. in section 5, we give evaluation results of baseline models and error analysis. ## related works in this section, we mainly introduce the various qa datasets which can be categorized as datasets with extractive answers, datasets with descriptive answers and datasets with multiple-choice questions. ## liveqa: dataset construction in this section we introduce how to construct liveqa from the raw hupu text and present the corpus statistics. the whole process of building liveqa mainly includes crawling the raw data and acquiring the game texts with corresponding quizzes. ## conclusion in this paper, we present liveqa, a question answering dataset constructed from play-by-play live broadcast. liveqa can evaluate a machine reading comprehension model in its ability to understand the timeline, track events and do mathematical calculation. it consists of 117k questions, which are timedependent and need math inference. due to the novel characteristics, it is hard for existing qa models to perform well on liveqa. we expect our dataset will stimulate the development of more advanced machine comprehension models.