Grounding NBA Matchup Summaries (2021)

Abstract: The present paper summarizes an attempt we made to meet a shared task challenge on grounding machine-generated summaries of NBA matchups ( https://github.com/ehudreiter/accuracySharedTask.git ). In the first half, we discuss methods and in the second, we report results, together with a discussion on what feature may have had an effect on the performance.

## introduction what led reiter and thomson (2020) to launch a shared task competition in 2020 was a concern that fact-checking automatically generated texts (machine texts, or m texts) in the context of data to text generation (wiseman et al., 2017), is hugely labor intensive, making it virtually impossible to run it at a scale. in an effort towards putting it under control, the project asks participants to find a way to do the assessment automatically, without any human intervention. the problem is set out as follows: you receive m texts, along with other external information such as box scores and human created summaries (or h texts). your goal is to locate factual mistakes in m texts and classify them according to a pre-defined scheme of error types ('word,' 'number,' 'name,', 'context,' 'not checkable'). ## method the following sections detail our approach, which in essence is a multi-pronged strategy. we deploy separate mechanisms to deal with different types of error. ## resolving coreference given the way utl works, it is important that we make explicit what a referring expression points to, in order for utl to successfully build a relq. to this end, we make use of neuralcoref 4.0, foot_7 which operates as an add-on functionality allow disallow w w @ w w # w w w w w w w w # w @ w @ w w w w # w w w w w w w w # @ w w w @ w w w # w w w w w w w w # w w @ for spacy. @xcite resolving coreferences with neural-coref (nc) results in every referring expression (r-expression, hereafter) in a text being replaced with a corresponding root entity (i.e. its canonical name). this can be troublesome though, because it may disrupt the way in which words are originally laid out, which we need to retain in order to report results conformant to the shared task format policy (which asks to report errors by indicating where they are in the original position). in response, we pursued an approach where we represented a text with a linked list structure in which each word is represented as a node which contains information on what node it is preceded by and what it is followed by, in addition to where it occurs relative to others. @xcite for each r-expression nc found, we replaced a token string held by a relevant node with its antecedent while keeping other information (occurrence site, forward/backward connections) in tact (fig. 2 ). furthermore, we restricted an r-expression subject to replacement, to be among 'their,' 'they,' 'he,' 'his,' 'its,' 'it,' and 'him.' ## setup and results the training data that ned used are sourced from part of the rotowire corpus (wiseman et al., 2017), foot_10 called 'train.json,' which contains 3,398 matchup results each with a summary manually ## conclusion this paper gave an overview of the approach we took to meet the shared task challenge for 2021, which is essentially a combination of hand crafted rules (wed) and machine learning (ned): it relied on rule based heuristics to identify errors in name and word while bringing in a neural model to locate number errors. the rule based part consisted of 'translating' a cue expression into a procedure to query box scores for its veracity, while the machine learning part was driven by a neural model, whose predictions allowed us to detect inconsistencies in number related statements in m texts. @xcite and crd on the 2021 test. @xcite response to organizers' question (which is about how we would cope with unknown player names, venues and teams). bringing up to date the database we use for wed, of players, teams and venues through wikipedia and sources devoted to nba conferences (e.g. box-office scores), could lessen a possible negative impact due to the lack of exposure to data not available at the time of training. if that does not work, we may go to some off-the-shelf ne tool such as one by spacy, though we expect it may hurt name error detection (due to its reduced accuracy), and to a lesser degree word error detection as it does not involve the recognition of venue names (player and team names can easily be picked by looking at box-office scores which we assume are available all the time). cue expressions we used were fairly generic. it is highly unlikely that they become less effective on summaries beyond the training data, though we recognize the need to expand the list.