Adapting Coreference Resolution to T witter Conversations (2020)

Abstract: The performance of standard coreference resolution is known to drop significantly on Twitter texts. We improve the performance of the (Lee et al., 2018) system, which is originally trained on OntoNotes, by retraining on manually-annotated Twitter conversation data. Further experiments by combining different portions of OntoNotes with Twitter data show that selecting text genres for the training data can beat the mere maximization of training data amount. In addition, we inspect several phenomena such as the role of deictic pronouns in conversational data, and present additional results for variant settings. Our best configuration improves the performance of the”out of the box” system by 21.6%.

## introduction and related work twitter messages present a discourse genre that includes noisy informal language with abbreviations and purposeful typos, use of nonstandard symbols such as # and @ signs, unintended misspellings, etc., which makes them challenging for nlp applications. we are here interested in the task of automated coreference resolution for nominal mentions in twitter conversations, i.e., threads of messages that specifically reply to one another. in addition to non-standard words, twitter conversations also show peculiar phenomena of referring, such as exophoric pointers to non-linguistic content in attached visual media, and mixed pronominal references to the same entity due to the nature of multi-user conversations @xcite . thus, tweets are a complicated genre for coreference resolution, but at the same time highly relevant for many applications that seek to extract information or opinions from users' messages. in this paper, we use a state-of-the-art resolution system built with the ontonotes corpus @xcite and experiment with adding annotated twitter conversations to the training data. next, we consider the different -spoken and writtengenres included in the ontonotes corpus. we thus conduct experiments with training on different portions, and we show that carefully selecting genre subsets beats the straightforward "taking as much as possible". overall, our best configuration improves the "out of the box" performance of the system by @xcite on twitter data by 21.6%. to our knowledge, there is no work specifically on adapting coreference resolution to twitter, other than the aforementioned study of aktas ȩt al. (2018), which showed a significant drop in performance when a system with ontonotes models is applied to twitter. more generally, one of the few studies on domain adaptation for coreference resolution is @xcite , which adapts the berkeley system @xcite to narrative stories. do et al. do not retrain the system but add linguistic features of narratives as soft constraints to the resolver. regarding ontonotes genre differences, @xcite and @xcite report varying performance in coreference resolution for distinct corpus sections; this work inspired our experiments reported in the following. section 2 describes our data sets, and section 3 the experiments. section 4 provides various additional analyses that shed light on the domain adaptation problem, and section 5 concludes. ## data for our experiments, foot_0 we use the english portion of the ontonotes benchmark used as training set in the conll-2012 shared task @xcite . it has texts from spoken and written registers, and contains gold annotations at different layers, including coreference chains, i.e., sets of mentions referring to the same entity. spoken data includes telephone conversations (tc), broadcast conversations (bc), and broadcast news (bn); written data contains magazine (mz), newswire (nw), pivot text (pt) and web blogs (wb). as shown in our second dataset is the twitter conversation corpus (tw) presented in @xcite . they are tree structures where each tweet has a parent (i.e. the tweet it is replied-to) except for the initial tweet starting the conversation. a tree can be shallow, with many replies on just one level, or it can be deep when participants interact with each other across several turns. the corpus holds 1756 tweets in 185 threads, defined as a path from the root to a leaf node of a conversation tree. foot_1 69% of the coreference chains in this dataset contain coreferential relations across tweets. hence, considering conversation context is important. we illustrate a thread structure with one example of coreference chain annotation in figure 1 . the original tw corpus was annotated with a scheme slightly different from that of ont. for systematic comparison, we modified the tw annotations so that they are conceptually parallel to ont; we thus call the dataset tw' here. ## experiments for our experiments, we chose 'e2e-coref' @xcite , an update of the end-to-end neural coreference resolver presented at emnlp 2017. it introduced a refined approach based on differentiable approximation to higher-order inference, and elmo embeddings @xcite for span scoring, which significantly improved performance on english ont. the approach achieved 73.0 f1, representing the 2018 state-of-art. due to its cost efficiency, speed and flexibility, it was later used as basis for several recent state-of-art models, including spanbert @xcite . ## additional analyses to gain further insight into the adaptation of coreference resolution to twitter, we quantitatively and qualitatively compare the results of the bestperforming test (d) to the baselines (see mention length for all tests, the average token length of mentions additionally predicted by the system (spurious predictions) is significantly longer (@xmath0.05) than that of the correct predictions. the higher the proportion of ont training data (whose mentions are on avg. 0.72 tokens longer than in tw'), the longer those predictions are. at the same time they are significantly shorter (@xmath1.05) than the missed gold predictions. hence there is a tendency to select longer spans (especially when training on ont), but these are also more error-prone. twitter-specific tokens hashtags and usernames caused many errors in test a. in tweets that are replies, user addresses are inserted at the beginning, so the majority of such tweet-initial usernames are not part of the syntax and have not been annotated. pronouns although they are relatively evenly distributed in the gold annotations, more 3rd person pronouns are resolved than 1st and 2nd ps. pronouns in test a, resulting in an overall f1 of 0.769. in test b with twitter training data, which is rich in pronouns, pronoun performance improves for 1st and especially 2nd ps., and remains the same for 3rd ps., improving the f1 to 0.917. in test d, pronoun performance is slightly worse (0.905). as the entire training data in b and d is conversational, which by nature has many 1st and 2nd ps. pronouns, we repeated all test with removing those chains containing only 1st and 2nd ps. pronouns. this is to make sure that improvement is not exclusively caused by easy detection of the pronouns. the results are in column f1 1 in verb annotations verb mentions are possible in ont if they co-refer with a nominal mention @xcite ), but they are not annotated in tw'. thus four predicted verb mentions in test a, of which two are correctly linked with the demonstrative pronoun that, are counted as erroneous predictions. after adding training data from tw' in test d however, no verbal mentions are predicted. to check the influence of this annotation difference, we also ran all tests with the verbal annotations removed from ont, which reduced mentions by 2.4% and chains by 3.6%. column f1 2 in chain linking the last section of ## conclusion we showed that the performance of a state-of-theart "standard" coreference resolution system run on twitter conversations can improve by 21.6% by adding in-domain training data. in fact, even small amounts of added in-domain data can have an impact. further, interestingly, for the out-domain 3 all gold mentions found, but also spurious mentions. training data (ont), the choice of genre can make a bigger difference than the bare amount of data. our additional analyses considered two more variants of the main experiment design: while all results given in in future work we plan to focus more on the specific kinds of training data portions and examine the influence of spoken versus written register, and on that of formal versus informal language (which need not necessarily coincide).