Variation in Coreference Strategies across Genres and Production Media (2020)

Abstract: In response to (i) inconclusive results in the literature as to the properties of coreference chains in written versus spoken language, and (ii) a general lack of work on automatic coreference resolution on both spoken language and social media, we undertake a corpus study involving the various genre sections of Ontonotes, the Switchboard corpus, and a corpus of Twitter conversations. Using a set of measures that previously have been applied individually to different data sets, we find fairly clear patterns of “behavior” for the different genres/media. Besides their role for psycholinguistic investigation (why do we employ different coreference strategies when we write or speak) and for the placement of Twitter in the spoken–written continuum, we see our results as a contribution to approaching genre-/media-specific coreference resolution.

## introduction research on strategies for producing referring expressions has often investigated the differences (if any) between spoken and written language, but as we will show in section 2, results have been inconclusive. sometimes, claims are simply contradictory, but the more important problem is that usually, the exact ways of measuring the properties of coreference chains are not being made transparent. in addition, the data that has been used can vary considerably, and it is not always clear how studies can be compared. our primary goal here is to shed light on coreference with respect to the spoken/written distinction, by undertaking a careful comparative corpus analysis and explicitly stating our methods of measurement. in an earlier study, we presented a quantitative study on different genre sections of the ontonotes corpus @xcite . we extend that study in two directions: we augment the rather small proportion of spoken data in ontonotes with the switchboard corpus @xcite , and we introduce new quantitative features for the comparative analysis. the secondary goal is to explore how the medium microblog, specifically twitter, relates to the spokenwritten spectrum for coreference strategies. in a recent study, we investigated this research question empirically through computational experiments on a state-of-the-art "standard" coreference resolution system @xcite . we showed that the choice of genre and the medium (spoken vs. written) in training data can make a bigger difference than the bare amount of data. in the current paper, we apply corpus-based empirical methods to situate the coreference strategies found in twitter conversations in the medium/genre spectrum. for this purpose, we finally extend the @xcite study by adding the comparison with twitter texts, thus achieving a wider range of production media. although switchboard and ontonotes have previously been used for investigating coreference, to our knowledge they have not yet been systematically compared. accordingly, an important part of our work is in harmonizing the data sets and the underlying annotation schemes, to enable a sensible analysis. we will show genre-specific distributional patterns of nominal referring expressions in terms of frequency of syntactic categories, heaviness of np structures, and relative distance between anaphors and antecedents in the text. most of our analyses lead to a common ranking and clustering of the genres based on the measures and results presented in section 4, as will be discussed in section 5. in addition to the theoretical value, we believe that the observed patterns can provide a basis for data-driven design of automated coreference resolution tools that perform better on spoken or twitter language than the current "out of the box" systems do. the following section presents related work. section 3 introduces data sources and corpus alignment strategies we applied to harmonize the data. section 4 presents the measures we used for the comparison, and the results of the analyses, which are further discussed in section 5, concentrating on the comparison of the genres and production media. section 6 summarizes and shows directions for future work. other phenomena that have been studied include the distribution of referring expressions in terms of their syntactic categories, i.e., pronouns vs. noun phrases (nps). @xcite argued that referential nps are generally more frequent in written texts than in spoken conversations (47% in written texts vs. 22% in conversations), whereas @xcite and @xcite found different characteristics: on the one hand, they confirm fox's finding that nps are more frequently used than pronouns in the written medium than in the spoken medium. but in written text, according to @xcite , nps are more frequent than pronouns as well (63% nps vs. 29% pronouns), which is not in line with @xcite . for length, they found that the avg. number of tokens of referring expressions in their written data is 3.42, compared to 2.58 for spoken data. another interesting finding on the distribution of syntactic categories is that narrative genres (fiction) show spoken-like characteristics in terms of pronoun usage @xcite @xcite @xcite . further quantitative metrics of coreference phenomena used in the literature are the number of referring expressions @xcite , number of referents @xcite , and chain length @xcite @xcite . we did not examine these features because in ontonotes, documents are artificially split in smaller parts, and because singletons are not annotated (see section 3); and in switchboard, there exist unannotated coreference chains in the data (see section 3.4). therefore, these metrics can create misleading results. although it is heavily used for coreference research, ontonotes has to our knowledge not been extensively examined for quantitative comparison of reference features across genres. among the exceptions is @xcite , who investigated how organisational entities are being referred to, and found a correlation between preferred reference type and genre (e.g., pronouns are more common in telephone conversations than newswire and broadcast news). @xcite used ontonotes for predicting 'notional anaphora' (i.e. pronouns disagreeing with their antecedents' grammatical categories for notional reasons, e.g., "the government ... they."), and found it to be more common in broadcast conversations than in newswire. zeldes used 20 linguistic features, and genre emerged as the third-most important one, indicating that differences between genres can have an impact on automatic classification. automatic coreference resolution for "non-standard" language. only few studies have addressed performance differences of coreference systems across the genres of ontonotes. @xcite reported that their system of choice showed better performance on telephone conversations (64%) than on news texts (56%) and broadcast news (59%). the authors evaluated which sections turned out to be the "easiest" but did not assess the possible reasons. another study that looks in detail at performance differences in ontonotes (and also in two other corpora) is that of @xcite , who compared the performance of domain-specific and generic models, for both knowledge-poor statistical systems and for implementations using hand-crafted linguistic features. similarly, coreference resolution specifically for spoken data has not received much attention. beside work on the rather domain-specific data of the trains corpus by @xcite , @xcite worked with the switchboard corpus and gave results for an algorithm resolving personal and demonstrative pronouns. they found that pronoun usage differs considerably from that in written text, because of a high frequency of both non-nominal antecedents and pronouns with no antecedent at all. @xcite refined the algorithm, and we are not aware of more recent work in this vein. while twitter has often been tackled for standard preprocessing tasks like pos tagging or ner, to our knowledge there are no previous studies investigating coreference resolution on twitter data except for our own studies @xcite . in @xcite , we report that a state-of-the-art coreference resolver trained on ontonotes data performs worse on twitter data (@xmath0.18 on twitter vs @xmath1.6 on original test data). in the same study, we conducted experiments with different training data settings and improved the performance of the tested system on the twitter data by 21.6%. in follow up work, we aim to further improve coreference resolution on the genre of twitter conversations by customizing the resolver according to the genre-based differences, including the contrasts that emerged from the comparative analysis presented in the current paper. ## corpora we use three english-language data sources for our empirical study. see ## data analysis we use frequency-, heaviness-and distance-metrics in the quantitative comparison of the (sub-)corpora surveyed in ## discussion our frequency-based analysis shows that the relative frequencies of pronouns and nps in the swbd/tc pair are close to each other, whereas nps are much more dominant in nw/wb. the comparison of distance-based features indicates that the textual distances between anaphoric pronouns and their antecedents are longer in swbd/tc than nw/wb. and lastly, nw/wb genres contain heavier nps than swbd/tc do. these inter-medium differences are statistically significant for all the measures except tbd, whereas for a number of cases, as demonstrated in section 4, intra-medium differences were not confirmed statistically. the significance levels can differ when parentheticals and discourse markers are excluded in the computation of features, but these differences do not affect the statistical significance levels between the swbd/tc and nw/wb pairs. the placement of the bc and tw genres in our genre ranking differs according to the measure. for instance, tw is located close to the swbd/tc pair with respect to np-length, but to nw/wb pair with respect to nbd. in addition to rankings derived from average-value orderings, we also used hierarchical clustering for grouping the genres based on the quantitative features. to obtain the clusters, we first normalized the data size, and used the complete linkage method for hierarchical clustering (i.e. the maximum distance between the members of clusters is taken as the cluster distance). for this purpose, we applied the hclust method in r with default settings. similar to the observations mentioned above, the swbd/tc and nw/wb pairs are always grouped together for all the metrics except nbd for swbd/tc and np-height for nw/wb. again, for bc and tw the grouping differs from feature to feature. for instance, in figure 3 , bc is clustered in the same branch with nw/wb, whereas in figure 4 , it is grouped together with the swbd/tc pair. the ranking-based and cluster-based groupings do not always overlap (as in the nbd case), but the proximity of the swbd/tc and nw/wb pairs is observed as a common pattern in both cases. a few exceptional cases of this pattern require more attention. the groups of bc and tw depend on features: bn genre is usually grouped with the nw/wb pair, as it is expected due to the edited content of the genre. the only exception is in hierarchical clustering with respect to the frequencies of personal pronouns, where bn is clustered together with swbd/tc. we consider these findings (i.e. existence of more rigid clusters as well as the floating genres between the groups) as supportive evidence for perceiving spoken/written media as a continuum rather than discrete concepts, and it turns out that twitter has a medium position in the continuum w.r.t. to the investigated features. the exact position differs from feature to feature. 6 conclusions and future work we presented an in-depth study of coreference strategies across genres that involve different production media (spoken, written). as a prerequisite, we harmonized the annotations (for syntax and coreference) of three corpora, and to our knowledge, this is the first systematic comparison of its kind. our findings and interpretation for the spoken-written dichotomy and the placement of twitter have been given in the previous section. some of the features used in this paper such as pronoun and np frequency, the distribution of personal pronouns, and the length of noun phrases are also considered in more general discussion of spoken and written language @xcite @xcite and our results are compatible with these previous findings. although we find the discussion about fine-grained characteristics of genre and medium (e.g. "involvement" vs "detachment" @xcite ) interesting and important, considering the scope of this paper and size constraints, we leave it as a future work. in addition, we see our results as potentially fruitful for adapting automatic coreference resolution to genres and media, in the light of small amounts of training data, where knowledge about the differences in strategies could be utilized. such experiments are a part of our future work, as is a psycholinguistic study (text production experiment) designed to validate the results on the spoken-written contrast from a perspective that is complementary to corpus analysis.