Multi-layered Annotation of Conversation-like Narratives in G erman (2023)

Abstract: This work presents two corpora based on excerpts from two novels with an informal narration style in German. We performed fine-grained multi-layer annotations of animate referents, assigning local and global prominence-lending features to the annotated referring expressions. In addition, our corpora include annotations of intra-sentential segments, which can serve as a more reliable unit of length measurement. Furthermore, we present two exemplary studies demonstrating how to use these corpora.

## introduction the rapid development of nlp increases the need for high-quality corpora and corpora of different registers and languages. however, most of the available corpora are in english, and on formal written genres such as wikipedia @xcite , and newspaper articles @xcite . but in order to study or generate more naturalistic, colloquial, and spoken language, corpora based on less formal registers must be created and investigated. due to the high complexity of handling spoken data, spoken corpora are less common than written corpora in nlp studies. using written corpora that resemble conversational language is one way to reduce the gap between colloquial and formal speech data. in the current work, we present two corpora based on excerpts from the german novel tschick foot_0 @xcite and the austrian novel auferstehung der toten foot_1 [henceforth adt] @xcite . these corpora both have a conversation-like narrative style. we are building a conversation-like corpus to study the choice of referring expressions (res) in naturalistic language use. our motivation to use a corpus other than the ones using formal language is that the register of a text can influence the choice of res. for instance, some referential forms are restricted to formal registers, whereas other forms occur more often in informal language. an example are the german demonstrative pronouns dieser and der, where dieser is more likely to occur in formal texts, and der in informal texts or spoken language @xcite . we find the creation of this corpus and the extensive annotations valuable for the following reasons: the structure of this paper is as follows: in section 2, we present an overview of available corpora for the study of reference. section 3 sets out the motivation for our annotations. in section 4, we introduce the corpora we are developing, followed by a detailed overview of our annotation practice in section 5. in section 6, we demonstrate the application of the annotation by presenting case studies. finally, we conclude the paper with discussion and conclusion in sections 7 and 8. ## related work there are numerous corpora that include annotations of referring expressions. according to @xcite , these corpora can be classified as either collected or found. collected corpora consist of data gathered in systematically designed experimental settings, whereas found corpora are composed of naturally occurring language data obtained in real-life situations, such as those found in newspapers or telephone conversations. most well-known collected corpora that include referring expression annotations are based on elicited language, using giver-director games (e.g. @xcite @xcite . therefore, they do not include a rich character / protagonist structure. also, these elicited corpora do not show a consistent, long-lasting narrative structure, but rather short exchanges about mostly inanimate entities. for instance, the scare corpus @xcite is based on spontaneous instruction-giving dialogues that were collected in a virtual reality game. the corpus, however, only contains annotations of res referring to inanimate entities such as a door, cabinet, and buttons that are entailed in the virtual reality world. the coconut corpus @xcite is another corpus including naturalistic language. it is based on computermediated dialogues collected in an experiment in which two human subjects collaborated via typed dialogue on the task of buying furniture to decorate two rooms of a house. the corpus includes only annotations of res that describe task objects. therefore, it only includes res referring to inanimate entities. also, the popular tuna corpus @xcite of elicited spoken english only includes res referring to inanimate entities. there is also a german pendant, the g-tuna corpus @xcite , which also does not include annotations of animate referents. in addition, there are two other corpora associated with the analysis of german res, namely the give-2 corpus @xcite and the pentoref corpus @xcite . both corpora rely on elicited naturalistic spoken language and only include annotations of res referring to inanimate entities. there are also a few narrative corpora that have been elicited through experiments, which offer the advantage of language production in a more "reallife" context. a shortcoming of this approach is that the elicited narratives usually describe a rather random topic (in order to ensure comparability) and are comparatively short and less complex. for instance, the inscript corpus @xcite provides simple english narratives that are centered around a specific scenario. the narratives were elicited by asking participants to describe a given scenario in narrative form, pretending to be explaining it to a child @xcite . the corpus includes coreference annotations of res referring to both inanimate and animate referents. but the corpus does not include detailed annotations of the referential form or additional syntactic and semantic features. in addition to the above-mentioned corpora that were collected in experiments, there are also various found corpora containing reference annotation. an example is the gnome corpus that consists of texts describing museum objects and patients' information leaflets @xcite . the corpus contains extensive annotation on the sentence and reference level. the annotation of referents contains information such as animacy, referential form, grammatical role, and gender. two other corpora which have been built specifically for investigating the form of referring expressions in context are grec-2.0 and grec-people @xcite . the data in the grec-2.0 corpus contains the introductory paragraphs of almost 2000 wikipedia articles classified into five categories: people, city, country, river and mountain. the grec-people corpus consists of 1,000 introductory sections of wikipedia articles in the category people, with subcategories chefs, composers and inventors. a limitation of these corpora is that in grec-2.0, only references to the main subject of the text have been annotated, and in the grec-people corpus, only references to human referents are marked. additionally, since the texts consist of only the introductory section of an article, they are relatively short. the narrative corpus @xcite includes conversational narratives, extracted from the demographically-sampled subcorpus of the british national corpus. however, the corpus does not include annotations of referring expressions, but rather of broader concepts such as speaker (social information on speakers), text (text ids, title, type of story, type of embedding, etc.), textual components (pre-/postnarrative talk, narrative, and narrative-initial/ final utterances), and utterance (participation roles, quotatives, and reporting modes). to build annotated reference corpora, various annotation schemes were also developed along the way. the gnome corpus is annotated using a comprehensive set of guidelines from the mate/gnome annotation scheme @xcite . extensions of this scheme facilitated additional reference annotations, including the annotation of abstract anaphora, i.e., cases where linguistic antecedents are verbal phrases, clauses, and discourse segment @xcite . reflex, as a more recent reference annotation scheme, facilitated the annotation of information status (including coreference and bridging) as well as lexical information status (semantic relations) of referents @xcite . ## linguistic motivation for annotations it is well known that the form of an re corresponds to the cognitive status of the discourse referent (e.g., @xcite . psycholinguistic research has shown that so-called prominencelending features (von heusinger and schumacher, 2019) influence the referential form of res and their interpretation (e.g., pronoun resolution of ambiguous pronouns). it has been shown that multiple local and global prominence-lending features contribute to the interpretation of res @xcite @xcite . for instance, for pronoun resolution, many cross-linguistic studies have examined the grammatical role of the previous mention as an influential feature @xcite . other studies have highlighted the importance of thematic roles @xcite as well as information structural cues at the discourse level and distance @xcite . also, perspectival features have been shown to influence the re form @xcite . ## our corpora the tschick corpus was formed from 9 chapters of the novel tschick @xcite : chapter 28 to 31, and 42 to 46. the novel can be described as a road novel @xcite or a coming-of-age novel @xcite . the adt corpus was formed from the first four chapters of the crime novel aufer-stehung der toten @xcite . both novels represent immensely successful contributions to contemporary german literature and have been recognized with awards. from a linguistic perspective, the novel tschick is interesting for two main reasons: first, the novel is characterized not only by a naturalistic and conversation-like narration style, but especially by the very authentic and timeless use of youth language. this allows the investigation of the use of res in a more ecologically valid setting. a side effect of its colloquial language is that tschick includes very explicit swearwords and invective. further, the novel consists largely of a dialogue structure, which is another factor supporting the naturalistic language of the novel. second, the novel is written from the point of view of the firstperson narrator maik and is thus characterized by an autodiegetic narrator, i.e. a first-person narrator is at the same time the main character, the narrator in a way tells his own story. the narration style of tschick and its characteristics is illustrated in example (1), where the protagonists try to steal fuel. from the example, the dominant dialogue structure of the novel becomes clear. square brackets and bold words indicate sentence segments and annotated res, respectively (cf. section 5 below). (1) [«was willst du mir erzählen?] [dass das wasser von unten nach oben läuft?»] [«du musst ansaugen.»] [«noch nie was von erdanziehung gehört (zero)?] [das läuft nicht nach oben.»] [«weil es ja danach nach unten läuft.] [es läuft ja insgesamt mehr nach unten,] [de-shalb.»] [«aber das weiß das benzin doch nicht,] [dass es nachher noch runtergeht.»] "what are you trying to tell me? that the water runs from the bottom to the top?" "you have to suck it in." "never heard of gravity? it doesn't run upwards." "because it's going down afterwards. it's running down more overall, that's why." "but the gasoline doesn't know that it's going down afterward." in the novel auferstehung der toten (but also all other brenner volumes), the events are narrated by an omnipresent, auctorial narrator, who never appears as a protagonist. at the same time, the private detective brenner is present almost exclusively and his thoughts, impressions, and feelings are described. the narrator always comments and evaluates what is going on. but most importantly, the narrator uses a style strongly reminiscent of oral language. the sentences are usually quite short and contain few embeddings, but they contain numerous left and right shifts, along with repeated omissions and sentence breaks. additionally, elliptical structures are used with notable frequency. moreover, the corpus is characterized by a simulated dialogicity @xcite , i.e. the narrator repeatedly addresses the reader directly by using the secondperson personal pronoun, which reinforces the oral language impression @xcite . by using these stylistic features, the author creates an artificial illustration of oral communication patterns. the following example (2) illustrates the characteristics mentioned. ## annotation practices in current work in the current work, we present two corpora based on excerpts from two novels with a very conversation-like narration style. although the two corpora are relatively short, they stand out for their extensive annotations. we annotated all res that refer to an animate referent and assigned specific grammatical and semantic features to them. additionally, the sentences were separated into segments to create a comparable sentence equivalent, since the length of the sentences often varied greatly. this approach is not often found in comparable corpora but becomes important when dealing with a text that contains very long sentences due to many insertions. ## studies in the following, we show examples of analyses that can be performed using our corpora. together, both corpora contain a total of 3264 res that refer to an animate referent. as it becomes clear from moving on to feature specification, the mosaic plots in figure 1 and figure 2 show the distribution of grammatical roles and thematic roles among the three main groups of re types. horizontally, the plots are divided into the three main re types: name (@xmath0, dp (@xmath1, and pronoun (@xmath2. vertically, the plots are divided into different classes of grammatical roles (figure 1 ) and thematic roles (figure 2 ). looking at figure 1 , it becomes clear that pronouns in subject position overall account for the largest share (55.1 %). in addition, when comparing the different grammatical roles (vertically), it can be noted that the grammatical role subject also accounts for the largest share of grammatical roles: 72.5 % of the res in the three main groups are in the subject position. by a large margin, the grammatical role oblique occurs second most frequently (8.1 %), followed by the grammatical roles direct object (7.6 %), res with no grammatical role (6.4 %), and indirect object (5.5 %). if we look at the distribution within the grammatical role subject, we see that with 76.0 %, pronouns have the largest share. vice versa, within the pronoun group, subjects have the largest share (75.6 %). looking at figure 2 , we see that pronouns in the proto-agent role comprise the majority of res (57.0 %) among the three main groups. when comparing the thematic roles, we see that the thematic role proto-agent accounts for the largest share among all thematic roles (75.5 %). the thematic role proto-patient is the second most frequent (12.1 %), followed by res with no thematic role (11.6 %), and the thematic role recipient (0.8 %). a look at the distribution of the thematic role protoagent shows that pronouns account for the largest group (75.6 %), and again vice versa, within the pronoun group, the thematic role proto-agent has the largest share (78.3 %). in addition to our corpus analysis, we conducted feature importance analyses to find out (1) which features contribute the most to the choice of the re form, and (2) how they affect this choice. in this analysis, our focus is solely on third-person anaphoric res within the adt corpus foot_3 . we first trained an xgboost model from the family of gradient boosting trees (chen and guestrin, 2016) using the features annotated in our corpus. concretely, we looked at the following features: the grammatical role of the current re and its antecedent (gm and prev_gm), the thematic role of the current re and its antecedent (tm and prev_tm), the segment distance between the current re and its antecedent (seg_dist), and the re form of the antecedent (prev_ref_type). to determine the importance ranking of the features, we compute the model-agnostic permutation-based variable importance of the model @xcite . in particular, we measure the extent to which performance changes when a particular feature is removed. figure 3 shows the change in performance for each feature in the case of a 3-way classification task (pronoun vs. proper name vs. dp). as shown in the figure, the distance calculated in the number of segments and the re form of the antecedent have the highest contribution. we then conducted a shap (shapley additive explanations) analysis to evaluate the positive and negative contributions of each feature to the prediction of each class. the shap analysis decomposes the predictions of the model into contributions that can be additively attributed to different variables (lundberg and lee, 2017). according to figure 4 , the segment distance with the value longdist (>6 segments) promotes the use of non-pronominal forms, i.e., name and dp, the most. interestingly, contrary to the variable importance graph, we see a significant contribution of the thematic role features to the choice of classes. ## discussion the goal of this work was to promote the development of more naturalistic and conversation-like corpora that reflect the nuances of colloquial speech. the analysis of res in informal language is particularly interesting, since the use of res may differ from that in formal language @xcite . moreover, this work offers fine-grained annotations of the res on local (referential form, grammatical role, thematic role) and global (referential chains, perspectival features, character name) prominence levels. although there are several corpora that include coreference annotations @xcite , only a few corpora include detailed information on the referential form @xcite ; additional annotations of prominence-lending features are even rarer. we have shown that in our narrative corpora, pronouns make up a very high proportion of the referential forms used. this large count of pronouns, especially personal pronouns, seems to be connected to the informal narrative structure. it appears that in (more) formal registers such as newspaper articles or in mixed collections of texts, the proportion of pronouns is radically lower than what we observed in our corpora. we examined the proportion of pronouns in the training set of two datasets from the corefud 1.1 collection: the english gum corpus @xcite , which includes texts from various genres, and the german potsdam commentary corpus @xcite , which contains commentaries on german newspaper articles. the former had 22% pronouns (7798 out of 35369 res), while the latter had only 14% pronouns (654 out of 4671 res). the significant variation in the distribution of re forms across different corpora highlights the importance of incorporating more diverse text registers, such as the narrative texts analyzed in this study. in addition, we have shown that our corpora can be used for modeling and predicting the referential form of res. however, since the referential forms in our corpora are unbalanced with a strong tendency towards pronouns, modeling attempts might be biased. as the next step, we will annotate more res and leverage state-of-the-art models like the german bert (gbert) to find out how reliably the re forms can be predicted. ## conclusion all in all, our two corpora show a comprehensive, diverse picture of the res that refer to animate referents. by annotating a variety of prominencelending features, a fine-grained characterization of the use of the res in the two corpora emerges. it is therefore worthwhile to expand the corpus annotations in the future to create a larger data set.