KOJAK : A New Corpus for Studying G erman Discourse Particle ja (2022)

Abstract: In German, ja can be used as a discourse particle to indicate that a proposition, according to the speaker, is believed by both the speaker and audience. We use this observation to create KoJaK, a distantly-labeled English dataset derived from Europarl for studying when a speaker believes a statement to be common ground. This corpus is then analyzed to identify lexical choices in English that correspond with German ja. Finally, we perform experiments on the dataset to predict if an English clause corresponds to a German clause containing ja and achieve an F-measure of 75.3% on a balanced test corpus.

## introduction predicting an author's belief, also called event factuality prediction (efp), has been studied extensively in the last decade. however, in addition to modeling their own beliefs, discourse participants develop a model of their audience's beliefs as well. it is well known since at least @xcite that a speaker or writer must be modeling the addressee's cognitive state in order to communicate felicitously, and the notion of "common ground" has received increased attention in cognitive science @xcite and philosophy @xcite . the task of predicting if a speaker believes a listener already knows a proposition, either because it has been established as common ground or connected to some shared reality, has been studied markedly less. this is at least in part due to a lack of corpora annotated for the task. this paper makes the following contributions: ## corpus creation our corpus is called kojak, which stands for "korpus für ja in kontext" (or "corpus for ja in context" in english). it contains roughly 3,000 examples of english sentences corresponding to german ja. we use the corrected and structured europarl corpus (costep), released by @xcite , as a base for constructing kojak. initially created for machine translation tasks, the europarl corpus contains roughly 30 million words paralleltranslated to 11 languages including english and german @xcite . they are sourced from proceedings of the european parliament starting as early as 1996 and contain additional languages as time moves on. there is a notable asymmetry in the realization of ja depending on the direction of translation. if the sentence containing ja was translated from german to some other language then we can be sure that the conception of common ground being expressed is that of the original speaker. however, if the german ja sentence was translated from some other language then the ja may be expressing the translator's belief regarding the speaker's belief of what the common ground is. we ignore the distinction in this paper but discuss possible improvements in section 7. our corpus is compiled in two steps. first, we create a filter for identifying sentences containing uses of ja as a discourse particle. we then use a heuristic for extracting only the clause containing the ja in question (i.e., the clause over which ja scopes). the latter step is motivated by an interest in the proposition which ja is modifying. ## statistical analysis with the relevant sentences now separated, our attention turns to how we can identify what items in english relate to the discourse particle usage of german ja. if the lexical choices between ja and na sentences are different, then we would ex-1-grams 2-grams 3-grams 4-gr num. for any token sequence of length n, we can count the number of times that n-gram appears in the ja and na sentences respectively. this is similar to creating two sets of samples and asking whether they are likely to be from different populations. intuitively, n-grams which are unique to ja sentences are probably related to the presence of the discourse particle. to investigate this we compute the counts of every n-gram for 2 ≤ @xmath2. these samples are used to perform a t-test at the 95% confidence level and then sorted by p-value. to ensure sequences which are reasonably robust, we discard any that did not appear more than 10 times in the data. the result, seen in to roughly evaluate the quality of the extracted n-grams, the second author, a native speaker of both german and english, performed an error analysis on the n-gram lists. we used the following categories: in fact, almost all examples refer explicitly to the european union or its political procedures including those of the parliament (e.g., madam president). the rating was performed only on the n-gram types rather than on occurrences of the n-grams, and no further context was provided. the goal is to provide a sense of the quality of the extracted n-grams, and we acknowledge the limitations of this study. the error analysis we present in section 6, in contrast, was based on actual full phrases. results are shown in overall, our simple statistical approach has extracted good n-grams, with a small number of bad ones. the results support the claim that the discourse meaning of german discourse particle ja is often preserved in translation equivalents. ## modeling & experiments we perform machine learning experiments to predict whether an english clause is the equivalent of a german ja clause or not. ## discussion we also investigated the use of sentence-level data, i.e. a version of kojak generated without extracting only the clauses over which ja scopes, but instead using the whole sentences in which ja occurs. when including this additional context both models performed worse, which supports the intuition for including only the clauses in scope. the use of ja is one way german speakers indicate they believe a proposition is already common ground, but it is not the only way. it is possible that the systems above are correctly identifying sentences in which this occurs but they correspond to a german sentence which does not contain ja. we analyzed 70 false positive errors of the statistical model, and found that 71% could plausibly be cases in which the speaker believes the hearer already believes the content of the clause, despite the absence of ja in the german clause. interestingly, another 11% look like cases in which the speaker is pretending as if the audience shares his or her beliefs, even though they probably do not (we both know you will clean the dishes now). ## future work in the relatively near future we hope to make improvements to kojak. while the corpus can currently only be used to study english, its underlying source provides data in many more languages. using a methodology similar to that which was described in section 4, we hope to expand kojak to support every language offered by europarl. on a similar note, costep also includes information about the original language for each utterance. if this were propagated, we could investigate the issue of translation direction mentioned in section 3 more closely by partitioning data along these lines. while the n-gram analysis discussed in section 4 roughly identifies sequences which correspond to ja, many artifacts (e.g. of the european union) persist in the output. one way to reduce these might be to perform a similar analysis but on the german text and discard sequences that correspond directly to the english n-gram list. these enhancements open up several directions for continuing work, the most conspicuous of which might be investigating the effectiveness of multitask learning, in which we exploit multiple languages, or related tasks such as factuality @xcite . it could also be interesting to use the n-grams identified in english and search for their german counterparts, which likely include more than just ja. we have only just scratched the surface of what is possible here. ## access to kojak the natural and balanced preparations of kojak are made available on github. foot_0 additional tooling