Topicalization in Language Models: A Case Study on J apanese (2022)

Abstract: Humans use different wordings depending on the context to facilitate efficient communication. For example, instead of completely new information, information related to the preceding context is typically placed at the sentence-initial position. In this study, we analyze whether neural language models (LMs) can capture such discourse-level preferences in text generation. Specifically, we focus on a particular aspect of discourse, namely the topic-comment structure. To analyze the linguistic knowledge of LMs separately, we chose the Japanese language, a topic-prominent language, for designing probing tasks, and we created human topicalization judgment data by crowdsourcing. Our experimental results suggest that LMs have different generalizations from humans; LMs exhibited less context-dependent behaviors toward topicalization judgment. These results highlight the need for the additional inductive biases to guide LMs to achieve successful discourse-level generalization.

## introduction building on the current success of neural language models (lms) in the field of natural language processing (nlp), much work has been conducted to test their linguistic knowledge, typically, syntactic generalizations in lms @xcite @xcite @xcite . although discourse is also an essential aspect of language production along with syntactic constructions, discourse-level knowledge in lms has been less explored or has been typically analyzed at the coarse level, for example, analyzing their sentence ordering abilities @xcite . as one step toward understanding the finegrained, discourse-level knowledge in lms, this study explores the generalization performance of 花瓶を昨日割った。 i broke a vase yesterday. ## data analysis: context effect in topicalization before our experiments, we preliminarily observed the characteristics of the collected data. if all the annotated preference remains unchanged regardless of the inter-sentential context, our data are not suitable for analyzing the discourse-level behaviors in lms. our analysis declines such a concern. 7 the japanese language has no explicit word boundary. as an approximation of word count, context c, sentence s, and nominative argument a have 56.4±31.7, 29.5±14.4, and 2.8±1.1 morphemes, respectively. we used a juman dictionary for morphological analysis @xcite . 8 we used the weighted krippendorff's alpha, where we assumed the distance scale as top ≺ "both okay" ≺ nom. ## experiments: comparing lms with humans do lms exhibit human-like topicalization preferences? to answer this question, we compared contrast the preferences of lms and humans. ## analysis our experiments showed that the topicalization decision of lms is unreasonably decontextualized. to further understand such non-human-like behaviors of lms, we investigated their behavior in situations with extremely limited context information. as introduced in section 3.4, we observed that humans struggle with topicalization judgment in the setting where the nominative constituent alone is shown (the a setting; language model preference. for each nominative argument a in the dataset, we computed the topicalization ratio when only the a is shown to lms. specifically, lms computed the generation probability of, for example, kabin-ga and kabinwa, regardless of any context, and subsequently, we compared these probabilities as per equation. 3. ## discussion contributions to linguistics. we posit that our dataset itself is also valuable for linguistic studies. topicalization in japanese has captured the attention in the field of linguistics for more than half a century @xcite @xcite , but resources created using a large-scale corpus and crowdsourcing have been limited. the observed relationship between mention frequency and topicalization preference ( testing coherence models. while we evaluated only the vanilla lms, there are several options for coherence modeling, such as entity-based coherence models @xcite and neural lms that are explicitly trained with coherence objectives @xcite . it would be interesting to investigate whether such models exhibit more human-like behaviors in terms of topicalization preference. ## conclusions we have compared the preferences of lms and humans for topicalization, an essential aspect of discourse. the results suggest that there exists a discrepancy between humans and lms with respect to the generalizations for topicalization. this implication leads us to future research: what type of inductive biases in model architecture or training objectives can lead to more human-like generalizations in discourse-level linguistic aspects? setting #labels class distribution top both okay nom c+s+a 1,853 65.7% 1.73% 32.5% s+a 1,900 54.9% 8.47% 36.6% a 1,732 2.02% 97.1% 0.88%