Analyzing Syntactic Generalization Capacity of Pre-trained Language Models on J apanese Honorific Conversion (2023)

Abstract: Using Japanese honorifics is challenging because it requires not only knowledge of the grammatical rules but also contextual information, such as social relationships. It remains unclear whether pre-trained large language models (LLMs) can flexibly handle Japanese honorifics like humans. To analyze this, we introduce an honorific conversion task that considers social relationships among people mentioned in a conversation. We construct a Japanese honorifics dataset from problem templates of various sentence structures to investigate the syntactic generalization capacity of GPT-3, one of the leading LLMs, on this task under two settings: fine-tuning and prompt learning. Our results showed that the fine-tuned GPT-3 performed better in a context-aware honorific conversion task than the prompt-based one. The fine-tuned model demonstrated overall syntactic generalizability towards compound honorific sentences, except when tested with the data involving direct speech.

## introduction the correct use of japanese honorifics is difficult because it requires both the knowledge of grammatical rules (i.e., verb conjugation) and contextual information (i.e., social relationships among the speaker, the hearer, and the people mentioned in a conversation) @xcite . we expect this syntactic and pragmatic ability for pre-trained large language models (llms), as they have shown high performance on natural language tasks @xcite . however, it remains unclear whether llms can handle japanese honorifics in a similar manner to humans, based on sentence structures and social context. several studies proposed datasets of japanese honorifics for classification @xcite and generation @xcite . @xcite introduced a task in which a model takes an honorific sentence as input and classifies its honorific level or the types of honorifics used in the sentence. someya and oseki (2022) provided a japanese acceptability classification dataset called jcola. in jcola, subject honorifics are categorized as sub-categories of subject-verb agreement tasks. however, these datasets aim to evaluate the syntactic performance of language models, and they do not analyze their pragmatic ability to understand honorifics by considering social relationships behind sentences. @xcite introduced an evaluation dataset for an honorific conversion task in which the input was a non-honorific sentence, and the output was an honorific sentence. @xcite mentioned the necessity of considering the information on social relationships among people in honorific conversion but did not clarify how such information should be processed in the task. in summary, the existing benchmark datasets of japanese honorifics focus on either syntactic or pragmatic knowledge required for honorific understanding, not both (appendix a). additionally, none of these existing studies discusses the generalization capacity toward various syntactic structures of honorific sentences. in this research, we introduce a new honorific conversion task that uses information on person's social relationships as additional input. in matsumoto et al. (2022)'s proposed honorific conversion, the input was only a non-honorific sentence. in our task, social relationships are expressed as a sentence and concatenated into an input sentence (section 2). this enables us to analyze whether llms could consider information on social relationships when executing honorific conversion. we also construct a dataset to investigate the syntactic generalizability of llms for this honorific task. we create hand-crafted templates and generate problems for the task by filling in the placeholders (section 3). by focusing on the syntactic generalization type target of respect and deference example subject honorifics (sh) agent sensei-ga hanako-o homete-irasshatta. teacher-nom hanako-acc praised-sh object honorifics (oh) patient (watashi-ga) sensei-no-tokoro-ni ukagau. (i-nom) teacher-poss-place-loc visit-oh ## dataset construction we construct a japanese honorific dataset by manually creating problem templates and filling their placeholders with vocabulary using dictionaries to evaluate llms' performance on the honorific task. our dataset construction method is shown in figure 3 . we take this approach instead of automatically collecting data from corpora for two reasons. first, it is difficult to create sentence data with complex structures, such as scrambling, from corpora in a controlled manner. this possibly makes it easier for a model to do honorific conversion than when the information is implicit. second, we need to prepare controlled settings for social relationship information to evaluate whether llms utilize it in honorific conversion; however, such information does not appear explicitly in the corpora. the second problem is related to the fact that words in argument positions are often dropped in japanese, especially in dialogue sentences. (1) and in this conversation, two ambiguous points must be clarified to determine the honorific relationship behind the conversation. the first point is that the junior answers with object honorifics to show deference in (2), but the target of deference is ambiguous without additional context. if snacks are something the boss originally brought to their office, the boss is the target of the junior's deference. however, if the snacks are prepared by some third person with a higher rank of job position than the junior, the deference must be towards them instead of the boss. the second point is that the subject is dropped in (2) (pro-drop), but we cannot determine whether ϕ refers to the junior or to a group of workers containing the junior. considering such language-specific phenomena, we take a templatebased approach instead of a corpus-based approach for dataset construction. ## conclusions and future work in this paper, we introduced an honorific conversion task that requires not only syntactic knowledge but also pragmatic knowledge, such as social relationships among people. we constructed a japanese honorific dataset using problem templates created manually and evaluated the syntactic generalization capacity of gpt-3 models on the task using our dataset. the experiments showed that the finetuned models could solve problems with simple structures but failed to generalize to problems with more complex structures, particularly with direct speech. regardless of the sentence structure, the prompt-based models did not successfully solve the problems with our current prompt setting. in future work, we plan to expand our dataset to include more diverse japanese honorific expressions, such as predicates other than verbs or honorific prefixes attached to nouns. for the promptbased experiments, we evaluated the models using zero-shot learning. it would be valuable to test them using few-shot learning by including simple examples in the prompts. we conducted experiments by explicitly providing information about social relationships. we will also continue to seek data construction methods to extract such information from the corpora, although we did not apply these corpora-based methods in this paper.