BERT -Based Simplification of J apanese Sentence-Ending Predicates in Descriptive Text (2020)

Abstract: Japanese sentence-ending predicates intricately combine content words and functional elements, such as aspect, modality, and honorifics; this can often hinder the understanding of language learners and children. Conventional lexical simplification methods, which replace difficult target words with simpler synonyms acquired from lexical resources in a word-by-word manner, are not always suitable for the simplification of such Japanese predicates. Given this situation, we propose a BERT-based simplification method, the core feature of which is the high ability to substitute the whole predicates with simple ones while maintaining their core meanings in the context by utilizing pre-trained masked language models. Experimental results showed that our proposed methods consistently outperformed the conventional thesaurus-based method by a wide margin. Furthermore, we investigated in detail the effectiveness of the average token embedding and dropout, and the remaining errors of our BERT-based methods.

## introduction lexical simplification (ls) is the process of replacing complex words into easy words or short phrases without changing their meaning to simplify text for an easier understanding @xcite . ls has also shown to be effective in improving the downstream natural language processing tasks, such as machine translation @xcite . popular ls methods use a paraphrase lexicon that pairs complex and easy words, built from various language resources @xcite . however, while the conventional ls methods are widely applicable to various sentences, they are not necessarily effective for tasks in which relatively longer linguistic spans should be treated comprehensively. the simplification of japanese sentence-ending predicates is one such task. these predicates comprise multiple parts, i.e., a particle, content part (mainly verbs, nouns and adjectives) and functional part (mainly postpositional particles and auxiliary verbs), foot_0 as shown in figure 1 . because the functional part tends to combine aspect, modality, and honorifics, the understanding of language learners and children may be hindered. figure 1 also shows an example of a manual simplification of the predicate; whole elements in the phrase are jointly rewritten. from this example, it is discovered that although the original meaning of the main verb "haichisuru (be arranged)" is reduced to the simpler general word "aru (there are)," the core meaning of the sentence can still be successfully conveyed. we assume that this interesting fact may generally hold for a certain type of text, namely, descriptive text, which explains the attributes and states of objects and situations @xcite . in descriptive sentences, core information is chiefly encoded in the arguments and their modifiers rather than the predicates. in fact, in many cases, if predicates are omitted from descriptive sentences, humans can reasonably complement alternative phrases by referring only to the remaining part. this observation motivated us to replicate these human intuitions by utilizing masked language models, which can predict the omitted tokens in a sentence. therefore, in this paper, we propose a bertbased simplification method for japanese sentenceending predicates. our key idea is to use a masked language model to sequentially restore a predicate that is masked entirely. our method also incorporate the mechanisms to partially encode the meaning of original predicates using average token embedding and dropout to generate various simple candidate phrases that are moderately related to the original one. we experimentally evaluated the performance of the proposed method by comparing it with a thesaurus-based ls method; the high ability of our method was revealed with detailed discussions and analyses. ## related work automatic text simplification (ats) is a task of rewriting a complex text as simpler text while maintaining its meaning. inspired by the machine translation (mt) task, recent ats approaches regard the text simplification as a monolingual mt task @xcite . although sequence-to-sequence neural mt has contributed to better simplification performance, a large amount of parallel data that pairs original-tosimplified text is required. in english, various types of such datasets are available, such as the pwkp dataset @xcite , which was constructed by automatically aligning sentences from english wikipedia and simple english wikipedia. 2 xu et al. (2015) introduced the newsela corpus, which comprises news articles rewritten by professional editors. several attempts have also been made to develop datasets for japanese simplification, such as one built by using crowdsourcing @xcite . however, compared with the resources available in english, they are not sufficiently large to build robust models. therefore, for languages with limited resources of aligned corpora for simplification, even including japanese, nmt-based ats methods are not necessarily feasible. ls is a subtask of ats that aims to substitute complex words with easy words. one of the most popular ls systems uses a simplified syn-2 https://simple.wikipedia.org/ onym lexicon extracted from wordnet and other resources (s. @xcite . simple-ppdb @xcite ) is a large-scale complex-to-simple paraphrase dataset built from ppdb @xcite . @xcite used word embeddings to obtain synonyms of the target word. a recent study @xcite proposed a bert-based ls method which produces simpler words that are both semantically and contextually similar to the target word. however, these ls methods adopt the word-byword substitution approach, and in principle, do not take longer units of sentences as input, such as japanese sentence-ending predicates. to manage longer text spans without recourse to sequence-to-sequence generation methods, leveraging the general-purpose monolingual models that are trained on large data might be effective. in recent years, various types of pre-trained models have been proposed and trained for many languages @xcite . for japanese, several models are available, such as bert (bidirectional encoder representations from transformers) @xcite , which is a transformer-based language model trained with two objectives: masked token prediction and next-sentence prediction. although bert has been applied to ls @xcite , these studies have focused on a single word as a simplification unit. the applicability of bert in simplifying longer spans of text remains to be investigated. ## conclusion in this paper, we proposed a bert-based method for generating simplified substitutions for japanese sentence-ending predicates in descriptive text. the principal feature of our method is that it can generate candidate predicates that conform to the target context and retain the core meaning of the original one, even though the candidates themselves may not always be exactly synonymous to the original one. furthermore, our method does not rely on conventional language resources such as thesauri but utilizes a large pre-trained language model and can generate an output for almost any input. the evaluation results showed that the proposed method generated simple, fluent candidates that conveyed the original meaning for 54% of difficult predicates and substantially outperformed the conventional thesaurus-based method. furthermore, we investigated the effectiveness of using average token embedding and dropout in the bert model and systematically analyzed the critical errors. in future studies, we plan to extend our architecture to manage idiomatic expressions and produce multiple content words to improve expressiveness. we will then develop a simplification support system integrating our proposed method that can return a list of candidates for human writers. we will also aim to examine our method for other text types, such as narrative and argumentative texts @xcite . in this regard, important information is encoded in predicates, which renders the task more challenging.