JADES : New Text Simplification Dataset in J apanese Targeted at Non-Native Speakers (2022)

Abstract: The user-dependency of Text Simplification makes its evaluation obscure. A targeted evaluation dataset clarifies the purpose of simplification, though its specification is hard to define. We built JADES (JApanese Dataset for the Evaluation of Simplification), a text simplification dataset targeted at non-native Japanese speakers, according to public vocabulary and grammar profiles. JADES comprises 3,907 complex-simple sentence pairs annotated by an expert. Analysis of JADES shows that wide and multiple rewriting operations were applied through simplification. Furthermore, we analyzed outputs on JADES from several benchmark systems and automatic and manual scores of them. Results of these analyses highlight differences between English and Japanese in operations and evaluations.

## introduction text simplification (ts) aims to rewrite texts for easier understanding. simplified texts can benefit children @xcite , non-native speakers @xcite , non-specialists @xcite , and people with cognitive disabilities @xcite . given the diverse users in various domains, automatic ts has been regarded as an important research area these years @xcite . however, the diversity, in turn, makes the evaluation of ts obscure. as @xcite stated, an appropriate simplification for one type of users will not be appropriate for another. therefore, the ideal ts system and evaluation is userdependent, but its specification is difficult to define. one step to the user-dependent ts could be focusing on a specific population. the validity of the simplification for a specific population can be evaluated using a targeted dataset. newsela @xcite , available in english and spanish, can be used in this way when using information about targeted grades on each article. japanese lacks such a dataset. snow @xcite is a japanese dataset for ts and limits vocabulary, which comprises the top 2000 words required to understand japanese. however, this criteria differs from targeting in that no specific populations are considered. for instance, they gave no simplification instructions on grammar to annotators. with a strictly limited vocabulary, this settings causes lengthy expressions. in addition, snow is problematic in that its original sentences are already short and simple. therefore, snow may not be suitable for simplifying daily texts such as news articles. building a targeted dataset requires criteria for specific populations. in japanese, japanese-language proficiency test (jlpt) published vocabulary and grammar profiles for grasping japanese on each level @xcite . these materials alleviate the difficulties in defining the specification and building a targeted dataset. in this paper, we introduce a new japanese ts dataset, jades foot_0 (japanese dataset for the evaluation of simplification). jades is targeted at non-native japanese speakers capable of everyday communications, following the specification of vocabulary and grammar. jades comprises 3,907 complex-simple parallel sentence pairs, which an expert of easy japanese manually simplifies. since obtaining manual simplification are costly, jades is oriented towards tuning and evaluation in size. we also implemented models as baselines on jades and rated their outputs automatically and manually. the contributions of this work include: ## our new dataset jades we create a new dataset, jades, for ts in japanese. jades contains manually simplified sentences targeted at independent non-native japanese speakers. jades comprises 3,907 complex-simple parallel sentence pairs and will help tune and evaluate ts models. ## evaluations we conducted ts in japanese with several models to investigate the characteristics of our dataset. we also evaluated models automatically and manually. ## conclusion we have introduced jades, a new dataset mainly for the evaluation of ts in japanese. simplified sentences in jades are targeted at non-native speakers and made by an expert. this setting may make operations more variant and induce drastic rewriting, as the manual operation identification shows. we can see the difference between english and japanese in operations and evaluation metrics, which emphasizes the need for manual datasets in diverse languages. since manual scores on automatic ts models are low, ts in japanese still has room for growth. with manual scores, jades can also be useful for investigating new evaluation metrics. we believe that jades facilitates ts in japanese and its application.