P arsi NLU : A Suite of Language Understanding Challenges for P ersian (2021)

Abstract: Despite the progress made in recent years in addressing natural language understanding (NLU) challenges, the majority of this progress remains to be concentrated on resource-rich languages like English. This work focuses on Persian language, one of the widely spoken languages in the world, and yet there are few NLU datasets available for this language. The availability of high-quality evaluation datasets is a necessity for reliable assessment of the progress on different NLU tasks and domains. We introduce ParsiNLU, the first benchmark in Persian language that includes a range of language understanding tasks—reading comprehension, textual entailment, and so on. These datasets are collected in a multitude of ways, often involving manual annotations by native speakers. This results in over 14.5k new instances across 6 distinct NLU tasks. Additionally, we present the first results on state-of-the-art monolingual and multilingual pre-trained language models on this benchmark and compare them with human performance, which provides valuable insights into our ability to tackle natural language understanding challenges in Persian. We hope ParsiNLU fosters further research and advances in Persian language understanding.1

## introduction in recent years, considerable progress has been made in building stronger nlu models, particularly supported by high-quality benchmarks @xcite @xcite the point of view of the authors are their own and not attributable to the company they work for. in this work, we focus on developing nlu benchmarks for persian (also known as "farsi"). this language has many attributes that make it distinct from other well-studied languages. in terms of script, persian is similar to semitic languages (e.g., arabic). linguistically, however, persian is an indo-european language @xcite and thus distantly related to most of the languages of europe as well as the northern part of the indian subcontinent. such attributes make persian a unique case to study in terms of language technologies. although persian is a widely spoken language @xcite , our ability to evaluate performance and measure the progress of nlu models on this language remains limited. this is mainly due to the lack of major language understanding benchmarks that can evaluate progress on a diverse range of tasks. in this work, we present parsinlu, a collection of nlu challenges for persian. foot_0 parsinlu contains challenges for reading comprehension, multiple-choice question-answering, textual entailment, sentiment analysis, question paraphrasing, and machine translation (examples in fig. 1 ). parsinlu offers data for tasks that have never been explored before in the context of the persian language. we are not aware of any publicly available dataset for persian question answering ( §3.2.2), reading comprehension ( §3.2.1), and paraphrasing ( §3.2.5). for the rest of the tasks, we improve at least one aspect of the existing datasets (e.g., better data construction, more comprehensive evaluation, and evaluation of less investigated genres or domains). to ensure the quality of the presented challenge tasks, we rely on the annotations from native persian speakers or novel data collection techniques, such as, search engine auto-complete ( §3.2.1), and past collegiate exams ( §3.2.2). to the best of our knowledge, this is the first comprehensive collection of its own, composed of a variety of persian nlu tasks. we conduct a collection of empirical work ( §4) to establish the difficulty of parsinlu. we benchmark each parsinlu task via collecting state-of-the-art multi-lingual and monolingual lms, as well as estimating the human upper bound scores. the gap between human and machine baselines indicate the need for further research and stronger models for persian. we hope that the release of parsinlu will encourage more research on persian nlp. ## related work cross-lingual benchmarks. there are several recent cross-lingual benchmarks; however, almost none includes persian: xnli @xcite for entailment, pwns-x @xcite for paraphrasing, xcopa @xcite for choice of plausible alternatives, xquad, mlqa, tydi and mkqa @xcite @xcite for reading comprehension. these datasets have also been integrated as part of multitask multilingual evaluation suites such as the xtreme @xcite and xglue @xcite . unfortunately, the persian portion of the former benchmark covers only two tagging tasks (pos and ner) and the latter does not cover persian. nlu benchmarks for other languages. benchmarks like glue @xcite encourage development of better and stronger models on a diverse set of challenges. there have been several efforts to create glue-like benchmarks for other languages; for example, clue for chinese @xcite , gluecos for hindi @xcite , and russian-superglue @xcite . we view parsinlu in the same family of benchmarks, dedicated to the persian language. nlu datasets for persian. prior work on creating evaluation resources for the persian language has focused on low-level tasks in narrow domains (e.g., datasets for pos @xcite , ner @xcite , parsing @xcite ). complementary to these efforts, we aim at providing an nlu evaluation benchmark for persian, consisting of a wide variety of tasks. below we mention several related works and how we build upon them. farstail @xcite ) is a concurrent work on the entailment task, where the dataset is constructed semi-automatically based on existing multiple-choice exams. different from this work, our entailment datasets are built with the annotations of native speakers of persian and some use of machine translation ( §3.2.4). therefore, we hypothesize our construction represents a slightly different distribution than that of farstail. there is a rich set of works on persian sentiment analysis. we build upon these works and differ from them in the following manners: (a) the existing work mainly focuses on document-level sentiment identification which does not capture the nuanced judgements with respect to aspects and entities of the context @xcite . in addition to such document-level annotations, we provide aspect-level sentiment annotations ( §3.2.3). (b) the majority of existing resources, such as mi-rasopinion (ashrafi asli et al., 2020) focus on binary or ternary sentiment classes. however, our annotations contain a more granular sentiment intensity with five labels ( §3.2.3). (c) compared to the aspect-level datasets hosseini et al. (2018); ataei et al. (2019), we cover two relatively less investigated domains: food & beverages and movies, each posing new challenges for persian sentiment analysis. ## experiments we experiment with several recent lms, to assess the difficulty of the parsinlu tasks (compared to human expert performance) and also to establish baseline performance of the state-of-the-art mono-and multi-lingual pre-trained models. all the baseline models used in this work are available online. 16 evaluation metrics. for each task, we pick a common set of existing metrics: for readingcomprehension, we use f1 between gold answer and the response string @xcite ; for question paraphrasing, textual entailment, multiple-choice question-answering, and sentiment analysis, we use accuracy. for the first two sub-tasks of sentiment analysis (documentlevel sentiment, aspect extraction), we use macro-f1. for the third sub-task (aspect-specific senti-ment) we use accuracy as our target evaluation metric @xcite . for machine translation we use sacre-bleu @xcite . task splits. for each task, we have provided statistics on eval, train, and dev splits in ## discussion we now discuss several limitations of the current dataset and the experiments. we then outline several directions for future work. beyond current models. as shown in the earlier experiments, for most of the tasks the current mid-sized models perform significantly worse than humans. this is particularly pronounced for the multiple-choice qa task where there is over 40% gap between the model and human performance, and increasing the model size (number of parameters) shows minimal benefits. we hypothesize that the difficulty of our multiple-choice questions (and other tasks, to some extent) for the models are partly due to the reasoning and abstraction needed to answer them. for example, the 'literature' questions often demand creating connection several pieces of poetry, based on abstract interpretations of their meanings. likewise, most of the 'math & logic' questions require several 'hops' of algebraic operations to get to the final answer. we hypothesizes these challenges (multi-hop reasoning over high-level abstractions of language) cannot be solely be addressed with more training data. and likely require a dramatic rethinking of our architectures design. for example, the poor performance on 'math & logic' questions might be due to models' inability to comprehend persian numbers and do logical reasoning with them, a topic that is briefly studied in english @xcite . there might also be value in exploring multi-task setups across our various tasks @xcite , which we delegate to the future work. we hope this benchmark will encourage more of such studies, especially in the context of the persian language. coverage of dialects. there are other dialects of persian, including dari and tajiki dialects, that are not covered by our dataset. we acknowledge this limitation and hope the future work will create broader and more inclusive collections. ## conclusion this work introduced parsinlu, a benchmark for high-level language understanding tasks in persian. we present a careful set of steps we have followed to construct each of the tasks with the help of native speakers ( §3.2). we have presented human scores to establish estimated upper-bounds for each task. this is followed by evaluating stateof-the-art models on each task and quantifying the human-machine gap ( §4). to the best of our knowledge, this is the first work that publishes a language understanding benchmark for persian language. we hope that parsinlu inspires more activity in the persian nlu tasks, as well as contributing to the latest efforts in multilingual nlu.