Towards A rabic Sentence Simplification via Classification and Generative Approaches (2022)

Abstract: This paper presents an attempt to build a Modern Standard Arabic (MSA) sentence-level simplification system. We experimented with sentence simplification using two approaches: (i) a classification approach leading to lexical simplification pipelines which use Arabic-BERT, a pre-trained contextualised model, as well as a model of fastText word embeddings; and (ii) a generative approach, a Seq2Seq technique by applying a multilingual Text-to-Text Transfer Transformer mT5. We developed our training corpus by aligning the original and simplified sentences from the internationally acclaimed Arabic novel Saaq al-Bambuu. We evaluate effectiveness of these methods by comparing the generated simple sentences to the target simple sentences using the BERTScore evaluation metric. The simple sentences produced by the mT5 model achieve P 0.72, R 0.68 and F-1 0.70 via BERTScore, while, combining Arabic-BERT and fastText achieves P 0.97, R 0.97 and F-1 0.97. In addition, we report a manual error analysis for these experiments.

## introduction text simplification (ts) is a natural language processing (nlp) task that aims to reduce the linguistic complexity of the text while maintaining its meaning and original information @xcite @xcite . according to @xcite definition, ts involves text transformation with new lexical items and/or rewriting sentences to ensure both its readability and understandability for the target audience @xcite . ts could be classified as a type of text style transfer (tst), where the target style of the generated text is "simple" @xcite . evidence suggests the importance of ts involves : (i) its usage in designing and simplifying the language curriculum for both second language and first language learners, in making text easy-to-read for first language early learners; in assisting first-language users with cognitive impairments and low literacy language level; (ii) being a fundamental pre-process in nlp applications such as text retrieval, extraction, summarization, categorization and translation @xcite ; and (iii) acting as a post-process step in automatic speech recognition. hence, there are various types of simplification systems based on the purpose and who is the end-user of the system. there are three key aspects of simple text that: (i) it is made up of frequent simple words, grammatically simple sentences, and direct language; (ii) unnecessary information is omitted ; (iii) it can be shorter by the number of words, but also with shorter sentences, which might lead to their increased number @xcite . @xcite approached ts differently as he came up with different opinion, that is a slightly simplified text for one user is generally simpler for any other users. but a more extensive simplification for a specific user, may lead to a more complex text for another user. most of ts techniques were borrowed from closely related nlp tasks such as machine translation @xcite . this has influenced our experiments to demonstrate the effectiveness of two different methods to address the sentence simplification (ss) task as follows: the main contribution of this paper is to examine different approaches for arabic sentence simplification task using automatic and manual evaluation. to our knowledge, this is the first available arabic sentence-level simplification system. ## corpus and tools the corpus used for training is a set of complex/simple parallel sentences that have been compiled from the internationally acclaimed arabic novel "saaq al-bambuu" which has an authorized simplified version for students of arabic as a second language @xcite . we assume that a successful sentence simplifier should be able to detect word/sentences in the original text that require simplification and simplify them in such a way as the original simple counterpart. the dataset consists of 2980 parallel sentences as illustrated in ## method one -classification approach the reference for this approach is the pipeline of the ls task, that focuses on ls by replacing com-plex vocabularies or phrasal-chunks with suitable substances @xcite . to reach this goal, we decided to implement three classification models: considering the definition of the four main steps applied in the pipeline for ls as follows: complex word identification [cwi] is the main first step performed at the top of the pipeline that employed to distinguish complex words from simple words in the sentence. substitution generation @xcite involves generating all possible substitutions but without including ambiguous substances that would confuse the system in the substitution selection step. substitution ranking [sr] is to order the new generated substitution list to ease the selection step by giving high probability of the most appropriate highly ranked word. substitution selection [ss] is responsible for selecting from the ordered sg's generated list the most appropriate substitute according to the context while preserving the same meaning and grammatical structure.taking into account the fact that, a word may have multiple meanings, and different meanings will have different relevant substitutions, then the ss task may generate a miss-substitution, which may lead to meaning corruption. the following part of this paper moves on to describe in greater detail the implementation of each step concentrating on employed methods and tools. ## method two: generative approach here, we employ a seq2seq approach adopting t5 "text-to-text transfer transformer". t5 is a bert-like transformer that takes input a text and training it on the model to generate target text of a different variety of nlp text-based tasks such as (summarization, translation, question answering and more) @xcite . the main difference between bert and t5 is that bert uses a masked language model (mlm) and an encoder-decoder, although t5 employs a unified seq2seq framework @xcite . t5 model initially targeted english-language nlp tasks. recent research extended the model to include more than 101 languages including the arabic language. a "multilingual text-to-text transfer transformer", multilingual t5, mt5 @xcite , a new variant of t5 and pre-trained on common crawl-based dataset. the pre-trained language model was very successful for the natural language understanding (nlu) task. considering the multilingual capabilities of mt5 and the suitability of the seq2seq format for language generation. this gives it the flexibility to perform any nlp task without having to modify the model architecture in any way. this experiment employs the 'mt5-for-conditional-generation' class that is used for language generation. training a ts model using "saaq al-bambuu" parallel sentences, over the mt5-base model. the system was developed in p ython3.8 environment with using other toolkits such as natural language processing toolkit n lt k and scikitlearn. our sentence corpus was randomly split into 80% for training and 20% for testing. ## evaluation likewise, most ts evaluation approaches have been driven from other similar nlp research areas. various evaluation methods have been applied across researches to measure the three main aspects of the newly generated text. these aspects are, i) fluency, referring to the grammatically wellformedness and structure simplicity; ii) adequacy, meaning preservation; iii) simplicity, more readable. all methods were evaluated on the same test dataset that consisted of 299 randomly chosen sentences excluded from training. we employed both automatic and manual evaluation comparing both systems. ## conclusion in this paper, we have presented the first modern standard arabic sentence simplification system by applying both classification and generative approaches. on the one hand, the classification approach focuses on lexical simplification. we looked at the different classification methods and showed that a combined method generates well-formed simple sentences. in addition, using word embeddings and transformers prove to produce a reasonable set of substitutions for the complex word more accurately than traditional methods such as wordnet. our interpretation of the limitation of the classification system arises from the fact that some of the generated sentence structures are not well-formed and that the system can misidentify what makes some complex words in the cwi step. even though this limitation reveals the limitations of the arabic cefr vocabulary list in identifying the complex word, the list is shown to be more useful in the substitution replacement step. on the other hand, while the generative seq2seq approach provides a less accurate simplified version in most cases, in some cases it outperforms the classification approaches by generating a simplified sentence, which can be even better than the target human simple sentence. nevertheless, one of the limitations of the generative approach concerns the trend to repeat identical patterns, which can be partly controlled by post-processing.