Neural Readability Pairwise Ranking for Sentences in I talian Administrative Language (2022)

Abstract: Automatic Readability Assessment aims at assigning a complexity level to a given text, which could help improve the accessibility to information in specific domains, such as the administrative one. In this paper, we investigate the behavior of a Neural Pairwise Ranking Model (NPRM) for sentence-level readability assessment of Italian administrative texts. To deal with data scarcity, we experiment with cross-lingual, cross- and in-domain approaches, and test our models on Admin-It, a new parallel corpus in the Italian administrative language, containing sentences simplified using three different rewriting strategies. We show that NPRMs are effective in zero-shot scenarios (~0.78 ranking accuracy), especially with ranking pairs containing simplifications produced by overall rewriting at the sentence-level, and that the best results are obtained by adding in-domain data (achieving perfect performance for such sentence pairs). Finally, we investigate where NPRMs failed, showing that the characteristics of the training data, rather than its size, have a bigger effect on a model’s performance.

## introduction due to its complexity, the style of italian administrative texts has been defined as "artificial" and "obscure" @xcite . during the last decades, italian institutions have fostered the use of a plain language in writing official acts and communications @xcite . however, the readability of italian administrative texts still remains an issue @xcite , and measuring their complexity can help institutions improve information accessibility, and guarantee a substantive equality of citizens @xcite . one way to tackle this problem is with technologies for automatic readability assessment (ara) that predict the complexity of texts @xcite . this task has been widely in-vestigated in the educational domain, usually classifying texts according to school grade levels or international frameworks for language proficiency. currently, most models for ara are based on neural networks @xcite , which are trained in a supervised fashion by fine-tuning pre-trained language models @xcite @xcite . however, this approach could require large amounts of monolingual indomain data, which is limited in specific sectorial languages like the one used in italian administrative texts, for which the available resources are quite scarce @xcite . in this paper, we tackle the data scarcity issue in two ways. first, we introduce admin-it (sec. 3), a parallel corpus in the italian administrative language with sentences that were simplified following three different styles of rewriting. then, we repurpose @xcite 's neural pairwise ranking model (nprm) to rank sentences (instead of documents) from the italian administrative language (sec. 4), because that model obtained better results than traditional classification and regression approaches in zero-shot cross-lingual set-ups. we evaluate the performance of nprms on admin-it in zero-shot settings (sec. 5), fine-tuning models with data from different languages (i.e., italian, english and spanish) and domains (i.e., administrative, educational, and news). we show that, overcoming the limitations of traditional ara system in cross-domain set-ups @xcite , nprms obtain good results in cross-domain and cross-lingual scenarios, especially when ranking sentences simplified via overall rewriting (sec. 6). finally, we conduct a qualitative analysis on the errors made by nprms (sec. 7), and observe how models deal with various kinds of simplification, such as overall rewriting versus the application of single operations of simplification (e.g., lexical substitution, splitting or deleting). to sum up, our main contributions are: while ara is normally a document-level task, we tackle it at the sentence level due to the characteristics of the datasets available in italian @xcite and the administrative domain @xcite , which mainly contain aligned sentences (see details in sec. 5.1). also, a sentencebased approach for readability could be more effective in detecting easy and complex to read texts, since complex documents may also contain easy-toread sentences @xcite @xcite . ## related work early ara techniques consisted in the so-called "readability formulae". such formulae were created for educational purposes and mainly considered shallow text features, like word and sentence length or lists of common words @xcite @xcite . however, longer words and sentences are not necessarily complex, and these formulae have been proved to be unreliable @xcite @xcite . in addition, traditional readability formulae should not be applied to fragments with less than 100 words, making them unsuitable to assess the readability of sentences, which is usually considered more difficult than predicting readability of documents @xcite . nlp and machine learning fostered the emergence of "ai readability" systems @xcite , leading to the creation of new techniques for both supervised and unsupervised approaches @xcite . traditional supervised techniques model 1 https://github.com/unipisa/admin-it ara as classification @xcite , regression @xcite , or ranking @xcite tasks, exploiting a wide range of linguistic features, at a lexical @xcite , syntactic @xcite , and discourse level @xcite @xcite . more recent systems are based on neural networks @xcite @xcite , exploiting contextual embeddings like bert @xcite to encode large quantity of linguistic knowledge. however, such models still need to be fine-tuned to be applied in downstream tasks. for some languages and domains, like italian administrative texts, this is not possible since there is not enough available data for a full supervised approach. for this reason, we adopted a cross-lingual approach and created our own resource for the italian administrative language (i.e., admin-it). recently, @xcite used neural models to address ara as a ranking task. their neural pairwise ranking model (nprm) ranks a group of documents by their readability, regardless of its size (i.e., the number of reading levels). their nprm obtained better results than classification and regression approaches for texts in english, spanish and french, in both monolingual and zeroshot cross-lingual set-ups. as such, we decided to exploit this architecture but for ranking sentences. furthermore, while @xcite found that the nprm struggles in a cross-domain setting, they did not deeply analyzed the behaviour of the model when dealing with data whose domains are wide apart (e.g., news and bureaucratic domains). in contrast, we study the impact on performances given both by the datasets used for fine-tuning the nprm and by the specific kind of simplification applied to the sentences being ranked. ## admin-it given the paucity of data in the italian administrative language for sentence readability and simplification, we decided to build admin-it, a parallel corpus of italian administrative language. the corpus comprises 736 sentence pairs corresponding to two readability levels: original and simplified. we organized the corpus in three subsets according to the different nature of the applied simplification: operations (admin-it op ): 588 pairs of sen-tences from the subsection of the simpitiki corpus @xcite related to the administrative domain. these pairs contain manual simplifications produced by rewriting original sentences using single operations, such as split, reorder, merge, lexical substitutions, among others. the authors report that most simplifications in this dataset involve lexical transformations at word (single terms) and phrase (e.g., multiword expressions) levels, whereas the merging operation is never applied. rewritten sents (admin-it rs ): new 100 pairs of original-simplified sentences. the original sentences were selected from websites of italian municipalities, foot_0 and from the longest phrases from the pawac corpus @xcite . we manually rewrote the sentences simplifying them both at lexical and syntactic levels. our simplification criteria were based on the thirty rules for good administrative writing by @xcite and by considering the typical traits of the administrative language @xcite . for example, some particularly frequent simplification operations are: the replacement of verbal phrases formed by verb + noun with the corresponding simple verbs (e.g., from apporre la firma [append a signature] to firmare @xcite ; from effettuare un pagamento [make a payment] to pagare [pay]) and the transformation of nouns in verbs, since nominalization is a typical trait of administrative language that affects its degree of readability. in addition, uncommon nouns and verbs were replaced by synonyms present in the basic italian vocabulary @xcite , which contains the most frequent terms of contemporary italian. an exception were the technical terms of the administrative language or its subsectors (e.g., catasto [real estate registry]; deroga[waive]; referendum abrogativo [abrogative referendum]). at the syntactic level, the number of subordinate clauses and parenthetical expressions was reduced, favoring coordination and shorter sentences. rewritten docs (admin-it rd ): 48 pairs of sentences selected from administrative texts, which were collected and simplified by @xcite and made publicly available. 3 this resource contains pairs of original-simplified documents rewritten according to linguistic simplification and communicative effectiveness criteria. we manually aligned selected sentences by choosing from the documents only those sentences in which the sim- in order to make admin-it publicly available, we masked potentially sensitive data mentioned in the sentences, such as bank account numbers, addresses, licence numbers, phones and emails. ## neural pairwise ranking for sentences in this section, we briefly describe the neural pairwise ranking model (nprm) of @xcite that ranks documents according to their readability, and then explain how we apply it to rank original-simplified sentence pairs. nprm for documents. the model's input is composed of a list of (v, r) tuples, such as @xmath0, r i ), ..., (v n , r n )], where v i is the vector representation of a document and r i is its readability score. by analyzing all permutations of pairs of documents in the list, the model aims at maximizing the probability that r @xmath1, i.e., that the readability score of a document is higher than the score assigned to the other document in the pair, so that the predicted scores p 1 ij , p 2 ij correspond to p 1 @xmath2, v j ) and p 2 @xmath3, v j ). the nprm is parametrized as n p @xmath4, v j ))), where f is a bert model and ψ is a fully connected layer. the adopted loss function is the pairwise logistic loss @xcite . nprm for sentences. in our setting, the input text is sentences instead of documents. even though the nprm can rank an arbitrary number of texts in each list of tuples, due to the characteristics of our data, we rank sentences in only two readability levels: complex and simple. therefore, the input is now a list of two tuples with the vector representations of the original (s o ) and simplified (s s ) versions of the same sentence, and their readabilities. that is x @xmath5, r o i ), (s s i , r s i )]. no further changes were made to the original model. to validate our adaptation of the model, we examined the performance of the nprm for ranking sentences in a monolingual setting for english. we fine-tuned it on the ose corpus (see sec. 5.1) via 5-fold cross validation with bert-base-uncased. the resulted ranking accuracy was quite high (0.96) and close to the one obtained by @xcite for the document-level setup in the same corpus (0.98). this supports using nprms for ranking sentences. foot_2 ## experimental settings we adapted the released code of @xcite 5 for our sentence-level task, but retained their parameter settings during the fine-tuning of the nprms and the training of the baselines. models were trained and fine-tuned on an nvidia gpu titan rtx . ## results and discussion we describe different zero-shot experiments, finetuning our models on combinations of monolingual, cross-lingual, in-domain and cross-domain data, and always using admin-it for testing. while the nprms showed variations in performance depending on the fine-tuning setting (as will be explained below), that was not the case for baseline l , perhaps due to the simplicity of the features extracted, i.e., the length of sentences expressed in characters. for this reason, in ## analysis we analyze where the nprms failed when ranking sentence pairs from admin-it rd and admin-it op . we focus on these two subsets of admin-it given the high results already obtained on admin-it rs . ## conclusions and future work in this paper, we investigated the behavior of a neural pairwise ranking model (nprm) for assessing the readability of sentences from the italian administrative language in zero-shot settings. to deal with data scarcity in this domain, we built admin-it, a corpus of original-simplified parallel sentences in the italian administrative language, containing three different styles of simplifications. this corpus allowed us to prove that nprms are effective in cross-domain and cross-lingual zeroshot settings, especially when simplifications were produced over single sentences and at several linguistic levels. we also conduced an error analysis and showed that the characteristics of the data used for fine-tuning rather than its size have an impact on a model's performance. in addition, we determined that simplifications where information was added are poorly handled by the models. in future work, we plan to analyze how nprms perform on sentences with the same simplification style (e.g., admin-it rs ) annotated for different degrees of complexity by humans. we also plan to improve admin-it rs to address the needs of specific targets, such as second language learners, who require the insertion of definitions of technical terms (not provided in the current version). to develop ara models in this setting, we could leverage the alignments of srikanth and li (2021) that focus on elaborative simplifications. furthermore, we plan to fine-tune models with in-domain data from languages with higher proximity to italian, e.g., with datasets similar to the one built for spanish by @xcite . moreover, we would like to apply our models in concrete applications, like evaluation of automatic simplifications. finally, we aim at extending our approach to other domains and languages besides the administrative one. farah nadeem and mari ostendorf. 2018. estimating linguistic complexity for science texts. in proceedings of the thirteenth workshop on innovative use of nlp for building educational applications, pages 45-55, new orleans, louisiana. association for computational linguistics. alessio palmero aprosio, sara tonelli, marco turchi, matteo negri, and di gangi mattia. 2019. neural text simplification in low-resource conditions using weak supervision. in workshop on methods for optimizing and evaluating neural language generation (neuralgen), pages 37-44. association for computational linguistics (acl). lucia c. passaro and alessandro lenci. 2015. extracting terms with extra. in proceedings of eu-rophras 2015, pages 188-196, malaga, spain. tradulex. sarah e. petersen and mari ostendorf. 2009. a machine learning approach to reading level assessment. computer speech & language, 23(1):89-106. emily pitler and ani nenkova. 2008. revisiting readability: a unified framework for predicting text quality. in proceedings of the 2008 conference on empirical methods in natural language processing, pages 186-195. nils reimers and iryna gurevych. 2019. sentence-bert: sentence embeddings using siamese bert-networks. in proceedings of the 2019 conference on empirical methods in natural language processing and the 9th international joint conference on natural language processing (emnlp-ijcnlp), pages 3982-3992. carolina scarton, gustavo paetzold, and lucia specia. 2018. simpa: a sentence-level simplification corpus for the public administration domain. in proceedings of the eleventh international conference on language resources and evaluation (lrec 2018). sarah e. schwarm and mari ostendorf. 2005. reading level assessment using support vector machines and statistical language models. in proceedings of the 43rd annual meeting of the association for computational linguistics (acl'05), pages 523-530. luo si and jamie callan. 2001. a statistical model for scientific readability. in proceedings of the tenth international conference on information and knowledge management, pages 574-576. neha srikanth and junyi jessy li. 2021. elaborative simplification: content addition and explanation generation in text simplification. in findings of the association for computational linguistics: acl-ijcnlp 2021, pages 5123-5137, online. association for computational linguistics. amalia todirascu, thomas françois, delphine bernhard, núria gala, and anne-laure ligozat. 2016. are cohesive features relevant for text readability evaluation? in 26th international conference on computational linguistics (coling 2016), pages 987-997. sara tonelli, alessio palmero aprosio, and francesca saltori. 2016. simpitiki: a simplification corpus for italian. in clic-it/evalita. sowmya vajjala. 2022. trends, limitations and open challenges in automatic readability assessment research. in proceedings of the language resources and evaluation conference, pages 5366-5377, marseille, france. european language resources association. sowmya vajjala and ivana lučić. 2018. on-estopenglish corpus: a new corpus for automatic readability assessment and text simplification. in proceedings of the thirteenth workshop on innovative use of nlp for building educational applications, pages 297-304, new orleans, louisiana. association for computational linguistics.