Using Eye-tracking Data to Predict the Readability of B razilian P ortuguese Sentences in Single-task, Multi-task and Sequential Transfer Learning Approaches (2020)

Abstract: Sentence complexity assessment is a relatively new task in Natural Language Processing. One of its aims is to highlight in a text which sentences are more complex to support the simplification of contents for a target audience (e.g., children, cognitively impaired users, non-native speakers and low-literacy readers (Scarton and Specia, 2018)). This task is evaluated using datasets of pairs of aligned sentences including the complex and simple version of the same sentence. For Brazilian Portuguese, the task was addressed by (Leal et al., 2018), who set up the first dataset to evaluate the task in this language, reaching 87.8% of accuracy with linguistic features. The present work advances these results, using models inspired by (Gonzalez-Garduño and Søgaard, 2018), which hold the state-of-the-art for the English language, with multi-task learning and eye-tracking measures. First-Pass Duration, Total Regression Duration and Total Fixation Duration were used in two moments; first to select a subset of linguistic features and then as an auxiliary task in the multi-task and sequential learning models. The best model proposed here reaches the new state-of-the-art for Portuguese with 97.5% accuracy 1 , an increase of almost 10 points compared to the best previous results, in addition to proposing improvements in the public dataset after analysing the errors of our best model.

## introduction readability is the ease of reading a text, not in its typographical aspects such as font size, but by measures such as its syntactic structure complexity, vocabulary frequency, content, style, and organisation that can be fitted to prior knowledge, reading skill, interest and motivation of the reader @xcite . tracking the automation of readability back to its origin, the first readability formulas can be found a century ago in the united states, aiming to help teachers, librarians and scholars to select reading material for classes @xcite . at that time, it was considered that complexity could be inferred by surface-level metrics of words and sentences, based on the frequency and size (number of letters) of the words and on the average number of words per sentence. since then, readability analysis has become a large area of multidisciplinary research, which has an ever growing body of literature, related tasks (e.g., text simplification task @xcite ) and text summarization task @xcite ), and has gained new computational approaches in this century using natural language processing (nlp) and machine learning methods @xcite . traditionally, the readability assessment task has been applied to the text level, assigning a grade (or level of proficiency ranking) for an entire document. however, in a document classified as simple, complex sentences can occur, just as there are simple sentences in a complex document. a sentence is an important unit that provides, in most cases, enough information to be able to infer and analyse its complexity. although the same approach can be used to assess the complexity of texts at the sentence level, @xcite demonstrated that a greater number of features are needed for readability prediction at the sentence level. a study conducted by @xcite has achieved state-of-the-art performance in readability prediction for english sentences, using multi-task learning and eye-tracking measures. an example of an application for the sentence level approach is the complexity checker tool, proposed by @xcite that analyses all sentences in a text, highlighting the complex ones to help with the simplification process. this paper presents a thorough evaluation of sentence readability prediction for brazilian portuguese (bp), starting by evaluating single-task methods, followed by a replication of the work developed by @xcite . at the end, we propose a new model based on the sequential transfer learning approach @xcite , which has achieved state-of-the-art performance in readability prediction of bp sentences. section 2 presents a literature review of the main works in readability prediction at sentence level (rpsl). section 3 describes the corpora and metrics used and section 4 presents the models evaluated and experimental results. section 5 presents an analysis of the main errors of our best model, followed by a revision of the evaluation dataset and the results of our final best model. section 6 draws the conclusions and proposes future research. ## readability prediction at sentence level the first studies on rpsl appeared in the last decade. therefore, we can consider them as a recent research task, which aims to individually analyse and evaluate the sentences of a text, allowing for more accurate information of their complex points. the first study to consider the rpsl task was @xcite , who compared its difficulty with readability at text level. however, a proposal of assessing the task was only consolidated by @xcite , leading to further studies to improve the results comparatively (see according to @xcite , sentence level readability is relevant because approaches to classifying text readability do not bring great advantages to the subsequent application of automatic simplification methods. furthermore, considering all sentences as complex in a text classified as complex can impair the training of methods, especially when these sentences are used to assess the task of predicting sentence complexity. this was demonstrated by @xcite when investigating the reasons for the low accuracy obtained from the wikipedia-simplewikipedia corpus, used without any sentence alignment method. @xcite and @xcite also explored the rpsl task using new metrics; the first study exclusively evaluated psycholinguistic metrics and the second one eye-tracking metrics. @xcite improved the results significantly by using a combinatory categorial grammar (ccg) parser, and (gonzalez-garduño and søgaard, 2018) achieved state-of-the-art performance in readability prediction for english sentences, using multi-task learning and eye-tracking measures combined with linguistic and psycholinguistic features. in addition, (gonzalez-garduño and søgaard, 2018) compared the performance of readability models that use eye-tracking data of native speakers with models using data from language learners. there was no significant drop in performance when replacing learners with natives, i.e. language learner difficulties can be efficiently estimated from native speakers. these findings are important since, in this paper we replicate the results of (gonzalez-garduño and søgaard, 2018), using an eye-tracking data with native speakers of brazilian portuguese, that we created to study predictability in reading. ## resources the porsimplessent corpus has three versions that include different approaches to sentences that have undergone the split operation. pss1 repeats the original sentence for each sentence resulting from the division; pss2 selects only the largest resulting sentence, which also has the greatest overlap of words, and pss3 contains only sentences that have not been divided, and thus is the smallest of the three. for the present study, version pss2 was chosen. it has 4,962 pairs of sentences, with original-natural, natural-strong and original-strong alignments, obtained in tsv format foot_1 . ## approaches and results the models developed and evaluated for the task are presented below. the first step was to validate whether the current linguistic features could predict the eye-tracking measures. only after proving that, the measures were used as a basis for feature selection, followed by a comparison among the singletask, multi-task and sequential transfer learning approaches. all models were evaluated with 10-fold cross validation and trained with an adam optimiser, implemented using the keras (chollet and others, 2015) and scikit-learn packages @xcite for the python language. as far as we could investigate, this work is the first to use sequential transfer learning in the rpsl task. ## error analysis after running the best model with 10-fold cross validation, all 151 pairs in which the model failed the prediction were manually annotated in a thoughtful error analysis, shown in figure 2 . this enabled us to verify that the second biggest cause of errors are problems in the dataset, possibly caused by the automatic alignment of sentences. there were problems in 23% of the pairs, such as just a capitalised letter, an added comma or spelling correction on the simplified side. the authors understand that these errors should be removed from the public dataset, as they do not represent real cases of simplification. the third major cause of errors are simplifications that go beyond the context of the sentence itself and would necessarily need the previous or subsequent sentences so that the model could automatically decide the correct class. however, it is important to note that the main cause of errors refers to simplifications at the lexical level, indicating the need to refine the metrics at this level and possibly to include new ones. to validate how the model behaves without the 54 pairs of sentences suggested for removal, it was run again in the new dataset, without these pairs, reaching the accuracy of 97.5% with an improvement of approximately 1 point as expected. the cleaned dataset was sent to the pss authors recommending them to publish it as a new revised version. ## conclusions this work reinforces the observations of (gonzalez-garduño and søgaard, 2018) on the importance of using eye-tracking measures, as well as models with transfer learning (multi-task and sequential learning) for the task of sentence readability assessment. moreover, it establishes the new state-of-the-art for the task of assessing sentence complexity in the brazilian portuguese language, with a substantial increase of almost 10 points over the best accuracy obtained so far. it also contributes to improving the pss2 dataset, identifying and proposing the elimination of alignment errors for future evaluations. the source codes with the implemented models are publicly available at https://github.com/ sidleal/simpligo-ranking. regarding future research, in order to mitigate errors at the lexical level, as mentioned in section 5, it is worth using a model combining word embeddings in an architecture with multi-view learning, as well as implementing and validating the recurring neural networks and attention-based architectures. another question that deserves further investigation is the difference observed when using the average and sum of eye-tracking measures, and why the average did not work well in our scenario. one hypothesis is that this may be related with the text genres of the eye-tracking dataset. we also intend to test the models proposed here in the two other versions of porsimplessent corpus.