Sentence Complexity in Context (2021)

Abstract: We study the influence of context on how humans evaluate the complexity of a sentence in English. We collect a new dataset of sentences, where each sentence is rated for perceived complexity within different contextual windows. We carry out an in-depth analysis to detect which linguistic features correlate more with complexity judgments and with the degree of agreement among annotators. We train several regression models, using either explicit linguistic features or contextualized word embeddings, to predict the mean complexity values assigned to sentences in the different contextual windows, as well as their standard deviation. Results show that models leveraging explicit features capturing morphosyntactic and syntactic phenomena perform always better, especially when they have access to features extracted from all contextual sentences.

## introduction from a human-based perspective, sentence complexity is assessed by measures of processing effort or performance in behavioral tasks. in this respect, a large part of studies has focused on reading single sentences and correlating syntactic and lexical properties with observed difficulty, being it captured by cognitive signals, such as eye-tracking metrics @xcite , or by explicit judgments of complexity given by readers @xcite . however, models of language comprehension underline the importance of contextual cues, such as the presence of explicit cohesive devices, in building a coherent representation of a text @xcite . this implies that a sentence can be perceived as more or less difficult according to the context in which it is presented. the effect of context on how humans evaluate a sentence has been investigated concerning its acceptability and grammaticality, two properties different from complexity, yet somehow related. in @xcite speakers were asked to evaluate the degree of acceptability of sentences from wikipedia, both in their original form and with some grammatical alterations artificially introduced by a process of round-trip machine translation. results showed that ill-formed sentences are evaluated as more acceptable when presented within context (i.e. along with their preceding or following sentence) rather than in isolation. more closely related to our study is the one by @xcite on readability assessment. in that work, authors gathered pairwise evaluations of reading difficulty on sentences presented with and without a larger context, training a logistic regression model to predict binary complexity labels assigned by humans. they observed that the context slightly modifies the perception of the readability of a sentence, although their predictive models perform better on sentences rated in isolation. our study aims to understand how the context surrounding a sentence influences its 'perceived' complexity by humans. as we consider linguistic complexity from the individual's perspective, following @xcite , we use the term complexity as a synonym of difficulty. also, we assume that sentence complexity is a gradient rather than a binary concept and we operationalize perceived complexity as a score on an ordinal scale. these scores were collected for a new dataset of sentences, where each sentence has been evaluated in three contextual windows, which change according to the position the sentence occupies within them. this enables us to deeply inspect the role of context, allowing us to determine if the perceived complexity of a sentence changes when the context is introduced, and also which contextual window may impact more. to do so, we consider the average complexity score assigned to each sentence as well as the degree of agreement among annotators, calculated in terms of standard deviation. we think that this measure is also relevant to comprehend perceived sentence complexity since this is a highly subjective task that cannot be tackled by following specific annotation guidelines. moreover, knowing that sentence length is a prominent predictor of sentence complexity, we study how complexity scores and annotators' agreement vary for sentences of the same length. finally, we run experiments to evaluate the accuracy of different regression models in predicting the mean complexity label and standard deviation assigned to a sentence. in particular, we compare models leveraging explicit linguistic features related to a wide set of morphosyntactic and syntactic properties of a sentence, to models exploiting the predictions of a state-of-the-art bidirectional transformer encoder, i.e. bert @xcite . to the best of our knowledge, the assessment of sentence complexity in context has never been tackled as a downstream task for evaluating the effectiveness of neural representations in modeling aspects of sentence complexity. despite the remarkable performance that neural language models have achieved so far in a variety of nlp tasks @xcite ) -also close to ours such as the prediction of perceived sentence acceptability in context (lau et al., 2020) -our results show that this is not the case as regards to the prediction of sentence complexity: models using explicit linguistic features perform better in all contextual windows, suggesting that information embedded in neural representations could be less effective in modeling the examined task, particularly when few labeled data will be made available. contributions. i) we release a new dataset of ∼2,900 english sentences rated with human judgments of complexity elicited by presenting sentences in their contexts; ii) we model a wide set of morphosyntactic and syntactic phenomena, extracted from the single sentence and contextual ones, and we study which of them are more corre-lated with sentence complexity judgments in different contextual windows; iii) we show that models relying explicitly on these features achieve higher performance in predicting complexity judgments than state-of-the art neural language models, proving the effectiveness of these features to address this task in particularly in a low resource scenario. all the data discussed here will be made available at: www.italianlp.it/resources/ . ## approach we first collected an appropriate corpus to evaluate the effect of context on the perception of sentence complexity. we started from the crowdsourced dataset by @xcite , which contains 1, 200 sentences annotated for perceived complexity on a 7-point likert scale. we similarly built a crowdsourcing task, asking native english speakers to read each sentence and rate its complexity on the same scale. however, while in brunato et al. all sentences were rated in isolation, in our task sentences were presented in three contextual windows, as illustrated in section 2.1. to study which linguistic phenomena may affect annotators' ratings, we represented sentences (the rated one and the contextual ones) with ∼100 linguistic features, based on those described in @xcite . these features model a wide range of sentence properties, which can be viewed as proxies of sentence complexity at different levels of linguistic annotation. the features were first used to study the influence of context-level and sentence-level phenomena on perceived complexity. we did this by analyzing the correlations between the features and the complexity ratings, and the correlations between the features and the standard deviation of complexity. then, we assessed the automatic prediction of sentence complexity and of the standard deviation of complexity judgments, evaluating if adding information from the context helps in the prediction. we tested two predicting approaches: one based on a linear svm regression model which leverages the linguistic features discussed so far, and one that employs bert, one of the most prominent pre-trained neural language model. we compared the accuracy of the models across various scenarios, considering their predictions both for sentences rated in different contextual windows and for sentences distinguished into same-length bins. ## correlation between linguistic features and complexity to detect which linguistic phenomena are more involved in the assessment of sentence complexity, and to verify whether these phenomena capture information about the sentence itself or about the context, we performed a correlation analysis between the complexity score assigned to each sentence and a wide set of linguistic features extracted from the sentence. for each sentence, we computed the spearman's rank correlation coefficient between the average complexity score and the value of each linguistic feature extracted from i) the rated sentence, ii) its preceding one and iii) its following one, according to the contextual window. we performed the correlation analysis on the sentences altogether and then dividing them into bins according to their length. the same process was repeated correlating the standard deviation of complexity scores with the linguistic features of each sentence. as stated in section 2, we focused on features that model a wide range of sentence properties extracted from different levels of linguistic annotation, from raw text features (i.e. sentence and word length) to morphosyntactic information (e.g. distribution of verbs according to morphological features such as tense, mood, person), to more complex aspects of the syntactic structure capturing global and local information (e.g. parse tree depth, length of dependency link, use of subordination). in what follows, we discuss the correlation results for the subset of sentences presented in the center window, since this is the only one in which the rated sentence was always surrounded by both a left and a right sentence, allowing us to compare the effect of the two context positions 4 . considering first the average complexity score, we found statistically significant correlations (p-@xmath0.05) with ρ ≥ ±0.20 for 103 features out of the whole set. among them, 44% belongs to the rated sentence (i.e. 45 features) and 56% to the contextual ones (i.e. 23 and 35 features to the left and the right sentence, respectively). although we could expect that many features extracted from the rated sentence were correlated to complexity judgments, these results also suggest that humans have paid attention to the whole context when rating the middle sentence, and especially to the following 4 we report in the appendix the whole tables of correlation results for all contextual windows. ## predicting sentence complexity the results of the correlation analysis have shown that linguistic information of the context affects the perception of sentence complexity and the extent to which this perception is shared by annotators. we thus proceed to assess the contribution of the context from a modeling standpoint. we built two regression tasks, one to predict the average complexity value assigned to each sentence, and one to predict the standard deviation of complexity for each sentence. in both scenarios, we employed two different models: the first is a linear svm regression model with standard parameters that leverages the explicit linguistic features presented in for every window, we carried out different runs of the models, varying the amount of contextual features to be considered. for the begin window (a) svm models (b) bert models figure 5: performance (mae) of svm regression models and bert models in the prediction of complexity standard deviation. in different windows of context and with different context spans, for all sentences and at different sentence lengths. and the end window we ran the models with i) the features of the single sentence (no context), ii) the features of the sentence + the features of the next sentence (right context) for the begin window, or + the features of the previous sentence (left context) for the end window, iii) the features of all the three sentences (full context, i.e. the whole window of context); for the center window, we trained the models with i) no context features, ii) left or right context features, iii) full context features. we measured the performance of the models in terms of mean absolute error (mae), evaluating their accuracy in predicting the same average judgment of complexity assigned by humans and the standard deviation of the complexity judgments. we then repeated the same experiments grouping the sentences according to their length. the baseline for the models evaluation was calculated (i) in the case of all sentences, by giving in input to the linear regression model only the length of the sentence as feature for the prediction, (ii) in the case of different lengths (binned sentences), by having the model always assigning the average complexity value (calculated on the whole set of sentences) to each sentence. figure 4 reports the results for the prediction of the average complexity, showing the average mae obtained after the 5-fold validation, both for svm models and bert models. the svm models with linguistic features outperform bert models overall. bert models remain close to the baseline in all cases, despite the amount of context considered and the length of the sentences. instead, the svm models show significant differences as appropriate. in the case of all sentences, the performances of the model are close to the baseline. adding contextual features slightly helps in the case of the begin and the end window, while performances worsen in the case of the center window. when considering sentences of the same length, the performance of the model is always helped by the presence of contextual features and best results are achieved when the full context is taken into account, for all the windows of context. this behavior confirms on one side that linguistic characteristics of the context are indeed very influential on complexity, on the other side that the length of the sentence plays an important role on the perception of complexity, as it is only by binning the sentences that we can exploit the effect of context in predicting complexity. figure 5 shows the results for the prediction of the standard deviation of complexity, for svm models and bert models. as in the previous case, bert models obtain results that are in line with the baseline and that are not influenced by different amounts of context. when looking at the results obtained with the explicit linguistic features, the outcome is quite different. for the all sentences case, the svm model cannot predict the standard deviation of complexity, although the error gets lower for the begin window and the end window when the full context is used. conversely, the model shows large improvement when working on sentences of the same length. in all windows and for all lengths, using the features of the whole context significantly decreases the error in the prediction of standard deviation. when running the model with the features of the single sentence (i.e. no context), the performances of the model are in general close to the ones of the baseline. this suggests that the context is particularly relevant in predicting how people will agree on their perception of complexity. overall, our results show that information about the complexity of a sentence is better encoded in its explicit linguistic features, thus its syntactic and morphosyntactic structures. on the other hand, although bert has been proven to embed a wide range of linguistic properties, including syntactic ones @xcite , our findings seem to suggest that this model does not exploit these kind of features to solve a downstream task like ours, for which few data are available. indeed, it has been shown that bert performs better on datasets larger than ours @xcite . thus, it is fair to assume that more data may be needed for bert to detect phenomenon about perceived complexity. moreover, our results show that the presence of context plays an important role on complexity. as the svm models are always helped by the contextual features, it is fair to assume that annotators have taken into account the whole context when expressing their judgment upon complexity, and that the presence of the context has strongly influenced their perception. also, contextual linguistic phenomena are the ones that impact more on the variation of complexity perception between annotators as they are the ones that help more in the prediction of this variation. ## conclusion we studied how the context surrounding a sentence influences the perception of its complexity by humans. starting from a newly collected dataset, we investigated which linguistic phenomena, among a wide set of lexical, morphosyntactic, and syntactic ones, are more correlated with complexity judgments and the degree of agreement between annotators. from a modeling standpoint, we observe that models using explicit linguistic features achieve higher accuracy than state-of-the-art neural language models in predicting the average complexity score assigned to a sentence, as well as the variation among scores. this is especially true when they use explicit linguistic features from all contextual sentences in addition to the linguistic features of the sole rated sentence. as many nlp applications are concerned with the analysis of linguistic complexity, particularly for text readability and text simplification purposes, we think that our results emphasize the importance of considering contextual information both in the creation of gold benchmarks, which are typically based only on data paired at sentence level and in the development of cognitively inspired evaluation systems driven by how people perceive complexity.