That Looks Hard: Characterizing Linguistic Complexity in Humans and Language Models (2021)

Abstract: This paper investigates the relationship between two complementary perspectives in the human assessment of sentence complexity and how they are modeled in a neural language model (NLM). The first perspective takes into account multiple online behavioral metrics obtained from eye-tracking recordings. The second one concerns the offline perception of complexity measured by explicit human judgments. Using a broad spectrum of linguistic features modeling lexical, morpho-syntactic, and syntactic properties of sentences, we perform a comprehensive analysis of linguistic phenomena associated with the two complexity viewpoints and report similarities and differences. We then show the effectiveness of linguistic features when explicitly leveraged by a regression model for predicting sentence complexity and compare its results with the ones obtained by a fine-tuned neural language model. We finally probe the NLM’s linguistic competence before and after fine-tuning, highlighting how linguistic information encoded in representations changes when the model learns to predict complexity.

## introduction from a human perspective, linguistic complexity concerns difficulties encountered by a language user during sentence comprehension. the source of such difficulties is commonly investigated using either offline measures or online behavioral metrics. in the offline framework, complexity ratings can be elicited either by assessing errors in comprehension tests or collecting explicit complexity judgments from readers. instead, in the online paradigm, cognitive signals are collected mainly through specialized machinery (e.g., mri scanners, eye-tracking systems) during natural or task-oriented reading. among the wide range of online complexity metrics, gaze data are widely regarded as reliable proxies of processing difficulties, reflecting both low and high-level complexity features of the input @xcite . eye-tracking measures have recently contributed to significant improvements across many popular nlp applications @xcite and in particular on tasks related to linguistic complexity such as automatic readability assessment (ara) @xcite @xcite , obtaining meaningful results for sentence-level classification in easy and hard-to-read categories @xcite @xcite . however, readability levels are conceptually very different from cognitive processing metrics since ara corpora are usually built in an automated fashion from parallel documents at different readability levels, without explicit evaluations of complexity by target readers @xcite . a different approach to complexity assessment that directly accounts for the perspective of readers is presented in the corpus by @xcite , where sentences are individually labeled with the perception of complexity of annotators, which may better reflect the underlying cognitive processing required by readers to parse the sentence. this consideration is supported by recent results highlighting the unpredictability of outliers in perceived complexity annotations, especially for sentences having complex syntactic structures @xcite . given the relation between complexity judgments elicited from annotators and online cognitive processing metrics, we investigate whether the connection between the two perspectives can be highlighted empirically in human annotations and language model representations. we begin by leveraging linguistic features associated with a variety of sentence-level structural phenomena and analyzing their correlation with offline and online complexity metrics. we then evaluate the performance of models using either complexity-related explicit features or contextualized word embeddings, focusing mainly on the neural language model al-bert @xcite . in this context, we show how both explicit features and learned representations obtain comparable results when predicting complexity scores. finally, we focus on studying how complexity-related properties are encoded in the representations of albert. this perspective goes in the direction of exploiting human processing data to address the interpretability issues of unsupervised language representations @xcite @xcite . to this end, we rely on the probing task approach, a recently introduced technique within the area of nlms interpretability consisting of training diagnostic classifiers to probe the presence of encoded linguistic properties inside contextual representations @xcite . we observe that fine-tuning on online and offline complexity produces a consequent increase in probing performances for complexityrelated features during our probing experiments. this investigation has the specific purpose of studying whether and how learning a new task affects the linguistic properties encoded in pretrained representations. in fact, while pre-trained models have been widely studied using probing methods, the effect of fine-tuning on encoded information was seldom investigated. for example, @xcite found that fine-tuning does not impact heavily the linguistic information implicitly learned by the model, especially when considering a supervised probe closely related to a downstream task. @xcite further demonstrated a positive correlation between the model's ability to solve a downstream task on a specific input sentence and the related linguistic knowledge encoded in a language model. nonetheless, to our knowledge, no previous work has taken into account sentence complexity assessment as a fine-tuning task for nlms. our results suggest that the model's competencies during training are interpretable from a linguistic perspective and are possibly related to its predictive capabilities for complexity assessment. contributions to our best knowledge, this is the first work displaying the connection between online and offline complexity metrics and studying how they are represented by a neural language model. we a) provide a comprehensive analysis of linguistic phenomena correlated with eye-tracking data and human perception of complexity, addressing ## data and preprocessing our study leverages two corpora, each capturing different aspects of linguistic complexity: eye-tracking for online complexity metrics, we used the monolingual english portion of geco @xcite , an eye-tracking corpus based on the novel "the mysterious case at styles" by agatha christie. the corpus consists of 5,386 sentences annotated at word-level with eyemovement records of 14 english native speakers. we select four online metrics spanning multiple annotation level linguistic feature description label raw text sentence length (tokens), word length (characters) n_tokens, char_per_tok words and lemmas type/token ratio ttr_form, ttr_lemma pos tagging distribution of ud and language-specific pos tags upos_dist_*, xpos_dist_* lexical density lexical_density inflectional morphology of auxiliaries (mood, tense) aux_mood_*, aux_tense_* dependency parsing syntactic tree depth parse_depth average and maximum length of dependency links avg_links_len, max_links_len number and average length of prepositional chains n_prep_chains, prep_chain_len relative ordering of main elements subj_pre, subj_post, obj_pre, obj_post distribution of dependency relations dep_dist_* distribution of verbal heads vb_head_per_sent distribution of principal and subordinate clauses princ_prop_dist, sub_prop_dist average length of subordination chains sub_chain_len relative ordering of subordinate clauses sub_post, sub_pre perceived complexity for the offline evaluation of sentence complexity, we used the english portion of the corpus by @xcite . the corpus contains 1,200 sentences taken from the wall street journal section of the penn treebank @xcite with uniformly-distributed lengths ranging between 10 and 35 tokens. each sentence is associated with 20 ratings of perceivedcomplexity on a 1-to-7 point scale. ratings were assigned by english native speakers on the crowd-flower platform. to reduce the noise produced by the annotation procedure, we removed duplicates and sentences for which less than half of the annotators agreed on a score in the range µ @xmath0, where µ n and σ n are respectively the average and standard deviation of all annotators' judgments for sentence n. again, we average scores across annotators to obtain a single metric for each sentence. ## analysis of linguistic phenomena as a first step to investigate the connection between the two complexity paradigms, we evaluate the correlation of online and offline complexity labels with linguistic phenomena modeling a number of properties of sentence structure. to this end, we rely on the profiling-ud tool @xcite to annotate each sentence in our corpora and extract from it ∼100 features representing their linguistic structure according to the universal dependencies formalism @xcite . these features capture a comprehensive set of phenomena, from basic information (e.g. sentence and word length) to more complex aspects of sentence structure (e.g. parse tree depth, verb arity), including properties related to sentence complexity at different levels of description. a summary of most relevant features in our analysis is presented in figure 1 reports correlation scores for features showing a strong connection (|ρ| > 0.3) with at least one of the evaluated metrics. features are ranked using their spearman's correlation with complexity metrics, and scores are leveraged to highlight the relation between linguistic phenomena and complexity paradigms. we observe that features showing a significant correlation with eyetracking metrics are twice as many as those correlating with pc scores and generally tend to have higher coefficients, except for total regression duration (trd). nevertheless, the most correlated features are the same across all metrics. as expected, sentence length (n_tokens) and other related features capturing aspects of structural complexity occupy the top positions in the ranking. among those, we also find the length of dependency links (max_links_len, avg_links_len) and the depth of the whole parse tree or selected sub-trees, i.e. nominal chains headed by a preposition (parse_depth, n_prep_chains). similarly, the distribution of subordinate clauses (sub_prop_dist, sub_post) is positively correlated with all metrics but with stronger effect for eye-tracking ones, especially in presence of longer embedded chains (sub_chain_len). interestingly, the presence of numbers (upos_num, dep_nummod) affects only the explicit perception of complexity while it is never strongly correlated with all eye-tracking metrics. this finding is expected since numbers are very short tokens and, like other functional pos, were never found to be strongly correlated with online reading in our results. conversely, numerical information has been identified as a factor hampering sentence readability and understanding @xcite . unsurprisingly, sentence length is the most correlated predictor for all complexity metrics. since many linguistic features highlighted in our analysis are strongly related to sentence length, we tested whether they maintain a relevant influence when this parameter is controlled. to this end, spearman's correlation was computed between features and complexity tasks, but this time considering bins of sentences having approximately the same length. specifically, we split each corpus into 6 bins of sentences with 10, 15, 20, 25, 30 and 35 tokens respectively, with a range @xmath1. figure 2 reports the new rankings of the most correlated linguistic features within each bin across complexity metrics (|ρ| > 0.2). again, we observe that features showing a significant correlation with complexity scores are fewer for pc bins than for eye-tracking ones. this fact depends on controlling for sentence length but also on the small size of bins for the whole dataset. as in the coarse-grained analysis, trd is the eye-tracking metric less correlated to linguistic features, while the other three (fxc, fpd, tfd) show a homogeneous behavior across bins. for the latters, vocabulary-related features (token-type ratio, average word length, lexical density) are always ranked on top (and with a positive correlation) in all bins, especially when considering shorter sentences (i.e. from 10 to 20 tokens). for pc, this is true only for some of them (i.e. word length and lexical density). at the same time, features encoding numerical information are still highly correlated with the explicit perception of complexity in almost all bins. interestingly, features modeling subordination phenomena extracted from fixed-length sentences exhibit a reverse trend than when extracted from the whole corpus, i.e. they are negatively correlated with judgments. if, on the one hand, we expect an increase in the presence of subordination for longer sentences (possibly making sentences more convoluted), on the other hand, when length is controlled, our findings suggest that subordinate structures are not necessarily perceived as a symptom of sentence complexity. our analysis also highlights that pc's relevant features are significantly different from those correlated to online eye-tracking metrics when controlling for sentence length. this aspect wasn't evident from the previous coarse-grained analysis. we note that, despite controlling sentence length, gaze measures are still significantly connected to length-related phenomena. this can be possibly due to @xmath2. ## predicting online and offline linguistic complexity given the high correlations reported above, we proceed to quantify the importance of explicit linguistic features from a modeling standpoint. lightweight yet effective alternative to bert @xcite for obtaining contextual word representations, using its last-layer [cls] sentence embedding as input for a linear regressor during fine-tuning and testing. we selected the last layer representations, despite having strong evidence on the importance of intermediate representation in encoding language properties, because we aim to investigate how final layers encode complexityrelated competences. given the availability of parallel eye-tracking annotations, we train albert using multitask learning with hard parameter sharing @xcite on gaze metrics. 2 from results in as a final step in the study of feature-based models, we inspect the importance accorded by the svm feats model to features highlighted in 2 additional information on parameters and chosen training approach is presented in appendix a. ## probing linguistic phenomena in albert representations to conduct the probing experiments, we aggregate three ud english treebanks representative of different genres, namely: ewt, gum and par-tut by @xcite ; @xcite ; @xcite , respectively. we thus obtain a corpus of 18,079 sentences and use the profiling-ud tool to extract n sentence-level linguistic features @xmath3, . . . , z n from gold linguistic annotations. we then generate representations a(x) of all sentences in the corpus using the last-layer [cls] embedding of a pretrained al-bert base model without additional fine-tuning, and train n single-layer perceptron regressors g i : a(x) → z i that learn to map representations a(x) to each linguistic feature z i . we finally evaluate the error and r 2 scores of each g i as a proxy to the quality of representations a(x) for encoding their respective linguistic feature z i . we repeat the same evaluation for alberts fine-tuned respectively on perceived complexity (pc) and on all eye-tracking labels with multitask learning (et), averaging scores with 5-fold cross-validation. results are shown on the left side of as we can see, albert's last-layer sentence representations have relatively low knowledge of complexity-related probes, but the performance on them highly increases after fine-tuning. specifically, a noticeable improvement is obtained on features that were already better encoded in base pretrained representation, i.e. sentence length and related features, suggesting that fine-tuning possibly accentuates only properties already well-known by the model, regardless of the target task. to verify that this isn't the case, we repeat the same experiments on albert models fine-tuned on the smallest length-binned subset (i.e. 10±1 tokens) presented in previous sections. the right side of the et-trained model confirms the same behavior, retaining strong but lower performances for length-related features. we note that, for all metrics, features that were highly relevant only for the svm predictions, such as those encoding verbal inflectional morphology or vocabulary-related ones ( to conclude, although higher probing tasks performances after fine-tuning on complexity metrics should not be interpreted as direct proof that the neural language model is exploiting newlyacquired morpho-syntactic and syntactic information, they suggest an importance shift in nlm representation, triggered by fine-tuning, that produces an encoding of linguistic properties able to better model the human assessment of complexity. ## conclusion this paper investigated the connection between eye-tracking metrics and the explicit perception of sentence complexity from an experimental standpoint. we performed an in-depth correlation analysis between complexity scores and sentence-level properties at different granularity levels, highlighting how all metrics are strongly connected to sentence length and related properties, but also revealing different behaviors when controlling for length. we then evaluated models using explicit linguistic features and unsupervised word embeddings to predict complexity, showing comparable performances across metrics. we finally tested the encoding of linguistic properties in the contextual representations of a neural language model, noting the natural emergence of task-related linguistic properties within the model's representations after the training process. we thus conjecture that a relation subsists between the linguistic knowledge acquired by the model during the training procedure and its downstream performances on tasks for which the morphosyntactic and syntactic structures play a relevant role. for the future, we would like to test comprehensively the effectiveness of tasks inspired by the human language learning as intermediate steps to train more robust and parsimonious neural language models.