Learning Syntactic Dense Embedding with Correlation Graph for Automatic Readability Assessment (2021)

Abstract: Deep learning models for automatic readability assessment generally discard linguistic features traditionally used in machine learning models for the task. We propose to incorporate linguistic features into neural network models by learning syntactic dense embeddings based on linguistic features. To cope with the relationships between the features, we form a correlation graph among features and use it to learn their embeddings so that similar features will be represented by similar embeddings. Experiments with six data sets of two proficiency levels demonstrate that our proposed methodology can complement BERT-only model to achieve significantly better performances for automatic readability assessment.

## introduction readability is the ease with which a reader can understand a written text 1 . predicting readability has been widely applied in education @xcite , book publishing @xcite , marketing @xcite , newspaper readership @xcite , and health information communication @xcite . ever since the first study by lively and pressey in 1923, many researchers have developed various popular readability formulas including flesch @xcite , fog @xcite and lexile @xcite . these traditional readability formulas are favored by domain applications due to their simplicity even though the formulas are mostly based on shallow features and known to lack accuracy @xcite @xcite . its strong reliance on expert knowledge is also a burden to adapt it to a new domain. machine learning approaches, which incorporate a broader set of morphological, lexical, syntactic, and discourse features, have shown to achieve better accuracy in readability assessment @xcite . figure 1 (a) describes a generic machine-learning framework for automatic readability assessment (ara) where manual feature engineering is an important step to extract important linguistic features for building readability classification models. to bypass the necessity of heavy feature engineering, deep learning strategies have been studied to automatically detect patterns or extract features related to readability (azpiazu and @xcite @xcite . figure 1 (b) provides a generic neural network structure of deep learning approach to ara. while neural network models take word embedding as input, they in general discard linguistic features traditionally used in machine learning models @xcite . if ever incorporated, linguistic features such as pos and morphological tags are only used to guide attention mechanism for embedding representation of the text @xcite . pre-trained models such as bert @xcite learn dense representations of text by informing the models with semantically neighboring words, sentences, or context. despite the attempts of recent research to assess bert's ability to implicitly capture the structural properties of language @xcite @xcite , it has been observed that bert "tends to rely more on semantic than structural differences during the classification phase and therefore performs better on problems with distinct semantic differences between classes" @xcite . there is clearly a lack of explicit consideration of syntactic (and structural) features in the current bert-based models for ara, which is known to be crucial. in this study, we address the problem of augmenting the ability of bert with widely used linguistic features in ara. to best integrate with bert, we create syntactic dense embedding as shown in figure 1(d ). an important problem we consider in this paper is the possible relationships between different features. linguistic features defined by linguistic experts may often be related. in this paper, we represent pairwise correlations between features as triplets with linguistic features as nodes and their correlations as edges. positive correlation implies that two features behave similarly in influencing the readability level of the text and should be represented with similar embeddings. the set of triplets forms a graph (as illustrated in figure 1(d) ). we then learn the dense representations of linguistic features with graphbased models. by encoding the similarity knowledge with dense embeddings, the ara classifier models will be better informed and gain predictive strength. our experiments on six datasets will confirm the effectiveness of this approach. we contribute to the research on automatic readability assessment in the following directions: ## methodology let 𝐹 = @𝑓 & , … , 𝑓 ' # b (where 𝑁 ( is the number of features) be a linguistic feature set designed for readability assessment. let matrix 𝒟 be a collection of the vector representations of 𝑁 ) documents with 𝑑 ! ∈ 𝑅 ' # , where 𝑑 ! = (𝑥 " , … , 𝑥 # ! ) $ , and 𝑥 " (1 ≤ 𝑗 ≤ 𝑁 ( ) is the value of feature 𝑓 " in 𝑑 ! . to construct the syntactic dense embeddings for document representation, we perform the following steps: to construct semantic dense embeddings for the documents, we learn the bert average embedding representations following the original procedures as shown in figure 2 , where the final bert representation is the average over all tokens. an alternative approach is to use the [cls] token embedding to represent the text and fine-tune it for prediction. in our pilot study, we experimented rigorously with different finetuning strategies for each of the six datasets. the best finetuning results as compared with the original bert average embeddings are reported in appendix a. the sizes of our corpora are small ranging from 326 to 2500 as described later in we first feed the bert embeddings into a fourlayer network and the syntactic dense embeddings into a two-layer network. we then concatenate the outputs of the two channels into combined syntactic-semantic dense embeddings as input into another two-layer network, with mlp and softmax layers for readability classification. the model architecture is provided in figure 3 . ## results and analysis we first present the comparison of bert-only dnn model with two traditional machine learning models of svm and logistic regression, and three other single channel dnn models. next, we compare bert-only model with the dual-channel dnn models with raw+bert and g-doc+bert. we find that augmenting bert with raw feature value vector or document vector based on gaussian embedding can slightly improve the performance of bert, showing that the raw linguistic features contain additional structural information of the text that are marginally but consistently useful to the neural models for all data sets. the performances of our proposed method are presented in the last two columns of comparing the last two columns of we may speculate that the differences in the improvement might be caused by two factors among many others: (1) how important the syntactic structure is for building the foundational knowledge in learning a certain language; and (2) how the semantic and syntactic knowledge of a certain language is organized throughout the learning process in order to lead the language learners through grasping the language. we construct the correlation graphs with positive correlation relationship only while we observe that there exist both positive and negative correlations among linguistic features. to investigate the effectiveness of learning embedding by considering negative correlation as well, we define an additional score function for negatively correlated features used in transe as: we present performance comparisons with gfe-transe+bert model in ## conclusions by combining the semantic dense embeddings and the syntactic dense embedding in a dualchannel neural network model, we propose a new methodology for readability assessment that capture both the semantic and the syntactic knowledge related to readability discrepancies. experiments with six data sets and two proficiency levels show that our dual-model is better than the semantic-alone and the syntactic-alone baselines. we prove that complementing semantic dense embeddings with syntactic dense embeddings learned with correlation graph of linguistic features can produce better-informed representations for readability assessment. we will further improve our research by studying other applicable algorithms and linguistic phenomena that could benefit from learning syntactic latent space and syntactic dense embedding representations. avg. num. characters per word (numchar) 18 avg. num. syllables per word (numsyll) syntactic features 19 mean length of a sentence 20 average number of words per punctuation-clause 21 number of punctuation-clauses per sentence 22 average number of subordinate clauses per punctuation clause 23 average number of subordinate clauses per sentence data set bert-finetuningonly bert-only (as in paper) chinese l1 0.353 0.3963 chinese l2 0.5353 0.6777 enew 0.8881 0.8425 weebit 0.8016 0.8348 onestopeng 0.8235 0.8157 cambridge 0.6687 0.696 3025 24 average number of co-ordinate phrases per punctuation clause 25 average number of co-ordinate phrases per sentence 26 average number of verb phrases per punctuation clause 27 average number of noun phrases per sentence 28 average number of verbal phrases per sentence 29 average number of prepositional phrases per sentence 30 average length of noun phrases 31 average length of verbal phrases 32 average length of prepositional phrases 33 average height of parse tree