TUM sebis at G erm E val 2022: A Hybrid Model Leveraging G aussian Processes and Fine-Tuned XLM - R o BERT a for G erman Text Complexity Analysis (2022)

Abstract: The task of quantifying the complexity of written language presents an interesting endeavor, particularly in the opportunity that it presents for aiding language learners. In this pursuit, the question of what exactly about natural language contributes to its complexity (or lack thereof) is an interesting point of investigation. We propose a hybrid approach, utilizing shallow models to capture linguistic features, while leveraging a fine-tuned embedding model to encode the semantics of input text. By harmonizing these two methods, we achieve competitive scores in the given metric, and we demonstrate improvements over either singular method. In addition, we uncover the effectiveness of Gaussian processes in the training of shallow models for text complexity analysis.

## introduction in this paper, we present a novel approach for the quantification of text complexity in the german language, as part of the text complexity de challenge 2022 @xcite . specifically, we emphasize a hybrid method for building a text complexity model, which combines a feature-based, shallow regression model with a fine-tuned xlm-roberta model. in doing so, we hope to capture to the fullest both the linguistic aspects that contribute to text complexity, as well as the semantic factors. in the following section 2, we briefly describe the task at hand, as well as the data used to train and test our models. next, section 3 introduces gaussian processes, which become central to our hybrid system. likewise, fine-tuning roberta for use in regression tasks is covered in section 4. these concepts are brought together in 6, which describes our overall model architecture for the task. before this, the feature set used to train both models is illustrated in section 5. in section 7, we present results from the training and validation * these authors contributed equally. ## dataset and task the dataset used in this task was first presented by @xcite . it consists of 1000 german language sentences sourced from 23 wikipedia articles. these articles have been classified into three different genres. these sentences are annotated with ratings of 1-7 in the category of complexity, understandability, and lexical complexity. these ratings are presented as a mean opinion score (mos), which is represented as the average scoring of the annotators. the sentences were scored by german language learners from a2 to b2 levels. for example, the sentence "eine seifenblase entsteht, wenn sich ein dünner wasserfilm mit seifenmolekülen vermischt." is scored with 2.9. for this challenge, one unified mos score was given. how this score was derived from the original three scores is not explained. the original 1000 sentences are the same. with this dataset, the task becomes to train a regression model that predicts the complexity of a sentence (i.e. mos) from the sentence text. this score is intended to aid in the quantification of text complexity for language learners, as well as in the evaluation of the simplified text. ## gaussian processes a popular area in machine learning, particularly near the turn of the century, rested in the study of "kernel machines", which include the popular support vector machines, but also the lesser known gaussian process models @xcite . at their core, gaussian processes (gps) are powerful in the way they incorporate probabilistic thinking into kernel machines, making them a particularly suitable tool for supervised machine learning in small data settings @xcite . literally, gaussian processes are built upon multivariate gaussian (normal) distributions, defined by a mean vector µ and covariance matrix σ, i.e.: this particular distribution has the useful property of being closed under marginalization (probability distribution of partitions) and conditioning (probability of one variable depending on another). the main goal of gaussian processes is to learn the underlying distribution of training data. key to this process is the utilization of bayesian inference, in which one assumes a prior and updates this hypothesis based upon new data. when modeling using gaussian processes, a prior with dimensionality equal to that of the unseen points is chosen. a kernel is used to generate the covariance matrix σ by evaluating it on all training points. in order to form the posterior distribution (i.e. train the model), the model observes training data, and conditions the current distribution based upon these new points. as new data comes in, the set of functions that the model can take is constrained, as only those functions exactly containing the new points are valid. the change in distribution induced by observing new points is reflected in an adjustment of the mean and standard deviation (achieved through marginalization). in addition, uncertainty in the data is modeled by adding an error term to the training points, modeled by ϵ ∼ n (µ, ψ 2 ). predictions from a trained gaussian process model are made simply by sampling from the distribution of the model. in this way, gaussian processes interestingly combine the ability to model (understand) the underlying distribution of the data at hand, as well as make accurate predictions from unseen instances. they, therefore, present a promising, powerful, and efficient method for tackling regression tasks @xcite . ## xlm-roberta for regression large pre-trained language models (plms) based on the transformer architecture @xcite have achieved state-of-the-art performance on a wide array of common nlp tasks. @xcite introduced bert as a powerful language representation model that learns deep contextual representations of words. roberta @xcite is a robustly optimized extension of the bert model. both of these models work primarily on english text, so we decided to use the multilingual model xlm-roberta (xlm-r), a variation of roberta trained on data written in one hundred languages @xcite . this model can recognize the language of an unseen textual input and achieves remarkable performance on a variety of non-english tasks, beating even the monolingual models optimized for specific languages. the key benefit of using plms is the ability to load the already pre-learned contextual word embeddings and then to fine-tune them for the specific downstream task at hand. we tried out different pre-trained models for xlm-r and the bestperforming ones for this task were the original xlmroberta-base, twitter-xlm-roberta-base, and xlmroberta-base-wikiann-ner. while plms like xlm-r are more commonly used for classification tasks, they can also be adapted for regression tasks. we achieved this by adding a new linear layer on top of the xlm-r. this linear layer had as its input the outputs of the final (12th) layer of xlm-r and learned what weights to assign to them. since our dataset contains only around 1000 examples, the process of fine-tuning had to be carried out carefully in order to prevent overfitting. hyperparameters used were: number of folds 5, number of epochs 3, batch size 16, max. length of 100, no weight decay. the starting learning rate was 10 -5 , after one third of all layers 5 • 10 -5 , and after two thirds it was 10 -4 . the idea behind this was that lower encoder layers can be understood as learning the lexical and syntactical features of the text, whereas higher layers model the semantic representation of it. for the task of text complexity, low-level features are more important so more emphasis was placed on them. ## feature selection the complete set of crafted features for model training is listed in feature description character-based * avg_chars average number of characters per token in sentence tokens_n number of tokens in sentence with @xmath0, i.e. number of tokens longest_word length of the longest word in a sentence commas number of commas in the sentence parentheses number of (open) parentheses characters digits number of numerical digits in the sentence quotes number of quotation characters (ór ) in the sentence avg_word_length average length of words in the sentence wordrank_score overall score calculated from german wiki frequency list 3 pos tags * pos_ratio ratio of (spacy.pos_) pos tags in sentence 4 tag_ratio ratio of (spacy.tag_) detailed pos tags in sentence 5 spacy features * dep_length cumulative length (width) of dependencies in sentence ne_length total length of all named entities in sentence ne number of named entities in sentence l2_norm l2 norm of spacy word vector representations vec_exists number of sentence tokens for which a spacy vector exists syntax tree syn_height height of syntax tree leaves number of leaves in syntax tree subtrees number of subtrees in syntax tree leaf_distance cumulative distance between the leave nodes in the sentence syllables tot_syl total number of syllables in sentence avg_syl average number of syllables per word single_syl number of single syllable words in sentence readability 6 flesch flesch reading ease score flesch_mod modified flesch score easy_words number of words in sentence @xmath1, † synset_exists number of lemmas in sentence for which a synset exists synset_depth cumulative maximum depth of all existing synsets in sentence hyponyms cumulative number of hyponyms for existing synsets senses cumulative number of word senses for existing synsets syn_def total length of synset definitions for all existing synsets avg_path average path length from one synset to the next, in sequential order experimental † scrabble_new scrabble score using the new german scrabble point values scrabble_old scrabble score using the old german scrabble point values 1 for @xmath2, 6, 7, 8, 10} 2 https://www.thegermanprofessor.com/top-500-german-words/ 3 https://github.com/gambolputty/dewiki-wordrank 4 for @xmath3, 'adp', 'adv', 'aux', 'noun', 'num', 'pron','propn', 'verb', 'x'} 5 for @xmath4, 'adjd', 'adv', 'appr', 'art', 'kon', 'kous', 'nn', 'prels', 'vafin', 'vvfin', 'vvpp'} 6 where applicable, scores are modified for single sentences (denoted by mod) 7 using the open german wordnet: https://github.com/hdasprachtechnologie/odenet * features in these categories are calculated on the logarithmic scale, with either add-1 or add-0.1 smoothing, where necessary † these features were not used in the final model (best test score) ## a hybrid system the development of the eventual final model took place in an iterative fashion. first, an array of popular shallow models were tested. in this process, the discovery of the effectiveness of gaussian processes for this specific task led the authors to choose these particular models for tuning. the kernel used was the sum of constant, matern, and white kernels, optimized with 10 restarts. as the training of gaussian process models seemed to hit a plateau, a deeper approach was pursued, namely using roberta. this achieved good results (see section 7), leading the authors to believe that some deep component was key to the task at hand. due to the documented success of stacking and ensemble methods @xcite , the authors considered a third approach in which the best shallow and deep models (gps and roberta) were to be stacked. concretely, the predictions of the two models could be harmonized in a way that combines the strengths of both. traditionally, stacking is performed by training a "meta-model", which learns the optimal way to combine the outputs of the "level 0" models. with this in mind, the authors took a simplified approach to stacking, in which the output predictions of the gaussian process model and the finetuned xlm-roberta were simply averaged. this resulted in the "meta" predictions, which were then used for submissions. in the development phase, this method proved to be the most effective, far outperforming both individual models. as such, a hybrid system was created, which was later utilized in the test phase. results from the development phase are outlined in section 7, where the performance of the individual and hybrid models are displayed. ## training and results in the following in ## discussion here, the authors reflect on lessons learned, useful findings, and possible future directions. a useful and somewhat surprising finding came with the excellent performance of gaussian processes, particularly during the development phase. in pondering why this occurred, one can look to the nature of gps in conjunction with the specifics of the text complexity task. at their core, gaussian process models aim to capture the underlying distribution of data that is complex. furthermore, gps seem to shine when the amount of training data is relatively small, e.g. under 1000 instances. as such, gps may have been a logical choice for this task, which comprised of a quite small dataset and whose goal represents a quite complex regression task. indeed, the quantification of text complexity proves to be challenging to reason about. nevertheless, the effectiveness shown by gps merits their consideration in future, related tasks. regarding xlm-roberta, it cannot be denied that its inclusion greatly strengthened the final model. it is interesting, though, that the model performance quite significantly varied based upon the particular pre-trained model that was chosen. in this light, a potential future direction would involve further investigation into better models, as well as why these might be superior. similarly, a more focused tuning of the hyperparameters for the fine-tuning process could have been performed. this likewise remains as future work. in the creation of the hybrid model which was eventually used in the test phase, an interesting lesson was learned regarding how to "stack" our two best models. a more systematic way of doing so would be to learn a meta-model, i.e. a simple linear regression model, to stack on top of our gp + roberta hybrid. as it turns out, learning such a model actually performed worse than a simple average of the two models' predictions. in this way, simplicity won the day in regards to the design of a well-performing hybrid regression model. the analysis of our feature set also produces interesting insights. as a cursory analysis, heatmaps illustrating the correlation amongst features and to the target values are presented in figures 1 and 2 in the appendix. the question then becomes whether more features should have been included, particularly those with a firm linguistic foundation. this notion is grounded in the authors' initial intuition that was used to produce the feature set. interesting findings were gained here, too, such as the relatively high correlation of complex linguistic concepts (e.g. passive voice, genitive case, depth of syntax tree) to the mos. we pose that such linguistic thinking is important for further improvements. as already alluded to, the complexity of the data itself, or rather the ability to describe it effectively via features, proved to be a central problem in tackling the task at hand. in the process of studying the datasets, the authors noticed particular characteristics that could be crucial to feature creation. of these, notable observations include the presence of rare-occurring symbols (e.g. §), as well as sentences that are clearly "simplified" sentences, rather than sourced directly from wikipedia. possible discrepancies between the dev and test sets were also observed. accounting for such factors in the features is likely key to model performance. ## conclusion in this paper, we discuss our novel approach to the quantification of text complexity in the german language. in particular, we present a hybrid model using gaussian processes and a fine-tuned xlm-roberta. we also provide our full feature set, which to be best of the authors' knowledge includes features not previously presented in the literature. finally, we discuss our results in the challenges and reflect upon their implications for future work.