Applying and Optimising a Multi-Scale Probit Model for Cross-Source Text Complexity Classification and Ranking in Swedish (2025)

Abstract: We present results from using Probit models to classify and rank texts of varying complexity from multiple sources. We use multiple linguistic sources including Swedish easy-to-read books and investigate data augmentation and feature regularisation as optimisation methods for text complexity assessment. Multi-Scale and Single Scale Probit models are implemented using different ratios of training data, and then compared. Overall, the findings suggest that the Multi-Scale Probit model is an effective method for classifying text complexity and ranking new texts and could be used to improve the performance on small datasets as well as normalize datasets labelled using different scales.

## introduction measuring or estimating text complexity is essential in various fields, including readability research and the adaptation and recommendation of texts for different audiences. in this paper, text complexity refers only to the linguistic characteristics that affect how easy or difficult a text is to read, without considering the interaction between the text and any particular reader. any comprehensive evaluation of text complexity must include three key components. first, linguistic features must be quantified, such as calculating the average sentence length. second, relevant linguistic features need to be selected for evaluation. third, the impact of each linguistic feature on text complexity must be assessed, for example, determining whether longer sentences increase or decrease complexity and to what extent. the distinction between effective and ineffective evaluations lies in the execution of these components. the selection of features and the methods employed to measure them significantly affect the quality of the evaluation @xcite . moreover, text complexity is not defined by a single superficial quality; rather, it results from an interplay of various features, each influencing complexity in distinct ways @xcite . understanding how and to what extent each linguistic feature contributes to overall text complexity poses an additional challenge. the approaches for identifying and selecting linguistic features vary, ranging from employing theoretical linguistic frameworks and reasoning about feature impacts @xcite to training machine learning models on specific features and assessing their performance @xcite , or even employing a combination of these methods. another aspect of the assessment of text complexity is the type of output that is produced. depending on the purpose of the evaluation, the results may be in the shape of a single binary classification of "easy to read" or "not easy to read". this type of evaluation is traditionally realised through simple linear functions, or more recently using machine learning models like the support vector machine (svm) that splits texts into two classes @xcite . another common evaluation method is to use one or a few linguistic features in a simple equation (often referred to as readability formulas) and computing a score to measure the complexity (e.g. the flesch reading ease formula @xcite ). these methods are beneficial in several ways, but all share a common downside. when using a few simple features or classifying texts in a binary manner, much nuance of text complexity is lost, and comparisons between texts are less informative @xcite . to solve these problems, we propose creating a model that uses many complex linguistic features and classifies or ranks texts into non-binary levels. this approach would, however, usually require data that are already labelled according to class or rank. the more features used in the model to increase the complexity of the evaluation, the more data is required in each class or rank @xcite . one method that has the potential to resolve many of the issues mentioned above is the multi-scale probit model, proposed and first implemented in @xcite . the probit model is a well established statistical model, introduced in the 1930s @xcite and used primarily for classification. it is closely related to the younger but somewhat more well known logit model, or logistic regression as it is often called in psychometric contexts, but it has some properties which make it especially suitable for bayesian modelling @xcite . the multi-scale probit model is a generalisation of the bayesian ordered probit model and is capable of training on data labelled into ordered levels, such as how hard a text is to read, from multiple text sources. these sources may use completely different scales, meaning that the levels need not correspond in any sense between sources apart from indicating text complexity. there is no requirement for a minimum amount of texts per level, which enables the use of data that would have to be discarded in other approaches. the key idea behind the model is the presence of a latent variable that is shared among all labelling schemes. in this context, that latent variable is text complexity, with the assumption that the different labelling schemes used across different data sources all represent measures of that latent variable. information about the latent variable is captured in the features, and the model learns how the latent variable is affected by the features, making it able to classify and even rank the text complexity of new texts. we explore how the multi-scale probit model performs when trained and evaluated on novel data, consisting of easy-to-read literature for children, teenagers, and adults 1 . ## text complexity analysis text complexity generally refers to characteristics of a text that make it more or less cognitively engaging during reading @xcite . quan-1 https://www.nyponochviljaforlag.se/om-oss/ titative and qualitative assessments of text complexity are of great value, as they can be used in many fields such as education (e.g. determining the appropriate material @xcite or automatic essay grading @xcite ), customisable text simplification (e.g. determining which texts to simplify @xcite ), or customising texts based on cognitive requirements (e.g. for readers with dyslexia (santini and jönsson, 2020)). pinpointing the properties of a text that tells us about its complexity has been proven to be a difficult and confusing task. the factors that make up the complexity of a text can themselves create a hyperplane that spans across a highly multidimensional space. classification is a simplified version of this with the purpose of assigning texts into one or more classes such as "easy to read". classification approaches consist of machine learning algorithms, statistical methods, and other nlp techniques. such approaches need to be trained on different text features or combinations of features and then evaluated on their performance in classifying texts accurately. as model performance becomes an indirect measurement of the relevance of the feature(s) to text complexity analysis, the features used to train models with better performance are chosen over the features of models with poorer performance @xcite . another category of classification algorithms is logistic regression and its variants. compared to svms and similar methods, cf. @xcite ; @xcite ; @xcite , the binary outcome is modelled as a probability between 0 and 1. for instance, a book could be classified as "easy to read" with a probability of 0.6, meaning that there is a 60% probability (according to the model) that the book is "easy to read". a common approach for such probability estimation is logistic regression @xcite , or the logit model, and in this paper we apply a version of the closely related probit model. ## text complexity features the most commonly used method for the analysis of text complexity is automatic evaluation using quantifiable features of texts, which are then used to compute one or more ratings of text complexity. these features measure different aspects of the text and can be categorised into four ordered levels of increasing analytical depth, as outlined below. shallow features: the features in the first category do not contain information about the content of the text. they simply consist of letter and word counts; very little or no knowledge of their meaning is necessary to measure or understand shallow features. nevertheless, they have been proven to be useful for measuring text complexity and are very simple to extract. the text is processed through tokenisation to create tokens out of words (and other components, e.g. delimiters). the tokens can be counted either as they are or by tallying the characters they contain. several traditional metrics are based on one or more of these features or variants thereof, cf. @xcite ; björnsson (1968). ## text complexity as a latent variable a key idea behind the use of statistical models for text complexity assessment is the assumption of a latent variable. as established, text complexity cannot be measured directly. instead, it is estimated using one or more linguistic features. furthermore, text complexity is assessed and labelled in various ways. for example, texts may be labelled as "easy to read" (with the implication that regular texts are less "easy to read"), rated on a scale of 1 to 7, or categorised into age groups, among other methods. although texts from different sources may use varying labels and methods to measure readability, we assume that they share the underlying latent variable of text complexity. in other words, variations in text complexity may be expressed differently, but the concept is consistently modelled across all sources. if data can be processed appropriately, it enables the latent variable to be statistically modelled and subsequently used to classify or rank texts. ## data the majority of data used in this research consisted of books from a corpus called nypon-vilja, consisting of books from nypon och vilja, the largest swedish publisher of easy-to-read literature for children, teenagers, and adults. swedish easy-to-read literature is catered to people with reading difficulties, beginner readers, or nonnative readers learning swedish. books from nypon and vilja are (generally) aimed at two different target groups; nypon at 'children and young' and vilja at 'young adults and adults'. the publisher uses separate scales (with their own naming schemes), each consisting of 6 levels, to indicate how easy or difficult a book is, where the first level (1 and x-small) is the easiest and the last level (6 and xx-large) the most difficult. before processing, all books were manually annotated based on their alignment with one of two narrativity dimensions: informational and narrative @xcite . informational text tends to be non-fictional, written to inform about or explain a specific topic. narrative text on the other hand is typically fictional and story-driven. in order to minimise the effects of variations in language use that affect text complexity between dimensions of narrativity, only books classified as narrative are used. finally, as level 6 from nypon contained only 2 books, they were merged with the books in level 5. this resulted in a dataset of 356 books with 5 levels in nypon and 6 levels in vilja, summarised in the stockholm-umeå corpus @xcite texts in 10 categories including newspaper reportage, popular lore, and imaginative prose, written for different audiences and with varying writing styles. the annotations contain information about linguistic, structural, and functional information. in this research, we used a free-foruse bag-of-sentences version (sucx 3.0) publicly available from språkbanken foot_0 . thus, no text features dependent on sentence order are included in our analysis. furthermore, in order to minimise the effects of variations in language use that affect text complexity between genres @xcite @xcite , only texts from the category 'imaginative prose' were extracted, giving a total of 127 texts from suc. this category was assumed to contain texts in a style the most similar to those extracted from nypon och vilja, being non-informational. the purpose of using suc is to obtain a composition of data at a level of text complexity above all books from nypon och vilja. this is a key assumption and is based on the rationale that texts from suc are written for typical adult readers and not with the express purpose of being especially easy to read, meaning text complexity can be assumed to be higher compared to the books from nypon och vilja. to extract all necessary linguistic features, all texts were processed using the stillett api service (sapis) @xcite . the api service allows for the tokenization, lemmatisation, part-of-speech tagging, and dependency parsing of any text input. it also allows for text complexity analysis through the scream module @xcite which computes related metrics. ## model implementation and evaluation the multi-scale model was implemented using a modified version of the framework developed by @xcite and executed using r (version 3.6.3) with rstudio (rstudio @xcite . the model uses a set of covariates as input. these covariates are the values of metrics extracted through the data processing step resulting in a total of 47 features, c.f. @xcite data containing values for all covariates in the feature set were split into 5 classes for the nypon scale and 6 classes for the vilja scale ordered according to their levels. the data were then first split into training and test sets 500 times, using different ratios for training and test data, creating 500 models. the training data were used to estimate the full joint posterior distribution described in section 4.2 through sampling according to the scheme described in @xcite . this step was completed to evaluate the performance of the models. then, instead of splitting the data into training and test sets, all data were used to run 20 chains of the gibbs sampler resulting in a combined set of samples of a full posterior distribution of the entire dataset. the number of chains was based on the number of cpu cores available, using one core per chain to speed up the sampling process. furthermore, the multi-scale model by definition uses multiple scales, meaning the posterior distribution is sampled using data from both nypon and vilja. however, since the multi-scale probit is a generalised version of an ordered probit model, which uses only one scale, its performance on either scale can be compared with a traditional ordered probit sampled using data from only that dataset. such models used the same implementation as the multi-scale model, but using one scale at a time. these models will be referred to as single scale models. after completing the sampling processes, parts of the resulting posterior distributions were visualised. the posterior was also used for classification and ranking, where the performance was evaluated by computing values for several evaluation metrics. for all evaluation metrics above, a higher positive value indicates better performance. for visualisation purposes and to enable easier comparison between model distributions, mode values are also plotted. the mode corresponds to the point with the highest probability density of a dis-tribution, i.e. equivalent to the value that appears most frequently in a discrete probability distribution. ## model performance the data was randomly separated into 500 different permutations of training and test data, and models were trained and evaluated for each such permutation. to assess whether one model generally outperforms the other we compute the difference between the posterior mean performance (f 1 -scores and kendall's τ b correlations, respectively, as well as the harmonic mean of them) between the models for each permutation. finally, to further examine how well the models perform given varying amounts of training data, two different split ratios were used: 2/3 training and 1/3 test data, and vice versa. with 2/3 of the data used for training, the mean posterior modes of the performance metrics f 1 and kendall's τ b can be seen in we can see that the choice of model makes little difference to the performance on the nypon dataset but has a noticeable impact for the vilja dataset. direct comparison of the models is done by computing the difference of the posterior mean f 1 and kendall's τ b for each model over the 500 data permutations. this shows that the multi-scale model outperforms the single scale model with respect to the f 1 -score 54.4% of the time on the nypon dataset and 73.4% of the time on the vilja dataset. the same comparison of kendall's τ b show that the multi-scale model is better 62% of the time on the nypon dataset and 89.2% of the time on the vilja dataset. figure 1 plots the distribution of differences in the harmonic mean of f 1score and kendall's τ b between the models over all 500 data permutations, showing that the multi-scale model is better in 58.4% and 85.4% of cases for nypon and vilja respectively when both performance metrics are considered. when the ratio of training to test data is reversed (i.e. 1/3 of the data used for training, the rest for testing), we see similar differences in overall performance on the vilja dataset but now, the the difference in overall performance on the nypon also shows a marked difference. figure 2 illustrates this for the harmonic mean of f 1 -score and kendall's τ b . however, as expected, the performance of both models is slightly lower with mean posterior modes, as seen in this implies that the multi-scale model is especially useful when the availability of training data is limited. meanwhile, figure 2 shows that the multi-scale model is better in 78.6% and 84.2% of cases for nypon and vilja respectively when both performance metrics are considered. to summarise, the results show that the multi-scale model generally outperforms the single scale model on both datasets, particularly on the vilja texts. furthermore, this performance difference was greater when using a data split of 1/3 training data and 2/3 test data compared to a 2/3 training and 1/3 test data split. this implies that the relative improvement of the multi-scale probit over the single scale ordered probit decreases with the size of the dataset available for training a single scale model. in other words, the multi-scale probit model is especially useful in contexts with sparse and diverse data for training. ## conclusion the purpose of this research was to utilise the multi-scale probit model in order to enable a standardised ranking and classification of text complexity, while also exploring how the model can be optimised. the assessment of text complexity can be used for a wide range of purposes, making its development pivotal in the field of natural language processing. the results from applying the multi-scale probit on easy-to-read swedish books have indicated that the model outperforms the single scale model in nearly all cases of classification and ranking, measured by f 1 -scores and kendall τ b correlations. furthermore, the results accentuate how the output from the multi-scale probit model can be used in a simple manner to classify and rank new texts in the same domain, or adapted to other domains by creating new models. through data augmentation and feature regularisation, the model can be optimised in terms of computational complexity and performance in specific contexts. the ability of the multi-scale probit model to utilise data from different sources, without the necessity of large data quantities per category, enables assessments of text complexity that have previously not been possible. this research has contributed to the goal of developing methods for classifying and ranking text complexity, with the broader aim of creating a more accessible society for readers with varying needs.