Predicting the Growth of Morphological Families from Social and Linguistic Factors (2020)

Abstract: We present the first study that examines the evolution of morphological families, i.e., sets of morphologically related words such as “trump”, “antitrumpism”, and “detrumpify”, in social media. We introduce the novel task of Morphological Family Expansion Prediction (MFEP) as predicting the increase in the size of a morphological family. We create a ten-year Reddit corpus as a benchmark for MFEP and evaluate a number of baselines on this benchmark. Our experiments demonstrate very good performance on MFEP.

## introduction lexical change is a prime indicator of topical dynamics in social media. when people or events attract the attention of a user community, this is reflected by the token frequency evolution of individual words. the burst in token frequency of the word "trump" in social media before the 2016 presidential election (see figure 1 ), e.g., mirrors the increasing presence of donald trump in public discourse during that time. however, token frequency is only one way of measuring changes in topical prominence. accompanying the increase in token frequency of "trump", there was a parallel increase in the number of words morphologically related to "trump", i.e., words like "trumpification", "antitrumpism", and "detrumpify" (see figure 1 , here, we present the first study of lexical change in social media that takes as its main unit of analysis the type frequency evolution of morphological families, i.e., changes in the number of morphologically related words such as "trump", "trumpification", "antitrumpism", and "detrumpify". we show that morphological families allow for a fresh view month words 04/2015 "trump" 05/2015 "trump", "trumpish", "trumpster", "trumpy" 06/2015 "trump", "trumpening", "trumper", "trumpish", "trumpness", "trumpology", "trumpster", "trumpy" 07/2015 "trump", "trumpening", "trumper", "trumpic", "trumpification", "trumpiness", "trumpish", "trumpism", "trumpistan", "trumpness", "trumpster", "trumpy" contributions. we introduce the novel task of morphological family expansion prediction (mfep), which aims at predicting whether a morphological family will increase in size or not. we publish a benchmark for mfep and show that the growth of morphological families can be successfully modeled using social and linguistic factors relating to the morphological parent. furthermore, our results add a new perspective to the growing body of research on the link between cultural and linguistic change in social media. 1 ## morphological families we define a morphological family as a set f of words w with a shared free morpheme. thus, "trump", "trumpification", "antitrumpism", and "detrumpify" are in the same morphological family because they share the free morpheme "trump". by contrast, "antitrumpism" and "antiprogressivism" are in different morphological families: even though both words have two morphemes in common ("anti" and "ism"), they do not belong to the same morphological family according to our definition since "anti" and "ism" are bound morphemes, not free morphemes like in the case of "trump", "trumpification", "antitrumpism", and "detrumpify". in this study, we only consider derivational morphology. foot_1 compounds such as "trumpwall" are not split into their parts, but they can become a parent (see below). thus, each word belongs to exactly one morphological family. the cardinality |f | of a family will be referred to as the morphological family size, a term also used in other studies @xcite @xcite . the morphological parent w * is the morphologically most basic word of a family f . the word "trump" is the parent of "antitrumpism", "trumpification", and "detrumpify". we denote the morphological family of w * as f (w * ). except for the parent, all members of a family are morphological children and form a subset f of the entire family f . the words "antitrumpism", "trumpification", and "detrumpify" are all morphological children. we further distinguish between old children fo , established words in the lexicon, and new children fn , innovative forms. while "trumpify" can still be considered a new child of "trump", "trumpster" is on its way to becoming an old child in the family. 3 as the example of "trumpster" shows, morpho-logical families are in constant flux. specifically, there are three types of change in morphological families: word birth ∅ fn , word entrenchment fn fo , and word death fn , fo ∅. the topic of this paper is word birth, i.e., we ask: given a set f of morphological families, which will increase in size during a specified time interval? this differentiates our work from previous research on lexical change in social media which has focused on word entrenchment (see section 7). one question we are particularly interested in is whether endogenous (language-internal) or exogenous (languageexternal) factors are better predictors of morphological family growth; these factors have been previously compared for changes in word token frequency @xcite , but not for changes in morphological type frequency. ## experimental data we develop mfep using reddit, a social media platform hosting discussions about a variety of topics. reddit is divided into smaller communities centered around a shared interest, so-called subreddits (srs), which are highly conducive to linguistic innovation @xcite . concretely, we draw upon the baumgartner reddit corpus, a collection of (almost) all publicly available comments posted on reddit since 2005. 4 a three-year slice of this corpus was used in a study on lexical change by @xcite . @xcite show that the corpus's coverage of reddit is not complete, but we do not expect this to affect our analysis. our study examines data from 2007 to 2018 in the four srs r/gaming, r/movies, r/nba, and r/politics. these srs were chosen because they are of comparable size, belong to the largest srs of reddit, and at the same time all reflect distinct areas of interest ( preprocessing. as in previous work @xcite , we filter posts for known bots and spammers. we remove abbreviations ("viz."), strings containing numbers ("b4"), references to users ("u/user") and srs ("r/subreddit"), and both computing morphological families. given a collection of texts s, we define the morphological families as follows. let v s be the vocabulary of s, i.e., all words occurring in it. we define the set of parents o s ⊂ v s as the 1,000 most frequent words in s, regardless of whether the word is decomposable or not. this means that parents are not necessarily morphological roots @xcite . 5 we attempt to segment all other words w using affixes from a representative list of productive prefixes and suffixes in english @xcite . we define the set c of candidate parents of w as follows. if @xmath0, then c(w) = {w}. otherwise, c(w) = @xmath1, where b(w) is the set of bases that remain when one of w's affixes is removed. for w * ∈ o s , we then define its morphological family as procedurally, families can be identified by a recursive bottom-up algorithm. the algorithm is sensitive to morpho-orthographic rules of english @xcite ; e.g., when "ness" is removed from "trumpiness", the result is "trumpy", not "trumpi". 5 in a situation where both "sense" and "sensation", e.g., fall above the frequency threshold, we get two separate morphological families of the parent "sense" (a root) with "nonsense", "sensitive", etc. and the parent "sensation" (not a root) with "sensational", "sensationalism", etc. (the children of "sensation" are not added to the family of "sense"). however, most morphological parents are in fact roots. ## predictors for mfep in a setup similar to @xcite , we formalize mfep as a binary classification task. given a context interval i (c) , a following temporally adjacent probe interval i (p) , and a morphological parent w * , we ask: what properties of here, we set the length of i (c) to 18 months and the length of i (p) to 6 months. morphological families are computed separately for each pair of i (c) and i (p) . the lowest frequency count of a parent in i (c) is 244. we define a number of predictors for family expansion that are measurements of properties of w * . all predictors are motivated by work in psycholinguistics and nlp. they fall into three natural classes: (i) a type frequency-based predictor (|f |), (ii) token frequency-based predictors (f r , z(p) ), and (iii) dissemination-based predictors (d u w , d t w , q w ). all predictors except for z(p) (which is measured in i (c) and i (p) ) are measured in i (c) . ## experimental setup finding growing families. we use two different notions of growth for mfep: absolute growth and relative growth. we define absolute growth as for both δ a and δ r , we define binary features based on thresholds l a and l r , i.e., we define a morphological family f to be a positive example if δ a (f ) > l a for a pair of i (p) and i (c) (in the case of absolute growth). we thus train two models: one for predicting whether δ a (f ) > l a , one for predicting whether δ r (f ) > l r . model. we use random forests (rf) to perform the classification @xcite . rf offers two main advantages in comparison with other models. firstly, as opposed to other tree-based models, rfs decorrelate trees, which is important if the features are correlated (as is the case here). secondly, the feature importance scores of a rf provide a transparent way to compare the predictive power of features. we do not use more complex albeit potentially better performing methods such as deep architectures since our primary goal is to compare various features and show that mfep is a feasible computational task. since the data contain considerably more negative than positive examples, we randomly sample one negative example for every positive example for the final data. the interval pairs from all srs were merged into one dataset, which was then split into 0.8 and 0.2 for train/dev and test sets. the train/dev set was split again into 0.8 and 0.2 for train and dev sets. thus, all sets contain a balanced sample of interval pairs from all srs. 6 we use a total of 68,000 pairs of intervals (i (c) , i (p) ) where i (c) is the context interval and i (p) the probe interval (see also we do not perform hyperparameter tuning and instead choose typical values for the hyperparameters of rf: 80 for the number of trees, and 20 for tree depth. for our initial mfep models, we set the thresholds l @xmath2.4 and l @xmath3.6, two values in the mid-range of existing values for δ a and δ r . we will later analyze the sensitivity to these hyperparameters in greater detail. ## results overall performance. as shown in the strongest predictor for both models is type frequency with a feature importance of 39.3% and 25.3%, respectively ( we now analyze how different design choices impact the performance of our model. thresholds l a and l r . we systematically vary l a and l r in the range 0.0 ≤ l @xmath6.8 and 1.0 ≤ l @xmath7.2 to examine their influence on per-formance. 7 we find that accuracies for predicting larger δ a and δ r are considerably higher than for smaller increases ( we then train single-feature models for varying l a and l r . the best predictor for all values of l a and l r is |f | ( to further analyze the relative importance of inidividual features, we examine the rf feature importance loadings for varying l a and l r ( these results indicate that while the family size is most predictive of morphological family growth in general, high growth rates are particularly likely for families of trending parents (most of which have initially small family sizes). an example for the second case is the burst in the "trump" family before the 2016 presidential election illustrated in section 1 (figure 1 , length of i (c) and i (p) . in previous experiments, the length of i (c) and i (p) was set to 18 and 6 months, respectively. we now analyze how the choice of the interval length, specifically of the length of i the performances of the two mfep models is comparable with |i (c) | = 18 ( ## related work morphological families and productivity. the concept of morphological families was introduced in psycholinguistic work on lexical processing @xcite @xcite . these studies show that response latencies in lexical decision are not only influenced by token frequency but also by type frequency, i.e., the size of their morphological family. morphological families have also been used for analyzing lexical change on historical time scales @xcite . however, this work is not comparable to our study since it relies on dictionaries, which typically exclude the transparently formed, non-entrenched words in fn we are interested in @xcite . the main question of our study (how can we predict the growth of morphological families?) is related to a long-standing problem in traditional linguistic scholarship, i.e., what factors influence morphological productivity. the productivity of a morpheme is defined as its propensity to be used in novel combinations and traditionally understood to refer to bound morphemes @xcite . @xcite highlight the fact that morphological productivity, just as morphology @xcite and other components of language @xcite in general, is heavily influenced by social variation. social groups differ in the morphological patterns they use and in the extent to which they extend these patterns to new words. this makes morphological productivity an exciting new area for future research in computational social science, and it further underscores the relevance of mfep for that field. derivational morphology in nlp. derivational morphology has received increasing attention in nlp recently. key challenges include segmenting @xcite @xcite , modeling the meaning @xcite @xcite , and predicting the form @xcite @xcite as well as morphological well-formedness @xcite of derivatives. whereas all these studies approach derivational morphology from a synchronic standpoint, mfep is to the best of our knowledge the first computational task that addresses diachronic aspects of derivation. lexical change in social media. language change @xcite is most visible on the lexical level. new words like "detrumpify" attract attention, often becoming the subject of public discourse @xcite . since innovations are taking place at a much faster rate on internet media @xcite , social media have become a central resource for studies on lexical change over the last decade @xcite @xcite @xcite @xcite . one central question in this field is: what factors determine whether a word will survive in the lexicon of an online community? usage frequency is a well-known factor that influences the evolution of a word at historical time scales @xcite . studies on lexical change in online groups have shown that this is also true for shorter time scales @xcite . another main factor is the dissemination of a word, i.e., how widely a word is spread across different social and linguistic contexts. generally, the more disseminated a word is, the more likely it is to grow. this holds for social dissemination across users and threads @xcite as well as linguistic dissemination across different lexical collocations @xcite . the studies mentioned so far focus on token frequency. an exciting new approach looks instead at the meaning of words using diachronic word embeddings @xcite , e.g., explore short-term meaning shifts on reddit and identify considerable changes even within a period of eight years. a main goal of this study is to add a third approach to studies on lexical change in social media besides word frequency and word embeddings: word families. from a linguistic point of view, these three approaches can be viewed to be complementary: whereas word frequency is contextindependent, both word embeddings and word families reflect context-sensitive measures. however, while word embeddings reflect proximity on the utterance level (which words are close to each other in spoken sentences?), word families reflect proximity on the system level (which words are close to each other in the mental lexicon?). ## conclusion in this paper, we have proposed mfep (morphological family expansion prediction), a new task that aims at predicting how morphological families evolve over time. we have shown that changes in morphological family size provide a fresh look at topical dynamics in social media, thus complementing token frequency as a metric. furthermore, we have presented a random forest model for mfep that achieves very good accura-cies, particularly in predicting extreme growth in morphological family size. the strongest predictor of growth is the morphological family size itself, an endogenous factor. however, the initial growth of small families is mainly driven by the trending behavior of the parent, an exogenous factor. this reflection of external events makes morphological families a promising tools for various fields drawing upon nlp techniques for tracing temporal dynamics in text (e.g., virality detection). overall, we see our study as an exciting step in the direction of bringing together computational social science and derivational morphology. in future work, we intend to further fine-tune our methodological apparatus for tackling mfep.