Length Dependence of Vocabulary Richness (2023)

Abstract: The relation between the length of a text and the number of unique words is investigated using several Swedish language corpora. We consider a number of existing measures of vocabulary richness, show that they are not length-independent, and try to improve on some of them based on statistical evidence. We also look at the spectrum of values over text lengths, and find that genres have characteristic shapes.

## introduction measures of lexical richness have several uses, including author identification, other forms of text classification, and estimating how difficult a text is. one of the simplest and most obvious measures of lexical richness is to compare the size of the vocabulary (that is, how many different words) to the size of the text (how many words in total). this can be done in several ways, most straightforwardly as the type-token ratio (henceforth ttr), u/n, where u is the number of unique words (types) and n is the total number of words (tokens). thus, for the sentence "this example is this example", there are three types and five tokens, so ttr is u/@xmath0.6. the obvious problem with ttr is that it changes with the length of the text. as we write a text, the more words we have already written, the more likely it is that the next word will be one that has already been used, so ttr goes down as the text grows longer. many attempts have been made to transform this measure into something independent of the length of the text, but many of those attempts were made in an age before "big data", or even before computers, and were based on a priori reasoning rather than statistical analysis @xcite . we will start by looking at some of these measures, and test them on a set of corpora to see how they hold up for a wide range of different n. after comparing some of the previous methods, we will briefly look into using the empirical data to come up with a better suggestion. the results give rise to another question: what if instead of aiming for a length-independent measure, we consider how the values change with the length? can that actually tell us new and interesting things? we find that if we analyse the type count for different sample lengths, we see clear and consistent differences between different types of text. this may be useful for genre classification, or for a more detailed description of the text complexity. although these measures are usually applied to specific texts, we here apply them to entire corpora. we will discuss the effects of this after seeing the results. ## data språkbanken (the swedish language bank) at the university of gothenburg (spraakbanken.gu.se) has a large collection of text corpora, mainly in swedish but including several other languages. in this study, we use swedish texts, focusing on large and homogeneous corpora, listed in the appendix. we extract the type count u for several different lengths n. words are case-independent but otherwise counted as written, without lemmatisation. for each n, we divide the corpus in chunks of length n, dropping any overflow at the end, and take the mean value of u for each of these chunks. (in some cases we remove the last value for being an outlier; presumably this is because it is the only value where a large part of the data is dropped due to overflow.) we use a pseudologarithmic scale for ease of reading, extracting values for @xmath1, 20, 50, 100, 200, 500, 1000... up to the maximum possible for each corpus; the largest go up to 500 million tokens. ## testing existing measures first of all, we can test and verify that ttr does go down. figure 1 shows ttr for 31 corpora. it seems likely that, as we compare different-size corpora, effects of size changes might be best described in terms of multiplicative changes rather than additive, so we might try looking at the logarithms of n and u. we see in figure 2 that the result looks fairly close to a straight line. the first obvious method, then, is to assume that this is indeed a straight line, and use the slope of that line as our presumed length-independent measure of richness, that is, log u/log n. this was proposed by @xcite . we see in figure 3 that the measure is decreasing quite steadily for all the texts. the six corpora used here are chosen partly for being large, and partly for having large differences in type count; many other corpora are not nearly as well separated. let us pause for a moment and consider what this figure illustrates. the fact that the measure decreases is not in itself a problem; although we are aiming for a near-constant, we should not expect it to be perfect. the amount of variation is also not relevant; we could change that by adding or multiplying by a constant. regardless of how large the variation is, we would also change the axes of the graph, so a glance at the variation of a single curve in the graph does not tell us whether the measure is near-constant in a relevant sense. what actually matters is comparing the curves. if the measure is to reliably compare different texts, regardless of the (sample) size for each text, what we need is to have the lines separated insofar as possible. if the lowest point of curve a is higher than the highest point of curve b, then we have successfully determined that a has a higher richness. we should also keep in mind that the first few points of the curve are not as important -we are probably not very interested in measuring richness for very short texts, so although the graphs go all the way from 10, we can mostly ignore values below 1000 or so. we would be content if the measure can separate the lines from that point on. as we see in figure 3 , this is not quite the case here. this measure works better than ttr, but the curves are still close enough that their ranges overlap. we will compare with a few other measures. guiraud @xcite ) proposed the measure u/ √ n, shown in figure 4 . this does not separate the curves particularly well, and does not seem to have any advantage over the previous method. dugast (1979) built on herdan by suggesting log u/log log n, seen in figure 5. we find no advantage with this method, and only added conceptual complexity with the double logarithm. @xcite proposed n ∧ (u -a ), where usually @xmath2.172. this is shown in figure 6 . this too is a fairly conceptually complicated method which shows no sign of improving the results. @xcite found another approach, with (log n-log u)/(log n) 2 , see figure 7 . this seems marginally more effective at separating the curves. 10 100 1000 1ᴇ4 1ᴇ5 1ᴇ6 1ᴇ7 1ᴇ8 0.01 0.02 0.03 0.04 0.05 0.06 0.07 maas sample length kalmar1910 wikipedia twitter2015 rd-prop gp1994 lb figure 7: maas's measure hultman and westman (1977) defined the ovix measure as log n log 2 -log u log n as we see, this also does a passable job, but there is a clear rising trend for most curves. this is confirmed by further testing on other corpora. ## improving measures by analysing the way these measures depend on n, we may be able to adjust and improve them. as noted, the fact that the curve of log u against log n is close to a line suggests that (log u/(log n) may be a constant, as per herdan. but that assumes that the line passes through (0, 0); if the line passes though (0, m) for some m, we should expect that (log um)/log n is constant. we find that for a subset of the corpora, the best-fitting line gives @xmath3.4, and we see in figure 9 that (log u -0.4)/log n does look a lot flatter. as before, we pay less attention to the values where @xmath4. on the other hand, we know that a text with one word certainly also has one unique word, so logically the curve of log u against log n must pass though (0, 0). figure 11: adjusted guiraud second, third and fourth order polynomials suggests that third is a good compromise. we find the best fit for six corpora, take the average for the quadratic and cubic terms, and get the adjusted measure you can see in figure 10 that this separates the curves considerably better than the pure herdan measure. from looking at the graph, this is probably the best option we have here, but we should note that the coefficients vary quite a bit between corpora (standard deviations are 0.015 and 0.0017), so this is not universal enough to adopt as some sort of standard measure. we can also consider the guiraud approach, and try to adjust it. we notice that while ttr (u/n) goes steadily down, guiraud (u/n 0.5 ) goes up. perhaps we can find a middle ground? figure 11 shows the results for u/n 0.75 , which looks overall much flatter and better separating the curves. this may not be a better result than the previous one, but it does have the advantage of not depending on experimentally determined coefficients. ## fixed-length measures is there another option, using only the length and the type count? yes, there is an option which is in principle completely independent of text length: measure the type count (or equivalently ttr) for a fixed length. one option would be to measure only the first n words of a text, but that could mean that a small part of the text has a large impact, so probably a better method is to cut the text into pieces of length n and take the average, exactly as we have done above. that leaves the question of choosing an n. very low values might give strange effects, very high values would make it unusable for shorter texts. other values were tested for comparison: @xmath5, while @xmath6, and beyond that we get mostly unremarkable results from just looking at the ranking. based on these limited results, @xmath7, and for short texts we can settle for @xmath8. ## spectrum comparison instead of considering type counts for only one n, what if we measure for many values of n, and look at the whole spectrum? this is essentially what we already did in all of section 3, and we could see that the curves for the different corpora certainly did have different shapes -some of them even crossed each other, which implies that any one number is not going to tell us the whole truth. to compare corpora instead of methods, we need to pick one method, one way to transform u based on n. using plain ttr as seen in figure 1 would make it difficult to tell the difference between shapes, and picking one of the tested methods seems like too arbitrary a choice. so for the purposes of this section, we will evade the problem. we normalise the type count (or equivalently ttr) for each n by subtracting the mean and dividing by the standard deviation. that is, the values on the vertical axis are in terms of standard deviations above the mean, counted for each separate value on the horizontal axis. (for high values, the mean/sd change erratically because of corpora dropping off. we adjust the normalisation to gradually change from actual to extrapolated mean/sd.) figures 13-22 show the spectra for each category. some curves are shorter because of limited data. figures 13 14 15 show different types of web-based texts, one set of blog texts and two different internet forums. we can see that each category is a little different, but all the curves share some characteristics -a short rise, then a drop, then flatter, and finally a small rise. most of them start slightly above the mean, and end below the mean. 10 100 1000 1ᴇ4 1ᴇ5 1ᴇ6 1ᴇ7 1ᴇ8 -0.6 -0.5 -0.4 -0.3 -0.2 -0.1 0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 sd above mean sample length fb-samhalle fb-ovrigt fb-flashback fb-livsstil figure 14: spectrum for the familjeliv forum figure 16 shows the "easy read" category. despite being unrelated, the curves share the same shape, which is clearly different from the web-based corpora -a drop, then a rise, peaking around 1000 without reaching the mean, then a drop. figures 17-18 show news texts, with figure 17 showing three newspapers from the early 1900s, 10 100 1000 1ᴇ4 1ᴇ5 1ᴇ6 1ᴇ7 1ᴇ8 -2 -1.8 -1.6 -1.4 -1.2 -1 -0.8 -0.6 -0.4 -0.2 0 0.2 0.4 0.6 sd above mean sample length fl-pappagrupp fl-kropp fl-planerarbarn figure 15: spectrum for the flashback forum 10 100 1000 1ᴇ4 1ᴇ5 1ᴇ6 1ᴇ7 1ᴇ8 -1.2 -1 -0.8 -0.6 -0.4 -0.2 0 0.2 0.4 0.6 0.8 sd above mean sample length attasidor lasbart klarsprak figure 16: spectrum for easy-read texts and figure 18 showing four more recent newspapers and one web-based news corpus. as with the blog/forum collection, we see that these two related categories have clear similarities: a slow rise up to between 10 000 and 100 000, and then a drop. but they are also visibly distinct, with the older newspapers having higher values and rising near the end. aside from some unpredictable behaviour for @xmath9, the curves in each category are remarkably similar in both shape and level. figures 19-20 show literary texts, with figure 19 showing regular novels and figure 20 showing children's fiction and young adult fiction. they are all comparatively straight and dropping slightly. children's literature is generally lower than young adult literature, and they both drop faster than the curves for books aimed at adults. 10 100 1000 1ᴇ4 1ᴇ5 1ᴇ6 1ᴇ7 1ᴇ8 -0.2 0 0.2 0.4 0.6 0.8 1 1.2 1.4 sd above mean sample length dn1987 gp1994 osterbott...2014 jakobsst...2000 webbnyheter figure 18: spectrum for recent newspapers figure 21 shows religious texts. we see two translations of the bible, with very similar curvesboth dropping, rising, levelling out, but unlike the easy read category they level out at about the same level where they started. also included is a book of church hymns, which happens to level out at a similar level, but starts with a large rise. finally, in figure 22 , we see three uncategorised corpora -one from a 1700s songwriter, one from a popular science magazine, and one from wikipedia. as expected, they show very different shapes and levels, and are clearly distinct from each other as well as all the other curves. explanations of the shapes are tentative at this point, but we can guess at the meaning of high richness in different regions of sample length: for low values (roughly 100-1000), it may indicate complex sentences with few function words; for medium values (around 10 000), complexity in topics, with many names etc.; and for high values, variety in topics. this may explain why newspapers peak in the middle (they address complex topics with many names, but return to the same topics), social media drop in the middle (they address simpler topics but with a wider variety in topic and style), and youth novels go down (they are on simple topics and consistent for entire books). further speculation is left for future work. 10 100 1000 1ᴇ4 1ᴇ5 1ᴇ6 1ᴇ7 1ᴇ8 -1.2 -1 -0.8 -0.6 -0.4 -0.2 0 0.2 0.4 0.6 sd above mean sample length barnlitteratur ungdomslitteratur figure 20: spectrum for youth novels ## applicability is it reasonable to apply measures like these on an entire corpus instead of just separate texts? first, "separate texts" is not necessarily well defined. is 10 100 1000 1ᴇ4 1ᴇ5 1ᴇ6 1ᴇ7 1ᴇ8 -3.5 -3 -2.5 -2 -1.5 -1 -0.5 0 0.5 1 1.5 sd above mean sample length fof bellman wikipedia figure 22: spectrum for some other texts a newspaper one text, or each article? books in a series? multiple entries on a web page? second, for low values of n, running the entire corpus at once should make little difference. for example, if @xmath10, only about 1% of samples would contain two texts, and the rest only one. for high values of n, using only separate texts would leave us with no data at all -it would be difficult to find singular coherent texts spanning hundreds of millions of words. this means that allowing corpora of multiple authors and topics is our only option for large n. but we can also look at the results. are the differences between the curves largely caused by differences in text length? if that was the case, we would expect that when a curve reaches the "critical n" where we go from a single text to multi-ple texts, the vocabulary richness should increase rapidly. the curve we would expect to see is one that starts out mostly flat (because hardly any texts are that short), then slowly decreases (as others reach their critical n and bring up the mean), then rapidly jumps up as it reaches its critical n, and then slowly decreases again. this is not a pattern that we see anywhere, so we can conclude that text length is not the driving factor of the curve shapes. ## conclusion the task of finding a length-independent measure of vocabulary richness is difficult at best. we have seen that many traditional measures are not satisfactory, and made some suggestions as to how they can be improved. perhaps the most obvious approach is to use average ttr over a sample length, with 10 000 words being a good sample length. the figures show that the curves have very different shapes, and often cross. thus, the ranking of corpora changes depending on the length of the text sample, so a perfect solution is not possible, or at least cannot be expressed as a single number. is this spectrum method useful for genre classification? it is perhaps rare that we need to analyse entire hundred-million-word corpora to see if they are made up of novels or newspapers, but we do see some differences even for much smaller lengths. we have also gained insight into what makes it difficult to find a good measure of vocabulary richness. most importantly, we have seen that there are notable differences between genres, and raised for future research the question of why.