A Three-Parameter Rank-Frequency Relation in Natural Languages (2020)

Abstract: We present that, the rank-frequency relation in textual data follows f \propto r^{-\alpha}(r+\gamma)^{-\beta} , where f is the token frequency and r is the rank by frequency, with ( \alpha , \beta , \gamma ) as parameters. The formulation is derived based on the empirical observation that d^2 (x+y)/dx^2 is a typical impulse function, where (x,y)=(\log r, \log f) . The formulation is the power law when \beta=0 and the Zipf–Mandelbrot law when \alpha=0 . We illustrate that \alpha is related to the analytic features of syntax and \beta+\gamma to those of morphology in natural languages from an investigation of multilingual corpora.

## introduction zipf's law @xcite is an empirical law to formulate the rank-frequency (r-f) relation in physical and social phenomena. linguistically, zipf's law can be observed on the distribution of words in corpora of natural languages, where the frequency (f ) of words is inversely proportional to its rank (r) by frequency; that is, f ∝ r - foot_0 . zipf's law is a special form of a general power law, that is, f ∝ r -α , with α = 1. the zipf's/power law is usually examined under a log-log plot of rank and frequency, where the data points lie on a straight line. the simple proportionality of the zipf's/power law can be observed on randomly generated textual data @xcite and it only roughly depicts the r-f relation in real textual data. a two-parameter generalization of the zipf's/power law is the zipf-mandelbrot law, where f ∝ (r + β) -α @xcite . @xcite considered the reversed rank of r max +1-r, where r max is the maximum of ranking index, and proposed a two-parameter formulation of f ∝ r -α (r max + 1 -r) β . as a straightforward observation, the coefficients of proportionality should be distinguished for common and rear words (powers, 1998; li ## proposed and related formulation under a logarithmic form, the zipf's law states that x + @xmath0, where (x, y) = (log r, log f ), and c is roughly a constant. we further investigate the property of @xmath1. the first and second-order differences on g(x) are calculated as here (x i , y i ) is the data point of the i-th frequent token, g @xmath2, and g 1 = g 1 = 0. 3 because the differences are intrinsically nonsmooth, bézier curves are applied for smoothing in the investigation. figure 2 shows examples of the smoothed g on english words and chinese characters from the same dataset used for fig. 1 . an artificial zipfian dataset generated in the manner of @xcite 4 is also used for comparison. it can be observed that the g on english words and chinese characters has an impulse, but not that on the artificial data. generally, the impulse becomes more obvious if the data are more non-zipfian. if we consider g as a general impulse function, then g is a general sigmoid function and g can be modeled by a general softplus function in the form of b log(exp(x -c) + 1). to replace x by a generalized linear form as ax + d, the obtained proportional form is a natural twocomponent extension of the power law and the 3 to avoid too many meaningless zeros in the differences, only the data point with the minimum x is used for data points with the same y, i.e., tokens with the same frequency. zipf-mandelbrot law. because the softplus function is a differentiable form of a rigid ramp function, eq. ( 3 ) can also be considered as a smoothed piecewise broken power law. as shown in fig. 1 , α and (α + β) depict the proportional coefficients at the two ends, and the proportional coefficients are switched smoothly around 2010 ) is also a two-component formulation. one more parameter (i.e., γ) in eq. ( 3 ) is used to identify the location of the impulse observed in g . under li's formulation, we obtain @xmath3, where c 1 and c 2 are constants. g is a monotonically decreasing function with @xmath4. therefore, li's formulation always has a steep tail and lacks the capacity to depict the switching of two stable proportional coefficients. figure 3 shows examples using li's formulation to fit data in fig. 1 . it can be observed that the non-zipfian chinese characters are fitted well, but not for the tail part in more zipfian english words. this can be explained from the shape of g in fig. 2 . it is reasonable to model the g of chinese characters using a monotonically decreasing function because the γ in eq. ( 3 ) is quite large (around r max ). however, it is not proper for english words, where a proper γ is required. ## experiment and discussion we used the proposed formulation to fit data of various european languages and typical asian languages. the europarl corpus @xcite and data from the second international chinese word segmentation bakeoff (icwb2) @xcite were mentioned in section 1. we also used english-japanese patent data from the 7th ntcir workshop @xcite . the europarl data and english data from ntcir were lower-cased and tokenized using the toolkit provided by moses foot_2 @xcite . fitting was performed under a logarithmic scale using the fit function foot_3 in gnuplot. 7 specifically, relation-frequency data were used to fit (α, β, γ) and c in @xmath5. for the initialization, (α, β, γ) = (1, 1, rmax 2 ) and @xmath6. from the nature of these languages, we consider that α can be explained as an axis of analysissynthesis on syntax and β + γ norm as that on morphology. a large α suggests a couple of extremely frequent words in the corpus. as typical examples, languages with a relatively large α, that is, romance and germanic, generally contain abundant prepositions, particles, and determiners to mark syntactic roles, whereas those with a smaller α, that is, slavic and uralic, tend to use complex declension and conjugation within words to afford syntactic information. interesting evidence is that bg, as a very analytic slavic language, has a larger α than other slavic languages. in another dimension, a large β + γ norm suggests a dramatic decrease in the frequency of rare words. hence, lan-greek (el), english (en), spanish (es), estonian (et), finnish (fi), french (fr), hungarian (hu), italian (it), lithuanian (lt), latvian (lv), dutch (nl), polish (pl), portuguese (pt), romanian (ro), slovak (sk), slovene (sl), and swedish (sv). 9 the non-typical germanic en, baltic lt and lv, and hellenic el are in gray. ro with a large β is excluded. 10 first principal component: -0.1α -0.7β -0.7γnorm, and second principal component: 1.0α + 0.1β -0.3γnorm. figure 5 provides further evidence. the data size of typical languages in europarl is gradu- ## conclusion and future work we have shown that f ∝ r -α (r + γ) -β for the rank-frequency relation in natural languages. this is an explainable extension of several related formulations, with α related to the analytic features of syntax and β + γ to that of morphology. a more general form, f ∝ k (r + γ k ) -β k , can be considered for further investigation. the k terms can depict k different proportional coefficients.