Disambiguatory Signals are Stronger in Word-initial Positions (2021)

Abstract: Psycholinguistic studies of human word processing and lexical access provide ample evidence of the preferred nature of word-initial versus word-final segments, e.g., in terms of attention paid by listeners (greater) or the likelihood of reduction by speakers (lower). This has led to the conjecture—as in Wedel et al. (2019b), but common elsewhere—that languages have evolved to provide more information earlier in words than later. Information-theoretic methods to establish such tendencies in lexicons have suffered from several methodological shortcomings that leave open the question of whether this high word-initial informativeness is actually a property of the lexicon or simply an artefact of the incremental nature of recognition. In this paper, we point out the confounds in existing methods for comparing the informativeness of segments early in the word versus later in the word, and present several new measures that avoid these confounds. When controlling for these confounds, we still find evidence across hundreds of languages that indeed there is a cross-linguistic tendency to front-load information in words.

## introduction the psycholinguistic study of human lexical access is largely concerned with the incremental processing of words-whereby, as individual sub-lexical units (e.g., phones) are perceived, listeners update their expectations of the word being spoken. one common tenet of such studies is that the disambiguatory signal contributed by units early in the word is stronger than that contributed lateri.e. disambiguatory signals are front-loaded in words. this intuition is derived from ample indirect evidence that the beginnings of words are more important for humans during word processingincluding, e.g., evidence of increased attention to word beginnings @xcite alia) or 0 2 4 6 8 position from start 2 4 forward celex wikipedia northeuralex 8 6 4 2 0 position from end 2 4 backward celex wikipedia northeuralex figure 1: forward and backward surprisals with lstm model from pimentel et al. (2020). the bottom plot has been flipped horizontally such that it visually corresponds to the normal string direction. to analyse this front-loading effect, researchers have investigated the information provided by segments in words. @xcite showed that, in dutch, a segment's position in a word is a very strong predictor of its conditional surprisal, with later segments being more predictable than earlier ones-a result which we show to arise directly from its definition in §3.3.1. recently @xcite and @xcite confirmed the effect on many more languages. their analysis, however, presents an inherent confound between the amount of conditional information available to a model and the surprisal of the subsequent segment-see fig. 1 for results illustrating this. using the lstm training recipes from @xcite ), foot_1 we calculated the conditional surprisal at each segment position within the words across all languages in three datasets. 3 the top-half of fig. 1 shows that, indeed, positions earlier in the string have higher surprisal than positions later in the string, supporting the thesis of higher informativity earlier in words. the bottomhalf shows that modelling the strings right-to-left instead of left-to-right reverses the resulting effect. this decouples conditional surprisal from the disambiguatory strength. to expose this decoupling, consider an artificial language where every word contains a copy of its first half, e.g., foofoo, barbar, foobarfoobar, etc. the first and second halves of these words have identical disambiguatory strength; they are the same so one could disambiguate the word as easily from its second half as from the first. in contrast, conditional surprisal would be nearly zero for the second halves of words because the second half is perfectly predictable from the first half. in natural languages, measuring conditional entropy in a left-to-right fashion inherently forces a reduction of conditional entropy in later segments because of a language's phonotactic constraints. however, the disambiguatory strength of later segments is not inherently less than that of earlier segments. for instance, in a language like turkish, which has vowel harmony, knowledge of any of the vowels in a word will provide information about the word's other vowels in a similar way. as such, knowledge of vowels towards the front of a word is as disambiguating as of vowels towards its end. the contributions of this paper are threefold. first, we document and demonstrate the shortcomings of existing methods for measuring the informativeness of individual segments in context, including the confound with the amount of conditional information discussed above. second, we introduce three surprisal-based measures that control for this confound and enable comparison of wordinitial versus -final positions in this respect: unigram, position-specific and cloze surprisal (see §3). finally, we find robust evidence across many languages of stronger disambiguatory signals in word initial than word-final positions. out of a total of 151 languages analysed across three separate collections, 82 of them present a higher cloze surprisal in word beginnings than in endings-with similar patterns arising with the other two measures. psycholinguistic evidence is key to understanding how lexical access works in human language processing, and can help us understand why lexicons may evolve to provide more disambiguatory signals earlier in words. 4 given the incremental nature of human lexical processing, however, such evidence cannot provide direct evidence of the nature of the lexicon uninfluenced by incrementality. computational evidence. to the best of our knowledge, van @xcite were the first to use computational methods coupled with an information theoretic definition of informativeness to investigate this question. they showed that segments in the beginning of words carry most of a word's information, as measured by their contextual surprisal using a plug-in tree structured probabilistic estimator. although assessing a less-biased sample of words than @xcite , 5 this study is also limited to a single language (dutch), hence cannot assess whether this is a general phenomenon or specific to that language. further, van @xcite use absolute word positions in their analysis. word length correlates strongly with frequency, hence while early positions are present in all words, later positions only exist for a much smaller sample of typically lower frequency words. thus this comparison amounts to asking if later positions in longer and infrequent words have lower surprisal than earlier positions in all (frequent or infrequent) words. we analyse this confounding factor in §6. wedel et al. (2019b) and king and wedel (2020 ) applied a methodology similar to that of @xcite to show, for many diverse languages, that more frequent words contain less informative segments in word initial positions, while less frequent types carry more informative ones. they further showed that segments in later word positions were less informative (given the previous ones) than average in rarer words. while controlling for length, king and wedel (2020) also compared words' forward and backward uniqueness points-nodes in a trie from which only one leaf node can be reached, i.e., where the word is uniquely identified-showing they happened earlier in forward strings. while these studies provide evidence from more diverse sets of languages, they follow van @xcite in studying closed lexicons. 6 as we show in §3.3.1, the use of probabilistic trie models on a closed lexicon yields a trivial effect of higher informativity at word initial positions. furthermore, such studies cannot account for out-of-vocabulary words (e.g., nonce, proper name or otherwise unknown words) or derivational morphology, which are key parts of lexical recognition. lexical access is also somewhat robust to segmental misordering @xcite and sounds later in a word help determine the perception of earlier ones @xcite . in contrast, a trie over a closed lexicon is deterministic. beyond this, @xcite showed in a corpus study that the probability of a word type being uniquely identifiable before its last segment was only 41%-and 19% of types were identified only by the end of word, being proper prefixes of other words, such as cat and cats. they conclude that uniqueness point statistics may only be useful for long word analysis. in @xcite , we analysed several languages' phonotactic distributions, focusing on presenting a trade-off between phonotactic entropy and word length across languages. as a control experiment we analysed the correlation between a segment's surprisal and its word position across 106 languages. we did not control for word length and did not run per-language experiments, thoughso we could have just been capturing the effect that later positions will mostly be present in languages with longer words (which, as we find, have lower information on average). 7 while this last work avoids many of the issues raised earlier in this section, it fails to control the key confound mentioned earlier: it relies on left-toright conditional probabilities to calculate surprisal. thus segments early in the word have less conditional information and hence are generally of lower probability-a trivial effect that does not indicate a segment's disambiguatory signal strength. ## character-level language models in this paper, we make use of character-level language models to model the probability distributions p θ and approximate the relevant cross-entropies. unigram. this might be the simplest language model still in use in natural language processing. we use its laplace-smoothed variant lstm. this architecture is the state-of-the-art for character-level language modelling @xcite . given a sequence of segments @xmath0, we use one hot lookup embeddings to transform each of them into a vector z @xmath1. we then feed these vectors into a k-layer in this equation, @xmath2. backward lstm. to get the backward surprisals we use models with the same architecture, but reverse all strings before feeding them to the models. as such, we get the similar equations transformer. transformers allow a segment to be conditioned on both future and previous symbols. our implementation starts similar to the lstm one, getting embedding vectors z t for each segment in the string @xmath3, except that we replace segment w t with a mask symbol. we then feed these vectors through k multi-headed selfattention layers, as defined by @xcite . finally, the representations from the last layer are linearly transformed and fed into a softmax position-specific transformer. to get positionspecific surprisal values, we again use a transformer architecture, but instead of replacing a single segment with a mask symbol, we replace all of them. this is equivalent to conditioning each segment's distribution on its position and the word length-i.e., estimating p θ (w t | t, |w|). ## data in order to estimate redundancy and informativeness of segments we use three different datasets, each with its own pros and cons. we focus on types instead of tokens-i.e., the datasets consist of lexicons-for a few different reasons. first, it is easier to get reliable samples of types than tokens for a language, specially low-resource ones. second, it is a well known result that token frequency correlates with both word length @xcite and phonotactic probability @xcite , so that would be a strong confound in the results. third, morphology is more easily modeled at the type level than at token level @xcite . foot_9 celex @xcite allows us to experiment exclusively on monomorphemic words, but covers only three closely related languages. it contains both morphological and phonetic annotations for a large number of words in english, dutch and german. we follow @xcite in using only words labeled as monomorphemic in our study, leaving us with 4,810 words in german, 6,206 words in english and 7,045 words in dutch. northeuralex @xcite spans 107 languages from 21 language families in a unified ipa format. this database is composed of concept aligned word lists for these languages, containing 1016 concepts, each of them translated in most languages. however, most of these languages are from eurasia, hence the collection lacks the typological diversity we would ideally like. wikipedia allows us to investigate a broader and more diverse set of languages, but has no phonetic information (only graphemes) and lexicons extracted from it may be "contaminated" with foreign words. we fetch the wikipedia for a set of 41 diverse languages, 11 and tokenise their text using language-specific tokenisers from spacy @xcite . when a language-specific tokeniser was not available, we used a multilingual one. we then filtered all non-word tokens-by removing the ones with any symbol not in the language's scripts-and kept only the 10,000 most frequent types in each language. ## experiments and results forward surprisal. we first replicate the results from van @xcite , @xcite , which show that surprisal decreases with as the words position advances. on average, forward surprisal, i.e. h θ (w t | @xmath4, could decrease for two reasons: (i) words indeed front-load disambiguatory signals; or (ii) the trivial fact that conditioning reduces entropy. for each word, we first get the forward surprisal for each segment in it. we then group surprisal values in two groups: word initial (when they are in the first half of its word) and final (when in the second half), ignoring mid positions in words with uneven lengths; we average these initial and final surprisals per word, getting a single value of each per word. this way we compare earlier vs. later word positions while ignoring any length effectwords with all lengths will possess segments in both groups. for each analysed language, we then use permutation tests (permuting word initial and final surprisals) to evaluate if one group is statistically larger than the other-using 100,000 permutations. all but one language in three analysed datasets had significantly larger surprisal in word initial positions 12 -the exception being abkhaz in northeuralex. these results can be seen in tab. 1 and in fig. 2 ## conclusions in this work, we analysed the distribution of disambiguatory information in word positions. we present an in-depth critique of previous work, showing several confounding effects in their analysis. we then proposed the use of three new methods which corrected for these biases-namely unigram, position-specific and cloze surprisal. these models controlled for the amount of conditional information across word positions, allowing for an unbiased analysis of the lexicon. using these models we show that the lexicons of most languages indeed front-load their disambiguatory signals. this effect, though, is not universal and the difference in disambiguatory information between word initial and final positions is much lower than previously estimated-ranging from 4% to 14%, depending on the used metric, instead of 31%.