Speakers Fill Lexical Semantic Gaps with Context (2020)

Abstract: Lexical ambiguity is widespread in language, allowing for the reuse of economical word forms and therefore making language more efficient. If ambiguous words cannot be disambiguated from context, however, this gain in efficiency might make language less clear—resulting in frequent miscommunication. For a language to be clear and efficiently encoded, we posit that the lexical ambiguity of a word type should correlate with how much information context provides about it, on average. To investigate whether this is the case, we operationalise the lexical ambiguity of a word as the entropy of meanings it can take, and provide two ways to estimate this—one which requires human annotation (using WordNet), and one which does not (using BERT), making it readily applicable to a large number of languages. We validate these measures by showing that, on six high-resource languages, there are significant Pearson correlations between our BERT-based estimate of ambiguity and the number of synonyms a word has in WordNet (e.g. \rho = 0.40 in English). We then test our main hypothesis—that a word’s lexical ambiguity should negatively correlate with its contextual uncertainty—and find significant correlations on all 18 typologically diverse languages we analyse. This suggests that, in the presence of ambiguity, speakers compensate by making contexts more informative.

## introduction linguistic structure and meaning are often underdetermined in the linguistic signal. in an extreme case this can lead to ambiguity: sentences might allow more than one valid syntactic structure, and pronouns could corefer to various antecedents. complementarily, linguistic signals can also overdetermine some aspect of the intended message-for instance, agreement patterns may require redundant marking, and word forms might occupy sparsely populated parts of the phonological space @xcite . in a tradition that goes back at least to zipf, it has been hypothesised that individuals maintain an efficient balance between over-and under-specifying an intended message. such balance is mediated by conflicting pressures for both clarity (the quality that allows the reconstruction of the intended message), and economy of expression (which allows for inexpensive and rapid encoding of the message in a linguistic signal). a recent instantiation of this idea is that in an efficient language, one expects economical words (which are short or phonotactically simple) to be associated with multiple unrelated meanings, so they can be more widely used @xcite . at first blush, this may appear to sacrifice clarity, increasing ambiguity and making it more difficult for a listener to resolve the linguistic signal. the emerging picture from psycholinguistics and pragmatics, however, is that individuals can fill in these ambiguous gaps, by tapping on additional linguistic or extra-linguistic cues @xcite @xcite . an obvious example is given by the role of contextual information in reducing the ambiguity associated with the meaning of a word form. for instance, the contexts which surround the word ruler in the sentences 'alice borrowed a ruler from her friends at school' and 'bob rose to power and became a ruthless ruler' each play a crucial role in disambiguating its intended underlying meaning. to remain robust in the presence of noise, we may expect the linguistic signal to be on average somewhat overdetermined by the speaker, leading to redundancy in how words and their contexts determine the intended meaning. 1 by analysing this redundant information-theoretically under the assumption that languages strike a balance between economy of expression and clarity, we derive that the 'amount' of lexical ambiguity in a given word type should negatively correlate with how uncertain on average the word is given its context (see §4). as communication unfolds, the efficiency of a particular word can only be modestly modified (e.g. by choosing clipped forms when available; @xcite . however, contexts can be enriched or demoted dynamically, so as to complement a word with the evidence needed for disambiguation. to investigate whether it is the case that the contexts in which a word appears are systematically adapted to enable disambiguation, we first provide an operationalisation of lexical ambiguity, grounded in information theory. we then provide two methods for estimating it, one using wordnet @xcite , and the other using multilingual bert's contextualised embeddings @xcite , which allows us to explore a large set of languages. we validate our lexical ambiguity measurements by comparing one to the other in six highresource languages from four language families (afro-asiatic: arabic; austronesian: indonesian; indo-european: english, persian and portuguese; uralic: finnish), and find significant correlations between the number of synsets in wordnet and our bert estimate (e.g. ρ = 0.40 in english), indicating that our annotation-free method for measuring lexical ambiguity is useful. we then test our main hypothesis-that the con-textual uncertainty about a word should negatively correlate with its degree of lexical ambiguity. first, we test this on the same set of six high-resource languages for which we have wordnet annotation, and find significant negative correlations on five of them. we then extend our evaluation, using our bert-based measure, to cover a much more representative set of 18 typologically diverse languages: afrikaans, arabic, bengali, english, estonian, finnish, hebrew, indonesian, icelandic, kannada, malayalam, marathi, persian, portuguese, tagalog, turkish, tatar, and yoruba. 2 in this set, we find significant negative correlations for all languages (see figure 1 ). ## ambiguity in language while the pervasiveness of ambiguity in language encumbers the algorithmic processing of natural language @xcite , people seamlessly overcome ambiguity through both linguistic and non-linguistic means. world knowledge, pragmatic inferences, and expectations about discourse coherence all contribute to rapidly decoding the intended message out of potentially ambiguous signals @xcite . while sometimes ambiguity might indeed result in an observed processing burden @xcite , which could lead communication astray, individuals can in response retrace and reanalyse their inferences (as it has been famously shown in garden-path sentences like "the horse raced past the barn fell"; @xcite . this outstanding capacity to navigate ambiguous linguistic signals calls for a reexamination of the presence of ambiguity found in language. if the linguistic signal was deterministically and uniquely decodable-as, for instance, in the universal language proposed by wilkins @xcite -then all of the para-linguistic evidence would be redundant, and the code underlying the signal would be substantially more cumbersome. on the other hand, if linguistic signals present individuals with too many compatible inferences, communication would break down. an extreme case is represented by louis victor leborgne, an aphasia patient described by paul broca @xcite . louis, in spite of immaculate comprehension and mental functions, was unable to utter anything else than the syllable "tan" in his attempts to communicate. the most influential explanation offered for why natural languages are seemingly far from both extremes derives from the seminal work of @xcite . in that work, zipf proposed several aspects of human cognition and behaviour could be derived from the principle of least effort. languages should aim to minimise the complexity and cost of linguistic signals as much as possible, under the sole constraint that the signal can be decoded efficiently. ## ambiguity and uncertainty we formulate both lexical ambiguity and contextual uncertainty information-theoretically. let m be a space of all lexical meaning representations, w be the space of all words and c be the space of all contexts. we denote the m-, w-, and c-valued random variables as m , w and c, respectively, and name elements of those sets m, w and c. we take m to be an either discrete or continuous mean-ing space, w to be the set of words in a language (excluding the beginning-of-and end-of-sequence symbols, bos and eos) and ## computation and approximation our information-theoretic operationalisation requires approximation. first, we do not know the true distributions over words, their meanings and their contexts. second, even if we did, eq. ( 3 ) and eq. ( 6 ) would likely be hard to compute. ## data we used wikipedia as the main data source for all our experiments. multilingual bert foot_9 was trained on the 104 languages with the largest wikipedias foot_10 -of these, we subsampled a diverse set of 18 for our experiments: afrikaans, arabic, bengali, english, estonian, finnish, hebrew, indonesian, icelandic, kannada, malayalam, marathi, persian, portuguese, tagalog, turkish, tatar, and yoruba. for each of these languages, we first downloaded their entire wikipedia, which we sentencized and tokenized using language specific models in spacy @xcite -our definition of a word here is, thus, a token as given by the spacy tokenizer. we then subsampled 1 million random sentences per language for our analysis and another 100,000 random sentences to train the model q θ . we run multilingual bert on the 1 million analysis sentences to acquire both h w,c and h c (eq. ( 11 ) and eq. ( 16 )) for each word in these corpora-discarding any word for which we do not have at least 100 contexts in which the word occurs. for the purpose of our analysis, we also discarded any word containing characters not in the individual scripts of the analysed language. the final number of word types used in our analysis can be found in tables 1 and 3 . ## discussion: wordnet vs. bert-based approximations the novel continuous (bert-based) approximation of lexical ambiguity has two important virtues over the alternative wordnet-based measure. on the practical side, it can be readily computed for many languages. since we are using multilingual bert for our continuous approximation, as discussed in §5, this quantity is easily obtainable for the 104 languages on which it was trained. second, on more theoretical grounds, the continuous representation of the space of meanings might better capture the gradient that goes from subtle but distinct senses of the same word to completely unrelated homophones @xcite p. 51). alternatively, the wordnet-based measure of lexical ambiguity is supported by expert human annotation and extensive research on its linguistic and psycholinguistic correlates, e.g. @xcite and @xcite . these differences notwithstanding, we expect both measures to correlate to a certain degree. to evaluate this, we run an experiment comparing both estimates in six languages from four different families for which wordnet is available: arabic, english, finnish, indonesian, persian, and portuguese. figure 2 and a potential underlying problem in the above study is that the number of senses a word has in wordnet might rely on word frequency (this beyond a true underlying relationship with it)-e.g. annotating senses for frequent words may be easier than for infrequent ones. furthermore, the number of samples a word has in our corpus will affect its sample density in the embedding space and thus its estimated bert entropy. as a second evaluation, we therefore train a multivariate linear regressor predicting our bert-based measure not only from the log of the number of senses a word has in word-net, but also the word's frequency (i.e. its number of occurrences in the corpus). this analysis is presented in ## lexical ambiguity correlates with contextual uncertainty comparing the top and bottom half of a quick visual inspection of figure 3 indicates this data might be heteroscedastic-it might have unequal variance across distinct ambiguity levels. to investigate this, we run @xcite test on the uncertainty-ambiguity pairs. this verifies the intuition that this distribution is heteroscedastic for both our wordnet and bert measures (@xmath0.01). future work should investigate the impact of this heteroscedasticity in lexical ambiguity. limitations this work focuses on proposing new information-theoretic approximations for both lexical ambiguity and bidirectional contextual uncertainty and on positing that these two measures should negatively correlate. in this experiment section, we tested the hypothesis on a set of typologically diverse languages. nonetheless, our experiments are restricted to wikipedia corpora. this data is naturally limited. for instance, while dialog utterances may rely on extra-linguistic clues, sentences in wikipedia cannot. furthermore, due to its ample audience target, the text in wikipedia may be over descriptive. future work should investigate if similar results apply to other corpora. ## conclusion in this paper we hypothesised that, were a language economical in its expressions and clear, then the contextual uncertainty of a word should negatively correlate with its lexical ambiguity-suggesting speakers compensate for lexical ambiguity by making contexts more informative. to investigate this, we proposed an information-theoretic operationalisation of lexical ambiguity, together with two methods of approximating it, one using wordnet and one using bert. we discuss the relative advantages of each, and provide experiments using both. with our wordnet approximation, we found significant negative correlations between lexical ambiguity and contextual uncertainty in five out of six high-resource languages analysed, supporting our hypothesis in this restricted setting. with our bert approximation, we then expanded our analysis to a larger set of 18 typologically diverse languages and found significant negative correlations between lexical ambiguity and contextual uncertainty in all of them, further supporting our hypothesis that contextual uncertainty negatively correlates with lexical ambiguity.