What company do words keep? Revisiting the distributional semantics of J . R . Firth & Zellig H arris (2022)

Abstract: The power of word embeddings is attributed to the linguistic theory that similar words will appear in similar contexts. This idea is specifically invoked by noting that “you shall know a word by the company it keeps,” a quote from British linguist J.R. Firth who, along with his American colleague Zellig Harris, is often credited with the invention of “distributional semantics.” While both Firth and Harris are cited in all major NLP textbooks and many foundational papers, the content and differences between their theories is seldom discussed. Engaging in a close reading of their work, we discover two distinct and in many ways divergent theories of meaning. One focuses exclusively on the internal workings of linguistic forms, while the other invites us to consider words in new company—not just with other linguistic elements, but also in a broader cultural and situational context. Contrasting these theories from the perspective of current debates in NLP, we discover in Firth a figure who could guide the field towards a more culturally grounded notion of semantics. We consider how an expanded notion of “context” might be modeled in practice through two different strategies: comparative stratification and syntagmatic extension.

## introduction we are in the world and the world is in us. @xcite if you have read any papers in computational linguistics in the past thirty years, you have likely come upon the following quote from british linguist j.r. @xcite : "you shall know a word by the company it keeps". cited in most major textbooks @xcite @xcite , several foundational papers, and hundreds of other nlp articles, this phrase has come to index a theoretical orientation in a field that is increasingly focused on computation, often at the expense of linguistic theory (on these trends in nlp see @xcite @xcite @xcite . together with american linguist zellig harris, firth is regularly called upon to justify a distributional theory of semantics, whereby the meaning of lexical units is conceived in terms of relative co-occurrence and shared contexts of use @xcite . while harris and firth are often invoked, their ideas are seldom closely engaged. hailing from disparate traditions, harris and firth had radically different ideas on the scope and context of linguistic analysis, and presented incongruent versions of the distributional method. drawing on the information theory pioneered by claude @xcite , harris was determined to work out a structuralist theory of language in terms of mathematical information @xcite . firth, on the other hand, came to linguistics via anthropology and borrowed heavily from pragmatic philosophies of language. for him, linguistic analysis always started with the "context of situation" and necessarily accounted for non-verbal actors and objects @xcite . considering the definition and extent of linguistic context is important for many reasons, as a spate of recent publications suggests @xcite @xcite @xcite . firstly, it touches upon the limits of current paradigms in nlp, where corpus linguistics is perfected through increasingly complex models trained on increasingly massive corpora @xcite . this approach may advance the identification of linguistic form, but might ultimately have little to say about the relation of meaning to the social world @xcite . secondly, even with more modest ambitions, several nlp applications-e.g., with spatial (mckenzie and adams, 2021) or historical @xcite data-require that linguistic patterns be related to other types of structure. thirdly, from sociological and sociolinguistic perspectives, meaning intrinsically varies as language is used in different settings and indexed to different social categories @xcite @xcite @xcite . finally, without a broader sense of context, nlp and language modeling in particular remains trapped in a paradigm where language is always treated as universal, making invisible both different speech communities (e.g. @xcite and the biases of language (e.g. @xcite @xcite . after a brief history of distributional semantics, we outline harris' and firth's research on distribution and, more broadly, on the scope of linguistic analysis. we look at some of the ways in which nlp has tried to account for broader context within the distributional paradigm. we suggest that existing strategies can be understood in terms of either "comparative stratification" or "syntagmatic extension." we conclude with thoughts on why rereading harris and, in particular, firth might aid the field of nlp with its current aporias. if words shall be known by the company they keep, then the question follows: what kind of company do they keep? are they found only alongside linguistic elements, or do they mingle with other types of entities? or, as firth himself wrote: "many different answers could be given to the question 'distribution of what, where and how?"' @xcite . ## background: distributional semantics and nlp distributional semantics has been an fundamental part of computational linguistics since the beginnings of the field, but in a discontinuous manner encompassing at least two distinct eras. firstly, during the 1950s and 60s, harris was integral to the mathematization of linguistics in the us after the second world war @xcite . firth was skeptical of efforts to mechanize linguistics, 1 but he nonetheless consulted for some of the early work on machine translation at cambridge language research group 1 he seemed to consider the idea orwellian @xcite ) and repeatedly attacked norbert wiener (e.g. @xcite , a pioneer whose work would later be considered foundational for connectionist ai @xcite . @xcite , which included firth's pupil, m.a.k. halliday @xcite and shortly later the nlp pioneer karen spärck jones @xcite . naturally, others also contributed to this first wave of distributional thinking, including @xcite with what might be considered one of the first language models, warren @xcite with an early proposal for distributional semantics, and martin @xcite with a statistical formulation of language as a symbolic system of conditional probabilities. secondly, when computational linguistics returned to its "empiricist" roots in probabilistic methods and information theory in the mid-80s and early 90s @xcite , firth and harris accompanied shannon among the authors who were invoked, in an acl "special issue on computational linguistics using large corpora," as foundational figures of a tradition that had been overshadowed for decades by the "rationalism" of characters such as noam chomsky and marvin minsky @xcite . during this "corpus turn," the rapid automation of linguistics was driven by a resumed connection with postwar computational linguistics and information theory @xcite . however, the 1990s wave of vector semantics papers that used methods like singular-value decomposition (svd) to produce early "dense vector" models of meaning like lsa @xcite @xcite and its derivatives @xcite , hal @xcite , or the models of schütze @xcite @xcite generally did not cite firth or harris, although a few papers from that period did @xcite . in short, while firth and harris were not regularly used as stand-ins for linguistic theory during the 1990s and early 2000s, a general revival of empiricism and distributional approaches to meaning signaled a potential resurgence of interest in their thinking. during the 2000s, the application of neural networks to language modeling tasks (e.g. @xcite and the development of selfsupervision techniques (e.g. @xcite set the stage for the word embedding breakthroughs of the early 2010s (e.g. @xcite . by the end of the decade, the introduction of attention @xcite and then of the transformer model @xcite made way for the next breakthrough, the large-language modeling revolution (e.g. @xcite @xcite . following the introduction of the word2vec model and its powerful but "static" embeddings, harris in particular was frequently cited @xcite @xcite , often (but not always) along with firth @xcite @xcite @xcite . however, despite an explosion of citations @xcite , this interest has not been very engaged. in fact, the canonization of firth and harris during this time is paradoxical. on the one hand, it seems that they are invoked to lend theoretical authority to a field that struggles to lift its gaze from the latest state-of-the-art numbers @xcite ). yet, the unspoken conclusion from the ascent of neural models and the language modeling revolution was that "learning from data made linguistic theories irrelevant" @xcite . in other words, just as nlp seemed to lose interest with linguistic theory, it elevated two pioneering theoreticians to canonical status, but seemingly without engaging closely with their work. in fact, it often seems as if firth and harris are referenced in such a cavalier manner that it deflects attention from the field's general lack of engagement with linguistic theory. meanwhile, firth and harris became figures who justify a relatively narrow conception of meaning, one that is predominantly intra-linguistic, without much to say about its usage in social life. this peculiar story has not been properly told. though léon discusses the contrast between harris and firth in the context of corpus linguistics @xcite and their influence on the history of computational linguistics (2021), her work does not address the differences in their distributional theories and conceptions of "context," nor the renewed and paradoxical significance of the two authors for language modeling. in our contribution, we emphasize the gap between the ideas of firth and harris as well as the insights a re-reading of their work offers for expanding the scope of computational semantics. ## harris's distributional structuralism few linguists contributed more to linguistic theory @xcite , and not just by serving as noam chomsky's doctoral advisor. in fact, the two came to share little in common @xcite . whereas chomsky's generative grammar repositioned linguistics as a cognitive science seeking to understand, in so few words, the idealized mental representations and structures enabling language acquisition and production (e.g. @xcite , harris' radically distributional approach to language effectively elevated the natural language corpus as the sole starting point from which linguistic theory could arise @xcite @xcite . this theory consisted of a linguistic structure segmentable into a finite set of formal objects characterized by constrained patterns of correspondence with one another @xcite @xcite . such patterns of correspondence can be observed only in language-in-use, that is, in natural language corpora. in his foundational paper "distributional structure," for example, harris @xcite @xcite provides a purely distributional account of how one might induce the semantic meanings of oculist, eyedoctor, and lawyer from the partial (in the case of oculist and lawyer) or nearly complete (in the case of oculist and eye-doctor) overlap in their observed "environments" of use. this approach applies to other levels of linguistic analysis, such as morphophonemics (e.g., @xcite . rather than producing a series of descriptive rules for the distribution of each phoneme, morpheme, or word, greater parsimony was sought by grouping these elements into structurally equivalent classescategorized by their relationships as "operators" and "arguments" in harris' later @xcite -sharing the same distributional rules, compounding elements in a hierarchical manner. in this section we draw attention to three aspects of harris's distributional linguistics: the relationship it posits between meaning and form; assumptions about heterogeneity among speakers of the same dialect; and the concept of sublanguages. ## firth's contextual semantics something similar to the sociolinguistic critique of harris could be articulated from a different perspective, namely, through the work of j.r. firth. firth (1890-1960)-professor of general linguistics at the university college of london and the first holder of a chair in that subject in britainindependently formulated a distributional theory of lexical semantics. however, unlike harris, firth refused to treat meaning separately from pragmatics, and words apart from their broader "context of situation." @xcite firth never published a fully articulated exposition of his general theory of language @xcite and today, all of his work is not only out of print but also mostly unavailable online. not understood by "the contemporary scientism" of american descriptivist linguistics and its pioneers like harris, firth was mostly ignored on the other side of the atlantic @xcite . unlike many of his american contemporaries, firth did not draw mainly from cognitive psychology and logic-the latter of which firth thought had "taken the heart out of language" @xcite -but from the work of polish anthropologist bronisław malinowski @xcite . here, we focus on the evolution of his thoughts on meaning and collocation, as well as his notions of context of situation and restricted language. ## discussion: words in mixed company often cited, together or separately, to justify a distributional appraoch to semantics, firth and harris nonetheless offer differing views on language and meaning. harris offers us a rigorous formalism that treats language as a "detached pattern"not a "code", but a particular system of meaning. firth, by comparison, left a much more scattered legacy that was only systematized by his students. firth and harris shared a concern about the lack of an external metalanguage of linguistics, but drew different conclusions from it. if harris responded to this conundrum by creating one hierarchically organized system without intrusion from extra-linguistic factors, firth called for an investigation of language as a "spectrum" @xcite with different modes of meaning that had to be addressed through multiple levels of analysisstarting with the context of situation and proceeding from there to decide which other levels are relevant @xcite ). firth's distributional theory has been described unfavourably as based on frequent co-occurrence, in contrast to the recursive dependencies developed by harris @xcite . for harris, the meaning of a word depends on its set (e.g. harris, 1991, 17) such that, for example, the words "divide" and "multiply" operate on the word "cell" (and vice versa) in the same way, producing essentially the same meaning @xcite . however, firth's final method of substitution and commutation also establishes complex, multidimensional criteria for distributional contrast as well as a framework for understanding polysemy. though less formalized and less obviously recursive than harris', firth's approach can, arguably, also be read as treating linguistic elements as operators and arguments defined by their sets in a complex hierarchy @xcite at a series of congruent levels" @xcite with different "bands of abstraction" @xcite , including the extra-linguistic context of situation. harris and firth both understand any linguistic analysis as incomplete, for harris always a pursuit of the least description" (harris, 1988, 3)-i.e., best "grammar" or model-for firth always grounded in the social construction of facts, without any possibility of "complete axiomatization" @xcite . from the perspective of empirical work, especially decades after their time, firth and harris also share similarities. both rejected the mentalism that was so prevalent during their time. they were both revived as empiricist originators during the rise of statistical learning in the 1990s, and their respective work on restricted languages and sublanguages largely conflated in service of the practical concerns of corpus linguistics @xcite . their theories both included in what we might call a "reality principle," a final arbiter of meaning outside of form: the pair test for harris, the context of situation for firth. the former grounds linguistics in the smallest possible unit of analysis as understood by the native speaker, the latter in social actions and objects. in light of recent calls to extend the "world scope" @xcite of nlp and to move towards pragmatic notions of meaning, it might make sense to balance harris's formalism and firth's pluralism. though firth warned us against overextending linguistics, he was generous with the company that words could keep. they mingled with each other, but also with events, objects, people, and indexical features such as time and space. and if nlp is ready to move beyond the corpus, then even harris might acknowledge that when modeling language in "the immediate situation"-whether in online interactions or face-to-face communication-words are not on their own, that to judge the meaning of a combination of words, we can summon "the aid of some of the extra-linguistic situational information" @xcite . in the following subsections, we consider two ways in which nlp is already doing this, in order to highlight some already existing strategies for broader contextualization. we call these strategies "comparative stratification" and "syntagmatic extension." ## conclusions this paper revisited the theories of the two most well-known progenitors of the distributional approach to meaning in nlp. recognizing the open question of how to bring nlp beyond the corpus, we offer a thorough account of the two distributional theories that are most often invoked to justify the modeling of meaning through departures from randomness in the company that words keep. comparing the work of harris and firth-who both published their major work before the rise of the internet and its corpora-we find two distinct theories of distribution: one formal and mathematical, treating language as a particular type of detached information, another more schematic and anthropological, treating language as a functional spectrum which always emanates from a particular context of situation. the legacies of both firth and harris can be seen in the current paradigm of corpus linguistics, but in the domain of distributional semantics, it is harris's ethos that dominates, despite firth providing its most famous tagline. moving forward, we suggest that semantic modeling take more inspiration from firth, and consider the context of situation and the wide variety of company that words can keep as crucial sites of innovation for the field. doing so may not involve following a finite set of steps or flowchart. rather, we humbly suggest that the field may be enriched by thoughtful and creative re-engagements with the intellectual traditions from which it has historically drawn. this does not imply abandoning the rigor provided by harris. on the contrary, we find that firth and harris would probably have agreed that any model or "grammar" is inevitably incomplete and partial. no universal model is possible, despite the large-language modeling fervor, nor will there be one theory of language to guide us. there are only the partial perspectives and the inevitable choice of adapting one.