Original: Whatkind of company do words keep? Revisiting the distributional semantics of J.R. Firth & Zellig Harris
Revised: What company do words keep? Revisiting the distributional semantics of J.R. Firth & Zellig Harris

Original: Anonymous ACL submission

Original: Abstract
Revised: Abstract

Original: The power of word embeddings is attributedto the linguistic theory that similar words willappear in similar contexts.
Revised: The power of word embeddings is attributed to the linguistic theory that similar words will appear in similar contexts.

Original: This idea is specifically invoked by noting that “words will beknown by the company they keep,” a quotefrom British linguist J.R. Firth who, along withhis American colleague Zellig Harris, is often credited with the invention of “distributional semantics.”
Revised: This idea is specifically invoked by noting that “you shall know a word by the company it keeps,” a quote from British linguist J.R. Firth who, along with his American colleague Zellig Harris, is often credited with the invention of “distributional semantics.”

Original: While both Firth and Harris are cited in all major NLP textbooks andmany foundational papers, the content and differences between their theories is seldom discussed.
Revised: While both Firth and Harris are cited in all major NLP textbooks and many foundational papers, the content and differences between their theories is seldom discussed.

Original: Engaging in a close reading with theirwork, we discover two distinct and in manyways divergent theories of meaning.
Revised: Engaging in a close reading of their work, we discover two distinct and in many ways divergent theories of meaning.

Original: One is“introverted,” focused on the internal workingsof language, while the otheris “extroverted”and invites us to consider words in new company, not just with other linguistic elementsbut also in a broader cultural  context.
Revised: One focuses exclusively on the internal workings of linguistic forms, while the other invites us to consider words in new company—not just with other linguistic elements, but also in a broader cultural and situational context.

Original: Contrasting these theories from the perspective ofcurrent debates in NLP, we discover in Firth afigure who could guide the field towards a moreculturally grounded notion of semantics.
Revised: Contrasting these theories from the perspective of current debates in NLP, we discover in Firth a figure who could guide the field towards a more culturally grounded notion of semantics.

Original: Weconsider how an expanded notion of “context”might be modeled in practice through two different strategies: comparative stratification andsyntagmatic extension.
Revised: We consider how an expanded notion of “context” might be modeled in practice through two different strategies: comparative stratification and syntagmatic extension.

Original:  Introduction
Revised:  Introduction

Original: We are  world 
Revised: We are in the world and the world is in us.

Original: Alfred North Whitehead 
Revised: Alfred North Whitehead (1938; cited in Firth 1957c, 29)

Original: If you have read any papers in computational linguistics in the past thirty years, it is likely thatyou have  come upon the following quote from British linguist J.R.Firth: “words shall be knownby the company they keep.”
Revised: If you have read any papers in computational linguistics in the past thirty years, you have likely come upon the following quote from British linguist J.R.Firth (1957c, 11): “You shall know a word by the company it keeps”.

Original: Cited in most major textbooks (Eisenstein, 2019; Jurafsky and Martin, 2021; Russell and Norvig, 2020; Manning and Schütze, 1999), several foundational papers, andhundreds of other NLP articles, these words havecome to index a theoretical orientation in a field that is increasingly focused on computation, often at the expense of linguistic theory (on these trends in NLP seeChurch and Liberman, 2021; Halevy et al., 2009;Henderson, 2020; Manning, 2015; Norvig, 2012).
Revised: Cited in most major textbooks (Manning and Schütze, 1999; Jurafsky and Martin, 2009; Eisenstein, 2019; Russell and Norvig, 2020), several foundational papers, and hundreds of other NLP articles, this phrase has come to index a theoretical orientation in a field that is increasingly focused on computation, often at the expense of linguistic theory (on these trends in NLP see Halevy et al., 2009; Manning, 2015; Norvig, 2012; Henderson, 2020; Church and Liberman, 2021).

Original: Together with American linguist Zellig Harris, Firth is often called upon to justify a distributional theory of semantics, where the meaning ofdifferent lexical units is conceived in terms of relative co-occurrence and shared contexts  (Sahlgren, 2008).
Revised: Together with American linguist Zellig Harris, Firth is regularly called upon to justify a distributional theory of semantics, whereby the meaning of lexical units is conceived in terms of relative co-occurrence and shared contexts of use (Sahlgren, 2008).

Original: While Harris and Firth are often invoked, their ideas are seldom closely engaged.
Revised: While Harris and Firth are often invoked, their ideas are seldom closely engaged.

Original: Hailing fromvery disparate traditions, Harris and Firth had radically different ideas on the scope and context of linguistic analysis and probably would not have agreed even on the question of  distributional semantics.
Revised: Hailing from disparate traditions, Harris and Firth had radically different ideas on the scope and context of linguistic analysis, and presented incongruent versions of the distributional method.

Original: Extending on the information theory of Claude Shannon (1948), Harris was determined to work out a structuralist theory of language in terms of mathematical information (Léon, 2011; Nevin, 1993).
Revised: Drawing on the information theory pioneered by Claude Shannon (1948), Harris was determined to work out a structuralist theory of language in terms of mathematical information (Léon, 2011; Nevin, 1993).

Original: Firth, on the other hand, came to linguistics through anthropology and borrowed heavily from pragmatic philosophies of language.
Revised: Firth, on the other hand, came to linguistics via anthropology and borrowed heavily from pragmatic philosophies of language.

Original: For him (1957b), linguistic analysis always started with the “context of situation” and necessarily accounted for non-verbal actors and objects.
Revised: For him, linguistic analysis always started with the “context of situation” and necessarily accounted for non-verbal actors and objects (Firth, 1957c, 9).

Original: Considering the definition and extent of linguistic context is important for many reasons, as a spate of recent publications suggests (Bisk et al., 2020;
Revised: Considering the definition and extent of linguistic context is important for many reasons, as a spate of recent publications suggests (Glenberg and

Original:  Bender and Koller, 2020; Trott et al., 2020; Tamari et al., 2020; Hovy, 2018).
Revised: Robertson, 2000; Hovy, 2018; Bender and Koller, 2020; Bisk et al., 2020; Tamari et al., 2020; Trott et al., 2020).

Original: First of all, it touches upon the limits of current paradigms in NLP, where corpus linguistics is perfected through increasingly complex models trained on increasingly massive corpora (Bender et al., 2021).
Revised: Firstly, it touches upon the limits of current paradigms in NLP, where corpus linguistics is perfected through increasingly complex models trained on increasingly massive corpora (Bender et al., 2021).

Original: This approach may advance the identification of linguistic form , but might ultimately have little to say about the relation of meaning to the social world (Bender and Koller, 2020; Bisk et al., 2020).
Revised: This approach may advance the identification of linguistic form , but might ultimately have little to say about the relation of meaning to the social world (Bender and Koller, 2020; Bisk et al., 2020).

Original: Secondly, even with more modest ambitions, several NLP applications—e.g. with spatial (McKenzie and Adams, 2021) or historical (Kutuzov et al., 2018) data—require thatlinguistic patterns are related to other types of structure.
Revised: Secondly, even with more modest ambitions, several NLP applications—e.g., with spatial (McKenzie and Adams, 2021) or historical (Kutuzov et al., 2018) data—require that linguistic patterns be related to other types of structure.

Original: Thirdly, from the perspective of socioculturalresearch, meaning intrinsically varies as languageis used in different settings and indexed to differentsocial categories (Labov, 1972; Silverstein, 2003;
Revised: Thirdly, from sociological and sociolinguistic perspectives, meaning intrinsically varies as language is used in different settings and indexed to different social categories (Labov, 1972; Bourdieu, 1984;

Original:  Eckert, 2018; Bourdieu, 1984; Hovy, 2018).
Revised: Silverstein, 2003; Eckert, 2008; Hovy, 2018).

Original: Finally, without a broader sense of context, NLP andlanguage modeling in particular remains trappedin a paradigm where language is always treated asuniversal, making invisible both different speechcommunities (e.g. Nguyen et al., 2021) and thebiases of language ( e.g. Lu et al., 2020; Blodgettet al., 2020; Sap et al., 2020).
Revised: Finally, without a broader sense of context, NLP and language modeling in particular remains trapped in a paradigm where language is always treated as universal, making invisible both different speech communities (e.g. Nguyen et al., 2021) and the biases of language (e.g. Blodgett et al., 2020; Lu et al., 2020; Sap et al., 2020).

Original: After a brief history of distributional semantics,we outlinework that Harris and Firth did on distribution and, more broadly, on the scope of linguistic analysis.
Revised: After a brief history of distributional semantics, we outline Harris and Firth’s research on distribution and, more broadly, on the scope of linguistic analysis.

Original: We look at some of the ways inwhich NLP has tried to account for broader contextwithin the distributional paradigm.
Revised: We look at some of the ways in which NLP has tried to account for broader context within the distributional paradigm.

Original: We suggest thatexisting strategies can be understood in terms ofeither “comparative stratification” or “syntagmaticextension.”
Revised: We suggest that existing strategies can be understood in terms of either “comparative stratification” or “syntagmatic extension.”

Original: We conclude with thoughts on howre-reading Harris and, in particular, Firth might aidthe field of NLP with its current aporias.
Revised: We conclude with thoughts on why rereading Harris and, in particular, Firth might aid the field of NLP with its current aporias.

Original: If wordsshall be known by the company they keep, thenthe question follows: what  company do they keep?
Revised: If words shall be known by the company they keep, then the question follows: what kind of company do they keep?

Original: Are they found only with words and other linguistic elements, or do they mingle with other typesof entities?
Revised: Are they found only alongside linguistic elements, or do they mingle with other types of entities?

Original: Or, as Firth himself wrote: “Manydifferent answers could be given to the question‘Distribution of what, where and how?”’
Revised: Or, as Firth himself wrote: “Many different answers could be given to the question ‘Distribution of what, where and how?”’

Original: (Firth,1957a, v ).
Revised: (Firth, 1957a, v ).

Original:  Background: Distributional Semantics 
Revised:  Background: Distributional Semantics and NLP

Original: and NLP

Original: Distributional semantics has been an integral partof computational linguistics since the beginningsof the field, but in a discontinuous manner that encompasses at least two distinct eras.
Revised: Distributional semantics has been an fundamental part of computational linguistics since the beginnings of the field, but in a discontinuous manner encompassing at least two distinct eras.

Original: Firstly, duringthe 1950s and 60s, Harris was integral to the mathematization of linguistics  after theSecond World
Revised: Firstly, during the 1950s and 60s, Harris was integral to the mathematization of linguistics in the US after the

Original:  War (Rubenstein and Goodenough, 1965; Léon,2021).
Revised: Second World War (Rubenstein and Goodenough, 1965; Léon, 2021).

Original: Firth was skeptical of efforts to mechanizelinguistics,{+mechanize linguistics, 
+} but he nonetheless consulted for someof the early work on machine translation at Cambridge (Léon, 2007, 410).
Revised: Firth was skeptical of efforts to {+mechanize linguistics, 
+} but he nonetheless consulted for some of the early work on machine translation at Cambridge Language Research Group

Original: Secondly, when computational linguistics returned to its “empiricist” rootsin probabilistic methods and information theory inthe mid-80s and early 90s (Norvig 2012, 30; Léon2021, 141), Firth and Harris were alongside Shannon among the authors who were evoked, in an
Revised: (Léon, 2007, 410), which included Firth’s pupil, M.A.K. Halliday (Léon, 2021, 144) and shortly later the NLP pioneer Karen Spärck Jones (Léon, 2021, 89). Naturally, others also contributed to this first wave of distributional thinking, including Shannon (1945; 1948) with what might be considered one of the first language models, Warren

Original:  ACL “Special Issue on Computational Linguistics Using Large Corpora,” as foundational figures of a tradition that had been overshadowed for decades by the “rationalism” of characters such as Noam 
Revised: Weaver (1952) with an early proposal for distributional semantics, and Martin Joos (1950) with a statistical formulation of language as a symbolic system of conditional probabilities. Secondly, when computational linguistics returned to its “empiricist” roots in probabilistic methods and information theory in the mid-80s and early 90s (Norvig 2012; Léon 2021, 141), Firth and Harris accompanied Shannon among the authors who were invoked, in an ACL “Special Issue on Computational Linguistics Using Large Corpora,” as foundational figures of a tradition that had been overshadowed for decades by the “rationalism” of characters such as Noam Chomsky and

Original: Chomsky and Marvin Minsky (Church and Mercer, 1993, 15).
Revised: Marvin Minsky (Church and Mercer, 1993, 15).

Original: During this “corpus turn,” the rapid automation of linguistics was driven by a resumed connection with postwar computational linguistics and information theory (Léon, 2021, 3).
Revised: During this “corpus turn,” the rapid automation of linguistics was driven by a resumed connection with postwar computational linguistics and information theory (Léon, 2021, 3).

Original: However,important papers on non-neural lexical semantics from the era (Landauer and Dumais, 1997; Deerwester et al., 1989, 1990) and other vector space models or their applications (Schutze, 1992; Schütze, 1993;
Revised: However, the 1990s wave of vector semantics papers that used methods like singular-value decomposition (SVD) to produce early “dense vector” models of meaning like LSA (Deerwester et al., 1989, 1990; Landauer and Dumais, 1997) and its derivatives (Hofmann, 1999;

Original: Schütze and Pedersen, 1993; Burgess, 1998; Blei et al., 2002) generally did not cite Firth orHarris.
Revised: Blei et al., 2002), HAL (Burgess, 1998), or the models of Schütze (Schütze, 1992, 1993; Schütze and Pedersen, 1993) generally did not cite Firth or

Original:  In short, while Firth and Harris were not regularly used as stand-ins for linguistic theory during this time of “count based” models (Baroni et al., 2014), a general revival of empiricism and distributional approaches to meaning did signal a potential resurgence of interest in their thinking.At the turn of the millennium, computational approaches to lexical semantics shifted toward predictive neural language modeling (Henderson, 2020).
Revised: Harris, although a few papers from that period did (Church and Hanks, 1989; Hindle, 1990). In short, while Firth and Harris were not regularly used as stand-ins for linguistic theory during the 1990s and early 2000s, a general revival of empiricism and distributional approaches to meaning signaled a potential resurgence of interest in their thinking.

Original: During the 2000s, the application of neural networks to language modeling tasks (e.g. Bengio et al., 2003) and the development of selfsupervision techniques (e.g. Raina et al., 2007) set the stage for the word embedding breakthroughs of the early 2010s (e.g. Mikolov et al., 2013).
Revised: During the 2000s, the application of neural networks to language modeling tasks (e.g. Bengio et al., 2003) and the development of selfsupervision techniques (e.g. Raina et al., 2007) set the stage for the word embedding breakthroughs of the early 2010s (e.g. Mikolov et al., 2013).

Original: By the end of the decade, the introduction of Long Short

Original: Term Memory networks (LSTMs) (e.g. Graves et al., 2013) and then of the Transformer model (Vaswani et al., 2017)had made way for the next breakthrough, the large-language modeling revolution (e.g.  Devlin et al., 2019; Peters et al., 2018).
Revised: By the end of the decade, the introduction of attention (Graves et al., 2013; Bahdanau et al., 2015) and then of the Transformer model (Vaswani et al., 2017) made way for the next breakthrough, the large-language modeling revolution (e.g. Peters et al., 2018; Devlin et al., 2019; Chowdhery et al., 2022).

Original: Following the introduction of the word2vec model and its powerful but “static” embeddings,
Revised: Following the introduction of the word2vec model and its powerful but “static” embeddings,

Original: Harris in particular was frequently cited (Bojanowski et al., 2017;
Revised: Harris in particular was frequently cited (Le and

Original: Le and Mikolov, 2014; Levy and Goldberg, 2014; Levy et al., 2015), often (but not always) along with Firth (Goldberg, 2017; Jurafsky and Martin, 2009, 2021; Hamilton et al., 2016;  Eisenstein, 2019; Lin and Dyer, 2010; Bruni et al., 2014).
Revised: Mikolov, 2014; Levy and Goldberg, 2014; Levy et al., 2015; Bojanowski et al., 2017), often (but not always) along with Firth (Bruni et al., 2014; Hamilton et al., 2016; Goldberg, 2017; Eisenstein, 2019; Jurafsky and Martin, 2021).

Original: However, despite an explosion of citations (Bisk et al., 2020, 8719), this interest has not been very engaged.
Revised: However, despite an explosion of citations (Bisk et al., 2020, 8719), this interest has not been very engaged.

Original: In fact, the canonization of Firth and Harris during this time is paradoxical.
Revised: In fact, the canonization of Firth and Harris during this time is paradoxical.

Original: On the one hand it seems that they are invoked tolend theoretical authority to a field that strugglesto lift its gaze from the latest state-of-the-art numbers (Manning, 2015; Bender and Koller, 2020).
Revised: On the one hand, it seems that they are invoked to lend theoretical authority to a field that struggles to lift its gaze from the latest state-of-the-art numbers (Manning, 2015; Bender and Koller, 2020).

Original: Yet,on the other hand, the unspoken conclusionfrom the ascent of connectionism and the languagemodeling revolutionin particular was that “learning from data made linguistic theories irrelevant”(Henderson, 2020, 6295).
Revised: Yet, the unspoken conclusion from the ascent of neural models and the language modeling revolution was that “learning from data made linguistic theories irrelevant” (Henderson, 2020, 6295).

Original: In other words, just as NLP seemed to lose interest with linguistic theory,it elevated two pioneering theoreticians to canonical status, but  withoutdeeply engaging  with theirwork.
Revised: In other words, just as NLP seemed to lose interest with linguistic theory, it elevated two pioneering theoreticians to canonical status, but seemingly without engaging closely with their work.

Revised: In fact, it often seems as if Firth and Harris are referenced in such a cavalier manner that it deflects attention from the field’s general lack of engagement with linguistic theory.

Revised: Meanwhile,

Original: Firth and Harris become figures who justifya relatively narrow conception of meaning, one thatis predominantly intra-linguistic, without much tosay about its uses in the world.
Revised: Firth and Harris became figures who justify a relatively narrow conception of meaning, one that is predominantly intra-linguistic, without much to say about its usage in social life.

Original:  Though Léon discusses the contrast between Harris and Firth in the context of corpus linguistics(2008) and their influence on the history of computational linguistics (2021), her work does notaddress the differences in their distributional theories and conceptions of “context,” nor the renewedand paradoxical significance of the two authorsfor language modeling.
Revised: This peculiar story has not been properly told. Though Léon discusses the contrast between Harris and Firth in the context of corpus linguistics (2008) and their influence on the history of computational linguistics (2021), her work does not address the differences in their distributional theories and conceptions of “context,” nor the renewed and paradoxical significance of the two authors for language modeling.

Original: In our contribution, weemphasize the gap between the ideas of Firth and Harris as well as thepotential insights a re-readingof their work offer for expanding the scope of computational semantics.
Revised: In our contribution, we emphasize the gap between the ideas of Firth and Harris as well as the insights a re-reading of their work offers for expanding the scope of computational semantics.

Original:  Harris’s distributional structuralism
Revised:  Harris’s distributional structuralism

Original: Few linguists contributed more to linguistic theorythan Zellig Harris (1909–1992), and not just forserving as  Chomsky’s doctoral advisor.
Revised: Few linguists contributed more to linguistic theory than Zellig Harris (1909–1992), and not just by serving as Noam Chomsky’s doctoral advisor.

Original: In fact, thetwo came to share little in common (Goldsmith,2005; Nevin, 2010); whereas Chomsky’s generative grammar repositioned linguistics as a cognitivescience seeking to understand, in so few words, theidealized mental representations and structures enabling language acquisition and production (e.g. Chomsky, 1972), Harris’ radically distributionalapproach to language saw the natural language corpus as the sole starting point from which a generallinguistic theory could arise.
Revised: In fact, the two came to share little in common (Goldsmith, 2005; Nevin, 2010). Whereas Chomsky’s generative grammar repositioned linguistics as a cognitive science seeking to understand, in so few words, the idealized mental representations and structures enabling language acquisition and production (e.g. Chomsky, 1972), Harris’ radically distributional approach to language effectively elevated the natural language corpus as the sole starting point from which linguistic theory could arise (Harris, 1951, 1; Harris et al., 1988, 2–3; Johnson, 2002, 143–144).

Original: This theory consisted of a linguistic structurebest understood as segmentable into a finite setof formal objects characterized bystructured, constrained patterns of correspondence with one another (Harris, 1951, 1954, 1991).
Revised: This theory consisted of a linguistic structure segmentable into a finite set of formal objects characterized by constrained patterns of correspondence with one another (Harris, 1951, 1954, 1991).

Original: Such patterns ofcorrespondence can be observed only in languagein-use, that is, in natural language corpora.
Revised: Such patterns of correspondence can be observed only in language-in-use, that is, in natural language corpora.

Original: In hisfoundational paper “Distributional Structure,” forexample, he provides a purely distributional account of how one might induce the semantic meanings of oculist, eye-doctor, and lawyer from the partial (in the case of oculist and lawyer) or nearly complete (in the case of oculist and eye-doctor) overlap in their observed “environments” of use (Harris, 1954, 156–157).
Revised: In his foundational paper “Distributional Structure,” for example, Harris (1954, 156–157) provides a purely distributional account of how one might induce the semantic meanings of oculist , eyedoctor , and lawyer from the partial (in the case of oculist and lawyer ) or nearly complete (in the case of oculist and eye-doctor ) overlap in their observed “environments” of use.

Original: Thisgeneral approach applies to other levels of linguistic analysis, such as morphophonemics (e.g., Harris, 1954, 155).
Revised: This approach applies to other levels of linguistic analysis, such as morphophonemics (e.g., Harris, 1954, 155).

Original: Rather than producing a series of descriptive rules for the distribution of each phoneme, morpheme, or word,the goal was to achieve greater parsimony  by grouping these elements into structurally equivalent classes sharing the same distributional rules, compounding elements in a hierarchical manner.
Revised: Rather than producing a series of descriptive rules for the distribution of each phoneme, morpheme, or word, greater parsimony was sought by grouping these elements into structurally equivalent classes— categorized by their relationships as “operators” and “arguments” in Harris’ later work (e.g. 1968; 1988; 1991)—sharing the same distributional rules, compounding elements in a hierarchical manner.

Original: In this section we draw attention to three aspects of Harris’s distributional linguistics: the relationship it posits between meaning and form; assumptions about heterogeneity among speakers of the same dialect; and the concept of sublanguages.
Revised: In this section we draw attention to three aspects of Harris’s distributional linguistics: the relationship it posits between meaning and form; assumptions about heterogeneity among speakers of the same dialect; and the concept of sublanguages.

Original: 3. Meaning and form
Revised: 3. Meaning and form

Original: A result of this vision of a linguistics—concerned above all with the structural and probabilistic constraints governing the combination of formal elements—is that the discipline would be fully autonomous, not only from biology and psychology but even from semantics, phonetics, and logic, “complete without intrusion of other features such as history or meaning” (Harris 1954, 146; Goldsmith 2005, 725–726).
Revised: A result of Harris’ vision of a linguistics— concerned above all with the structural and probabilistic constraints governing the combination of formal elements—is that the discipline would be fully autonomous, not only from biology and psychology but even from semantics, phonetics, and logic, “complete without intrusion of other features such as history or meaning” (Harris 1954, 146; Goldsmith 2005, 725–726).

Original:  Harris’s reasoning drew on the particular status he gave to linguistics among all the sciences.
Revised: 2 Harris’s reasoning depended on the particular status he gave to linguistics among all the sciences.

Original: Taking language as its object of inquiry, linguistics lacks—unlike other sciences— a metalanguage external to language, i.e., to its object of inquiry (Nevin, 1993, 358–359).
Revised: Taking language as its object of inquiry, linguistics lacks—unlike other sciences—a metalanguage external to language, i.e., to its object of inquiry (Harris, 1991, 4–5; Nevin, 1993, 356).

Original: Even if some sort of symbolic system is used, “those symbols will have to be defined ultimately in a natural language” (Harris, 1991, 274), as the surging demand for interpretability and explainability in NLP has made evident (see e.g. Danilevsky et al., 2020).
Revised: Even if some other symbolic system is used, “those symbols will have to be defined ultimately in a natural language” (Harris, 1991, 274), as the surging demand for interpretability and explainability in NLP has made evident (see e.g. Danilevsky et al., 2020).

Original: Language is consequently not a “code” of “forms” that correlate with some meanings outside of it.
Revised: Language is consequently not a “code” of “forms” that correlate with some meanings outside of it.

Original: It has no “one-to-one” conformity “with some independently discoverable structure of meaning” (Harris, 1954, 152).
Revised: It has no “one-to-one” conformity “with some independently discoverable structure of meaning” (Harris, 1954, 152).

Original: Instead, it is a system related to, but also independent from, thought (Nevin, 1993, 361–363).
Revised: Instead, it is a system related to, but also independent from, thought (Harris, 1991, 383–384; Nevin, 1993, 361–363).

Original: While all human activity is meaningful, the particular meanings of language are constituted by its form, not correlated to it (Nevin, 1993, 394).
Revised: While all human activity is meaningful, the particular meanings of language are constituted by its form, not correlated with it (Nevin, 1993, 394).

Original: Meaning thus understood is about departures from equiprobability in the distribution of theseconstraints (Harris, 1991, 23).
Revised: Meaning thus understood is about departures from equiprobability in the distribution of these constraints (Harris, 1991, 23).

Revised: These departures “define a range of meaning for each morpheme, which includes its meaning in each occurrence”.

Revised: Nonetheless, shared environments do not necessarily imply shared meanings: “bumped into a pole can be said after a minor accident or after a chance meeting with an East European” (Harris, 1951, 191).

Original: Linguistic inquiry is,in the final instance, about evaluating the efficiencyof different grammars and their ability to model thestatistical constraints imposed upon the distributionof different linguistic elements.
Revised: The ultimate goal of Harris’s linguistic inquiry is to evaluate the efficiency of different grammars and their ability to model the statistical constraints imposed upon the distribution of different linguistic elements (Goldsmith, 2005, 723–725).

Original: Harris held thatlanguage was a “detached pattern” (Harris, 1941,295)—information that was  socially transmissibleand hence constitutive of new types of sociallyshared and conventional meaning (360) (Nevin,1993, 360, 365)—and linguistics could at best discover different incomplete grammars.
Revised: Harris held that language was a “detached pattern” (Harris, 1941, 295)—information that was public and socially transmissible and hence constitutive of new types of socially shared and conventional meaning (Harris, 1991, 342–345, 377–382; Nevin, 1993, 360, 365)—and linguistics could at best discover different incomplete grammars (Harris, 1991, 31– 36).

Original: Though linguistics might provide insights about meaning anddiscourse, or about cultural practices, such findingswould not bear directly on linguistics per se .
Revised: Though linguistics might provide insights about meaning and discourse, or about cultural practices, such findings would not bear directly on linguistics per se (Goldsmith, 2005, 725–726).

Original: Indeed, while Harris acknowledges that our sense ofword meaning is aided by “extra-linguistic situational information,” words “beyond the immediatesituation” are “on their own” (Harris, 1991, 368).
Revised: Indeed, while Harris acknowledged that our sense of word meaning is aided by “extra-linguistic situational information,” words “beyond the immediate situation” are “on their own” (Harris, 1991, 368).

Original: However, Harris’s method is not completely detached: Searching for a method to segment speech,he notes that the similarity of elements “reducesultimately to the similarity of sound segments under repetition,” implemented through “the pair test”in which native speakers are asked to discriminatebetween sound segments (Harris, 1954, 158–159),producing an observational primitive that is “moreeasy to control than data on meaning” (e.g. Harris,1951, 20).
Revised: However, Harris’s method is not completely detached. Searching for a method to segment speech, he notes that the similarity of elements “reduces ultimately to the similarity of sound segments under repetition,” implemented through “the pair test” in which native speakers are asked to discriminate between sound segments (Harris, 1954, 158–159), producing an observational primitive that is “more easily controlled than data on meaning” (e.g. Harris, 1951, 20).

Original:  Variation, or lack thereof
Revised: 3. Variation, or lack thereof

Original: Harris’s view of language and linguistics, isolatedfrom the vagaries of social interaction and variation,is obviously difficult to reconcile with a sociolinguistic perspective.
Revised: Harris’s view of language and linguistics, isolated from the vagaries of social interaction and variation, is obviously difficult to reconcile with a sociolinguistic perspective.

Original: As Harris writes in hisMethods of Structural Linguistics (Harris, 1951, 9), hisapproach is meant to describe a homogeneous dialect, which “[i]n most cases...presents no problem,since the whole speech of the person or communityshows dialectal consistency.”
Revised: As Harris writes in his Structural Linguistics (1951, 9), his approach is meant to describe a homogeneous dialect, which “[i]n most cases...presents no problem, since the whole speech of the person or community shows dialectal consistency.”

Original: Referencing this passage,  dialectology pioneer William Labov (2006,5) argues that “the inconsistency found in most
Revised: Referencing this passage, sociolinguist and dialectology pioneer William Labov (1966/2006, 5) argues that “the inconsistency found in most

Original: New York City idiolects is so great that the firstalternative of Harris is impossible, and the secondimplausible.”
Revised: New York City idiolects is so great that the first alternative of Harris is impossible, and the second implausible.”

Original: In other words, even at the level ofthe speaker, Harris’s idealized, unvarying idiolectdoes not hold up to empirical scrutiny.
Revised: In other words, even at the level of the speaker, Harris’s idealized, unvarying idiolect does not hold up to empirical scrutiny.

Original: Rigorousconsideration of factors that Harris would deemextra-linguistic (class, race, interactional roles, etc.) are indeed essential to produce a systematic description of linguistic structure.
Revised: Rigorous consideration of factors that Harris would deem extra-linguistic (class, race, interactional roles, etc.) are indeed essential to produce a systematic description of linguistic structure (Labov, 1972).

Original: From the sociolinguistic perspective, Harris’s vision of a linguistic science fully isolated from the “intrusion” of non-verbal social life would never obtain the systemicity to which it aspired.
Revised: From the sociolinguistic perspective, Harris’s vision of a linguistic science fully isolated from the “intrusion” of non-verbal social life would never obtain the systemicity to which it aspired.

Original: The critiquefrom sociolinguistics highlightsmore fundamental issues in Harris’sstructuralist perspective.
Revised: This sociolinguistic critique highlights fundamental limitations of Harris’s perspective.

Original: Language is viewed primarily through the distributional restrictions imposed by convention, rather than  “stylistic practice” and the ways in which speakers “make social-semiotic moves, reinterpreting variables and combining and recombining them in a continual process” (Eckert, 2012, 94).
Revised: Language is viewed primarily through the distributional restrictions imposed by convention, rather than by “stylistic practice” and the ways in which speakers “make social-semiotic moves, reinterpreting variables and combining and recombining them in a continual process” (Eckert, 2012, 94).

Original: We can study changes in discourse, as Harris himself did in an impressive volume on structures in immunological theory over time (Harris et al., 1988), but not how people make those changes, or indeed the way in which “variation itself  is systemic” (Deleuze and Guattari, 1987, 93).
Revised: We can study changes in discourse, as Harris himself did in an impressive volume on structures in immunological theory over time (Harris et al., 1988), but not how people make those changes, or indeed the way in which it is “the variation itself that is systemic” (Deleuze and Guattari, 1987, 93).

Original: 3. Sublanguages
Revised: 3. Sublanguages

Revised: As noted above, Harris’ revival in the 1990s was driven by the new interest in “corpus linguistics” of large corpora, a research paradigm that was partly derived from Harris’ notion of “sublanguages” (Léon, 2021).

Original: Harris introducedthe concept of sublanguages in his book Mathematical Structures of Language , defined as “certain proper subsets of a language [which] may be closed under some or all of the operations defined in the language, and [which] thus constitute a sublanguage of it” (Harris, 1968, 152).
Revised: Harris introduced sublanguages in his book Mathematical Structures of Language , defined as “[c]ertain proper subsets of a language [which] may be closed under some or all of the operations defined in the language, and [which] thus constitute a sublanguage of it” (Harris, 1968, 152).

Original: The sublanguage is a set of sentences which are a subset of the sentences of the “whole” language.
Revised: A sublanguage is a set of sentences which are a subset of the sentences of the “whole” language.

Original: However, the grammatical constraints of the sublanguage are not necessarily those of the whole language; rather, their grammars intersect (Harris 1968, Ch 11; Kittredge and Lehrberger 1982, 1).
Revised: However, the grammatical constraints of the sublanguage are not necessarily those of the whole language; rather, their grammars intersect (Harris 1968, Ch 11; Kittredge and Lehrberger 1982, 1).

Original: In application the term has come to refer primarily to the grammar and vocabulary unique to or characteristic of a particular professional or scientific field (e.g. Harris, 1988), an influential concept for early information retrieval research (Sager, 1975, 1981).
Revised: In application the term has come to refer primarily to the grammar and vocabulary unique to or characteristic of a particular professional or scientific field (e.g. Harris, 1988), an influential concept for early information retrieval research (Sager, 1975, 1981).

Original:  Harris believed that sublanguages could be neatly identified using the distributional methods of his general linguistic program.
Revised: 4 Harris believed that sublanguages could be neatly identified using the distributional methods of his general linguistic program.

Original:  Firth’s contextual semantics
Revised:  Firth’s contextual semantics

Original: Something similar to the sociolinguistic critique towards Harris could be articulated from another perspective as well, namely through the work of J.R. Firth.
Revised: Something similar to the sociolinguistic critique of Harris could be articulated from a different perspective, namely, through the work of J.R. Firth.

Original: Firth (1890–1960)—professor of General Linguistics at the University College of London and the first holder of a chair in that subject at any British university—independently formulated a distributional theory of lexical semantics.
Revised: Firth (1890–1960)—professor of General Linguistics at the University College of London and the first holder of a chair in that subject in Britain— independently formulated a distributional theory of lexical semantics.

Original: However, unlike Harris, Firth refused to treat semanticsseparate from pragmatics, and words apart fromtheir broader “context of situation.”
Revised: However, unlike Harris, Firth refused to treat meaning separately from pragmatics, and words apart from their broader “context of situation.”

Original: (Robins, 1997,205–208)
Revised: (Robins, 1997, 205–208)

Original: Firth never published a fully articulated exposition of his general theory of language (Robins1997, 216; Thomas 2011, 180) and today, mostof his work is  out of print and mostly unavailableonline.
Revised: Firth never published a fully articulated exposition of his general theory of language (Robins 1997, 216; Thomas 2011, 180) and today, all of his work is not only out of print but also mostly unavailable online.

Original: Not understood by “the contemporary scientism” of American descriptivist linguistics andits pioneers like Harris, Firth was mostly ignoredon the other side of the Atlantic (Pandit 1970, 280; Palmer 1968, 2).
Revised: Not understood by “the contemporary scientism” of American descriptivist linguistics and its pioneers like Harris, Firth was mostly ignored on the other side of the Atlantic (Palmer 1968, 2; Pandit 1970, 280).

Original: Unlike many of his Americancontemporaries, Firth did not draw mainly fromcognitive psychology and logic—the latter  which Firth thought had “taken the heart out of language”(Firth, 1957a, 186)—but from the work of Polish anthropologist Bronislaw Malinowski (Robins,1997, 211).
Revised: Unlike many of his American contemporaries, Firth did not draw mainly from cognitive psychology and logic—the latter of which Firth thought had “taken the heart out of language” (Firth, 1957a, 186)—but from the work of Polish anthropologist Bronisław Malinowski (Robins, 1997, 211).

Original: Here, we focus on the evolution of histhoughts on meaning and collocation, as well ashis notions of context of situation and restrictedlanguage .
Revised: Here, we focus on the evolution of his thoughts on meaning and collocation, as well as his notions of context of situation and restricted language .

Original: 4. Meaning by collocation
Revised: 4. Meaning by collocation

Original: To Firth, the purpose of linguistics is to “studymeaning in its own terms” (Firth 1968a, 145 ; Senis 2015, 289).
Revised: To Firth, the purpose of linguistics is to “study meaning in its own terms” (Firth 1968b, 145; Senis 2015, 289).

Original: The famous phrase about the company that words keep concerned a particular “modeof meaning,” “meaning by collocation”.
Revised: The famous phrase about the company that words keep concerned a particular “mode of meaning”: “meaning by collocation” (Firth, 1957b, 194).

Original: Anticipating vocabulary now ubiquitous in NLP, Firththought that this level of meaning could be foundby examining the “habitual collocations” of wordsand the “word material” in which they are “mostcharacteristically embedded” (Firth, 1957b, 11–12).
Revised: Anticipating vocabulary now ubiquitous in NLP, Firth thought that this level of meaning could be found by examining the “habitual collocations” of words and the “word-material” in which they are “most characteristically embedded” (Firth, 1957c, 11–12).

Original: Meaning by collocation was an abstraction of syntagmatic relations (Firth 1957b, 12; Oyelaran 1967,444) that went beyond “mere juxtaposition,” stating instead “an order of mutual expectancy”
Revised: Meaning by collocation was an abstraction of syntagmatic relations (Oyelaran 1967, 444) that went beyond “mere juxtaposition,” stating instead “an order of mutual expectancy ” and “mutual prehension” (Firth, 1957c, 12).

Original: (Firth,1957b, 12) and “mutual prehension.” While mutualexpectancy could be understood similarly to Joos’s(1950) conditional probabilities of occurrence orthe concept of PMI (Fano, 1949), the notion of“prehension” originates in the work of philosopherand mathematician Alfred North Whitehead (1957,1968, see also Butt 2013) and concerns the manner in which one entity grasps another and makesit part of its own experience (Christian 1959, 12; Bryant 2011, 136).
Revised: While mutual expectancy could be understood similarly to Joos’s (1950) conditional probabilities of occurrence or the concept of Pointwise Mutual Information (Fano, 1961), the notion of “prehension” originates in the work of philosopher and mathematician Alfred North Whitehead (1938, 1957; see also Butt 2013) and concerns the manner in which one entity grasps another and makes it part of its own experience (Christian 1959, 12; Bryant 2011, 136).

Original: Drawing on Whitehead’s “modes of thought”(Whitehead 1968; Butt 2001, 1812; Butt 2019, 28),
Revised: Drawing on Whitehead’s “modes of thought” (1938; see also Butt 2001, 1812, Butt 2019, 28),

Original: Firth advocated a type of “polysystemic” linguistic analysis that was interested in different, congruent modes of meaning, whether phonetic, phonological, syntactic, or semantic, but always situated in broader social context (Firth 1957b, 27, 30 ; Robins 1997, 214).
Revised: Firth advocated a type of “polysystemic” linguistic analysis that was interested in different, congruent modes of meaning, whether phonetic, phonological, syntactic, or semantic, but always situated in broader social context (Firth 1957c, 27, 30 ; Robins 1997, 214).

Original: In stark contrast to Harris, Firth explicitly rejected any efforts to create “unity in linguistics”  or one system of analysis.
Revised: In stark contrast to Harris, Firth explicitly rejected any efforts to create “unity in linguistics” (Firth, 1968d, 48) or one system of analysis.

Original: Citing the later Wittgenstein (1953; Firth 1957b, 11), Firth was mainly interested in the concrete use of language, reversing theSaussurian schema (De Saussure, 2011) in which language ( la langue ) is a system “external to and on a different plane from individual phenonema,” including the concrete instances ( la parole ) of language use (Firth1950, 44–45; Firth 1949, 144).
Revised: Citing the later Wittgenstein (1953, Firth 1957c, 11), Firth was mainly interested in the concrete use of language, reversing the schema of Ferdinand de Saussure (1916/2011) in which language ( la langue ) is a system “external to and on a different plane from individual phenonema,” including the concrete instances ( la parole ) of language use (Firth 1949, 400; Firth 1950, 44–45).

Original: While his final ideas matured considerably later, Firth initially articulated his ideas about semantics in two 1935 papers, one on semantics (Firth, 1935a) and one on phonology (Firth, 1935b), using the term “contextual distribution” in both (see in particular Firth 1935a, 55).
Revised: While his final ideas matured considerably later, Firth initially articulated his ideas about semantics in two 1935 papers, one on semantics (Firth, 1935a) and one on phonology (Firth, 1935b), using the term “contextual distribution” in both.

Original: However, Firth (1957b, 18) ultimately disavowed this  distributional theory—which was not too dissimilar from Harris’s—as “useful” but inadequate to act as the “main principle” in a theory “of structures involving the statement of the values of the elements of structure by reference to systems.”
Revised: However, Firth (1957c, 18) ultimately disavowed this initial distributional theory—which was not too dissimilar from Harris’s—as “useful” but inadequate to act as the “main principle” in a theory “of structures involving the statement of the values of the elements of structure by reference to systems.”

Original: By distinguishing between system (syntagm) and structure (paradigm), Firth wanted to highlight two operational principles necessary for meaning by collocation: (1) substitution within “the same level of abstraction,” and (2) commutation across different levels (Robins, 1953, 140).
Revised: By distinguishing between system (syntagm) and structure (paradigm), Firth wanted to highlight two operational principles necessary for meaning by collocation: 1) substitution within “the same level of abstraction,” and 2) commutation across different levels (Robins, 1953, 140).

Original: Only substitution that does not produce commutation in a sequence, indicates similarity of value or function (Firth 1968b, 23; Firth 1957b, 15).
Revised: Only substitution that does not produce commutation in a sequence, indicates similarity of value or function (Firth 1957c, 5; Firth 1968c, 23) 5 .

Original: I.e. two words are only substitutable—and hence similar in function and meaning—ifwe ensure that their values do not commute across a particular sentence.
Revised: Two words are only substitutable—and hence similar in function and meaning—if their values do not commute across a particular sentence.

Original: Substitution, then, does not equal synonymy.
Revised: Substitutability, then, does not equal synonymy.

Original: Take, for example, the following two phrases containing (a) prepositional and (b) adverbial uses of the word “by” :
Revised: Take, for example, the following two phrases containing a) prepositional and b) adverbial uses of the word “by”:

Original: (a) They go by night.
Revised: (a) They go by night.

Original: (b) They go by night after night.
Revised: (b) They go by night after night.

Original: Now, “by” could be replaced by the word “past” without commuting the meaning of the other words in (b).
Revised: Now, “by” could be replaced by the word “past” without commuting the meaning of the other words in (b).

Original: However, replacing “by” with “past” would commute with the rest of (a) in an impossible way (Firth, 1968b, 23–24).
Revised: However, replacing “by” with “past” would commute with the rest of (a) in an impossible way (Firth, 1968c, 23–24).

Original: This demonstrates how substitution concerns the relationship between “by” and “past” as two elements at the same level ofanalysis—i.e., lexical units—but in order to account for commutation, we need to look beyondthis level to other levels of abstraction.
Revised: This demonstrates how substitution concerns the relationship between “by” and “past” as two elements at the same level of analysis—i.e., lexical units—but in order to account for commutation, we need to look beyond this level to other levels of abstraction.

Revised: Firth’s conception of collocation and his frequent nods to Whitehead were part of his “monistic” approach that rejected the division between mind and body (Firth 1957c, 2; Palmer 1968, 5) and all the other dualities—language and thought, word and idea, signifier and signified, expression and content (Firth, 1951, 86)—that characterized the structuralist linguistics of his time.

Revised: He similarly rejected any notion of linguistics as “a theory of universals for general linguistic description ” (Firth, 1957c, 21).

Revised: Anticipating contemporary concerns about language diversity in NLP (e.g. Bender, 2019), Firth called for the Western scholar to “de-Europeanize himself” and the English scholar, due to the universal use of his language, to “de-Anglicize himself” (Firth 1968a, 96; Senis 2015, 274).

Original: 4. Context and connection
Revised: 4. Context and connection

Original: Diverging from structuralist linguistics, Firth suggested that a text should always be given a “renewalof connection with experience” (Firth, 1957b, 29).
Revised: Diverging from structuralist linguistics, Firth suggested that a text should always be given a “renewal of connection with experience” (Firth, 1957c, 29).

Original: This notion of meaning was influenced by Malinowski, for whom Firth worked as an assistant earlyin his career (Plug, 2008, 346) and from whom heborrowed the notion of “context of situation” (Firth1935a, Robins 1997, 211).
Revised: This notion of meaning was influenced by Malinowski, for whom Firth worked as an assistant early in his career (Plug, 2008, 346) and from whom he borrowed the notion of “context of situation” (Firth 1935a, Robins 1997, 211).

Original: In Malinwoski’s view,meaning was more than just a dyadic relationshipbetween a word and its referent, “a multidimensional and functional set of relations between theword in its sentence and the context of its occurrence” (Robins, 1971, 35).
Revised: In Malinwoski’s view, meaning was more than just a dyadic relationship between a word and its referent, “a multidimensional and functional set of relations between the word in its sentence and the context of its occurrence” (Robins, 1971, 35).

Original: However, while Malinowski’s view on meaning was entirely functionaland hyper-local, Firth employed the notion of “context of situation” as a necessary abstraction, not asa shorthand for things in themselves (Firth 1950, 7; Palmer 1968, 6).
Revised: However, while Malinowski’s view on meaning was entirely functional and hyper-local, Firth employed the notion of “context of situation” as a necessary abstraction, not as a shorthand for things in themselves (Firth 1950, 43, Palmer 1968, 6).

Original: Context of situation is derivedfrom an analytical choice, “a set of categories inordered relations abstracted from the life of man inthe flux of events, from personality in society,”  prehending something of importance and bracketingthe rest.
Revised: Context of situation is derived from an analytical choice, “a set of categories in ordered relations abstracted from the life of man in the flux of events, from personality in society,” (Firth, 1957c, 30) prehending something of importance and bracketing the rest.

Original: It is, then, not necessarily about restrictingthe meaning of every utterance to a specific timeand place, but about defining “an abstract set ofsemantically relevant categories, abstracted frommultitudes of actual situations, to which unique particulars could be referred.”
Revised: It is, then, not necessarily about restricting the meaning of every utterance to a specific time and place, but about defining “an abstract set of semantically relevant categories, abstracted from multitudes of actual situations, to which unique particulars could be referred.”

Original: (Robins, 1971, 41–42)
Revised: (Robins, 1971, 41–42)

Original: Firth called for the linguist to focus on “attestedlanguage text duly recorded” (Firth, 1957b, 29–30),accounting for both the interior relations of the textas well as its associated context of situation and itsinterior relations.
Revised: Firth called for the linguist to focus on “attested language text duly recorded”, accounting for a text’s associated context of situation and its interior relations.

Original: (Firth, 1957b, 30)
Revised: (Firth, 1957c, 29–30)

Original: Firth was famously opaque with the exact operationalization of his concepts, including context ofsituation, but he did provide a detailed list of thedifferent contextual elements that a linguist shouldbring into relation during analysis (Firth, 1950, 43):
Revised: Firth was famously opaque with the exact operationalization of his concepts, including context of situation, but he did provide a detailed list of the different contextual elements that a linguist should bring into relation during analysis (Firth, 1950, 43).

Original: A. The relevant features of participants: persons,
Revised: These include the relevant features of participants (persons, personalities); their verbal and non-verbal actions; the relevant objects; and the effects of verbal action.

Original: personalities.
Revised: During Firth’s lifetime, the most thorough work that put his notion of context of situation to work was an ethnographic study by his student T.F. Mitchell in former Cyreneica (today Libya) on the language of buying and selling at the local markets of different cities and villages in the region.

Original: (i)
Revised: For

Original: The verbal action of  the participants. (ii) The non-verbal action of  the participants. B. The relevant objects.
Revised: Mitchell (1957, 32–33), contexts that might “correlate” with particular types of text included: the spatio-temporal situation of persons in the context; the activities of participants; the attitudes of the participants; their “personalities” such as specific trade of profession, geographical and class origins, educational standard, inter-relationship, and so on.

Original: C. The effect of  the verbal action.
Revised: It is worth noting that both Mitchell and M.A.K. Halliday—Firth’s student who synthesized much of his theory—used words such as “correlation,” “inference,” and “prediction” to describe the relationship between a text and its situational context, implying that a statistical extension of their approaches would not be completely unreasonable. In fact, Halliday himself suggested as much, when he in the early 1990s made efforts to bridge his branch of linguistics with the nascent field of corpus linguistics (Halliday, 1991).

Original: In conclusion: Firth’s famous quote itself refers to collocation, while his notion of “context” implies something much broader, “the whole conceptual meaning” (Firth, 1957b, 11).
Revised: In conclusion: Firth’s famous quote itself refers to collocation, while his notion of “context” implies something much broader, “the whole conceptual meaning” (Firth, 1957c, 11).

Original: Context is the ground against which the figure of the text must be understood, no matter (per Harris) how “detached” its pattern might be (e.g. Auer, 1996).
Revised: Context is the ground against which the figure of the text must be understood, no matter (per Harris) how “detached” its pattern might be (e.g. Auer, 1996).

Original: Without context, collocation captures only one narrow “mode of meaning.”
Revised: Without context, collocation captures only one narrow “mode of meaning.”

Original: 4. Restricted languages
Revised: 4. Restricted languages

Revised: Like Harris, Firth’s revival in connection with 1990s corpus linguistics was related to his attempts to respond to practical needs of empirical research.

Original: Expanding upon his functional understanding of language, Firth developed his notion of “restricted languages” in the 1950s (1957b, 10; Léon 2007, 7).
Revised: Expanding upon his functional understanding of language, Firth developed his notion of “restricted languages” in the 1950s (Léon 2007, 7).

Original: In a posthumously published essay, he(1968c, 207) describes social actors as “collect[ing] a varied repertory of interlocking roles” corresponding to a “constellation of restricted languages.”
Revised: In a posthumously published essay, he describes social actors as “collect[ing] a varied repertory of interlocking roles” corresponding to a “constellation of restricted languages” (Firth, 1968e, 207).

Original: As people shift between locally contextualized roles, they draw upon their “repertory” of restricted languages with specialized vocabulary and discursive styles that both reflect and constitute these contexts.
Revised: As people shift between locally contextualized roles, they draw upon their “repertory” of restricted languages with specialized vocabulary and discursive styles that both reflect and constitute these contexts.

Original: Thus one might speak of a “restricted language of science, sport, defense, industry, aviation, military services, commerce, law and civil administration, politics, literature, etc.”
Revised: Thus one might speak of a “restricted language of science, sport, defense, industry, aviation, military services, commerce, law and civil administration, politics, literature, etc.”

Original: (Léon, 2008, 261).
Revised: (Léon, 2008, 261).

Original: As such, the concept of restricted language is now generally seen as a precursor to the concept of “register,” which was taken up by subsequent sociolinguists and linguistic anthropologists (e.g. Halliday, 1968; Gumperz and Hymes, 1972; Agha, 2005).
Revised: As such, the concept of restricted language is now generally seen as a precursor to the concept of “register,” which was taken up by subsequent sociolinguists and linguistic anthropologists (e.g. Halliday, 1968; Gumperz and Hymes, 1972; Agha, 2005).

Original: In proposing restricted languages as the proper object of descriptive linguistic analysis, Firth was making a broader theoretical point against, on the one hand, “the monosystemic view of language” of neo-Bloomfieldians like Harris and “pointless discussions on metalanguage” on the other, for metalanguage could be reanalyzed as a “restricted language of linguistics” itself (Léon, 2007, 9).
Revised: In proposing restricted languages as the proper object of descriptive linguistic analysis, Firth was making a broader theoretical point against, on the one hand, “the monosystemic view of language” of neo-Bloomfieldians like Harris and “pointless discussions on metalanguage” on the other, for metalanguage could be reanalyzed as a “restricted language of linguistics” itself (Léon, 2007, 9).

Original: Simply put, a descriptive linguistics which privileges restricted languages also necessarily privileges contexts of situation as an essential dimension of variation that allows social meaning to inhere in language.
Revised: Simply put, a descriptive linguistics which privileges restricted languages also necessarily privileges contexts of situation as an essential dimension of variation that allows social meaning to inhere in language.

Original:  Discussion: Words in new company
Revised:  Discussion: Words in mixed company

Original: Often cited, together or separately, with the same purpose of justifying a distributional view on se-  mantics, Firth and Harris nonetheless present vastlydifferent views on language and meaning.
Revised: Often cited, together or separately, to justify a distributional appraoch to semantics, Firth and Harris nonetheless offer differing views on language and meaning.

Original: In Harriswe find a rigorous formalism that treats language asa “detached pattern.” Language is not a “code”, butfunctions instead as a particular system of meaning.
Revised: Harris offers us a rigorous formalism that treats language as a “detached pattern”— not a “code”, but a particular system of meaning.

Original: Firth, by comparison, left a much more scatteredlegacy and many of his insights were only systematizedand formalized by his students.
Revised: Firth, by comparison, left a much more scattered legacy that was only systematized by his students.

Original: Firth and Harris shared a concern about the dependence between the languages of and under description, butdrew different conclusions regarding the tensionbetween form and meaning in language.
Revised: Firth and Harris shared a concern about the lack of an external metalanguage of linguistics, but drew different conclusions from it.

Original: If Harris responded to this conundrum by creating onesystem without intrusion from extra-linguistic factors, Firth called for an investigation of languageas a “spectrum”  with different modes of meaningthat had to be addressed through multiple levelsofanalysis, without a “unity of linguistics.” Analysisshould start by stating the context of situation andproceed from there to decide which other levelsofanalysis are relevant.
Revised: If Harris responded to this conundrum by creating one hierarchically organized system without intrusion from extra-linguistic factors, Firth called for an investigation of language as a “spectrum” (Firth, 1951, 76) with different modes of meaning that had to be addressed through multiple levels of analysis— starting with the context of situation and proceeding from there to decide which other levels are relevant (Firth, 1950, 44).

Original: Firth’s distributional theoryhas been contrasted unfavourably as “frequencybased” to the “recursive dependencies” developedby Harris (Habert and Zweigenbaum, 2002, 205).
Revised: Firth’s distributional theory has been described unfavourably as based on frequent co-occurrence, in contrast to the recursive dependencies developed by Harris (Habert and Zweigenbaum, 2002, 205).

Original: For Harris, the meaning of a word depends on itsset (Harris, 1991, 17) such that, for example, thewords “divide” and “multiply” operate on the word“cell” (and vice versa) in the same way, producing essentially the same meaning (Harris 1988, 62;see also Harris 1991, 5–6).
Revised: For Harris, the meaning of a word depends on its set (e.g. Harris, 1991, 17) such that, for example, the words “divide” and “multiply” operate on the word “cell” (and vice versa) in the same way, producing essentially the same meaning (Harris 1988, 62).

Original: However, Firth’s final method of substitution and commutation  establishes complex, multidimensional criteria fordistributional contrast as well as a framework forunderstanding polysemy.
Revised: However, Firth’s final method of substitution and commutation also establishes complex, multidimensional criteria for distributional contrast as well as a framework for understanding polysemy.

Original: Though less formalizedthan Harris, Firth’s approach ultimatelly also treatslinguistic elements as operators with specific setsof elements in a complex hierarchy, including theextra-linguistic context of situation. 
Revised: Though less formalized and less obviously recursive than Harris’, Firth’s approach can, arguably, also be read as treating linguistic elements as operators and arguments defined by their sets in a complex hierarchy (Firth, 1950, 44; Firth, 1951, 76 “at a series of congruent levels” (Firth, 1957c, 29–30) with different “bands of abstraction” (Firth, 1968d, 49), including the extra-linguistic context of situation. Harris and Firth both understand any linguistic analysis as incomplete, for Harris always a pursuit of the least description” (Harris, 1988, 3)—i.e.

Revised: , best “grammar” or model—for Firth always grounded in the social construction of facts, without any possibility of “complete axiomatization” (Firth, 1968d, 44–45).

Original: Nonetheless, from the perspective of empiricalwork, especially decades after their time, Firth and Harris also share similarities.
Revised: From the perspective of empirical work, especially decades after their time, Firth and Harris also share similarities.

Original: Both rejected thementalism that was so prevalent during their time.
Revised: Both rejected the mentalism that was so prevalent during their time.

Original: They were both revived as empiricist originatorsduring the rise of statistical learning in the 1990sand their respective work on restricted languagesand sublanguages was important for the rise ofcorpus linguistics and  largely conflated in serviceof practical concerns  (Léon, 2008).
Revised: They were both revived as empiricist originators during the rise of statistical learning in the 1990s, and their respective work on restricted languages and sublanguages largely conflated in service of the practical concerns of corpus linguistics (Léon, 2008).

Original: They both included intheir theories what we might call a “reality principle,” a final arbiter of meaning outsideof form: the pair test for Harris, the context of situation for Firth.
Revised: Their theories both included in what we might call a “reality principle,” a final arbiter of meaning outside of form: the pair test for Harris, the context of situation for Firth.

Original: However, as NLP tries to move beyond the written corpus, we might reassess the details and differences between Firth and Harris in order to understand the practical limits and possibilities of the distributional view on language.
Revised: The former grounds linguistics in the smallest possible unit of analysis as understood by the native speaker, the latter in social actions and objects.

Original: In light of recent calls to extend the “world scope” (Bisk et al., 2020) of NLP and to move towards pragmatic notions of meaning, we might find Harris’s rigorous but rigid system somewhat limiting and Firth’s appeals to different contexts helpful.
Revised: In light of recent calls to extend the “world scope” (Bisk et al., 2020) of NLP and to move towards pragmatic notions of meaning, it might make sense to balance Harris’s formalism and Firth’s pluralism.

Original: Though Firth warned us against overextending linguistics, he was generous with the company that words could keep.
Revised: Though Firth warned us against overextending linguistics, he was generous with the company that words could keep.

Original: They mingled with each other, but also with events, objects, people, and indexical features such as time and space.
Revised: They mingled with each other, but also with events, objects, people, and indexical features such as time and space.

Original: And if NLP is ready to move beyond the corpus, then even Harris might acknowledge that when modeling language in “the immediate situation”—whether
Revised: And if NLP is ready to move beyond the corpus, then even Harris might acknowledge that when modeling language in “the immediate situation”—whether

Original: in online interactions or face-to-face communication—words are not on their own, that to judge the meaning of a combination of words, we can summon “the aid of some of the extra-linguistic situational information” (Harris, 1991, 368).
Revised: in online interactions or face-to-face communication—words are not on their own, that to judge the meaning of a combination of words, we can summon “the aid of some of the extra-linguistic situational information” (Harris, 1991, 368).

Original: In the next two subsections, we consider the ways in which NLP is already doing this, in order to highlight some  existing strategies for broader contextualization.
Revised: In the following subsections, we consider two ways in which NLP is already doing this, in order to highlight some already existing strategies for broader contextualization.

Revised: We call these strategies “comparative stratification” and “syntagmatic extension.”

Original: 5. Comparative stratification
Revised: 5. Comparative stratification

Original: Corpus linguistics emerges from the question of what kind of company words keep, depending on their context.
Revised: Corpus linguistics emerges from the question of what kind of company words keep, depending on their context.

Original: The issue of context was motivated  the introduction of corpus linguistics both for students of Firth—who considered restricted languages as a way of handling context—and for Harris’s sublanguages,“concrete discourse structures, which could be applied to real scientific texts, contextually situated and suitable for being processed automatically” (Léon, 2021, 149–150).
Revised: The issue of context was motivated by the introduction of corpus linguistics both for students of Firth—who considered restricted languages as a way of handling context—and for Harris’s sublanguages, which were “contextually situated and suitable for being processed automatically” (Léon, 2021, 149–150).

Original: However, beyond just studying the restricted corpus, we might also consider the ways in which large datasets can be “stratified”—a type of implicit factorization, if you will—systematically dividing them into sub-corpora that arerigorously and statistically studied in relation to each other.
Revised: However, beyond just studying the restricted corpus, we might also consider the ways in which large datasets can be “stratified,” systematically dividing them into subcorpora that are studied in relation to each other.

Original: Here, the company that words keep among each other is limited for analytical purposes, but in a manner that implicitly tell us what kind of context different kinds of words keep as company.
Revised: Here, the company that words keep among each other is limited for analytical purposes, but in a manner that implies a relationship between that “company” and situational context.

Original: Diachronic embeddings are especially representative of this approach.
Revised: Diachronic embeddings are especially representative of this approach.

Original: By stratifying timestamped data into a number of intervals, traininga separate model for eachinterval and then aligning the embeddings using either “second-order embeddings”or methods such as linear transformations (Kutuzov et al., 2018), the analyst can effectively givewords a temporal “context of situation.”
Revised: By stratifying timestamped data into a number of intervals, training separate models for each and then aligning the embeddings using either “second-order embeddings” or methods such as linear transformations (Kutuzov et al., 2018), the analyst can effectively represent a temporal “context of situation.”

Original: Thisgeneral approach works with both static (Hamiltonet al., 2016) and contextual embeddings (Martincet al., 2020).
Revised: This approach works with both static (Hamilton et al., 2016) and contextual embeddings (Martinc et al., 2020).

Original: While  most commonly used in theliterature on semantic shifts (e.g., Garg et al., 2018;
Revised: While variants of this approach are most commonly used to study semantic shifts (e.g., Garg et al., 2018;

Original: Kozlowski et al., 2019; Mendelsohn et al., 2020),it could plausibly be used to stratify a dataset usingother variables such as space (e.g. Yan et al., 2017),online communities (Lucy and Bamman, 2021),persons (Yao et al., 2020), or domains (Spindeet al., 2021).
Revised: Kozlowski et al., 2019; Mendelsohn et al., 2020; Brunila and LaViolette, 2021), it could plausibly be used to stratify a dataset according to other variables such as space (e.g. Bamman et al., 2014; Gong et al., 2020), online communities (Lucy and Bamman, 2021), persons (Yao et al., 2020), or domains (Spinde et al., 2021).

Original: Words still only keep the companyof other words, but by limiting their company wecan implicitly introduce other participants in theanalysis.
Revised: Words still only keep the company of one another, but by limiting their company we implicitly introduce other participants in the analysis.

Original: 5. Syntagmatic extension
Revised: 5. Syntagmatic extension

Original: Recall that for Firth, meaning by collocation andconsiderations of a “typical context of situation”were exercises in abstraction, with collocation being an abstraction at the syntagmatic level.
Revised: Recall that for Firth, meaning by collocation and considerations of a “typical context of situation” (Firth, 1950, 44) were exercises in abstraction, with collocation being an abstraction at the syntagmatic level.

Original: Insteadof restricting the company words keep, we mightfollow Firth’s recommendation to consider them inwider company “of the same abstract nature”
Revised: Instead of restricting the company words keep, we might follow Firth’s recommendation to consider them in wider company “of the same abstract nature”

Original: (Firth,1950, 7).
Revised: (Firth, 1950, 7).

Original: In vector semantics, this would imply thatwe explicitly introduce different contextual factorsin the same vector space with our words, giving allparticipants in the company ontological equality.
Revised: In vector semantics, this would imply that we explicitly introduce different contextual factors in the same vector space with our words, endowing them all with ontological equality.

Original: The paragraph vectors introduced by Le and
Revised: The paragraph vectors introduced by Le and

Original: Mikolov (2014) as an extension of their skipgramalgorithm (Mikolov et al., 2013) are representativefor this approach.
Revised: Mikolov (2014) as an extension of the earlier Skipgram algorithm (Mikolov et al., 2013) are representative for this approach.

Original: In practice, this method extendsthe syntagmatic chain of words by introducing avector for the document as a new paradigmatic element.
Revised: In practice, this method extends the syntagmatic chain of words by introducing a vector for the document as a new paradigmatic element.

Original: This type of “global context” (Grbovic and Cheng, 2018) couldin principle be anything and include several paradigmatic elements, as we can seein the research on multi-modal embeddings (e.g. Baroni, 2016).
Revised: In principle, this type of “global context” (Grbovic and Cheng, 2018) could be anything and include several paradigmatic elements, as we can see in the research on multi-modal embeddings (e.g. Baroni, 2016) and generative modeling (e.g. Ramesh et al., 2022).

Original: For example, models have been developed that include demographic (Garimella et al. ,2017) or persona (Li et al., 2016) vectors in the embedding space, such that intra-textual relations areaccompanied by information about speakers’ so- cial categories.
Revised: 8 Models have been developed that include demographic (Garimella et al., 2017) or persona (Li et al., 2016) vectors in the embedding space, such that intra-textual relations are accompanied by information about speakers’ social categories.

Original: However, implemented with static embeddings and without some additional grammar restrictions, these context vectors essentially add only a “bag of contexts.”
Revised: However, implemented with static embeddings and without some additional grammar restrictions, these context vectors essentially add only a “bag of contexts.”

Original: For static embeddings, additional grammar constraints could be introduced, as had been done in research on Pointof-Interest (POI) data in the the domain of geosemantics, where researchers constrained contextual vectors using spatial variograms (Yan et al., 2017).
Revised: For static embeddings, additional grammar constraints could be introduced, as was done in research on Point-of-Interest (POI) data in the the domain of geosemantics, where researchers constrained contextual vectors using spatial variograms (Yan et al., 2017).

Original: Beyond static embeddings, large-language models and their dynamic embeddings could either be pretrained (with the appropriate dataset) orat least finetuned on data with text associated to different contextual variables.
Revised: Beyond static embeddings, large-language models and their dynamic embeddings could either be pretrained (with the appropriate dataset) or finetuned on data with text associated with different contextual variables.

Original: This would realize the proposal that Halliday made in the early 1990s when he suggested an extension of the language modeling schema from the early work of Shannon, to a model with “global probabilities, those of the grammar of English, and the locally conditioned probabilities, those of this or that particular register” (Halliday, 1991, 37).
Revised: This would realize the proposal that Halliday made in the early 1990s when he suggested an extension of the language modeling schema from the early work of Shannon, to a model with “global probabilities, those of the grammar of English, and the locally conditioned probabilities, those of this or that particular register” (Halliday, 1991, 37).

Original:  Conclusions
Revised:  Conclusions

Original: This paper revisited the theories of the two most well-known progenitors of  distributional semantics.
Revised: This paper revisited the theories of the two most well-known progenitors of the distributional approach to meaning in NLP.

Original: Recognizing the ongoing debate on how to bring NLPin general and language modeling in particular beyond the corpus, we provide a thorough account of the  distributional theories that are most frequently evoked in order to justify the modeling of meaning through departures from equiprobability in the company that words keep.
Revised: Recognizing the open question of how to bring NLP beyond the corpus, we offer a thorough account of the two distributional theories that are most often invoked to justify the modeling of meaning through departures from randomness in the company that words keep.

Original: Comparing the work of Harris and Firth—who both published their major work before the rise of the internet and its corpora—we find two distinct theories of distribution: one formal and mathematical, treating language as a particular type of detached information, another more schematic and anthropological, treating language as a functional spectrum which always emanates from a particular context of situation.
Revised: Comparing the work of Harris and Firth—who both published their major work before the rise of the internet and its corpora—we find two distinct theories of distribution: one formal and mathematical, treating language as a particular type of detached information, another more schematic and anthropological, treating language as a functional spectrum which always emanates from a particular context of situation.

Original: The legacies of both Firth and Harris can be seen in the current paradigm of corpus linguistics, but in the domain of distributional semantics, it is Harris’s ethos that dominates, despite Firth providing its most famous tagline.
Revised: The legacies of both Firth and Harris can be seen in the current paradigm of corpus linguistics, but in the domain of distributional semantics, it is Harris’s ethos that dominates, despite Firth providing its most famous tagline.

Original: Moving forward, we suggest that semantic modeling take more cues from Firth, and consider the context of situation and the wide variety of company that words can keep as crucial challenges with which the field must grapple.
Revised: Moving forward, we suggest that semantic modeling take more inspiration from Firth, and consider the context of situation and the wide variety of company that words can keep as crucial sites of innovation for the field. Doing so may not involve following a finite set of steps or flowchart.

Revised: Rather, we humbly suggest that the field may be enriched by thoughtful and creative re-engagements with the intellectual traditions from which it has historically drawn.

Revised: This does not imply abandoning the rigor provided by Harris.

Revised: On the contrary, we find that

Revised: Firth and Harris would probably have agreed that any model or “grammar” is inevitably incomplete and partial.

Revised: No universal model is possible, despite the large-language modeling fervor, nor will there be one theory of language to guide us.

Revised: There are only the partial perspectives and the inevitable choice of adapting one.

Revised: Acknolwedgements

Revised: We would like to thank Dan Jurafsky and Jacqueline Léon for their gracious and helpful comments which greatly improved the paper.

Revised: Mikael Brunila is funded by the Kone Foundation, and Jack LaViolette is funded by the Mellon Foundation.

Revised: We thank them for their generous support.