That’s Optional: A Contemporary Exploration of “that” Omission in E nglish Subordinate Clauses (2024)

Abstract: The Uniform Information Density (UID) hypothesis posits that speakers optimize the communicative properties of their utterances by avoiding spikes in information, thereby maintaining a relatively uniform information profile over time. This paper investigates the impact of UID principles on syntactic reduction, specifically focusing on the optional omission of the connector “that” in English subordinate clauses. Building upon previous research, we extend our investigation to a larger corpus of written English, utilize contemporary large language models (LLMs) and extend the information-uniformity principles by the notion of entropy, to estimate the UID manifestations in the usecase of syntactic reduction choices.

## introduction exploiting the expressive richness of languages, speakers often convey the same messages in multiple ways. a body of research on uniform information density (uid) puts forward the hypothesis that speakers tend to optimize the communicative effectiveness of their utterances when faced with multiple options for structuring a message. the uid hypothesis @xcite @xcite suggests that speakers tend to spread information evenly throughout an utterance, avoiding large fluctuations in the perunit information content of an utterance, thereby decreasing the processing load on the listener. the uid hypothesis has been used as an explanatory principle for phonetic duration @xcite , the choice between short-and long-form of words that can be used interchangeably, such as "info" and "information" @xcite , and word order patterns @xcite @xcite . our work studies how uid principles affect the phenomenon of syntactic reduction -the situation where a speaker has the choice of whether marking a subordinate clause in sentence with an optional subordinate conjunction (sconj) "that" or leave it unmarked, as in "my daughter mentioned @xcite he looked good". the only study that tested the uid hypothesis computationally in the context of syntactic reduction is @xcite , followed by @xcite , who studied the effect of multiple factors on the speaker choice of explicit or implicit "that" conjunction. investigating sentences with main clause (mc, e.g., "my daughter mentioned") and subordinate clause (sc, e.g., "[that] he looked good"), connected by the optional sconj, the authors found that uid optimization was the most prominent factor affecting a speaker choice of "that" omission. specifically, @xcite investigated 6700 sentences extracted from the switchboard spoken english dataset, and operationalized the uid principle by computing the surprisal (non-predictability) of the sc opening word (sc onset) using a statistical bigram language model computed from the corpus itself. our work studies the role of uid principle in syntactic reduction in multiple differing ways. first, we extend the investigation to a much larger corpus of informal written english collected from social media. second, we use contemporary large language models (llms) to estimate the operationalizations of information uniformity in syntactic reduction, suggesting the robustness of our findings. finally, inspired by the information-theoretic nature of uid and prior art @xcite , we extend the sc onset surprisal uid manifestation with the notion of sc onset entropy -the information entropy of llm distribution over sc opening word, conditioned on the main clause -factor that turns out to have a complementary and significant effect. the contribution of this work is, therefore, twofold: first, we collect and release a large and diverse corpus of nearly 100k sentences, where main and subordinate clauses are connected by the optional sconj "that". foot_0 second, we go above and beyond prior work by using transformer-based llms @xcite , thereby providing a sound empirical evidence for uid principles associated with syntactic reduction decision, shedding a new and interesting light on the manifestation of uid in spontaneous written language. ## methodology we define a set of factors that we were found to affect syntactic reduction choices @xcite , and further study the magnitude of their predictive power by casting the usecase as a classification scenario. we harness the power of contemporary llms for reliable computation of sc onset surprisal, as well as for computation of its complementary predictor: sc onset entropy. we define the following predictors: main clause (mc) length previous work suggested that the conjunction is likely to be spelled out explicitly in longer sentences; in particular after a longer main clause. this predictor is computed by the number of tokens preceding the (explicit or implicit) sconj. as an example, in the sentence "do you realize [that] i've never actually seen him at the office?", mc length will be assigned 3. subordinate clause (sc) length similar intuition suggests that the length of a subordinate clause (and more generally, the rest of the sentence) can be used as another predictor. in the example sentence above, sc length will be assigned 9. main verb frequency jaeger (2010) found negative correlation between the main clause verb frequency and the tendency to spell out "that" sconj. we compute the frequency of main verbs in all sentences as their relative count in the entire corpus of over 480k sentences (see section 2). sc subject distance this predictor is defined as the number of words at the sc onset up to and including the sc subject. multiple studies found positive correlation of this factor with the tendency to spell out sconj @xcite @xcite . we extract the sc subject using the nsubj annotation assigned by spacy's dependency parser to the subordinate clause subject. sc onset information density (id) @xcite and @xcite computed this factor by using the simplest possible estimation, where the information of the sc onset is only conditioned on the main verb, and is operationalized by the notion of surprisal: -log p(sc onset | main verb). all counts (and probabilities) were calculated from the dataset at hand. harnessing the power of modern pretrained llms, we define this predictor as the probability of sc onset, conditioned on entire main clause, namely -log p(sc onset | mc). notably, @xcite trained the bigram model in a controlled setting where all "that" conjunctions had been omitted. without this control, results may be circular, e.g., in cases where "that" is explicitly spelled out, the computation -log p(sc onset | mc) could be self-evident because "that" is normally inserted between mc and sc onset (recall that sc onset denotes the opening word of the subordinate clause, "that" excluded). since training a language model from scratch on corpora with omitted scs is often impractical, we marginalize out the presence of "that", re-defining the sc onset surprisal to be: this refined definition of sc onset surprisal eliminates the need to re-train a language model on a corpus where the sc "that" had been omitted. ## experimental results and discussion experimental setup we use the opt-125m autoregressive pretrained transformer model @xcite , roughly matching the performance and sizes of the gpt-3 class of models, for computation of sc onset surprisal and entropy. given a sentence prefix, we first extract next token logits and convert them to a probability distribution over the lexicon by applying the softmax function. sc onset surprisal was computed by applying the natural log on the sc onset token probability given the relevant sentence prefix. sc onset entropy was computed by applying the entropy equation (see section 3) on the outcome probability distribution. 3 estimating the contextual surprisal (or entropy) per word with decoder llms operating at the subword level is hard; we, therefore, approximate these metrics by computing the surprisal (or entropy) over the subwords. @xcite show that this is practically equivalent to computing a lower bound on the true contextual measurements. finally, logistic regression is used as a predictive model due to its effectiveness and intrepretability. ## conclusions we study the uid hypothesis manifestation in syntactic reduction using a large, diverse and carefully compiled corpus of english sentences with explicit or implicit "that" subordinate conjunction. harnessing the power of contemporary pretrained llms, we show that sc onset surprisal and entropy are the main factors affecting a speaker's choice to spell out the optional conjunction "that". last but not least, a large body of linguistic literature has studied the conditions under which complementizers (like "that" subordinate conjunction) can or cannot be omitted (inter alia erteschik-shir (1997); @xcite ). we believe that future work in this field should better engage with this literature, incorporating insights for more linguistically-informed approach to the task of syntactic reduction analysis.