Speech Disfluencies occur at Higher Perplexities (2020)

Abstract: Speech disfluencies have been hypothesized to occur before words that are less predictable and therefore more cognitively demanding. In this paper, we revisit this hypothesis by using OpenAI’s GPT-2 to calculate predictability of words as language model perplexity. Using the Switchboard corpus, we find that 51% of disfluencies occur at the highest, second highest, or within one token of the highest perplexity, and this distribution is not random. We also show that disfluencies precede words with significantly higher perplexity than fluent contexts. Based on our results, we offer new evidence that disfluencies are more likely to occur before less predictable words.

## introduction speech disfluencies occur naturally in spontaneous speech. disfluencies such as filled pauses @xcite , repetitions ('about about eight months ago'), and repairs ('about eight days i mean months ago') are estimated to occur in 6% of words in spoken english @xcite . @xcite hypothesized a relationship between disfluencies and the likelihood of the next word. he proposed that speakers have habitual ways of speaking, and the more unexpected a word given the context, the greater the likelihood of a disfluency. lounsbury did not test this hypothesis, saying that calculating the probability of every word in every context was, at the time, "an impossible task" @xcite ). since then, several studies have found that disfluencies occur before less predictable words @xcite @xcite . in this short paper, we revisit lounsbury (1954)'s hypothesis with newer nlp technology. using openai's gpt-2 @xcite , a neural language model, we calculate the predictability of words in disfluent sentences using language model perplexity. on the switchboard corpus @xcite , a large-scale spoken language dataset, we find that 22% of disfluencies precede the word with the highest perplexity (i.e. the lowest probability), 51% of disfluencies occur either at the highest, second highest, or within one token of the highest perplexity, and this distribution is not random. we also find that words preceded by a disfluency have significantly higher perplexity than words in fluent contexts. based on these findings, we offer new evidence of a relationship between disfluencies and less predictable words and conclude with suggested applications in nlp. ## related works cognitive load has often been studied as a factor that affects disfluencies @xcite . disfluencies are found more often before longer sentences @xcite , in new or unfamiliar contexts @xcite , and when speakers are performing more challenging tasks @xcite . @xcite suggested that the likelihood of a word also affects disfluencies. early studies evaluating disfluencies and the probability of a word used the shannon guessing technique @xcite or the cloze procedure @xcite . in these techniques, a spoken text was transcribed and given to judges with missing words. in the shannon guessing technique, judges guessed each word given the preceding context. in the cloze procedure, every nth word was deleted, so judges had both left and right context. the probability of a word was based on how many judges could correctly guess it. with the shannon guessing technique, @xcite found that words preceding a silent pause had lower probability than words without a pause. using the cloze procedure, studies found that disfluencies occurred before words with lower probability than fluent contexts @xcite . @xcite compared word frequency and contextual probability and found that disfluencies were more likely to occur before words with lower contextual probability, even when word frequency was held constant. with the release of larger-scale datasets, contextual probability no longer needed to be hand-annotated but could be calculated in a large enough corpus. the probability of a word given its previous word was estimated by counting the number of times both words occurred together divided by the number of times the first word occurred. this could be extended to the first n words in an n-gram language model. using large corpora and n-gram language models, studies again found that disfluencies occur before words with significantly lower probability @xcite , probability varies for different disfluency types and positions @xcite , and disfluencies tend to be longer before lower probability words @xcite . in related psychology works, eye-tracking studies have shown that listeners are primed to anticipate low frequency words when hearing a disfluency @xcite and speakers are more likely to hesitate before low frequency words @xcite . many psychology studies, however, focus more on word frequency rather than contextual probability. ## method to calculate the probability of a word, we use openai's gpt-2 @xcite , a large-scale neural language model that has achieved state-of-the-art results on various nlp tasks. gpt-2 uses a transformer-based architecture with 1.5 billion parameters and is trained on 8m documents. to our knowledge, we are the first to evaluate the relationship between disfluencies and contextual probability with a language model of this scale. to calculate probability, we use the perplexity returned by gpt-2. perplexity is the inverse probability of a sequence normalized by the number of words. due to the inverse, the lower the probability of a sequence, the higher the perplexity. the perplexity of a sequence of words w is calculated with the joint probabilities p of each word w using the formula: we use the implementation of gpt-2 available from huggingface @xcite . since gpt-2 is trained on written text and we experiment with spoken language, we fine-tune our gpt-2 model with the objective of predicting the next word given the previous words on 150k examples from switchboard @xcite ) for 2 epochs. this prevents our language model from predicting high perplexities for phrases that are common in spoken language but not in written language (e.g. "bye bye"). ## dataset for our experiemnts, we use the switchboard corpus @xcite , which was built by asking volunteers to speak to each other on the telephone about a topic assigned by a computer operator. we use the version released by @xcite foot_0 . we modify this dataset by including 'uh' and 'um' as disfluencies, which are included in the transcriptions but not labeled as disfluencies. from this dataset, we use 150k examples to fine-tune our gpt-2 model. we hold out 10,000 disfluent sentences from the model for our experiments. we filter this held-out disfluent set by: 1) removing sentences that are fewer than 5 words or longer than 15 words long (excluding disfluencies), as these are often incomplete (e.g. "oh uh") or run-on sentences, 2) removing sentences where the disfluency is the first or last word of the sentence since our experiments require left and right context to measure perplexity, and 3) removing sentences with non-consecutive disfluencies. this is done for simplicity and because a majority (70%) of sentences contain consecutive disfluencies. statistics about our disfluent set are shown in ## experiments for each utterance in our disfluent set, we create a fluent version by removing the disfluencies. using the fluent versions, we calculate the perplexity of each substring starting with the first two tokens of the sentence and adding one token at a time until the sentence is complete. an example is shown in first, we evaluate how often disfluencies occur at the most unpredictable word. given the list of perplexities for each sentence, we measure how often the maximum perplexity occurs at the word following the disfluency. we find that only 22% of disfluencies occur before the highest perplexity. we next calculate two more lenient measures of highest perplexity. we measure both how many disfluencies occur before the second highest perplexity and how many occur within one token of the highest perplexity. we find that 15% of disfluencies occur before the second highest perplexity, and 23% occur within one token of the highest perplexity. taken together, 51% of disfluencies occur either before the highest perplexity, the second highest perplexity, or within one token of the highest perplexity. given random chance, we would expect a disfluency at the highest perplexity 25% of the time, but we see it closer to 30%, showing that our results are likely not a result of random chance. is there a bias that causes this pattern? 41% of utterances in our dataset have a disfluency at the second word, and disfluencies are known to occur more often near the beginning of the sentence @xcite . 46% of our highest perplexities are also at the second word. to test if we have just identified a sentence-initial bias, we run our calculations excluding examples where the disfluency or maximum perplexity is at the second word. in this filtered dataset, we find that 18% of disfluencies occur before the maximum perplexity, and 41% occur before the highest, second highest, or within one token of the highest perplexity. while these numbers are lower than on the full dataset, they still show the same pattern. this suggests that our results are not just due to a bias, and may suggest that this pattern in perplexity is related to higher planning demands at the beginning of a sentence. finally, we measure if disfluencies occur at higher points of statistical uncertainty compared to fluent contexts. to calculate this, we normalize the perplexities of each sentence by dividing by the sum of all perplexities in the sentence. we compare the distribution of perplexities with no disfluency against perplexities with a disfluency. the results are shown in figures 4 and 5 and ## conclusion in this paper, we provide new evidence using a large-scale neural language model that disfluencies occur more often before less predictable words. we consider this relationship between perplexity and disfluencies useful for applications in nlp and see the following areas as promising future directions: