Information-theoretic analysis of disfluencies in speech
Abstract
This study proposes and examines an information-theoretic measure of planning inincremental speech production, and investigates the effects of planning, predictability, and interference-based measures on lexical substitution errors.
We then presenta rate-distortion theoretic model of speech production that explicates how thesefactors affect the production of lexical substitution errors
Introduction
Spontaneous speech is punctuated with disfluencies that reflect delays or interruptions in languageproduction mechanisms [1, 2, 3, 4].
One common kind of disfluency is a lexical substitution, where aspeaker says one word (which we term a distractor ) and then corrects it to another word (which weterm a target )
For example, see the following naturally-occurring utterance from the Switchboardcorpus [6]:(1) and I believe that mental acuity is easy to sustain maintain if you just simply continue toexercise your mind
Here the distractor is sustain , which the speaker corrects to maintain .
More generally, severalprominent models of speech production have attributed hesitations and disfluencies of various kindsto difficulties in selecting what word to say next [7, 8, 9].
Here we develop an information-theoretic analysis of lexical substitution errors in a dataset ofnaturally-occurring disfluencies in the Switchboard corpus, applying a new measure of planning.
Weleverage large language models to estimate the relevant information quantities, and perform a targetedanalysis that predicts which word is likely to appear as a distractor in each context.
Finally, weexplicate our results within the framework of a rate–distortion theoretic model of speech production.
1.1 Background and related work
Computational characterizations of lexical selection have emphasized the effects of word frequencyand predictability on the ease of retrieving the appropriate word from memory.
Low frequency wordswere not only associated with delayed production in naming experiments [10, 11, 12], but were alsomore likely to be preceded by disfluencies in naturalistic speech [13, 14].
Similarly, words thatwere less predictable conditional on the past (low forward predictability) or within the surroundingcontext (low contextual predictability) were also more likely to be preceded by hesitations, repetitions,and corrections [1, 15, 13, 16, 17].
Intriguingly, disfluencies and slowdowns in speech were alsoassociated with backward transitional probability: the probability of a word given its followingcontext [2, 18]; this is usually considered to be related to a speaker’s plans for future production.
Inaddition to being a strong predictor of fillers and corrections [19, 20], backward predictability has also been associated with delays caused by the reactivation of past material to aid retrieval of a targetcompatible with the planned future [21].
A complementary line of experimental research in word production has also demonstrated thatdelays or errors in selecting a target can arise due to the activation accrued by its semantic and/orphonological neighbors [22, 23, 24].
The semantic interference effect, in particular, has been arobust finding in the Picture Word Interference paradigm, where the degree of semantic relatednessbetween a pictorial target and an orthographic distractor predicts a delay in retrieving the target [25].
Similarly, the presentation of a phonologically-related distractor has been associated with an increasein activation of the segments that the distractor shares in common with the target [26].
Hence, higherproximity between the target and distractor along semantic and/or phonological dimensions correlatesto a greater degree of co-activation during processing.
1.2 Planning measure
Integrating perspectives from both lines of inquiry, our analyses of lexical substitution errors incorporate the effects of frequency, incremental production, and planning along with distance-basedmetrics such as phonological and semantic distance.
We propose an information-theoretic measure ofplanning based on Pointwise Mutual Information (PMI) [27].
In particular, we use the PMI of a wordx with the future context c f given the past context c p : pmiFP = ln p ( x | c f ,c p ) p ( x | c p ) , where p ( x | c f , c p )is the probability of x given both the preceding and following contexts and p ( x | c p ) is its forwardpredictability.
Unlike backward predictability, pmiFP does not assume that the future is plannedindependent of the past.
Rather it serves as a measure of planning that (i) does not commit to aparticular planning order and (ii) uses the information provided by the past context to estimate theassociation between the word and the future context.
Therefore, pmiFP can accommodate both linear[28, 29] and hierarchical planning [30, 31].
Previous works investigating backward predictabilityhave not considered the integration of past and future context represented in pmiFP.
In Section 4, wegive a more detailed theoretical justification for the use of this measure.
Methods
We develop a regression model that predicts whether a given word x , as opposed to any other wordin the vocabulary, is the distractor that the speaker selects and eventually overrides after production.
Specifically, we examine how proximity to the target, frequency, incremental or forward predictability,and planned production guide the selection of a particular distractor given an utterance context.
Measures
Frequency :
We use the frequencies from the SUBTLEXus corpus [32] to estimate unigram probability p ( x ) of a given word x
Predictability-based measures : We estimate the forward, backward, and contextual probabilitiesof targets and distractors using XLNet [33], a large transformer-based model trained on both causaland masked modeling objectives.
We select utterances where the length of both the preceding andfollowing contexts exceed one word.
To estimate p ( x | c p ) and p ( x | c f ) for a continuation x , weprovide as input the past context c p and the future context c f (in reverse) respectively.
To estimatep ( x | c p , c f ) , we provide the entire bidirectional utterance context ( c p , c f ) to the model with a maskapplied to x in order to indicate the word to be predicted (see Appendix A for examples).
Distance measures : We estimate phonological distance between the target and distractor based onthe Soundex algorithm [34] and semantic distance using cosine distance between pretrained GloVeembeddings [35].
Materials
For our analyses, we use Switchboard Annotations [36], a human-parsed subset of the Switchboardcorpus of conversational speech [37] with annotations marking reparanda and repairs.
In order toextract utterances with lexical substitutions, we identify two distinct signatures characteristic of theseerrors, namely (i) where the distractor or reparandum is immediately followed by the target or repair(1) and (ii) where a single word is substituted within a repeated phrase (Appendix A).
Model
We use a generalized linear mixed model (GLMM) [38] with a binary dependent variable isDistractorand the target identity as a random effect:
[Equation]
For each instance of a lexical substitution error (N = 1368), the model above is fit to predict whichword out of a subset of nontarget words (N = 150) selected from the vocabulary with varying degreesof semantic and phonetic relatedness to the target is the chosen distractor.
We hypothesize that a nontarget x that is more frequent, and has higher forward predictability andpmiFP will have a higher likelihood of being the distractor that receives the activation required forselection.
A word likely to be the distractor will also be co-activated with the target x T by virtueof their common semantic and/or phonological features.
Hence, we predict that the most probablecandidate for the distractor will also have lower semantic and/or phonological distance from x T .
Results
Both predictability and distance-based metrics emerged as significant predictors in the isDistractormodel.
As hypothesized, we observe a strong positive effect of frequency or unigram probability( β = 0 .
240 , p < 0 .
001 ), forward predictability ( β = 0 .
269 , p < 0 .
001 ), and pmiFP ( β = 0 .
223 , p <0 .
In contrast, we find a strong negative effect of semantic ( β = − 0 .
695 , p < 0 .
001 ) andphonological distance ( β = − 0 . 103 , p < 0 . 001 ).
We analyze the effects of individual predictors by examining the goodness-of-fit of the isDistractormodel with incrementally added features.
We observe that a model with pmiFP as the planningmeasure provides a better fit than one with backward predictability ( χ 2 = 800 .
5 , p < 0 .
001 ), themeasure used in previous work.
We also find that when both these predictors are included as planningmeasures, backward predictability has a negligible contribution to the increase in log likelihood(Figure 1).
Model sketch
To explicate our results, we present a sketch of a model of speech production within a rate–distortionframework, deriving an optimal probabilistic policy p g ( x | c p ) for the selection of a word x givena communicative goal g and current state
c p consisting of a sequence of previous words, subject toa constraint on the policy’s usage of information about the goal g [39, 40, 41, 42, 43, 44, 45, 46].
Following Todorov [39]’s KL-control framework, a policy is selected to maximize an averagefuture-discounted value-to-go
[Equation]
(1) where α ∈ [0 , 1] is a future-discount parameter, ⟨·⟩ p indicates an average over distribution p , p ∗ g ( x | c p )is an optimal policy, and ℓ ( x | c p ) is the local value of an action x given goal g and state c p .
The local value is given by the communicative value of a word x minus a control cost term which reflectsthe KL divergence between the policy p g and an automatic policy p 0 which is not conditional on thegoal g :
[Equation]
Communicative value R g is meant to signify how well the word x conveys the speaker’s intendedmessage to a listener.
Viewed within a rate-distortion theoretic framework, maximizing the communicative value corresponds to minimizing the distortion subject to the rate or control cost, whichquantifies the amount of information about the goal used to determine the optimal action.
Given thissetting, the policy that maximizes average value-to-go as derived by [39] is
[Equation]
We propose that the selection of the next word in speech is determined by Eq. 3.
The policy predictsthat what matters for the selection of a word x is (1) predictability given past context c p , (2) thecommunicative value R g of the word with respect to the current goal and state, and (3) the expectedvalue of words following x , a kind of planning effect.
4.1 Application to lexical substitution errors
We consider lexical substitution errors to reflect cases where there are two words (the target and thedistractor) that both receive high probability under Eq.
In that case, a word is likely to appear as adistractor whenever any of the three terms inside Eq.
3 are high.
We will see that these correspond toforward predictability, semantic and phonetic distance, and pmiFP respectively.
To understand the effect of the communicative value of word x , we consider the difference incommunicative value between the distractor x and target x T , ∆ R g
= R g ( x | c p ) − R g ( x T | c p ) .
This value differential ∆ R g corresponds to a negative communicative cost for saying x instead ofx T .
This cost should reflect both semantic distance, because semantically similar words will sharemany of their features relevant for communicative goals, and phonetic distance, because phoneticallysimilar words may be indistinguishable to a listener.
In order to draw out predictions from Eq.
3, we make three simplifying assumptions.
First, weassume production consists of a word x followed by a second word representing the entire future ofthe utterance, c f .
Second, we assume that the production of the future c f is deterministic given thecommunicative goal, and third, that the communicative value of c f is independent of the choice of x .
Under these assumptions, the policy in Eq.
3 simplifies to
[Equation]
Rewriting the probability of the future c f using Bayes’ Rule in terms of the probability of the currentword x given the future c f , p 0 ( x | c p , c f ) , we get
[Equation]
where the three terms correspond to factors that were shown to predict which words appear asdistractors in our corpus studies, including pmiFP.
See Appendix B for full derivations.
In addition,the frequency effects that we observe may be accommodated by adding an additional control cost foruse of information about the state c p , but we leave this modeling question to future work.
Conclusion
We have presented an analysis of lexical substitution errors in speech that predicts which wordssurface as distractors, and shown how the results can be accommodated in a rate–distortion controlframework.
The work opens the way for information-theoretic models of speech production that aretightly linked with rate–distortion models in other fields such as neuroscience and psychophysics.
A Appendix: Lexical substitution examples and data processing
Example of a lexical substitution within repeated material:1 so until i see the entire quote old guard of the soviet military of the soviet governmentcompletely roll over and disappear preferably buried i still consider them a threat
Example of utterance with multiple substitution errors preprocessed into contexts with single substitution errors:
A it depends on whether you whether we figure that we have a defense oriented military or anaggressive aggression oriented military
a it depends on whether [ you / we ] figure that we have a defense oriented military or anaggression oriented military b it depends on whether we figure that we have a defense oriented military or an [ aggression / aggressive ] oriented military
Examples of preprocessed XLnet inputs for estimating forward, backward, and masked probabilities:
A.3 it depends on whether <mask>
A.4 military oriented aggression an or military oriented defense
a have we that figure <mask>
A.5 it depends on whether <mask> figure that we have a defense oriented military or an aggres-
sion oriented military
Code for preprocessing, calculating metrics, and analysis can be found at: InfoTheoreticDisfModel
B Appendix: Derivation of pmiFP from the speech production model
Starting with policy of Eq.
3, we can get to Eq.
5 by assuming (1) production consists of a wordx followed by a second word representing the entire future of the utterance, c f , (2) production ofthe future c f is deterministic given the communicative goal, and (3) that the communicative valueof c f is independent of the choice of x .
Under these assumptions, the policy in Eq.
The firstassumption means that we only need to consider a finite time horizon with one future action.
Thesecond assumption means that the policy p ∗ g ( c | c p , x ) =
δ cc f .
The third assumption means that
R g ( c f | x, c p ) is the same for all x .
These assumptions represent a scenario where the speaker hashigh certainty about what they will say next, and where the next part of an utterance is relativelyindependent of the current part, for example at the end of a phrase or clause.
We start with the policy probability (setting α = 1 to save writing):
[Equation]
First, we rewrite R g ( x | c p )
=
R g
( x T |
c p ) + ∆ R g .
Because R g ( x T | c p ) is not a function of theaction under consideration x , it can be absorbed into the normalizing constant of Eq. 3, givingp
[Equation]
x |
[Equation]
p
[Equation]
∝
[Equation]
|
[Equation]
+
[Equation]
x ) ⟩
[Equation] Now using assumption (1), we can rewrite the policy in Eq.
3 as:
[Equation]
Using p ∗ g ( c | c p , x ) =
δ cc f for the expectation over future actions (assumption 2), we get
[Equation]
Because R g ( c f | c p , x ) is invariant to x (assumption 3), it can be absorbed into the implicit normalizing constant of Eq. 9, giving
[Equation]
Now applying Bayes’ rule to the term ln p
( c f | c p , x ) , and then applying logarithm rules, we get
[Equation]
Again, the last term is invariant to x , so it can be absorbed into the normalizing constant, leaving uswith the policy considered in the main text
[Equation]