Prosodic segmentation for parsing spoken dialogue (2021)

Abstract: Parsing spoken dialogue poses unique difficulties, including disfluencies and unmarked boundaries between sentence-like units. Previous work has shown that prosody can help with parsing disfluent speech (Tran et al. 2018), but has assumed that the input to the parser is already segmented into sentence-like units (SUs), which isn’t true in existing speech applications. We investigate how prosody affects a parser that receives an entire dialogue turn as input (a turn-based model), instead of gold standard pre-segmented SUs (an SU-based model). In experiments on the English Switchboard corpus, we find that when using transcripts alone, the turn-based model has trouble segmenting SUs, leading to worse parse performance than the SU-based model. However, prosody can effectively replace gold standard SU boundaries: with prosody, the turn-based model performs as well as the SU-based model (91.38 vs. 91.06 F1 score, respectively), despite performing two tasks (SU segmentation and parsing) rather than one (parsing alone). Analysis shows that pitch and intensity features are the most important for this corpus, since they allow the model to correctly distinguish an SU boundary from a speech disfluency – a distinction that the model otherwise struggles to make.

## introduction parsing spoken dialogue poses unique difficulties: spontaneous speech is full of disfluencies, including false starts, repetitions, and filled pauses. in addition, speech transcripts lack punctuation, which would otherwise help signal the boundaries of sentence-like units (sus). 1 because of these difficulties, current parsers struggle to accurately parse english speech transcripts, even when they handle other english text well. however, research has shown that prosody can help with at least one of these problems, improving parsing performance for speech that contains disfluencies @xcite . in this work, we hypothesize that incorporating prosodic features from the speech signal can actually help with both of these problems: not only parsing disfluent speech, but also parsing speech that isn't segmented into sus. other researchers have augmented parsers with prosodic features, but always with the assumption that the parser has access to gold su boundaries, which cannot be assumed in a deployed speech application. for example, @xcite ; @xcite and @xcite incorporated prosody into statistical parsers or parse rerankers, with mixed results. more recently, @xcite and @xcite found that prosody improved an end-to-end neural parser, with the most significant gains in disfluent sentences. parsing without access to gold su boundaries is much more difficult: @xcite showed that parsing quality depends on the quality of the sentence segmentation. furthermore, finding su boundaries is not as simple as finding long pauses in speech, as we demonstrate below. we hypothesize that access to prosodic features will help an english parser that has to both parse and correctly identify su boundaries (which we call su segmentation). we test this hypothesis by inputting entire dialog turns to a neural parser without gold su boundaries. we call this the turnbased model, and compare it to an su-based model, which assumes gold su boundaries and parses one su at a time. we use turns as our input unit because they resemble the input a dialog agent would receive from a user. following @xcite and others, we use a human-generated gold transcript instead of an automatic speech recognition ## background: prosody and syntax prosodic signals divide speech into units @xcite . the location and type of these prosodic units are determined by information structure @xcite , disfluencies @xcite , and to some extent, syntax @xcite . some psycholinguistic research shows that in experimental conditions, speakers can use prosody to predict syntax -for example, that english speakers can use prosody to determine where to attach a modifier or prepositional phrase, or how to correctly group coordinands (e.g., @xcite ; @xcite ; @xcite ). however, @xcite argues that english speakers often "fail to exploit" this prosodic information even when it is present, so it isn't actually a signal for syntax in practice. many computational linguists have experimented with this possible link between syntax and prosody by incorporating prosody into syntactic parsers (e.g., @xcite ; @xcite ; @xcite ; tran et al. ( 2018) ). these models have had mixed success: for example, @xcite found that prosody was at best a neutral addition to their model, while @xcite found that prosody helped rerank pcfg output. one possible reason that prosody is only somewhat effective in previous research is that prosodic units below the level of the su do not always coincide with traditional syntactic constituents @xcite . 2 in fact, the only prosodic boundaries that consistently coincide with syntactic boundaries are the prosodic boundaries at the ends of sus @xcite . the prosodic boundaries at the end of sus are more distinctive (i.e., tending to correspond to longer pauses and more distinctive pitch and intensity variations) and less likely appear in any other location. these features make prosody a reliable signal for su boundaries, even though it is an unreliable signal for syntactic structure below the su level. some researcheres have used this correlation between prosody and su boundaries to help in su boundary detection. examples of su segmentation models that found prosodic cues were important include @xcite ; @xcite ; @xcite ; @xcite , who all used traditional statistical models (e.g., hmms, finite state machines, and decision trees), and @xcite , who used a neural model. @xcite and @xcite also looked at downstream parsing accuracy on the same corpus we use. like us, @xcite don't use gold su boundaries, but direct comparison is impossible because they use asr output instead of human transcriptions and a different metric for parse performance (sparseval; @xcite ). however, they show that having access to gold su boundaries increases the sparseval score from 78.5 to 82.3, which shows that parsing without gold su boundaries is difficult. however, in some research areas, prosody is less frequently used for su detection. some asr corpora and applications segment at relatively arbitrary boundaries such as long silences or even regular intervals (e.g., @xcite ). other applications, such as speech translation, do require syntactically coherent input, but even there, systems targeting sus have often used only textual features @xcite . systems for restoring punctuation from asr output must identify su boundaries to correctly insert sentence-final punctuation, but these systems are typically evaluated on rehearsed monologues (such as ted talks) or read speech, which largely lack disfluencies (e.g., @xcite ). here, we show that prosody is primarily helpful for distinguishing su boundaries from disfluencies, so although some of these systems have used prosody (e.g., @xcite ), text-only systems are very competitive (e.g., @xcite ; @xcite ). even when su boundaries are already known, other research in parsing conversational speech has shown that prosody helps identify and correctly handle disfluencies. @xcite found that prosody only modestly affects parsing of fluent sus, but has a marked effect on disfluent sus. this accords with other previous work that has found that prosody is helpful in disfluency detection @xcite we discuss the relationship between prosody and disfluencies in greater detail in section 6, including how prosody helps the model not to confuse disfluencies and su boundaries, as shown in figure 1 above. ## task and data we use the american english corpus switchboard nxt (henceforth swbd-nxt) @xcite . we choose this corpus mainly so we can compare performance with @xcite and @xcite , as well as other earlier proba-bilistic models such as @xcite . swbd-nxt comprises 642 dialogues between strangers conducted by telephone. these dialogues are transcribed and hand-annotated with penn treebankstyle constituency parses. we preprocess the transcripts to remove punctuation and lower-case all letters, making the input more like an asr transcript that would be used in a deployed application. the transcript divides the corpus into sus and turns. since these sus may be sentences or other syntactically independent units such as sentence fragments, we use the generic term 'sentence-like unit' (su). a turn is a contiguous span of speech by a single speaker. turns are hand-annotated in swbd-nxt, but for a deployed dialog agent, a turn is simply whatever contiguous input the user gives. not all turns in the swbd-nxt contain more than one su: of a total 60.1k turns, 35.8k consist of a single su. the remaining 24.3k contain more than one su; the majority (52.4 percent) of these contain just two sus. the average number of sus per turn is 1.82. we follow the general approach of @xcite , but where they parse a single su at a time, we give our parser a single dialog turn at a time for our turn-based model. the model returns constituency parses for the turn in the form of penn treebank (ptb)-style trees. in order to keep the output in the form of valid ptb trees, we add a top-level constituent, labelled turn, to all turns, however many sus they consist of. this example shows how the two sentences in (1) would be fused ## model we use the parser described in @xcite , directly extending the code base described in their paper. 3 the model is a neural end-to-end constituency parser based on kitaev and klein (2018)'s textonly parser, with a transformer-based encoder and a chart-style decoder based on @xcite and @xcite . this encoder-decoder is augmented with a cnn on the input side that handles prosodic features @xcite . for further description of the model and hyperparameters, see @xcite . the text is encoded using 300-dimensional glove embeddings @xcite . 4 of the four types of prosodic features described in section 3, pause and duration features are already token-level. however, pitch and intensity features are extracted from the speech signal at the frame level. in order to map from these frame-level features to a token-level representation, the pitch and intensity features pass through a cnn, and are then concatenated with the token-level pause and duration features. we follow @xcite in training each model 10 times with different random seeds. for the development set, we report the mean of these 10 models' performance. we then select the median model by development set performance, and use it to calculate test set results. for any further experiments, such as those discussed in section 6, we use the random seed for this median model. each model is trained for 50 epochs and use the epoch with highest development set performance. in addition to this end-to-end approach, we also report results for a pipeline approach. for the pipeline, we first segment the speech into sus using a modified version of the parser architecture: we keep the encoder the same, but we change the decoder so that it only does sequence labelling, and we frame the su segmentation task as a sequence labelling task. we then use the su-based parser to parse the resulting sus. we report the model's performance with and without prosodic features during the segmentation and parsing steps. ## results we compare the turn-based f1 performance of our parser to a replication of the su-based performance described in @xcite and @xcite . the turn-based parser's task is also more complex: it has to perform both su segmentation and parsing, rather than parsing alone. this gives the turn-based parser novel ways to make errors by splitting a turn into the wrong number of sus. however, prosody brings the turn-based parser up to the level of the su-based parser, even though the turn-based model's task is more complex. ## distinguishing disfluencies and su boundaries one effect of disfluencies is that the text-only model tends to confuse certain kinds of disfluencies for su boundaries, as illustrated in figure 1 . ## conclusion our experiments show that parsing english speech transcriptions without gold su boundaries is difficult for our parser: its f1 score drops by about 4 percentage points compared to a model with gold su boundaries. incorrect su segmentation causes a large part of this damage, though other errors in tree construction also play a role. we show that we can undo this damage by giving our parser prosodic information. importantly, prosody helps by allowing the parser to distinguish disfluencies from su boundaries. these results argue for giving prosodic information to parsers in deployed applications, where no su boundary annotations are available, including dialog agents. furthermore, our experiments show that even limited prosodic features help a great deal: for our english data, pitch information alone is not significantly worse than pitch, intensity, pause, and word duration information combined. this means that incorporating the right kind of prosodic information can potentially lead to significant gains.