The role of context in neural pitch accent detection in E nglish (2020)

Abstract: Prosody is a rich information source in natural language, serving as a marker for phenomena such as contrast. In order to make this information available to downstream tasks, we need a way to detect prosodic events in speech. We propose a new model for pitch accent detection, inspired by the work of Stehwien et al. (2018), who presented a CNN-based model for this task. Our model makes greater use of context by using full utterances as input and adding an LSTM layer. We find that these innovations lead to an improvement from 87.5% to 88.7% accuracy on pitch accent detection on American English speech in the Boston University Radio News Corpus, a state-of-the-art result. We also find that a simple baseline that just predicts a pitch accent on every content word yields 82.2% accuracy, and we suggest that this is the appropriate baseline for this task. Finally, we conduct ablation tests that show pitch is the most important acoustic feature for this task and this corpus.

## introduction prosody is a rich information source with the potential to improve performance in many spoken nlp tasks @xcite . in order to make prosodic information available to downstream tasks, many models have been proposed to predict which words in an utterance carry pitch accents-word-level prosodic prominences signaled by a deviation from the speaker's usual pitch, duration, intensity, or some combination of these three features. identifying pitch accents is helpful since they are often used to signal important or unexpected information. for example, pitch accents in english typically fall on content words, which are generally more informative. when a pitch accent falls on a function word, it indicates that it is unusually informative, as in the sentence, they ran out of toilet paper even before the quarantine, where before is more informative because it contrasts with what might be a default assumption (e.g., during). previous pitch accent prediction models include rule-based models @xcite , traditional machine learning models @xcite @xcite , and neural models @xcite @xcite . @xcite and @xcite (henceforth, svs18) showed that neural methods can perform comparably to traditional methods using a relatively small amount of speech contextjust a single word on either side of the target word. however, since pitch accents are deviations from a speaker's average pitch, intensity, and duration, we hypothesize that, as in some non-neural models (e.g. @xcite ; rosenberg and hirschberg 2009), a wider input context will allow the model to better determine the speaker's baseline for these features and therefore improve its ability to detect deviations. in addition, we hypothesize that a recurrent model (rather than the cnn used by svs18) will also improve performance, since it is better adapted to processing long-distance dependencies. in this paper, we test these hypotheses by building a new neural pitch accent prediction model that takes in prosodic speech features, text features, or both. our main contribution is showing that these context-enhancing innovations in the speechonly model improve performance on a corpus of american english speech, yielding higher accuracy than svs18 and all previous models on this dataset. we also find that a baseline of simply labeling all content words with pitch accents is very robust, matching the performance of the text-only model. we argue that this more robust contentword baseline is the correct baseline for this task. we find that our speech-only model is able to outperform this baseline by detecting some of the cases where a speaker deviates from the predictions of the content-word baseline, and we provide an analysis of which acoustic features yield the most benefit. ## models we build models to predict which words carry a pitch accent, given input of either prosodic speech features, text features, or both. the variants are shown in figure 1 and described below. all models are implemented in pytorch @xcite . 1 we also describe the ways in which we varied the amount of context available to the speech-only model in particular. speech-only model. like svs18's model, our speech encoder begins with several cnn layers that take a series of frames f 1 , f 2 , ..., f n as input, where each frame f i is a vector of 6 acoustic-prosodic features (see §3). these frames are encoded by the cnn, which reduces the overall number of frames by passing a kernel over the input with a stride of size 2, resulting in frames f 1 , f 2 , ..., f k . however, rather than predicting the label for a single token at a time, as svs18 do, our model labels the whole sequence at once. in order to divide the output of the cnn into word tokens, we use the token timestamps provided in the corpus to divide the frames at places corresponding to word boundaries in the input, similar to the approach taken in @xcite . each resulting subdivision of the frames [f i , f i+1 , ..., f j ] contains different numbers of frames, since tokens are of various lengths. to obtain token representations of identical size, we sum across all frames for a given token: t @xmath0, f i+1 , ..., f k ). each token embedding t 1 , ..., t m is then passed into a bidirectional lstm, and finally a feed forward layer that outputs a prediction for each token. the model's hyperparameters are described in detail in appendix a.3. our full model takes an entire utterance as input and predicts all labels at once, but we also experiment with using only three or one token(s) as input. in these cases, the model only predicts the label for the central input token. the three-token scenario is designed to be most similar to svs18's model. text-only model. the text-only model is a simple bidirectional lstm. an embedding for each token is passed to the bilstm and a prediction is made at each timestep. we followed svs18 in using pretrained 300d glove word embeddings @xcite , although using pre-trained embeddings did not improve performance much over randomly initialized embeddings. speech+text model. the speech-only and textonly models both include a bidirectional lstm, so for the combined model, we just concatenate the embedding for each token generated by the cnn encoder with the pretrained text embedding for that token before passing them to the lstm. baselines. in addition to a majority class baseline, we also report results on a content-word baseline, where all content words (non-stopwords as identified by nltk) are labelled as carrying a pitch accent. we also report a duration-only baseline, where the input features to the speech-only model are all replaced with the value 1-so the model can only tell how many frames each token contains. ## data and experimental setup we train and test all models using data from the boston university radio news corpus (hereafter burnc) foot_1 , a speech corpus of general american english that is partially annotated with prosodic information. the annotated subsection of the corpus that we use includes five speakers, three female, and two male, all of them trained radio journalists reading pre-written news segments. this kind of read speech from trained speakers is different from spontaneous speech and so the conclusions we reach here can only confidently be applied to this genre. the data we use amount to approximately 2.75 hours of speech, consisting of 1721 utterances. these come from a total of 398 news segments. there are 28,489 word tokens, 15,544 of which carry pitch accents. though this is a limited amount of data, this corpus is one of very few corpora with available prosodic annotations and enables us to compare with previous studies that use this resource, including svs18. for the speech-only model, we follow svs18 in using the opensmile toolkit @xcite to extract six features, which fall into three broad categories: pitch features (smoothed f0), intensity features (rms energy, loudness), and voicing features (zero-crossing rate, voicing probability, and harmonics-to-noise ratio). these features are extracted from frames of varying sizes (following schuller et al. ( 2013 )), and frames are offset by 10ms. the speech-only model has no access to phone-level or spectral information that might allow it to make predictions based on word identity. the transcription of the speech in this corpus includes marked breaths, which we use to segment the corpus into utterances. note that there are no explicit correlates of duration in this feature set, though the model has access to the absolute duration of each token via the number of input frames per token. in future, we could follow tran et al. ( 2018 ) by giving an explicit feature for the duration of a given token normalized by the average duration of that token in the corpus. for the text-only model, we follow svs18 in removing contractions (e.g. we'll -→ we), though we diverge in leaving hyphenated tokens in place (e.g. eighty-eight remains eighty-eight). we perform tenfold cross-validation of all experiments and report the average performance. for a detailed description of how we divided data into train, development, and test sets for cross-validation, see appendix a.2. in order to test for repeatability, we furthermore initialize our model architecture with five distinct random seeds and repeat the tenfold cross-validation procedure for each of these five model initializations. our reported test set results are the average performance of all these five model initializations. we report accuracy as our primary metric since this task is a balanced binary classification task. we train for 25 epochs and we report the highest development set accuracy of these 25 epochs. we use this same epoch to report test set accuracy. ## results and discussion development set results from the speech-only model using different input contexts and architectures are shown in overall, our best speech-only model outperforms the previous work (svs18) as well as the text-only model and baselines on the test set (see speech: text: speech: text: if we only consider these tokens where the speaker's production deviates from the contentword baseline, the speech-only model achieves 66.7 percent accuracy, vs. only 38.2 percent for the textonly model. in addition to the evaluations described above, where all utterances are randomly assigned to train, development, and test sets, we do speakerindependent evaluation of the speech+text model. that is, we hold out a single speaker for testing and use all the other speakers for training and development. these results are shown in ## conclusions this work demonstrates some important principles for predicting pitch accent from text and speech. first, we show that a speech-only model benefits from having utterance-level context. second, we show that both the text and the speech-only model derive at least some of their performance from being able to distinguish function words from content words. in fact, our bilstm-based text model can hardly outperform a content-word baseline. finally, we show that a speech-only model can successfully predict pitch accent in cases where a text-only model cannot, and that combining text and speech provides only a tiny benefit. these results indicate that the speech-only model uses information available in the prosodic features to surpass the contentword baseline, and that knowing the actual words doesn't provide much further useful information. jeffrey pennington, richard socher, and christopher d. manning. 2014. glove: global vectors for word representation. in empirical methods in natural language processing (emnlp), pages 1532-1543. ina roesiger, sabrina stehwien, arndt riester, and thang vu. 2017. improving coreference resolution with automatically predicted prosodic information. in proceedings of the workshop on speech-centric natural language processing, pages 78-83. andrew rosenberg and julia bell hirschberg. 2009. detecting pitch accents at the word, syllable and vowel level. in proceedings of naacl/hlt. björn schuller, stefan steidl, anton batliner, alessandro vinciarelli, klaus scherer, fabien ringeval, mohamed chetouani, felix weninger, florian eyben, erik marchi, et al. 2013. the interspeech 2013 computational paralinguistics challenge: social signals, conflict, emotion, autism. in proceedings inter-speech 2013, 14th annual conference of the international speech communication association, lyon, france. sabrina stehwien and ngoc thang vu. 2017. prosodic event recognition using convolutional neural networks with context information. in proc. interspeech 2017, pages 2326-2330. sabrina stehwien, ngoc thang vu, and antje schweitzer. 2018. effects of word embeddings on neural network-based pitch accent detection. in proc. 9th international conference on speech prosody 2018, pages 719-723. trang tran, shubham toshniwal, mohit bansal, kevin gimpel, karen livescu, and mari ostendorf. 2018. parsing speech: a neural approach to integrating lexical and acoustic-prosodic information. in proceedings of the 2018 conference of the north american chapter of the association for computational linguistics: human language technologies, volume 1 (long papers), pages 69-81, new orleans, louisiana. association for computational linguistics. colin w wightman and mari ostendorf. 1994. automatic labeling of prosodic patterns. ieee transactions on speech and audio processing, 2(4):469-481.