Was That a Question? Automatic Classification of Discourse Meaning in S panish (2023)

Abstract: This paper examines the effectiveness of different feature representations of audio data in accurately classifying discourse meaning in Spanish. The task involves determining whether an utterance is a declarative sentence, an interrogative, an imperative, etc. We explore how pitch contour can be represented for a discourse-meaning classification task, employing three different audio features: MFCCs, Mel-scale spectrograms, and chromagrams. We also determine if utilizing means is more effective in representing the speech signal, given the large number of coefficients produced during the feature extraction process. Finally, we evaluate whether these feature representation techniques are sensitive to speaker information. Our results show that a recurrent neural network architecture in conjunction with all three feature sets yields the best results for the task.

## introduction the aim of this study is to investigate the efficacy of feature representations of audio data in accurately classifying discourse meaning in spanish. the task involves determining whether an utterance is a declarative sentence, an interrogative, an imperative, etc. since there does not seem to be an agreed upon name for this task, we will refer to it as discourse meaning (rather than referring to a broader sense of this term). in human perception, this process involves the comprehension of the relationship of words, phrases, and clauses used in a sentence, as well as their overall contribution to the intended meaning of the sentence. we focus on the prosodic features of different discourse meanings in spanish. pitch, or the perceived highness or lowness of a sound, can play a role in distinguishing between different discourse meanings. for example, declarative sentences typically present a falling pitch contour, indicating that the statement is complete, while interrogatives usually have a rising contour, signaling that a question is being asked. in contrast to tonal languages such as mandarin chinese, thai, or punjabi, which mark the phonological contrast of pitch at the lexical level (word level), intonational languages such as spanish or english mark the phonological contrast of pitch at the utterance level. for spanish, pitch movements are mainly used to signal discourse meaning or to mark focus. the properties that govern production in intonation are structurally analogous to those that govern lexical tones and morphological paradigms @xcite . this means that a declarative statement like maría viene 'maría is coming' and its interrogative counterpart ¿maría viene? 'is maria coming?' differ only in the intonational contour with which they are produced, since both are syntactically and lexically identical. ## rq 1: exploring audio feature representations the first research question (rq1) investigates the effectiveness of the three representations of the audio signal: mfccs, mel-scale spectrograms, and chromagrams. our goal is twofold: 1) to investigate whether the three audio features are effective in capturing the necessary information to classify pitch based on discourse meaning, and 2) to assess whether the use of mean values, as opposed to all values, is a more efficient method for capturing this information. since the number of frames produced by stft varies based on the length of each audio file, the exact number of all the coefficients for each feature set varied accordingly. therefore, to ensure uniformity, we padded with zero values such that each file had the same number of coefficients as the longest file. specifically, we set the number of coefficients to @xmath0,840; mel @xmath1,936; chro-@xmath2,812 (see ## rq2: comparing information content of audio features rq2 investigates whether the three audio feature representations convey different types of information. while mfccs have been shown to be the most effective for the audio classification tasks for rq1, their reliance on a low-rank linear projection of the mel spectrum may lead to information loss. thus, we explore the possibility of enhancing the discriminatory power of mfccs by incorporating additional representations, such as mel spectrograms or chromagrams, which may convey complementary information. if the combinations of audio features provides a full set of information, we expect increased classification results. we focus on means for each feature type since their use resulted in higher accuracy for rq1. we replicate the methodology of the previous experiment using the same data split as above. ## rq3: analyzing speaker effects rq3 investigates the impact of speaker effects on the classification of discourse meaning. our objective is to examine whether there exist individual variations in how people generate intonation curves for different types of sentences and whether these differences are captured by the three feature representations. we replicate the previous experiments while employing a leave-one-out cross-validation approach where each fold corresponds to one speaker. since the model has not seen any data from the test speaker, a deterioration in this setting will indicate that the features types include speaker specific information. ## conclusion and future work in this paper, we investigated the efficacy of various audio input representations for accurately classifying discourse meaning in spanish. we explored pitch contour representation using three audio features and compared the efficiency of utilizing means with different algorithms. we also evaluated if these features convey different information and their generalizability across speakers. our findings suggest that using a combination of the three features with a recurrent neural network architecture provides the best results for our discoursemeaning classification task. we also found that there is speaker specific information represented in the features, and that that combination of mlp and mfccs is much more robust in a setting where we test on an unknown speaker than the other combinations. we will need to have a closer look to understand better why this is the case. we are also planning on extending the corpus to include more speakers, and to balance it for dialects.