Towards end-2-end learning for predicting behavior codes from spoken utterances in psychotherapy conversations (2020)

Abstract: Spoken language understanding tasks usually rely on pipelines involving complex processing blocks such as voice activity detection, speaker diarization and Automatic speech recognition (ASR). We propose a novel framework for predicting utterance level labels directly from speech features, thus removing the dependency on first generating transcripts, and transcription free behavioral coding. Our classifier uses a pretrained Speech-2-Vector encoder as bottleneck to generate word-level representations from speech features. This pretrained encoder learns to encode speech features for a word using an objective similar to Word2Vec. Our proposed approach just uses speech features and word segmentation information for predicting spoken utterance-level target labels. We show that our model achieves competitive results to other state-of-the-art approaches which use transcribed text for the task of predicting psychotherapy-relevant behavior codes.

## introduction speech interfaces have seen a widely growing trend and this has brought about increasing interest in advancing computational approaches to spoken language understanding (slu). @xcite @xcite . slu systems often rely on automatic speech recognition (asr) for generating lexical features. the asr output is then used for the target natural language understanding task. furthermore, end-2-end slu systems for various applications, including speech synthesis @xcite , asr tasks @xcite @xcite and speech-2-text translation @xcite have shown promising results. recently @xcite propose a method for learning audio-linguistuc embedding but that too depends on using transcribed text. lower part shows our proposed approach where we predict behavior codes without using transcripts due to the nature of the speech processing pipeline, natural language understanding tasks suffer from two major problems, 1) error propagation through asr leading to noisy lexical features 2) loss of rich information which supplement lexical features, such as prosodic and acoustic expressive speech patterns. in this paper, we propose a framework to address the problem of predicting behavior codes directly from speech utterances. we focus on data from motivational interviewing (mi) sessions, a type of talk-based psychotherapy focused on behavior change. in psychology research and clinical practice, behavioral coding is often used to understand process mechanisms and therapy efficacy and outcomes. behavior codes are annotated by an expert at an utterance level (or interaction level) by listening to the session. examples of utterance level behavior codes include if there was a simple of complex reflection by the therapist of their patient's previous utterance(s). several approaches have been proposed for automatic prediction of behavior codes, mainly using lexical features and/or linguistic features such as information from dependency trees @xcite @xcite @xcite . recent works @xcite reveal that using acoustic and prosodic features in addition to lexical features outperforms single modality models. speech2vec @xcite has shown that high quality word representations can be learnt by just using speech features. it learns word representations in an unsupervised manner using an objective similar to the skipgram objective of word2vec @xcite (a word representation should be representative of its context words) and sequence-to-sequence framework. however, speech2vec only aims to learn word representations which are averaged spoken-word representations of that word in the corpus. our proposed approach aims to exploit speech signal to word encoder learnt using an architecture similar to speech2vec as lower level dynamic word representations for the utterance classifier. thus, our system never actually needs to know what word it is but only word segmentation information. we hypothesize that word segmentation information can be obtained with cheaper tools, e.g. a supervised word segmentation system @xcite or a heuristics based system based on acoustic and prosodic cues @xcite . we plan to investigate the effect of noise in word boundaries on encoder quality in the future. our end-2-end transcription-free approach is similar and perhaps even motivated some of the previous works. there have been some works @xcite which perform prediction tasks directly from speech signals but lack in capturing the underlying linguis-tic structure of a language (sentences break into words for semantics). we believe capturing some of the important linguistic units (e.g. words) are important for spoken language understanding. @xcite is most similar to our work in terms of overall architecture as they also first get word level representations and then use the encoder for utterance level prediction. however @xcite uses transcribed word transcriptions but we only use word boundaries for asr-free end-2-end spoken language understanding. as shown in figure 1 , most previous works follow the upper pipeline. they start with a transcript (manually generated or through an asr), which is first segmented into utterances. they then use word-embeddings for each word in the transcript before feeding it into a classifier to predict target behavior codes. our approach shows competitive results when compared to state-of-the-art models which use transcribed text. our target application domain in this work is psychotherapy. while utterance level behavior coding is a valuable resource for psychotherapy process research, it is also a labor intensive task for manual annotation. our proposed method which does not rely on transcripts should help with cheaper and faster behavioral annotation. we believe this framework can be a promising direction to directly perform classification tasks given a spoken utterance. ## our approach we first learn a word-level speech signal to word encoder using a sequence-to-sequence framework. speech-2-vector follows the learning objective similar to skipgram architecture of word2vec. we then use the pre-trained encoder to predict behavior codes. ## training details speech-2-vector encoder: we implemented the model with pytorch @xcite . similar to @xcite , we also adopted the attention mechanism which enables the decoder to condition every decoding step on the last hidden state of the encoder @xcite . the window size was set to 4. we train the model using stochastic gradient descent (sgd) with learning rate of 1e * -3 and batch size of 64 (spokenword, context) pairs. we experimented with hyperparameter combinations for: using bidirectional or unidirectional rnns, using gru vs lstm cell, number of lstm hidden layers and learning rates. we found there was not a big difference in encoder output quality with higher dimensions. therefore, we use a 50 dimensional lstm cell, thus the resulting encoder output becomes 100 (bidirectional last hidden states) + 100 (cell state) = 200 dimensions. utterance classifier: the chosen batch size was 40 utterances. the lstm hidden state dimension is 50. we use dropout at the embedding layer with drop probability 0.3. the dense layer is of 100 dimensions. the model is trained using the adam optimizer (kingma and ba, 2014) with a learning rate of 0.001 and an exponential decay ## experiments & results speech2vec vs word2vec: results confirm two main observations: 1) it is better to learn/fine-tune the word embeddings on an in-domain dataset. 2) speech2vec that learns word embeddings based on different spoken variations of word provides better results for behavior code prediction. this result is consistent with findings from @xcite where it is shown that acoustic-prosodic information can provide complementary information for predicting behavior codes and hence, produce better results. one challenge is that sswe and speech2vec generally needs large amount of transcribed data to learn high quality word embeddings. therefore, we first train sswe on a general speech corpus (here, librespeech (libre)) before fine-tuning it on our classifier training data (results with * show this experiment). transcriptions vs. no transcriptions: methods discussed above still rely on transcriptions to know what the word is. however, our proposed method does not use any explicit transcription but only the word segmentation information. results in ## conclusions we show that comparable results can be achieved for behavior code prediction by just using speech features and without any asr or human transcriptions. our approach still depends on word segmentation information, however, we believe obtaining word segmentation from speech is comparatively easier than building a high quality asr. the evaluation results show the application significance of an end-2-end speech to behavioral coding for psychotherapy conversations. this allows for building systems that do not include explicit transcriptions, an attractive option for privacy reasons, when the end goal (as determined by the behavioral codes) is to characterize the overall quality of the clinical encounter for training or quality assurance.