Growing Trees on Sounds: Assessing Strategies for End-to-End Dependency Parsing of Speech (2024)

Abstract: Direct dependency parsing of the speech signal –as opposed to parsing speech transcriptions– has recently been proposed as a task (Pupier et al. 2022), as a way of incorporating prosodic information in the parsing system and bypassing the limitations of a pipeline approach that would consist of using first an Automatic Speech Recognition (ASR) system and then a syntactic parser. In this article, we report on a set of experiments aiming at assessing the performance of two parsing paradigms (graph-based parsing and sequence labeling based parsing) on speech parsing. We perform this evaluation on a large treebank of spoken French, featuring realistic spontaneous conversations. Our findings show that (i) the graph based approach obtain better results across the board (ii) parsing directly from speech outperforms a pipeline approach, despite having 30% fewer parameters.

## introduction dependency parsing is a central task in natural language processing (nlp). in the nlp community, it has mostly been addressed on textual data, either natively written texts or sometimes speech transcriptions. yet, speech is the main form of communication between humans, as well as arguably one of the most realistic types of linguistic data, which motivates the design of nlp systems able to deal directly with speech, both for applicative purposes and to construct corpora annotated with linguistic information. when parsing speech transcriptions, most prior work has focused on disfluency detection and removal @xcite @xcite @xcite , in an effort to 'normalize' the transcriptions and make them suitable input for nlp systems trained on written language. using only transcriptions as input is a natural choice from an nlp perspective: it makes it possible to use offthe-shelf nlp parsers 'as is'. however, predicted transcriptions can be very noisy, in particular for speech from spontaneous conversations. furthermore, transcriptions are abstractions that contain much less information than the speech signal. the prosody, and the pauses in the speech utterances are very important clues for parsing @xcite that are completely absent from transcriptions. hence, we address speech parsing using only the speech signal as input. with the popularization of self-supervised method and modern neural network architecture (pretrained transformers), both speech and text domains now use similar techniques @xcite . this convergence of methodology has raised interest in other applications of speech models to go beyond 'simple' speech recognition. thus, addressing classical nlp tasks directly on speech is a natural step and design nlp tools able to deal with spontaneous speech, arguably the most realistic type of linguistic production. in short, our contributions are the following: we define speech parsing as the task of predicting a dependency tree from an audio signal corresponding to a spoken utterance. 2 our parser is composed of 2 modules (figure 1a ): (i) an acoustic module that is used to predict transcriptions and a segmentation of the signal in words and (ii) a parsing module that uses the segmentation to construct audio word embeddings and predict trees. word level representations from speech to extract representations from the raw speech, we use a pre-trained wav2vec2 model trained on seven thou-sand hours of french speech: lebenchmark7k foot_2 @xcite . parsing requires word-level representations. we use the methodology of @xcite to construct audio word embeddings from the implicit frame level segmentation provided by the ctc speech recognition algorithm @xcite . the method consists in combining the frame vectors corresponding to a single predicted word with an lstm. ## dataset we use the cefc-orféo treebank @xcite , a dependency-annotated french corpus composed of multiple subcorpora @xcite @xcite @xcite @xcite @xcite @xcite , and released with the audio recordings. the treebank consists of various types of interactions, all of which feature spontaneous discussions, except for the french oral narrative corpus (audiobooks). orféo features many types of speech situations (eg. commercial interactions, interviews, informal discussions between friends) and is the largest french spoken corpus annotated in dependency syntax. the annotation scheme has been designed specifically for orféo @xcite and differs from the universal dependency framework in many re-gards (in particular: its pos tagset is finer-grained, whereas the syntactic function tagset has only 14 relations). the syntactic annotations of orféo were done manually for 5% of the corpus and automatically for the rest of the corpus. the train/dev/test split we use makes sure that the test section only contains gold annotations. nevertheless, the subcorpora with gold syntactic annotations correspond to low-quality recordings, which makes them a very challenging benchmark. ## experiments experimental settings our experiments aim at: (i) comparing our graph-based parser to the seq2label model, (ii) comparing to pipeline approaches with text-based parsers, and (iii) assessing the robustness of word representations with control experiments: using word boundaries (provided in the corpus) as input for the audio models and gold transcriptions for the text-based model. we compare the following settings (illustrated in figure 1 ): both pipeline and text settings use a french bert model: camembert-base foot_4 @xcite to extract contextualized word embeddings. for pipeline and text settings, on top of our implementations, we use hops @xcite , an external state-of-the-art graph-based parser. the hops parser uses a character-bi-lstm in addition to bert to produce word embeddings, whereas our implementation does not (in an effort to make both versions of our parser, text-based and audio-based, as similar as possible). each parsing method for each modality is trained with the same number of epochs, the same hyperparameters (see ## conclusion we introduced a graph-based speech parser that takes only the raw audio signal as input and assessed its performance in various settings and in several control experiments. we show that a simple graph-based approach with wav2vec2 audio features is on a par with or outmatches a more complex pipeline approach that requires two pretrained models. from control experiments (oracle), we show that acquiring good quality word representations directly from speech is the main challenge for speech parsing. we will focus future work on improving the quality of word segmentation on the speech signal.