Textual S upervision for V isually G rounded S poken L anguage U nderstanding (2020)

Abstract: Visually-grounded models of spoken language understanding extract semantic information directly from speech, without relying on transcriptions. This is useful for low-resource languages, where transcriptions can be expensive or impossible to obtain. Recent work showed that these models can be improved if transcriptions are available at training time. However, it is not clear how an end-to-end approach compares to a traditional pipeline-based approach when one has access to transcriptions. Comparing different strategies, we find that the pipeline approach works better when enough text is available. With low-resource languages in mind, we also show that translations can be effectively used in place of transcriptions but more data is needed to obtain similar results.

## introduction spoken language understanding promises to transform our interactions with technology by allowing people to control electronic devices through voice commands. however, mapping speech to meaning is far from trivial. the traditional approach, which has proven its effectiveness, relies on text as an intermediate representation. this so-called pipeline approach combines an automatic speech recognition (asr) system and a natural language understanding (nlu) component. while this allows us to take advantage of improvements achieved in both fields, it requires transcribed speech, which is an expensive resource. visually-grounded models of spoken language understanding @xcite were recently introduced to extract semantic information from speech directly, without relying on textual information (see figure 1 for an illustration). the advantages of these approaches are twofold: (i) expensive transcriptions are not necessary to train the system, which is beneficial a boy in a green shirt on a skateboard on a stone wall with graffiti. ## methodology the architecture and the training procedure used in this paper are inspired by the improved version of the visually-grounded spoken language understanding system of @xcite . appendix a.1 provides details on the choice of hyperparameters. encoder speech image encoder encoder (a) speech-image model encoder text image encoder encoder (b) text-image model encoder image text speech encoder encoder + decoder (c) pipeline models (pipe-ind and pipe-seq) shared encoder decoder dedicated encoder dedicated encoder image speech text encoder (d) mtl-transcribe/mtl-translate model shared encoder encoder dedicated encoder dedicated encoder image speech text encoder (e) mtl-match model ## conclusion in this paper, we investigated the use of textual supervision in visually-grounded models of spoken language understanding. we found that the improvements reported in @xcite and @xcite are a low estimate of what can be achieved when textual labels are available. among the different approaches we explored, the more traditional pipeline approach, trained sequentially, is particularly effective and hard to beat with endto-end systems. this indicates that text is a very powerful intermediate representation. end-to-end approaches tend to perform better only when the amount of textual data is limited. we have also shown that written translations are a viable alternative to transcriptions (especially for unwritten languages), though more data might be useful to compensate for the harder task.