Text-Free Image-to-Speech Synthesis Using Learned Segmental Units (2021)

Abstract: In this paper we present the first model for directly synthesizing fluent, natural-sounding spoken audio captions for images that does not require natural language text as an intermediate representation or source of supervision. Instead, we connect the image captioning module and the speech synthesis module with a set of discrete, sub-word speech units that are discovered with a self-supervised visual grounding task. We conduct experiments on the Flickr8k spoken caption dataset in addition to a novel corpus of spoken audio captions collected for the popular MSCOCO dataset, demonstrating that our generated captions also capture diverse visual semantics of the images they describe. We investigate several different intermediate speech representations, and empirically find that the representation must satisfy several important properties to serve as drop-in replacements for text.

## introduction although there are over 7,000 languages spoken worldwide @xcite , only several dozen have enough data available to support supervised speech recognition, and many languages do not even employ a writing system @xcite . in contrast, most people learn to use spoken language long before they learn to read and write, suggesting that linguistic annotation is not a prerequisite for speech processing systems. this line of reasoning motivates research that aims to discover meaningful linguistic abstractions (phones, words, etc.) directly from the speech signal, with the intention that they could reduce the reliance of spoken language systems on text transcripts. a rich body of work has recently emerged investigating representation learning for speech using visual grounding objectives @xcite @xcite @xcite @xcite @xcite @xcite @xcite , as well as how word-like and subword-like linguistic units can be made to emerge within these models @xcite @xcite @xcite @xcite . so far, these efforts have predominantly focused on inference, where the goal is to learn a mapping from speech waveforms to a semantic embedding space. generation of speech conditioned on a point in a semantic space has been less explored, and is what we focus on in this work. we hypothesize that generative approaches offer interesting advantages over relying solely on inference. for example, prior works have demonstrated the capability of recognizing visually descriptive words, but have not been shown to learn non-visual words or grammar. our experiments show that these aspects of spoken language are learned to some degree by a visually-grounded generative model of speech. specifically, we introduce a model capable of directly generating fluent spoken audio captions of images without the need for natural language text, either as an intermediate representation or a form of supervision during training (figure 1 ). tremendous progress has been made recently in natural language image caption generation @xcite @xcite @xcite @xcite @xcite and naturalistic text-tospeech synthesis (tts) @xcite @xcite @xcite . combining these models provides a means for generating spoken image descrip-a person in a blue jacket is on a snowboard on a snow covered slope a snowboarder is snowboarding on the side of the mountain a snowboarder is snowboarding on the side of the mountain same unit sequence, different speakers different unit sequences, same speaker figure 1 : spoken image captions generated from the proposed model, with diversity in both linguistic content and acoustic properties, controlled through the i2u and the u2s models, respectively. transcriptions are provided only for illustration. audio samples are available at https://wnhsu.github.io/image-to-speech-demo . ## experiments we design experiments to address three questions: first, how can we measure the performance of an image-to-speech system? our system can fail to produce a good caption if the i2u model fails to encode linguistic/semantic information into the unit sequence, or if the u2s model fails to synthesize an intelligible waveform given a unit sequence. to better localize these failure modes, we evaluate the full i2s system as well as the u2s system in isolation. we evaluate the u2s system by using it as a vocoder to synthesize unit sequences inferred from real speech and soliciting human judgements in the form of mean opinion score (mos) and side-by-side (sxs) preference tests ( to evaluate the i2s system, we can use any method that measures the semantic information contained in the generated speech. we consider two sets of end-to-end metrics: word-based and retrieval-based, and one set of proxy unit-based metrics. word-based metrics transcribe a generated spoken caption into text (manually or with an asr system) and then measure word-based captioning metrics against a set of reference captions, such as bleu-4 @xcite (adjusted ngram precision), meteor (denkowski and lavie, 2014) (uni-gram f-score considering word-to-word alignment), rouge @xcite (n-gram recall), cider @xcite (tf-idf weighted n-gram cosine similarity), and spice @xcite (f-score of semantic propositions in scene graphs). this enables comparison between image-to-speech systems with a text "upperbound", but is not applicable to unwritten languages. retrieval-based metrics include image-to-speech and speech-to-image retrieval @xcite , which require a separately trained crossmodal retrieval model for evaluation. such metrics are text-free, but they cannot measure other aspects of language generation such as syntactic correctness (partially captured by bleu-4) and scope of the learned vocabulary. lastly, unit-based metrics are similar to text-based, but replace words with units when computing n-gram statistics. however, systems built on different units are not directly comparable, and second, can be inflated if duration is modeled using unit repetition. second, what properties must learned units have to be a drop-in replacement for text? the most essential differences between text and speech are the amount of information encoded and the sequence lengths. beyond text, speech also encodes prosody, speaker, environment information and the duration for each phone, all of which are minimally correlated with the conditioned images. we hypothesize that learned speech units should discard such information in order to seamlessly connect the i2u and u2s modules. to verify it, we pay particular attention to the variations of the learned units in frame rate (vq2/vq3), encoding of duration information (rle or not), and robustness to domain shift (wvq/vq3). units are run-length encoded by default. third, how should language generation models be evaluated more generally? we examine evaluation of the i2s model using beam searchbased decoding as well as sampling-based decoding. we find that because evaluation metrics that are reliant on beam search-based decoding only evaluate the mode of a model's posterior, they do not reflect the ability of a model to generate diverse linguistic content. furthermore, we show that it is possible for a model's posterior mode to be linguistically meaningless, and yet meaningful language can still be generated with sampling-based decoding. towards this end, we introduce a novel multihypothesis evaluation metric (m-spice), which uses sampling-based decoding (instead of beam search) to generate a set of captions. we can then compute the overall coverage of this caption set against a reference; see section 4.4 for details. ## conclusion in this paper, we presented the first model capable of generating fluent spoken captions of images without relying on text, which almost matches the performance of early text-based image captioning models. our comprehensive experiments demonstrated that learned units need to be robust, of low framerate, and encoding little or none duration information to be a drop-in replacement for text. we also identified the caveats of mode-based evaluation and proposed a new metric to address semantic diversity. as part of this work, a novel dataset of over 600k spoken captions for the mscoco dataset is introduced, which we will make publicly available to the research community. future work should investigate applying the proposed method to additional languages, devising improved speech unit representations, and jointly training the speech unit model with the i2s model. this would offer the opportunity to explore new analysis-by-synthesis training objectives. bolei zhou, agata lapedriza, jianxiong xiao, antonio torralba, and aude oliva. 2014. learning deep features for scene recognition using places database. in proc. neural information processing systems (neurips).