  In this paper we present the first model for directly synthesizing fluent,
natural-sounding spoken audio captions for images that does not require natural
language text as an intermediate representation or source of supervision.
Instead, we connect the image captioning module and the speech synthesis module
with a set of discrete, sub-word speech units that are discovered with a
self-supervised visual grounding task. We conduct experiments on the Flickr8k
spoken caption dataset in addition to a novel corpus of spoken audio captions
collected for the popular MSCOCO dataset, demonstrating that our generated
captions also capture diverse visual semantics of the images they describe. We
investigate several different intermediate speech representations, and
empirically find that the representation must satisfy several important
properties to serve as drop-in replacements for text.
