  We propose direct multimodal few-shot models that learn a shared embedding
space of spoken words and images from only a few paired examples. Imagine an
agent is shown an image along with a spoken word describing the object in the
picture, e.g. pen, book and eraser. After observing a few paired examples of
each class, the model is asked to identify the "book" in a set of unseen
pictures. Previous work used a two-step indirect approach relying on learned
unimodal representations: speech-speech and image-image comparisons are
performed across the support set of given speech-image pairs. We propose two
direct models which instead learn a single multimodal space where inputs from
different modalities are directly comparable: a multimodal triplet network
(MTriplet) and a multimodal correspondence autoencoder (MCAE). To train these
direct models, we mine speech-image pairs: the support set is used to pair up
unlabelled in-domain speech and images. In a speech-to-image digit matching
task, direct models outperform indirect models, with the MTriplet achieving the
best multimodal five-shot accuracy. We show that the improvements are due to
the combination of unsupervised and transfer learning in the direct models, and
the absence of two-step compounding errors.
