Catplayinginthesnow: Impact of Prior Segmentation on a Model of Visually Grounded Speech (2020)

Abstract: The language acquisition literature shows that children do not build their lexicon by segmenting the spoken input into phonemes and then building up words from them, but rather adopt a top-down approach and start by segmenting word-like units and then break them down into smaller units. This suggests that the ideal way of learning a language is by starting from full semantic units. In this paper, we investigate if this is also the case for a neural model of Visually Grounded Speech trained on a speech-image retrieval task. We evaluated how well such a network is able to learn a reliable speech-to-image mapping when provided with phone, syllable, or word boundary information. We present a simple way to introduce such information into an RNN-based model and investigate which type of boundary is the most efficient. We also explore at which level of the network’s architecture such information should be introduced so as to maximise its performances. Finally, we show that using multiple boundary types at once in a hierarchical structure, by which low-level segments are used to recompose high-level segments, is beneficial and yields better results than using low-level or high-level segments in isolation.

## introduction and prior work visually grounded speech (vgs) models whether cnn-based @xcite @xcite or rnn-based @xcite became recently popular as they enable to model complex interaction between two modalities, namely speech and vision, and can thus be used to model child language acquisition, and more specifically lexical acquisition. indeed, these models are trained to solve a speech-image retrieval task. that is, given a spoken input description, they are trained to retrieve the image that matches the description the best. this task requires the model to identify lexical units that might be relevant in the spoken input, detect which objects are present in the image, and finally see if those objects match the detected spoken lexical units. their task is thus very close to that of a child learning its mother tongue, who is surrounded by a visually perceptible context and tries to match parts of the acoustic input to surrounding visible situations. research in language acquisition have put forward that children do not build their lexicon by segmenting the spoken input into phonemes and then building up words, but rather adopt a topdown approach @xcite and start by identifying and memorising whole words @xcite or chunks of words @xcite and then segment the spoken input into smaller units, such as phonemes. this suggests that the most efficient way of segmenting the spoken input to map a visual context to its description is at word level. from a more technological point of view, speech-based models lag behind their textual counterparts. for example, speech-image retrieval performs worse than text-image retrieval, despite being trained on the same data, the only changing factor being the modality where text or speech is used as a query. this begs the question: what makes text inherently better than speech for such applications? is it because text is made up of already-segmented (discrete) units which lack internal variation, or because these discrete units (usually tokens) stand for full semantic units, or a combination of both? since the pioneering computational modelling work of lexical acquisition by @xcite , neural network enabled an even tighter interaction between the visual and the audio modalities. recent works suggest that networks trained on a speech-image retrieval task perform an implicit segmentation of their input. whether cnnbased approaches or rnn-based approaches are employed, all seem to segment individual words from the inputted spoken utterance @xcite @xcite @xcite . this result stands also for languages other than english, such as hindi or japanese @xcite @xcite . @xcite and @xcite , however, observed that not all layers encode wordlike units, suggesting that some layers specialise in lexical processing whereas some other do not encode such information. contributions our research question can be framed as follows: what is the segmentation that maximises the performance of an audio-visual network if speech were to be segmented? to answer this question we investigate how it is possible to give speech boundary information to a neural network and explore which type of boundary (phone, syllable, or word) is the most efficient. we also explore where such information should be provided, that is, at which layer of the architecture is the addition of this information the most beneficial? architecture the models we train in our experiments all have the same architecture and are based on that of @xcite . 1 as all models of vgs, be they cnn-based or rnn-based, this architecture has two main components: an image encoder and a speech encoder. such models are trained to solve a speech-image retrieval task, that is, given a query in the form of a spoken description, they should retrieve the closest matching image fitting the description. the image encoder is a simple linear layer that reduces pre-computed vgg image vectors to the desired dimension. the speech encoder, which receives mfcc vectors as input, consists of a 1d convolutional layer, followed by five stacked recurrent layers with residual connections, followed by an attention mechanism. we use uni-directional recurrent layers and not bi-directional recurrent layers even though it has been shown they lead to better results @xcite . indeed, we aim at having a cognitively plausible model: humans process speech in a left-to-right fashion, as speech is being gradually uttered, and not from both ends simultaneously. we use the same loss function as initially used by @xcite : hyperparameters for both coco and flickr8k we use 1d convolutions with 64 filters of length 6 and a stride of 1 to preserve the original time resolution (and hence, boundary position). we use 512 units per recurrent layer for coco and 1024 for flickr8k. all models were trained using adam optimiser and an initial learning rate of 0.0002. for our experiments we use the pre-computed mfcc vectors and pre-computed vgg vectors provided by @xcite . 2 we also use the same training, validation and testing splits. figure 1a shows a vanilla gru. figure 1b shows gru pack. in the all condition where all the vectors produced at each time step are passed on to the next layer. 1c shows gru pack. in the keep condition where only the last vector of a segment is passed on to the next layer, thus resulting in a output sequence shorter than the input sequence. the red crosses inscribed in a square ( ) signal that the output vector computed at a given timestep is not passed on to the next timestep and that the initial state h 0 is passed on instead. the red crosses inscribed in a circle ( ) signal that the output vector computed at a given timestep is not passed on to the next layer. dotted line group vectors belonging to a same segment (either phone, syllable-connected, syllable-word, or word). note that h 0 is only passed on to the next state at the end of a segment, thus effectively materialising a boundary by manually resetting the history. also note that the x 1 , x 2 , ..., x t figured in this representation could either be the original input sequence (in our case, acoustic vectors) or could also be the output of the previous recurrent layer. ## towards hierarchical segmentation in our current approach, only one out of the five recurrent layers is a gru pack. layer, which handles only one type of segment. however, we can stack as many gru pack. as desired, provided they are supplied with boundary information. stacking such layers enables us to not only integrate bound-the layer 2 and 3 of our architecture are those that benefit the most from adding linguistic information, and confirm the fact that the upper layers (such as the fifth) do not take as much advantage of this information as the lower layers. introducing structure allowed us to remove two recurrent layers without a big loss of performance (r@1 = 8.1 for a three-layered architecture with two gru pack. layers) while the baseline architecture with only three layers performs poorly. phones, syllables, and words: we also explore an architecture with 3 gru pack. layers, to which we provide phone, syllable-word and word boundaries. as in our previous experiments, we vary the number of layers (from 3 to 5), and test all possible configurations. the results of this experiment are presented in our results show that the more structure we introduce into the network, the better it performs. additionally, introducing structure enables us to reduce the number of layers (and hence the number of computations) while increasing the performances compared to the baseline. overall, it is better to use boundary information in coordination in a hierarchical structure than using them in isolation.