Original: L EARNING H IERARCHICAL D ISCRETE L INGUISTIC U NITS FROM V ISUALLY -G ROUNDED S PEECH
Revised: L EARNING H IERARCHICAL D ISCRETE L INGUISTIC U NITS FROM V ISUALLY -G ROUNDED S PEECH

Original: Anonymous authors

Original: Paper under double-blind review

Original: A BSTRACT
Revised: A BSTRACT

Original: In this paper, we present a method for learning discrete linguistic units by incorporating vector quantization layers into neural models of visually grounded speech.
Revised: In this paper, we present a method for learning discrete linguistic units by incorporating vector quantization layers into neural models of visually grounded speech.

Original: We show that our method is capable of capturing both word-level and sub-word units, depending on how it is conﬁgured.
Revised: We show that our method is capable of capturing both word-level and sub-word units, depending on how it is conﬁgured.

Original: What differentiates this paper from prior work on speech unit learning is the choice of training objective.
Revised: What differentiates this paper from prior work on speech unit learning is the choice of training objective.

Original: Rather than using a reconstruction-based loss, we use a discriminative, multimodal grounding objective which forces the learned units to be useful for semantic image retrieval.
Revised: Rather than using a reconstruction-based loss, we use a discriminative, multimodal grounding objective which forces the learned units to be useful for semantic image retrieval.

Original: We evaluate the sub-word units on the ZeroSpeech 2019 challenge, achieving a 27.3% reduction in ABX error rate over the top-performing submission, while keeping the bitrate approximately the same.
Revised: We evaluate the sub-word units on the ZeroSpeech 2019 challenge, achieving a 27.3% reduction in ABX error rate over the top-performing submission, while keeping the bitrate approximately the same.

Original: We also present experiments demonstrating the noise robustness of these units.
Revised: We also present experiments demonstrating the noise robustness of these units.

Original: Finally, we show that a model with multiple quantizers can simultaneously learn phone-like detectors at a lower layer and word-like detectors at a higher layer.
Revised: Finally, we show that a model with multiple quantizers can simultaneously learn phone-like detectors at a lower layer and word-like detectors at a higher layer.

Original: We show that these detectors are highly accurate, discovering 279 words with an F1 score of greater than 0.5.
Revised: We show that these detectors are highly accurate, discovering 279 words with an F1 score of greater than 0.5.

Original:  I NTRODUCTION
Revised:  I NTRODUCTION

Original: By 8 months of age, human infants learn to recognize not only the names of their caregivers and common objects, but also the contrast between the different vowels and consonants which comprise these words (Dupoux, 2018).
Revised: By 8 months of age, human infants learn to recognize not only the names of their caregivers and common objects, but also the contrast between the different vowels and consonants which comprise these words (Dupoux, 2018).

Original: Nearly all toddlers learn to carry a conversation long before they can read and write.
Revised: Nearly all toddlers learn to carry a conversation long before they can read and write.

Original: Humans learn to model the discrete, hierarchical, and compositional nature of their native language not from written text, but from speech audio - a continuous, time-varying waveform which is the product not only of the underlying words which were spoken, but also the physical properties of the speaker’s vocal tract, the speaker’s health and emotional state, and the noise and reverberation present in the environment.
Revised: Humans learn to model the discrete, hierarchical, and compositional nature of their native language not from written text but from speech audio - a continuous, time-varying waveform which is the product not only of the underlying words which were spoken, but also the physical properties of the speaker’s vocal tract, the speaker’s health and emotional state and the noise and reverberation present in the environment.

Original: The question of how such a complex symbolic system is inferred from continuous and noisy sensory input data is of interest not only to the cognitive science community, but also to machine learning researchers who aim to reproduce this ability with computers.
Revised: The question of how such a complex symbolic system is inferred from continuous and noisy sensory input data is of interest not only to the cognitive science community, but also to machine learning researchers who aim to reproduce this ability with computers.

Original: A more comprehensive understanding of human language acquisition has practical signiﬁcance in real-world applications, such as automatic speech recognition (ASR) and natural language understanding (NLU) systems.
Revised: A more comprehensive understanding of human language acquisition has practical signiﬁcance in real-world applications, such as automatic speech recognition (ASR) and natural language understanding (NLU) systems.

Original: In the past several decades, enormous progress has been made in speech recognition research, and nowadays ASR systems are able to achieve human-level accuracy in many domains (Chiu et al., 2018).
Revised: In the past several decades, enormous progress has been made in speech recognition research, and nowadays ASR systems are able to achieve human-level accuracy in many domains (Chiu et al., 2018).

Original: Unfortunately, the techniques that have been developed to achieve these levels of performance are extremely data-hungry, requiring many thousands of hours of speech audio recordings for training.
Revised: Unfortunately, the techniques that have been developed to achieve these levels of performance are extremely data-hungry, requiring many thousands of hours of speech audio recordings for training.

Original: Since supervised machine learning algorithms form the basis of ASR training, the data also needs to be annotated by expert humans.
Revised: Since supervised machine learning algorithms form the basis of ASR training, the data also needs to be annotated by expert humans.

Original: Due to the immense cost of collecting and annotating speech data, ASR technology currently exists for approximately 120 (Google, 2019) out of the nearly 7,000 (Lewis et al., 2016) human languages spoken worldwide.
Revised: Due to the immense cost of collecting and annotating speech data, ASR technology currently exists for approximately 120 (Google, 2019) out of the nearly 7,000 (Lewis et al., 2016) human languages spoken worldwide.

Original: It is highly unlikely that purely supervised machine learning techniques will be able to scale to include all human languages, necessitating the development of alternative methods by researchers which are able to function with far fewer annotations, or even no annotations at all.
Revised: It is highly unlikely that purely supervised machine learning techniques will be able to scale to include all human languages, necessitating the development of alternative methods by researchers which are able to function with far fewer annotations, or even no annotations at all.

Original: Because human beings provide an existence proof of language acquisition from speech completely without language supervision, it is plausible that this ability could be replicated by a machine learning algorithm.
Revised: Because human beings provide an existence proof of language acquisition from speech completely without language supervision, it is plausible that this ability could be replicated by a machine learning algorithm.

Original: In this paper, we present a method for discovering discrete and hierarchical representations of speech units both at the sub-word level and the word level.
Revised: In this paper, we present a method for discovering discrete and hierarchical representations of speech units both at the sub-word level and the word level.

Original: The key difference between our work and previously proposed linguistic unit discovery methods is the fact that we employ a discriminative, visual-semantic groundingobjective rather than a signal reconstruction objective.
Revised: Previously proposed linguistic unit discovery methods have only leveraged the speech audio modality in isolation, relying on objective functions that attempt to capture statistical regularities within the speech signal. The key innovation in our work is that we discover units by training models with explicit discretization layers to associate speech waveforms with visual images using a cross-modal grounding objective.

Original: This forces our models to learn representations which capture semantic information at the output layer of the network.
Revised: This forces our models to learn representations which capture semantic information at the highest layers of the network.

Original: Because semantics are predominantly carried by words, and words are composed of sub-word units (such as phones and syllables), the visual grounding objective indirectly forces the model to learn speaker- and noise-invariant representations of speech units.
Revised: Because semantics are predominantly carried by words, and words are composed of subword units (such as phones and syllables), the visual grounding objective indirectly forces the model to learn speaker- and noise-invariant representations of speech units.

Original: By incorporating trainable quantization layers into our networks, we are able to capture these units in discrete inventories.
Revised: By incorporating trainable quantization layers into our networks, we are able to capture these units in discrete inventories.

Original: Whether these units correspond to word-like or sub-word units depends on where the quantization layers are inserted, and how they are trained.
Revised: Whether these units correspond to word-like or sub-word units depends on where the quantization layers are inserted, and how they are trained.

Original:  R ELATED W ORK
Revised:  R ELATED W ORK

Original: Prior work on unsupervised modeling of the speech signal has generally focused on learning representations which either disentangle or isolate the latent factors that are of interest for downstream tasks.
Revised: Prior work on unsupervised modeling of the speech signal has generally focused on learning representations which either disentangle or isolate the latent factors that are of interest for downstream tasks.

Original: In most cases the primary latent factor of interest is the phonetic or lexical identity of a given segment of speech, but other factors, such as the identity of the speaker, are sometimes of interest as well.
Revised: In most cases the primary latent factor of interest is the phonetic or lexical identity of a given segment of speech, but other factors, such as the identity of the speaker, are sometimes of interest as well.

Original: Because the factors of interest are often inherently discrete (e.g. words and phones), many of the proposed approaches attempt to perform segmentation and clustering of the surface features in one way or another.
Revised: Because the factors of interest are often inherently discrete (e.g. words and phones), many of the proposed approaches attempt to perform segmentation and clustering of the surface features in one way or another.

Original: One family of techniques is based upon Segmental Dynamic Time
Revised: One family of techniques is based upon Segmental Dynamic Time

Original: Warping (S-DTW) (Park & Glass, 2005; 2008; Jansen et al., 2010; Jansen &  Durme, 2011), which uses a selfcomparison algorithm to identify relatively long duration (on the order of a second) patterns which frequently reoccur in a speech corpus; these patterns tend to capture words or short phrases.
Revised: Warping (S-DTW) (Park & Glass, 2005; 2008; Jansen et al., 2010; Jansen & Van Durme, 2011), which uses a self-comparison algorithm to identify relatively long duration (on the order of a second) patterns which frequently reoccur in a speech corpus; these patterns tend to capture words or short phrases.

Original: A different line of work employs probabilistic graphical models to jointly segment and cluster the speech signal (Varadarajan et al., 2008; Zhang & Glass, 2009; Gish et al., 2009; Lee & Glass, 2012;
Revised: A different line of work employs probabilistic graphical models to jointly segment and cluster the speech signal (Varadarajan et al., 2008; Zhang & Glass, 2009; Gish et al., 2009; Lee & Glass, 2012;

Original: Siu et al., 2014; Lee et al., 2015; Ondel et al., 2016; Kamper et al., 2016; 2017a).
Revised: Siu et al., 2014; Lee et al., 2015; Ondel et al., 2016; Kamper et al., 2016; 2017a).

Original: With an appropriately designed model, it is possible to learn multiple, hierarchical categories of speech units.
Revised: With an appropriately designed model, it is possible to learn multiple, hierarchical categories of speech units.

Original: However, in order to enable efﬁcient inference, the conditional distributions of these models tend to be simple and therefore have limited modeling power.
Revised: However, in order to enable efﬁcient inference, the conditional distributions of these models tend to be simple and therefore have limited modeling power.

Original: Deep neural network models have been successfully used to learn powerful speech representations using weakly or unsupervised objectives (Thiolliere et al., 2015; Kamper et al., 2015; Hsu et al., 2017a;b; Hsu & Glass, 2018; Holzenberger et al., 2018; Milde & Biemann, 2018; van den Oord et al., 2018;
Revised: Deep neural network models have been successfully used to learn powerful speech representations using weakly or unsupervised objectives (Thiolliere et al., 2015; Kamper et al., 2015; Hsu et al., 2017a;b; Hsu & Glass, 2018; Holzenberger et al., 2018; Milde & Biemann, 2018; van den Oord et al., 2018;

Original: Chung et al., 2019; Pascual et al., 2019).
Revised: Chung et al., 2019; Pascual et al., 2019).

Original: These representations have predominantly been continuous in nature, as discrete latent variables are not trivially compatible with backpropagation.
Revised: These representations have predominantly been continuous in nature, as discrete latent variables are not trivially compatible with backpropagation.

Original: To obtain discrete representations, a post-hoc clustering step can be applied to the continuous representations (Kamper et al., 2017b;
Revised: To obtain discrete representations, a post-hoc clustering step can be applied to the continuous representations (Kamper et al., 2017b;

Original: Feng et al., 2019).
Revised: Feng et al., 2019).

Original: More recently, several papers have proposed ways of directly incorporating discrete variables into neural network models, including using
Revised: More recently, several papers have proposed ways of directly incorporating discrete variables into neural network models, including using

Original: Gumbel-Softmax (Eloff et al., 2019b) or straight-through estimators (van den Oord et al., 2017;
Revised: Gumbel-Softmax (Eloff et al., 2019b) or straight-through estimators (van den Oord et al., 2017;

Original: Chorowski et al., 2019; Razavi et al., 2019).
Revised: Chorowski et al., 2019; Razavi et al., 2019).

Original: A different method for learning meaningful representations of speech is via a multimodal grounding objective, which encourages the learning of speech representations that are predictive of the contextual information contained in a separate but accompanying modality, such as vision.
Revised: A different method for learning meaningful representations of speech is via a multimodal grounding objective, which encourages the learning of speech representations that are predictive of the contextual information contained in a separate but accompanying modality, such as vision.

Original: Visual grounding of speech is a form of self-supervised learning (Virginia de Sa, 1994), which is powerful in part because it offers a way of training models with a discriminative objective that does not depend on traditional transcriptions or annotations.
Revised: Visual grounding of speech is a form of self-supervised learning (Virginia de Sa, 1994), which is powerful in part because it offers a way of training models with a discriminative objective that does not depend on traditional transcriptions or annotations.

Original: The ﬁrst work in this direction relied on phone strings to represent the speech (Roy & Pentland, 2002; Roy, 2003), but more recently this learning has been shown to be possible directly on the speech signal (Synnaeve et al., 2014; Harwath & Glass, 2015;
Revised: The ﬁrst work in this direction relied on phone strings to represent the speech (Roy & Pentland, 2002; Roy, 2003), but more recently this learning has been shown to be possible directly on the speech signal (Synnaeve et al., 2014; Harwath & Glass, 2015;

Original: Harwath et al., 2016).
Revised: Harwath et al., 2016).

Original: Subsequent work on visually-grounded models of speech has investigated improvements and alternatives to the modeling or training algorithms (Leidal et al., 2017; Kamper et al., 2017c; Havard et al., 2019a; Merkx et al., 2019; Chrupała et al., 2017; Scharenborg et al., 2018; Kamper et al., 2019b;a; Sur´ıs et al., 2019; Ilharco et al., 2019; Eloff et al., 2019a), application to multilingual settings (Harwath et al., 2018a; Kamper & Roth, 2017; Azuh et al., 2019;
Revised: Subsequent work on visually-grounded models of speech has investigated improvements and alternatives to the modeling or training algorithms (Leidal et al., 2017; Kamper et al., 2017c; Havard et al., 2019a; Merkx et al., 2019; Chrupała et al., 2017; Scharenborg et al., 2018; Kamper et al., 2019b;a; Sur´ıs et al., 2019; Ilharco et al., 2019; Eloff et al., 2019a), application to multilingual settings (Harwath et al., 2018a; Kamper & Roth, 2017; Azuh et al., 2019;

Original: Havard et al., 2019a), analysis of the linguistic abstractions, such as words and phones, which are learned by the models (Harwath & Glass, 2017; Harwath et al., 2018b; Drexler & Glass, 2017; Alishahi et al., 2017; Harwath et al., 2019; Harwath & Glass, 2019; Havard et al., 2019b), and the impact of jointly training with textual input (Holzenberger et al., 2019; Chrupała, 2019; Pasad et al., 2019).
Revised: Havard et al., 2019a), analysis of the linguistic abstractions, such as words and phones, which are learned by the models (Harwath & Glass, 2017; Harwath et al., 2018b; Drexler & Glass, 2017; Alishahi et al., 2017; Harwath et al., 2019; Harwath & Glass, 2019; Havard et al., 2019b), and the impact of jointly training with textual input (Holzenberger et al., 2019; Chrupała, 2019; Pasad et al., 2019).

Original: Representations learned by models of visually grounded speech are also well-suited for transfer learning to supervised tasks, being highly robust to noise and domain shift (Hsu et al., 2019).
Revised: Representations learned by models of visually grounded speech are also well-suited for transfer learning to supervised tasks, being highly robust to noise and domain shift (Hsu et al., 2019).

Original:  D ATA AND M ODELS 3. D ATASET
Revised:  D ATA AND M ODELS 3. D ATASET

Original: For training our models, we utilize the MIT Places 205 dataset (Zhou et al., 2014) and their accompanying spoken audio captions (Harwath et al., 2016; 2018b).
Revised: For training our models, we utilize the MIT Places 205 dataset (Zhou et al., 2014) and their accompanying spoken audio captions (Harwath et al., 2016; 2018b).

Original: The caption dataset contains approximately 400,000 spoken audio captions, each of which describes a different Places image.
Revised: The caption dataset contains approximately 400,000 spoken audio captions, each of which describes a different Places image.

Original: These captions are free-form spontaneous speech, collected from over 2,500 different speakers and covering a 40,000 word vocabulary.
Revised: These captions are free-form spontaneous speech, collected from over 2,500 different speakers and covering a 40,000 word vocabulary.

Original: The average caption duration is approximately 10 seconds, and each caption contains on average 20 words.
Revised: The average caption duration is approximately 10 seconds, and each caption contains on average 20 words.

Original: For vetting our models during training, we use a held-out validation set of 1,000 image-caption pairs.
Revised: For vetting our models during training, we use a held-out validation set of 1,000 image-caption pairs.

Original: 3. N EURAL M ODELS OF V ISUALLY -G ROUNDED S PEECH
Revised: 3. N EURAL M ODELS OF V ISUALLY -G ROUNDED S PEECH

Original: We base our model upon the Residual Deep Audio-Visual Embedding network (ResDAVEnet) architecture (Harwath et al., 2019), which contains two branches of fully convolutional networks, one for images and the other for audio.
Revised: We base our model upon the Residual Deep Audio-Visual Embedding network (ResDAVEnet) architecture (Harwath et al., 2019), which contains two branches of fully convolutional networks, one for images and the other for audio.

Original: Each branch encodes samples of the corresponding modality into a d -dimensional space, regardless of the original dimensionality of the samples.
Revised: Each branch encodes samples of the corresponding modality into a d -dimensional space, regardless of the original dimensionality of the samples.

Original: This is achieved by applying global spatial mean pooling and global temporal mean pooling to the image branch output and the audio branch output, respectively.
Revised: This is achieved by applying global spatial mean pooling and global temporal mean pooling to the image branch output and the audio branch output, respectively.

Original: The image branch is adapted from ResNet50 (He et al., 2016), where the ﬁnal softmax layer and the preceding fully-connected layers are removed, replaced with a 1x1 linear convolutional layer in order to project the feature map to the desired dimension.
Revised: The image branch is adapted from ResNet50 (He et al., 2016), where the ﬁnal softmax layer and the preceding fully-connected layers are removed, replaced with a 1x1 linear convolutional layer in order to project the feature map to the desired dimension.

Original: To model the audio inputs, a 17-layer fully convolutional network with residual connections is used.
Revised: To model the audio inputs, a 17-layer fully convolutional network with residual connections is used.

Original: The input is a log Mel-frequency spectrogram with 40 frequency bins and 25 ms-wide, Hammingwindowed frames with a shift of 10 ms.
Revised: The input is a log Mel-frequency spectrogram with 40 frequency bins and 25 ms-wide, Hammingwindowed frames with a shift of 10 ms.

Original: The ﬁrst layer of this network is a 1-D convolution that spans the entire frequency axis of the spectrogram, while the remaining 16 convolutional layers are 1-D across the time axis.
Revised: The ﬁrst layer of this network is a 1-D convolution that spans the entire frequency axis of the spectrogram, while the remaining 16 convolutional layers are 1-D across the time axis.

Original: These 16 layers are divided into four residual blocks of 4 layers each, and downsampling between these blocks is accomplished by applying the ﬁrst convolution of each block with a stride of 2.
Revised: These 16 layers are divided into four residual blocks of 4 layers each, and downsampling between these blocks is accomplished by applying the ﬁrst convolution of each block with a stride of 2.

Original: For full details of the model, refer to Harwath et al.
Revised: For full details of the model, refer to Harwath et al.

Original: 3. L EARNING H IERARCHICAL D ISCRETE U NITS WITH V ECTOR Q UANTIZING L AYERS
Revised: 3. L EARNING H IERARCHICAL D ISCRETE U NITS WITH V ECTOR Q UANTIZING L AYERS

Original: Previous analyses reveal that ResDAVEnet-like models learn linguistic abstractions at different levels, including words (Harwath & Glass, 2017) and robust phonetic features (Harwath & Glass, 2019;
Revised: Previous analyses reveal that ResDAVEnet-like models learn linguistic abstractions at different levels, including words (Harwath & Glass, 2017) and robust phonetic features (Harwath & Glass, 2019;

Original: Hsu et al., 2019).
Revised: Hsu et al., 2019).

Original: To explicitly learn hierarchical discrete linguistic units within this framework, we propose to incorporate multiple vector quantization (VQ) layers (van den Oord et al., 2017) into the
Revised: To explicitly learn hierarchical discrete linguistic units within this framework, we propose to incorporate multiple vector quantization (VQ) layers (van den Oord et al., 2017) into the

Original: ResDAVEnet audio branch
Revised: ResDAVEnet audio branch

Original: ; we refer to this new architecture as ResDAVEnet-VQ.
Revised: ; we refer to this new architecture as ResDAVEnet-VQ.

Original: VQ layers can be understood as a type of bottleneck, which constrain the amount of information that can ﬂow through.
Revised: VQ layers can be understood as a type of bottleneck, which constrain the amount of information that can ﬂow through.

Original: While these layers have been used to learn discrete sub-word units (van den
Revised: While these layers have been used to learn discrete sub-word units (van den

Original: Oord et al., 2017; Chorowski et al., 2019; Razavi et al., 2019), previous work injects VQ layers into autoencoders that are trained with a reconstruction loss.
Revised: Oord et al., 2017; Chorowski et al., 2019; Razavi et al., 2019), previous work injects VQ layers into autoencoders that are trained with a reconstruction loss.

Original: As a result, the embedding dimension of each code and the number of codes need to be carefully tuned (Liu et al., 2019).
Revised: As a result, the embedding dimension of each code and the number of codes need to be carefully tuned (Liu et al., 2019).

Original: When the embedding dimension is too low or the codebook size too small, the model does not have enough expressive power to capture linguistic variability.
Revised: When the embedding dimension is too low or the codebook size too small, the model does not have enough expressive power to capture linguistic variability.

Original: When it is too large, the model starts to encode non-linguistic information in order to improve reconstruction.
Revised: When it is too large, the model starts to encode non-linguistic information in order to improve reconstruction.

Original: In contrast, the learning signal of
Revised: In contrast, the learning signal of

Original: ResDAVEnet-VQ is provided by the visual-semantic grounding objective.
Revised: ResDAVEnet-VQ is provided by the visual-semantic grounding objective.

Original: Rather than encoding as much information about input as possible, the learned codes in ResDAVEnet-VQ only need to capture semantic information.
Revised: Rather than encoding as much information about input as possible, the learned codes in ResDAVEnet-VQ only need to capture semantic information.

Original: Since semantics in speech are predominantly transmitted by words, and words are composed of sub-word units like phones, the grounding objective places pressure on the model to robustly infer both from speech.
Revised: Since semantics in speech are predominantly transmitted by words, and words are composed of sub-word units like phones, the grounding objective places pressure on the model to robustly infer both from speech.

Original: Since words and phones are inherently discrete symbols, representing them with learned discrete units may not even hurt the grounding performance.
Revised: Since words and phones are inherently discrete symbols, representing them with learned discrete units may not even hurt the grounding performance.

Original: Figure 1 illustrates the proposed ResDAVEnet-VQ model.
Revised: Figure 1 illustrates the proposed ResDAVEnet-VQ model.

Original: We add a quantization layer after each of the ﬁrst two residual blocks of the ResDAVEnet-VQ model, denoted as VQ2 and VQ3, respectively, with the intention that they should capture discreteword-like and sub-word-like  units.
Revised: We add a quantization layer after each of the ﬁrst two residual blocks of the ResDAVEnet-VQ model, denoted as VQ2 and VQ3, respectively, with the intention that they should capture discrete sub-word-like and word-like units.

Original: A VQ layer is deﬁned as E ∈ R K × D , where K represents the codebook size, and D represents the output dimensionality of the input features to the codebook.
Revised: A VQ layer is deﬁned as E ∈ R K × D , where K represents the codebook size, and D represents the output dimensionality of the input features to the codebook.

Original: Denoting the t th temporal frame of the input to the quantization layer as x t , quantization is performed according to q t =
Revised: Denoting the t th temporal frame of the input to the quantization layer as x t , quantization is performed according to q t =

Original: E k, : , where k = arg min j || x t − E j, : || 

Revised: E k, : , where k = arg min j || x t − E j, : ||

Original: The quantized output is then fed as input to the subsequent residual block.
Revised: The quantized output is then fed as input to the subsequent residual block.

Original: As in van den Oord et al.
Revised: As in van den Oord et al.

Original: (2017), we use the straight-through estimator (Bengio et al., 2013) to compute the gradient passed from q t to x t .
Revised: (2017), we use the straight-through estimator (Bengio et al., 2013) to compute the gradient passed from q t to x t .

Original: We use the exponential moving average (EMA) codebook updates proposed by van den Oord et al.[Figure] Conv Res
Revised: We use the exponential moving average (EMA) codebook updates proposed by van den Oord et al.

Original: VQ

Original: [Figure]

Original: Res

Original: VQ

Original: Res

Original: Res x e x e e  x 1 e

Original: Bypass

Original: [Figure]
Revised: [Figure]

Original: 3. C ODEBOOK L EARNING S CHEDULES
Revised: 3. C ODEBOOK L EARNING S CHEDULES

Original: We include multiple VQ layers in the ResDAVEnet-VQ model, each of which can be independently enabled or bypassed without changing the rest of the architecture conﬁguration.
Revised: We include multiple VQ layers in the ResDAVEnet-VQ model, each of which can be independently enabled or bypassed without changing the rest of the architecture conﬁguration.

Revised: When all model weights, including the VQ codebooks, are trained jointly in a single training run we call this a “coldstart” model.

Original: A model can be warm-started by copying the weights from another trained model that has fewer  VQ layers enabled, and randomly initializing the codebook of the newly activated VQ layer(s).
Revised: Alternatively, a model can be “warm-started” by copying the weights from another trained model that has fewer (or no) VQ layers enabled, and randomly initializing the codebook of the newly activated VQ layer(s).

Original: This gives rise to the questions of how many quantizers should be used and in what order they should be enabled.
Revised: This gives rise to the questions of how many quantizers should be used and in what order they should be enabled.

Original: It is unclear whether models with  same VQ layers activated would learn the same representation at each layer regardless of the training curriculum.
Revised: It is unclear whether models with the same VQ layers activated would learn the same representation at each layer regardless of the training curriculum.

Original: Let A m denote a subset of all VQ layers, and A m − 1 ⊂ A m .
Revised: Let A m denote a subset of all VQ layers, and A m − 1 ⊂ A m .

Original: We use “ A 1 → ...
Revised: We use “ A 1 → ...

Original: → A M ” to denote a model that is obtained by sequentially training models “ A 1 → ...
Revised: → A M ” to denote a model that is obtained by sequentially training models “ A 1 → ...

Original: → A m ” initialized from “ A 1 → ...
Revised: → A m ” initialized from “ A 1 → ...

Original: → A m − 1 ”, where the model A 1 is initialized from scratch, and the ﬁnal model would have VQ layers in A M activated.
Revised: → A m − 1 ”, where the model A 1 is initialized from scratch, and the ﬁnal model would have VQ layers in A M activated.

Original: For instance, a model initialized from scratch with no VQ layers enabled is denoted as “ ∅ ”, and a model initialized with that and with both layers enabled is denoted as “ ∅ → { 2 , 3 } ”.
Revised: For instance, a model initialized from scratch with no VQ layers enabled is denoted as “ ∅ ”, and a model initialized with that and with both layers enabled is denoted as “ ∅ → { 2 , 3 } ”.

Original: 3. T RAINING WITH THE T RIPLET L OSS
Revised: 3. T RAINING WITH THE T RIPLET L OSS

Original: We train our models using the same loss function as Harwath et al.
Revised: We train our models using the same loss function as Harwath et al.

Original: This loss function blends two triplet loss terms (Weinberger & Saul, 2009), one based on random sampling of negative examples, and the other based on semi-hard negative mining (Jansen et al., 2018), in order to ﬁnd more challenging negative samples.
Revised: This loss function blends two triplet loss terms (Weinberger & Saul, 2009), one based on random sampling of negative examples, and the other based on semi-hard negative mining (Jansen et al., 2018), in order to ﬁnd more challenging negative samples.

Original: Speciﬁcally, let the sets of output embedding vectors for a minibatch of B audio/image training pairs respectively be A
Revised: Speciﬁcally, let the sets of output embedding vectors for a minibatch of B audio/image training pairs respectively be A

Original: = { a 1 , . .
Revised: = { a 1 , . .

Original: . , a B } and I =
Revised: . , a B } and I =

Original: { i 1 , . .
Revised: { i 1 , . .

Original: , i B } .
Revised: , i B } .

Original: To compute the randomly-sampled triplet loss term, we select impostor examples for the j th input according to ¯ a j ∼ UniformCategorical ( { a 1 , . .
Revised: To compute the randomly-sampled triplet loss term, we select impostor examples for the j th input according to ¯ a j ∼ UniformCategorical ( { a 1 , . .

Original: . , a B }\ a j ) and ¯ i j ∼ UniformCategorical ( { i 1 , . .
Revised: . , a B }\ a j ) and ¯ i j ∼ UniformCategorical ( { i 1 , . .

Original: , i B }\ i j ) .
Revised: , i B }\ i j ) .

Original: The randomly-sampled triplet loss is then computed as:
Revised: The randomly-sampled triplet loss is then computed as:

Original: [Equation]
Revised: [Equation]

Original: For the semi-hard negative triplet loss, we ﬁrst deﬁne the sets of impostor candidates for the j th example as ˆ A j = { a ∈ A | i Tj a < i 
Revised: For the semi-hard negative triplet loss, we ﬁrst deﬁne the sets of impostor candidates for the j th example as ˆ A j = { a ∈ A | i T j a < i T j a j } and ˆ I

Original: Tj a j } and ˆ I j = { i ∈ I | i T a j < i 
Revised: j = { i ∈ I | i T a j < i T

Original: Tj a j } .
Revised: j a j } .

Original: The semi-hard negative loss is then computed as:
Revised: The semi-hard negative loss is then computed as:

Original: [Equation]
Revised: [Equation]

Original: Finally, the overall loss function is computed as the sum of the two above losses, L = L s + L h .
Revised: Finally, the overall loss function is computed by combining the two above losses, L = L s + L h , which was found by (Harwath et al., 2019) to outperform either loss on its own.

Original: 3. I MPLEMENTATION D ETAILS
Revised: 3. I MPLEMENTATION D ETAILS

Original: All of our models were trained for 180 epochs using the Adam optimizer (Kingma & Ba, 2014) with a batch size of 80.
Revised: All of our models were trained for 180 epochs using the Adam optimizer (Kingma & Ba, 2014) with a batch size of 80.

Original: We used an exponentially decaying learning rate schedule, with an initial value of 2e-4 that decayed by a factor of 0.95 every 3 epochs.
Revised: We used an exponentially decaying learning rate schedule, with an initial value of 2e-4 that decayed by a factor of 0.95 every 3 epochs.

Original: Following van den Oord et al.
Revised: Following van den Oord et al.

Original: (2017), we use an EMA decay factor of γ = .
Revised: (2017), we use an EMA decay factor of γ = .

Original: 99 for training each VQ codebook.
Revised: 99 for training each VQ codebook.

Original: Our core experimental results all use a codebook size of 1024 vectors for all quantizers, but in the supplementary material we include experiments with smaller and larger codebooks.
Revised: Our core experimental results all use a codebook size of 1024 vectors for all quantizers, but in the supplementary material we include experiments with smaller and larger codebooks.

Original: Following Chorowski et al.
Revised: Following Chorowski et al.

Original: (2019), the jitter probability hyperparameter for each quantization layer was ﬁxed at 0.12.
Revised: (2019), the jitter probability hyperparameter for each quantization layer was ﬁxed at 0.12.

Original: While we do not apply data augmentation to the input spectrograms, during training we perform standard data augmentation techniques to the images.
Revised: While we do not apply data augmentation to the input spectrograms, during training we perform standard data augmentation techniques to the images.

Original: We resize each raw image so that its smallest dimension is 256 pixels, and then we apply an Inception-style random crop which is resized to 224 pixels square.
Revised: We resize each raw image so that its smallest dimension is 256 pixels, and then we apply an Inception-style random crop which is resized to 224 pixels square.

Original: During training, we also ﬂip each image horizontally with a probability of 0.5.
Revised: During training, we also ﬂip each image horizontally with a probability of 0.5.

Original: During evaluation, the center 224 pixel square crop is always taken from the image.
Revised: During evaluation, the center 224 pixel square crop is always taken from the image.

Original: Finally, the RGB pixel values are mean and variance normalized.
Revised: Finally, the RGB pixel values are mean and variance normalized.

Original: We trained each model on the Places audio caption train split, and computed the image and caption recall at 10 (R@10) scores on the validation split of the Places audio captions after each training epoch.
Revised: We trained each model on the Places audio caption train split, and computed the image and caption recall at 10 (R@10) scores on the validation split of the Places audio captions after each training epoch.

Original: The model snapshot that achieved the highest average R@10 score on the validation set from each training is used for all evaluation.
Revised: The model snapshot that achieved the highest average R@10 score on the validation set from each training is used for all evaluation.

Original: To extract embeddings and units from our models, we simply perform a forward pass through the speech branch of the ResDAVEnet-VQ network and retain the outputs from the target layer at a uniform frame-rate.
Revised: To extract embeddings and units from our models, we simply perform a forward pass through the speech branch of the ResDAVEnet-VQ network and retain the outputs from the target layer at a uniform frame-rate.

Original: The frame-rate is determined by the downsampling factor at the target layer relative to the input.
Revised: The frame-rate is determined by the downsampling factor at the target layer relative to the input.

Original: For non-quantized layers, these outputs will be continuous embeddings.
Revised: For non-quantized layers, these outputs will be continuous embeddings.

Original: For quantized layers, these will be quantized embedding retrieved from the assigned entry in the codebook.
Revised: For quantized layers, these will be quantized embedding retrieved from the assigned entry in the codebook.

Original:  E XPERIMENTS 4. S UB -W ORD U NIT L EARNING ON THE Z ERO S PEECH 2019 ABX T ASK
Revised:  E XPERIMENTS 4. S UB -W ORD U NIT L EARNING ON THE Z ERO S PEECH 2019 ABX T ASK

Original: Evaluation metrics Learning unsupervised speech representations that are indicative of phonetic content is of high interest to the speech community, and recently has been the focus of the ZeroSpeech Challenge (Versteegh et al., 2015; Dunbar et al., 2017; 2019).
Revised: Evaluation metrics Learning unsupervised speech representations that are indicative of phonetic content is of high interest to the speech community, and recently has been the focus of the ZeroSpeech Challenge (Versteegh et al., 2015; Dunbar et al., 2017; 2019).

Original: One of the core evaluations is themodel-free, minimal-pair ABX task (Schatz et al., 2013), which aims to benchmark representations in terms of their discriminability between different sub-word speech units.
Revised: One of the core evaluations is the minimal-pair ABX task (Schatz et al., 2013), which aims to benchmark representations in terms of their discriminability between different sub-word speech units.

Original: In this task, three speech waveform segments denoted by A , B , and X are presented to a model.
Revised: In this task, a model is tasked with extracting representations for a triplet of speech waveform segments denoted by A , B , and X .

Original: A and B are constrained to be a triphone minimal pair; that is, both segments capture three phones, but differ only in the identity of their center phone.
Revised: A and B are constrained to be a triphone minimal pair; that is, both segments capture three phones, but differ only in the identity of their center phone.

Original: The third segment, X is chosen to contain the same underlying triphone sequence as A .
Revised: The third segment, X is chosen to contain the same underlying triphone sequence as A .

Original: Supposing f ( · ) is a function to be evaluated that maps a waveform segment to a feature segment, the ABX error rate under a given similarity metric S ( · , · ) is deﬁned as the fraction of ABX triples in which S ( f ( A ) , f ( X )) > S ( f ( B ) , f ( X )) .
Revised: Supposing f ( · ) denotes the model’s mapping function from a waveform segment to a sequence of embedding vectors, the ABX error rate under a given similarity metric S ( · , · ) is deﬁned as the fraction of ABX triples in which S ( f ( A ) , f ( X )) > S ( f ( B ) , f ( X )) .

Original: An ABX error rate of 50% indicates random assignment, while an ABX of 0% reﬂects perfect phone discriminability.
Revised: An ABX error rate of 50% indicates random assignment, while an ABX of 0% reﬂects perfect phone discriminability.

Original: In the ZeroSpeech challenge, dynamic time warping (DTW) is used to measure similarity between a pair of feature sequences for ABX evaluations, with several options available for computing the distance between a pair of feature frames.
Revised: In the ZeroSpeech challenge, S ( · , · ) is implemented using Dynamic Time Warping (DTW) with various distance measures (cosine, KL, etc.). In our evaluation, we use the cosine distance.

Original: The ZeroSpeech 2019 challenge in particular emphasizes on discovering an inventory of discrete sub-word units, rather than continuous representations.
Revised: The ZeroSpeech 2019 challenge in particular emphasizes on discovering an inventory of discrete sub-word units, rather than continuous representations.

Original: Therefore, in addition to an ABX error rate, a bitrate is also computed for each model which reﬂects the amount of information carried by the learned units.
Revised: Therefore, in addition to an ABX error rate, a bitrate is also computed for each model which reﬂects the amount of information carried by the learned units.

Original: A lower bitrate can be achieved by having a more compact inventory of learned units or having a smaller number of codes per second.
Revised: A lower bitrate can be achieved by having a more compact inventory of learned units or having a smaller number of codes per second.

Original: The full details of the evaluation can be found in Dunbar et al.
Revised: The full details of the evaluation can be found in Dunbar et al.

Original: To be clear, all of our ResDAVEnet-VQ models were not trained on the  ZeroSpeech training data, but instead on the Places audio captions, thus there is a domain mismatch between training and testing these models.
Revised: To be clear, all of our ResDAVEnet-VQ models were not trained on the [Table] ZeroSpeech training data, but instead on the Places audio captions, thus there is a domain mismatch between training and testing these models.

Original: In addition to the frame-based bitrate and ABX scores computed by the ZeroSpeech 2019 evaluation toolkit, we implement our own extensions to these metrics.
Revised: In addition to the frame-based bitrate and ABX scores computed by the ZeroSpeech 2019 evaluation toolkit, we implement our own extensions to these metrics.

Original: Because it is common for successive frames to be assigned to the same codebook entry and phonetic information is not encoded at a ﬁxed frame rate, lossless run length encoding (RLE) can be a more reasonable measure of the bitrate of a frame-based model.
Revised: Because it is common for successive frames to be assigned to the same codebook entry and phonetic information is not encoded at a ﬁxed frame rate, lossless run length encoding (RLE) can be a more reasonable measure of the bitrate of a frame-based model.

Original: RLE does not change the ABX score since it can be trivially inverted, but it does change the bitrate.
Revised: RLE does not change the ABX score since it can be trivially inverted, but it does change the bitrate.

Original: For computing the RLE bitrate, we modify the bitrate calculation speciﬁed in Dunbar et al.
Revised: For computing the RLE bitrate, we modify the bitrate calculation speciﬁed in Dunbar et al.

Original: so that a unique symbol is deﬁned as the tuple (unit, length) where length is[Table] the number of frames assigned to a given unit with in a segment.
Revised: so that a unique symbol is deﬁned as the tuple (unit, length) where length is the number of frames assigned to a given unit with in a segment.

Original: We also consider segment-based
Revised: We also consider segment-based

Original: ABX and bitrate, which is similar to the RLE metrics except in this case we outright discard the frame length information.
Revised: ABX and bitrate, which is similar to the RLE metrics except in this case we outright discard the frame length information.

Original: This typically results in an even greater reduction in bitrate, but also an accompanying deterioration in ABX score.
Revised: This typically results in an even greater reduction in bitrate, but also an accompanying deterioration in ABX score.

Original: Baseline models
Revised: Baseline models

Original: In Table 1, we compare our results to those derived from two of the topperforming submissions to the ZeroSpeech 2019 challenge: a re-implementation of WaveNetVQ (Chorowski et al., 2019) provided by Cho et al.
Revised: In Table 1, we compare our results to those derived from two of the topperforming submissions to the ZeroSpeech 2019 challenge: a re-implementation of WaveNetVQ (Chorowski et al., 2019) provided by Cho et al.

Original: and FHVAE-DPGMM (Feng et al., 2019).
Revised: and FHVAE-DPGMM (Feng et al., 2019).

Original: Using the code accompanied with the WaveNet-VQ submission, we were able to train their model on the set of 400,000 Places audio captions to make a fairer comparison with our ResDAVEnet-VQ models in terms of the amount of speech data used.
Revised: Using the code accompanied with the WaveNet-VQ submission, we were able to train their model on the set of 400,000 Places audio captions to make a fairer comparison with our ResDAVEnet-VQ models in terms of the amount of speech data used.

Original: In addition, when trying to reproduce the reported WaveNet-VQ results, we obtain better performance than previously reported by training for more steps.
Revised: In addition, when trying to reproduce the reported WaveNet-VQ results, we obtain better performance than previously reported by training for more steps.

Original: Table 1 shows that WaveNet-VQ achieves similar bitrates regardless of the training data.
Revised: Table 1 shows that WaveNet-VQ achieves similar bitrates regardless of the training data.

Original: However, ABX deteriorates from 19.98 to 24.87, implying the model cannot utilize data of a larger scale but out-of-domain relative to the test set.
Revised: However, ABX deteriorates from 19.98 to 24.87, implying the model cannot utilize data of a larger scale but out-of-domain relative to the test set.

Original: A similar degradation when testing on out-of-domain data with FHVAE models was observed in Hsu et al.
Revised: A similar degradation when testing on out-of-domain data with FHVAE models was observed in Hsu et al.

Original: We did not re-train the model submitted by Feng et al.
Revised: We did not re-train the model submitted by Feng et al.

Original: and instead compare against the scores reported in Dunbar et al.
Revised: and instead compare against the scores reported in Dunbar et al.

Original: ABX discrimination without using quantization Our ﬁrst experiment investigates exactly which layer in the ResDAVEnet-VQ model is most suited for ABX phone discrimination, and would thus make a good candidate for learning of quantized sub-word units. 
Revised: ABX discrimination without using quantization Our ﬁrst experiment investigates exactly which layer in the ResDAVEnet-VQ model is most suited for ABX phone discrimination, and would thus make a good candidate for learning of quantized sub-word units. The leftmost plot in Figureshows that layers 2 and 3 of a ResDAVEnet-VQ model without any quantization enabled perform [Figure] X

Revised: E rr o r

Original: The leftmost plot in Figureshows that layers 2 and 3 of a ResDAVEnet-VQ model without any quantization enabled perform the best in terms of ABX error rate on the ZeroSpeech 2019 English test set; the exact numbers for this model are displayed in the caption of Figure 2.
Revised: R a t e Model “ ∅ ” layer1 layer layer3 layer Model “ { 2 } ” Model “ { 3 } ”

Revised: Epoch

Original:  Because layers 2 and 3 achieve the lowest ABX error rates without quantization, we focus our attention on the impact of quantization there.
Revised: [Figure] Epoch Epoch the best in terms of ABX error rate on the ZeroSpeech 2019 English test set; the exact numbers for this model are displayed in the caption of Figure 2. Because layers 2 and 3 achieve the lowest ABX error rates without quantization, we focus our attention on the impact of quantization there.

Original: Quantizing one layer When quantizing only one layer, we examine quantization of layer 2 vs. layer 3, and using cold-start training vs. warm-start initialization from model “ ∅ ”.
Revised: Quantizing one layer When quantizing only one layer, we examine quantization of layer 2 vs. layer 3, and using cold-start training vs. warm-start initialization from model “ ∅ ”.

Original: The ABX and bitrate results for these models, as well as the R@10 scores on the Places validation set, are shown in
Revised: The ABX and bitrate results for these models, as well as the R@10 scores on the Places validation set, are shown in

Original: Table 1.
Revised: Table 1.

Original: In all cases, quantization applied at the output of layer 2 achieves a better ABX score than quantization at layer 3, but VQ3 achieves a better bitrate.
Revised: In all cases, quantization applied at the output of layer 2 achieves a better ABX score than quantization at layer 3, but VQ3 achieves a better bitrate.

Original: Quantization barely impacts the performance of layer 2, whose ABX score very slightly rises from 11.35 to 11.79.
Revised: Quantization barely impacts the performance of layer 2, whose ABX score very slightly rises from 11.35 to 11.79.

Original: Warm-start initialization is beneﬁcial to R@10 and ABX score in both cases, but we notice an intriguing anomaly when applying cold-start quantization to layer 3: the ABX score deteriorates signiﬁcantly, rising from 10.[Figure] a [Figure]
Revised: Warm-start initialization is beneﬁcial to R@10 and ABX score in both cases, but we notice an intriguing anomaly when applying cold-start quantization to layer 3: the ABX score deteriorates signiﬁcantly, rising from 10.

Original: layer3 layer

Original: Model “ { 2 } ”

Original: Model “ { 3 } ”

Original: [Figure]

Original: Epoch

Original: [Figure]

Original: Epoch

Original: Epoch in the case of the non-quantized model to 38.21.
Revised: in the case of the non-quantized model to 38.21.

Original: This indicates that while VQ2 is capable of learning a ﬁnite inventory of units that are highly predictive of phonetic identity from either a warm-start or cold-start initialization, cold-start training of VQ3 results in very little phonetic information captured by the quantizer.
Revised: This indicates that while VQ2 is capable of learning a ﬁnite inventory of units that are highly predictive of phonetic identity from either a warm-start or cold-start initialization, cold-start training of VQ3 results in very little phonetic information captured by the quantizer.

Original: Interestingly, this model is still learning to infer visual semantics from the speech signal, as evidenced by a high R@10 score; we later show in Section 4.2 that the reason for this anomaly is because cold-start training of VQ3 results in the learning of word detectors.
Revised: Interestingly, this model is still learning to infer visual semantics from the speech signal, as evidenced by a high R@10 score; we later show in Section 4.2 that the reason for this anomaly is because cold-start training of VQ3 results in the learning of word detectors.

Original: In all cases except for model “ { 3 } ”, we note that the ABX scores achieved by our models are signiﬁcantly better than the baselines.
Revised: In all cases except for model “ { 3 } ”, we note that the ABX scores achieved by our models are signiﬁcantly better than the baselines.

Original: Our best model in terms of ABX (“ ∅ → { 2 } ”) achieves a 41.0% reduction in ABX over the WaveNet-VQ baseline, at a cost of a 132.3% increase in RLE bitrate; however, model “ ∅ → { 3 } ” achieves a 24.7% reduction in ABX error rate with only a 2.4% increase in RLE bitrate.
Revised: Our best model in terms of ABX (“ ∅ → { 2 } ”) achieves a 41.0% reduction in ABX over the WaveNet-VQ baseline, at a cost of a 132.3% increase in RLE bitrate; however, model “ ∅ → { 3 } ” achieves a 24.7% reduction in ABX error rate with only a 2.4% increase in RLE bitrate.

Original: These results do not constitute a fair comparison, however, because the WaveNet-VQ and
Revised: These results do not constitute a fair comparison, however, because the WaveNet-VQ and

Original: ResDAVEnet-VQ models were trained on different datasets; when training the WaveNet-VQ model on the same set of audio captions used to train ResDAVEnet-VQ (but without the accompanying images, since WaveNet-VQ is not a multimodal model), the ABX error rate increases to 24.87%, tipping the results even more in favor of the ResDAVEnet-VQ models.
Revised: ResDAVEnet-VQ models were trained on different datasets; when training the WaveNet-VQ model on the same set of audio captions used to train ResDAVEnet-VQ (but without the accompanying images, since WaveNet-VQ is not a multimodal model), the ABX error rate increases to 24.87%, tipping the results even more in favor of the ResDAVEnet-VQ models.

Original: Quantizing two layers Quantizing multiple layers at once offers the possibility of learning a hierarchy of units.
Revised: Quantizing two layers Quantizing multiple layers at once offers the possibility of learning a hierarchy of units.

Original: Thus, we aim to capture phonetic information in a lower layer quantizer and word-level information at a higher layer quantizer.
Revised: Thus, we aim to capture phonetic information in a lower layer quantizer and word-level information at a higher layer quantizer.

Original: Cold-start training of two quantizers (“ { 2 , 3 } ”) results in a signiﬁcant drop in ABX performance for both VQ2 and VQ3, but also a drop in R@10 on the Places validation set.
Revised: Cold-start training of two quantizers (“ { 2 , 3 } ”) results in a signiﬁcant drop in ABX performance for both VQ2 and VQ3, but also a drop in R@10 on the Places validation set.

Original: We see much better results in terms of R@10 and ABX for the remaining 3 models which were initialized from the “ ∅ ” model or a model with only one quantizer enabled; for example, model “ { 3 } → { 2 , 3 } ” achieves an ABX of 14.52 with an RLE bitrate of 136.11, representing a 27.3% ABX improvement over the best baseline while keeping the bitrate approximately the same.
Revised: We see much better results in terms of R@10 and ABX for the remaining 3 models which were initialized from the “ ∅ ” model or a model with only one quantizer enabled; for example, model “ { 2 } → { 2 , 3 } ” achieves an ABX of 14.52 with an RLE bitrate of 136.11, representing a 27.3% ABX improvement over the best baseline while keeping the bitrate approximately the same.

Original: We see in model “ { 3 } → { 2 , 3 } ” that the same phenomenon observed with model “ { 3 } ” persists: VQ3 achieves relatively poor ABX, despite a high overall R@10 and strong ABX with VQ2 at 13.55%.
Revised: We see in model “ { 3 } → { 2 , 3 } ” that the same phenomenon observed with model “ { 3 } ” persists: VQ3 achieves relatively poor ABX, despite a high overall R@10 and strong ABX with VQ2 at 13.55%.

Original: We conﬁrm in Section 4.2 that the VQ3 layer of model “ { 3 } → { 2 , 3 } ” does indeed capture word-level information, indicating that this model has successfully localized phonetic unit identity in the second layer and lexical unit identity in the third layer.
Revised: We conﬁrm in Section 4.2 that the VQ3 layer of model “ { 3 } → { 2 , 3 } ” does indeed capture word-level information, indicating that this model has successfully localized phonetic unit identity in the second layer and lexical unit identity in the third layer.

Original: Overall, our results suggest that when learning hierarchical quantized representations with a ResDAVEnet-VQ model, the nature of the representations learned is highly dependent on the training curriculum.
Revised: Overall, our results suggest that when learning hierarchical quantized representations with a ResDAVEnet-VQ model, the nature of the representations learned is highly dependent on the training curriculum [Table] [Table]

Original: Training and testing on noisy data In Hsu et al.
Revised: Training and testing on noisy data In Hsu et al.

Original: (2019), it was shown that representations learned by a ResDAVEnet model were far more robust to train/test domain mismatch in terms of background noise, channel characteristics, and speaker identity than standard spectral features when training a supervised speech recognizer.
Revised: (2019), it was shown that representations learned by a ResDAVEnet model were far more robust to train/test domain mismatch in terms of background noise, channel characteristics, and speaker identity than standard spectral features when training a supervised speech recognizer.

Original: Here, we examine whether this robustness is also exempliﬁed by the quantized versions of this model.
Revised: Here, we examine whether this robustness is also exempliﬁed by the quantized versions of this model.

Original: We construct three additional test sets using the ZeroSpeech 2019 English testing data by adding noise sampled from the AudioSet (Jansen et al., 2018) dataset.
Revised: We construct three additional test sets using the ZeroSpeech 2019 English testing data by adding noise sampled from the AudioSet (Jansen et al., 2018) dataset.

Original: For each ZeroSpeech testing waveform, we randomly sampled an AudioSet waveform of the same duration and performed linear mixing with a signal-to-noise ratio (SNR) selected randomly within[Table] a speciﬁed range.
Revised: For each ZeroSpeech testing waveform, we randomly sampled an AudioSet waveform of the same duration and performed linear mixing with a signal-to-noise ratio (SNR) selected randomly within a speciﬁed range.

Original: We construct low, medium, and high noise testing sets, corresponding to SNRs of 20-30 dB, 10-20 dB [Table] and 0-10 dB. We then perform the ABX discrimination task on these noisy waveforms, displaying the results in Table 2.
Revised: We construct low, medium, and high noise testing sets, corresponding to SNRs of 20-30 dB, 10-20 dB, and 0-10 dB. We then perform the ABX discrimination task on these noisy waveforms, displaying the results in Table 2.

Original: We ﬁnd that for all models, a worsening SNR results in a deterioration in ABX performance.
Revised: We ﬁnd that for all models, a worsening SNR results in a deterioration in ABX performance.

Original: However, the ResDAVEnet-VQ models prove to be far more noise robust than the Wavenet-VQ model; even in the high noise testing set, the best ResDAVEnetVQ model achieves an ABX of 23.96%, while the WaveNet-VQ models degrade to nearly-random
Revised: However, the ResDAVEnet-VQ models prove to be far more noise robust than the Wavenet-VQ model; even in the high noise testing set, the best ResDAVEnetVQ model achieves an ABX of 23.96%, while the WaveNet-VQ models degrade to nearly-random

Original: ABX scores of 42.55% and 42.67%.
Revised: ABX scores of 42.55% and 42.67%.

Original: Given that a ResDAVEnet-VQ model trained on the “clean” Places Audio captions is highly robust to additive noise on the ABX discrimination task, we investigated whether adding noise to the Places
Revised: Given that a ResDAVEnet-VQ model trained on the “clean” Places Audio captions is highly robust to additive noise on the ABX discrimination task, we investigated whether adding noise to the Places

Original: Audio captions themselves would result in an even higher degree of noise robustness.
Revised: Audio captions themselves would result in an even higher degree of noise robustness.

Original: To that end, we followed a similar data augmentation approach to create a noisy version of the Places Audio captions, where the SNR of each caption was randomly chosen to sit within the range of 0-30 dB.
Revised: To that end, we followed a similar data augmentation approach to create a noisy version of the Places Audio captions, where the SNR of each caption was randomly chosen to sit within the range of 0-30 dB.

Original: The bottom half of Table 2 shows the results of training several ResDAVEnet-VQ models on the noisy Places Audio captions and testing on the clean and noisy ZeroSpeech ABX tasks.
Revised: The bottom half of Table 2 shows the results of training several ResDAVEnet-VQ models on the noisy Places Audio captions and testing on the clean and noisy ZeroSpeech ABX tasks.

Original: In general, we observe a degradation ABX score in the clean conditions, but with a signiﬁcantly higher degree of noise robustness in the noisier conditions.[Figure] jhchshuwyeyihiyrehuherdxahaefthmnnggtpkszsiloyawaaayao owwlvhhbdhqd T
Revised: In general, we observe a degradation ABX score in the clean conditions, but with a signiﬁcantly higher degree of noise robustness in the noisier conditions.

Original: I M I T P h o n e L a b e l s Visualization of learned units To better measure the correspondence between the VQ units and
Revised: Visualization of learned units To better measure the correspondence between the VQ units and

Original: English phones, we compute corpus-level co-occurrence statistics (at the frame-level) across the
Revised: English phones, we compute corpus-level co-occurrence statistics (at the frame-level) across the

Original: TIMIT training set, excluding the sa dialect sentences.
Revised: TIMIT training set, excluding the sa dialect sentences.

Original: To facilitate visualization, we use the “ ∅ → { 2 } ” model with a codebook size of 128.
Revised: To facilitate visualization, we use the “ ∅ → { 2 } ” model with a codebook size of 128.

Original: We display the conditional probability matrix
Revised: We display the conditional probability matrix

Original: P ( phone | unit ) in Figure 3, with the rows and columns ordered via spectral co-clustering withclusters in order to group together phones that share similar sets of VQ codes.
Revised: P ( phone | unit ) in Figure 3, with the rows and columns ordered via spectral co-clustering with [Figure] uweyihyiytpkgayaaoyawwowaolzsshchjhsilmnngehahdxuheraerthfdhqdhhbv T

Revised: I M I T

Revised: P h

Original:  Visually, there is a strong mapping between TIMIT phone labels and ResDAVEnet-VQ codes.
Revised: o n e L a b e l s clusters in order to group together phones that share similar sets of VQ codes. Visually, there is a strong mapping between TIMIT phone labels and ResDAVEnet-VQ codes.

Original: In some cases, redundant codes are used for the same phone label (this is especially the case for the silence label) [Table] and in other cases we see that phones belonging to the same manner class often tend to share codebook units.
Revised: In some cases, redundant codes are used for the same phone label (this is especially the case for the silence label), and in other cases we see that phones belonging to the same manner class often tend to share codebook units.

Original: We can numerically quantify the mapping between the phone and unit labels with the normalized mutual information measure (NMI), which we found to be .378 in this case.
Revised: We can numerically quantify the mapping between the phone and unit labels with the normalized mutual information measure (NMI), which we found to be .378 in this case.

Original: We also include several  spectrograms with their time-aligned unit sequences in  the supplementary material.[Table]
Revised: We also include several caption spectrograms with their time-aligned unit sequences in Figures 5, 6, and 7 in the supplementary material.

Original: [Table]
Revised: [Table]

Original: 4. F ROM P HONES TO W ORDS : L EARNING A H IERARCHY OF U NITS
Revised: 4. F ROM P HONES TO W ORDS : L EARNING A H IERARCHY OF U NITS

Original: As shown in Table 1, all of the ResDAVEnet-VQ models which underwent cold-start training of VQ exhibited a similar phenomenon in which the ABX error rate of that layer was particularly high, despite the model performing well at the image-caption retrieval task.
Revised: As shown in Table [Table] all of the ResDAVEnet-VQ models which underwent cold-start training of VQ exhibited a similar phenomenon in which the ABX error rate of that layer was particularly high, despite the model performing well at the image-caption retrieval task.

Original: We hypothesized that this could be due to VQ3 learning to recognize higher level linguistic units, such as words.
Revised: We hypothesized that this could be due to VQ3 learning to recognize higher level linguistic units [Table] such as words.

Original: To examine this empirically, we inferred the VQ3 unit sequence for every audio caption in the Places Audio training set according to several different models.
Revised: To examine this empirically [Table] we inferred the VQ3 unit sequence for every audio caption in the Places Audio training set according to several different models.

Original: Using the estimated word-level transcriptions of the utterances (provided by the Google SpeechRecognition API), we computed precision, recall, and F1 scores for every unique (word, VQ3 code) pair for a given model and quantization layer.
Revised: Using the estimated word-level transcriptions of the utterances (provided by the Google SpeechRecognition API) [Table] we computed precision, recall, and F1 scores for every unique (word, VQ3 code) pair for a given model and quantization layer.

Original: We then ranked the VQ codes in descending order according to their maximum F1 score for any word in the vocabulary.
Revised: We then ranked the VQ codes in descending order according to their maximum F1 score for any word in the vocabulary.

Original: Table 3 shows a sampling of these statistics for model “ { 3 } → { 2 , 3 } ”, Table 
Revised: Table 3 shows a sampling of these statistics for model “ { 3 } → { 2 , 3 } ”. In the supplementary material, we include many more examples for this model in Table 7, as well as [Figure] [Figure]

Revised: [Figure]

Original: 1.0 Precision 0.
Revised: [Figure] [Figure] Precision [Figure]

Original: R e c a ll
Revised: R e c a ll

Original: Words of Top 250 F1-Scores "{2} {2,3}" "{3} {2,3}" "{3}" 0.
Revised: Words of Top 250 F1-Scores "{2} {2,3}" "{3} {2,3}" "{3}" 0.

Original: 1.0 F1 Threshold 

Revised: 1.0 F1 Threshold

Original: N u m b e r o f W o r d s
Revised: N u m b e r o f W o r d s

Original: Number of Words above F1 Threshold "{2} {2,3}" "{3} {2,3}" "{3}" "{2,3}" " {2,3}" " {3}"  for model “ { 2 } → { 2 , 3 } ”.
Revised: Number of Words above F1 Threshold "{2} {2,3}" "{3} {2,3}" "{3}" "{2,3}" " {2,3}" " {3}" examples for the “ { 2 } → { 2 , 3 } ” model (which did not learn VQ3 word detectors) in Table 8.

Original: It should be emphasized that these models are exactly the same in all respects, except for the order in which their quantizers were trained.
Revised: It should be emphasized that these models are exactly the same in all respects, except for the order in which their quantizers were trained.

Original: We examine the overall performance of VQ3 as a word detector for these models in Figure 4.
Revised: We examine the overall performance of VQ3 as a word detector for these models in Figure 4.

Original: The right hand side of Figure 4 displays the number of VQ3 codes whose maximum F1 score is above a given threshold, while the left hand side shows the distribution of precision and recall scores for the top 250 words ranked by F1.
Revised: The right hand side of Figure 4 displays the number of VQ3 codes whose maximum F1 score is above a given threshold, while the left hand side shows the distribution of precision and recall scores for the top 250 words ranked by F1.

Original: This gives an approximate indication of how many VQ3 codes have learned to specialize as detectors for a speciﬁc word.
Revised: This gives an approximate indication of how many VQ3 codes have learned to specialize as detectors for a speciﬁc word.

Original: We see that the VQ3 layer of model “ { 3 } → { 2 , 3 } ” learns 279 codebook entries with an F1 score above 0.5.
Revised: We see that the VQ3 layer of model “ { 3 } → { 2 , 3 } ” learns 279 codebook entries with an F1 score above 0.5.

Original: In contrast, the VQ3 layer of model “ { 2 } → { 2 , 3 } ” learns only a handful of word-detecting codebook entries with an F1 of greater than 0.5.
Revised: In contrast, the VQ3 layer of model “ { 2 } → { 2 , 3 } ” learns only a handful of word-detecting codebook entries with an F1 of greater than 0.5.

Original: This experiment supports the notion that the reason for the poor ABX performances of the VQ3 layer in models “ { 3 } ” and “ { 3 } → { 2 , 3 } ” is in fact due to its specialization for detecting speciﬁc words, and that this specialization only emerges when the VQ3 layer is learned before the 
Revised: This experiment supports the notion that the reason for the poor ABX performances of the VQ3 layer in models “ { 3 } ” and “ { 3 } → { 2 , 3 } ” is in fact due to its specialization for detecting speciﬁc words, and that this specialization only emerges when the VQ3 layer is learned before the VQ2 layer.

Original: VQ2 layer. More extensive tables detailing the words learned by many more VQ3 codes for these models can be found in the supplementary material.
Revised: Section A.2 in the supplementary material examines this phenomenon in greater experimental detail.

Original:  C ONCLUSIONS
Revised:  C ONCLUSIONS

Original: In this paper, we demonstrated that the neural vector quantization layers proposed by van den Oord et al.
Revised: In this paper, we demonstrated that the neural vector quantization layers proposed by van den Oord et al.

Original: (2017) can be integrated into the visually-grounded speech models proposed by Harwath et al.
Revised: (2017) can be integrated into the visually-grounded speech models proposed by Harwath et al.

Original: This resulted in the ability of the speech model to directly represent speech units, such as phones and words, as discrete latent variables.
Revised: This resulted in the ability of the speech model to directly represent speech units, such as phones and words, as discrete latent variables.

Original: We presented extensive experiments and analysis of these learned representations, demonstrating signiﬁcant improvements in phone discrimination ability over the current state-of-the-art models for sub-word speech unit discovery.
Revised: We presented extensive experiments and analysis of these learned representations, demonstrating signiﬁcant improvements in phone discrimination ability over the current state-of-the-art models for sub-word speech unit discovery.

Original: We demonstrated that these units are also far more robust to noise and domain shift than units derived from previously proposed models.
Revised: We demonstrated that these units are also far more robust to noise and domain shift than units derived from previously proposed models.

Original: These results supported the notion that semantic supervision via a discriminative, multimodal grounding objective has the potential to be more powerful than reconstruction-based objectives typically used in unsupervised speech models.
Revised: These results supported the notion that semantic supervision via a discriminative, multimodal grounding objective has the potential to be more powerful than reconstruction-based objectives typically used in unsupervised speech models.

Original: We also showed how multiple vector quantizers could be employed simultaneously within a single
Revised: We also showed how multiple vector quantizers could be employed simultaneously within a single

Original: ResDAVEnet-VQ model, and that these quantizers could be made to specialize in learning a hierarchy of speech units: speciﬁcally, phones in the lower quantizer and words in the upper quantizer.
Revised: ResDAVEnet-VQ model, and that these quantizers could be made to specialize in learning a hierarchy of speech units: speciﬁcally, phones in the lower quantizer and words in the upper quantizer.

Original: Our analysis showed that hundreds of codebooks in the upper quantizer learned to perform as word detectors, and that these detectors were highly accurate.
Revised: Our analysis showed that hundreds of codebooks in the upper quantizer learned to perform as word detectors, and that these detectors were highly accurate.

Original: Our experiments also revealed that this behavior only emerged when VQ3 was trained before VQ2.
Revised: Our experiments also revealed that this behavior only emerged when VQ3 was trained before VQ2.

Original: These results suggest the importance of the learning curriculum, which should be more deeply investigated in future work.
Revised: These results suggest the importance of the learning curriculum, which should be more deeply investigated in future work.

Original: Future work should attempt to make explicit what kind of compositional rules are implicitly encoded by these models when mapping sequences of codes from the lower quantizer to word-level units in the upper quantizer; the automatic derivation of a sub-word unit inventory, vocabulary, and pronunciation lexicon could serve as the starting point for a fully unsupervised speech recognition system.
Revised: Future work should attempt to make explicit what kind of compositional rules are implicitly encoded by these models when mapping sequences of codes from the lower quantizer to word-level units in the upper quantizer; the automatic derivation of a sub-word unit inventory, vocabulary, and pronunciation lexicon could serve as the starting point for a fully unsupervised speech recognition system.

Original: Future work should also investigate whether layers above VQ3 could be made to learn even higher-level linguistic abstractions, such as grammar, syntax, and compositional reasoning.
Revised: Future work should also investigate whether layers above VQ3 could be made to learn even higher-level linguistic abstractions, such as grammar, syntax, and compositional reasoning.

Original:  A. V ARYING THE CODEBOOK SIZE .
Revised: A A PPENDIX A. V ARYING THE CODEBOOK SIZE .

Original: In Table 5, we examine the impact of varying the codebook size of model “ ∅ → { 2 } ” fromthrough 2048.
Revised: In Table 4, we examine the impact of varying the codebook size of model “ ∅ → { 2 } ” fromthrough 2048.

Original: We ﬁnd that the ABX score is best for 1024 codebook vectors, although the performance is quite good for all models.
Revised: We ﬁnd that the ABX score is best for 1024 codebook vectors, although the performance is quite good for all models.

Original: Unsurprisingly, models with smaller codebooks also achieve lower bitrates.
Revised: Unsurprisingly, models with smaller codebooks also achieve lower bitrates.

Original: [Table]
Revised: [Table]

Original: A. U NIT V ISUALIZATION ON TIMIT S PECTROGRAMS
Revised: A. T HE I MPACT OF THE VQ T RAINING C URRICULUM ON THE L OCALIZATION OF W ORD D ETECTORS

Original:  In Figure 5, we visualize two unit sequences for two different TIMIT utterances from different speakers containing the same underlying word sequence.
Revised: [Table] In Section 4. [Table] we showed that cold-start training of the VQ layer caused its codebook vectors to specialize as word detectors, whereas warm-start training did not.

Revised: Our subsequent experiments (Table 6) revealed that when adding a third quantization layer at the Res4 position to a model that did not learn word detectors at VQ3, the VQ4 layer did in fact learn many word detectors.

Revised: This suggests implicit word recognition ability can be localized at different layers in the ResDAVEnet audio model, and exactly where it emerges depends upon the VQ training curriculum.

Revised: We hypothesize that this is due in part to two factors:

Revised: In a warm-start model, whatever type of information (subword-like, word-like) the continuous model learned to encode at a particular convolutional layer (or residual block) does not change after a quantizer is appended to that layer.

Revised: In a cold-start model, each active quantization layer forms a potential bottleneck, restricting the amount of information that is able to pass through to subsequent layers.

Original: Unit sequences are shown for the VQ layer  for  the “ ∅ → { 2 } ” model with a codebook size of 
Revised: According to this hypothesis, if word-level recognition tends to emerge at a particular layer in an unconstrained network with no quantization bottleneck, it will stay there when quantization is introduced for ﬁne-tuning. However, when a quantization bottleneck is introduced from the very beginning of training, the gradient ﬂowing down into the lower network layers is more constrained during the initial training epochs (when the gradient tends to be the largest). This may have the effect of steering the optimizer into a different part of the parameter space, in which word recognition occurs at a different layer than it otherwise would. We present results from two experiments that support this view.

Original: [Figure]
Revised: In Table 5, we show the ABX scores of the Res3 layer prior to quantization (if present) during the course of three different training curricula resulting in a “ { 2 , 3 } ” ﬁnal model. We observe that the ABX error rate changes very little within each individual curriculum. This indicates that the initial model sets the stage for which layer learns to capture phonetic information with the highest salience, and that subsequent training steps do not tend to move this information elsewhere.

Original: [Table]
Revised: [Table]

Original: [Figure]
Revised: Table 6 displays the number of word detectors learned by various VQ layers across different training curricula. Here, we claim that a codebook vector belonging to a particular VQ layer has learned to be a word detector if its F1 score for any word appearing in the test set exceeds 0.5 (as measured on the test set). There are several interesting things to note here. First, we only observe a signiﬁcant number of word detectors at the VQ3 layer when that layer is trained from a cold-start.

Original: [Table] [Figure]
Revised: Even when adding a second VQ layer as in the “ { 3 } → { 2 , 3 } ” model, these detectors remain at VQ3.

Original: [Table]
Revised: Jointly training VQ2 and VQ3 together from a cold-start results in the word detectors being divided between those layers. While this experiment demonstrates that it is possible to jointly train two quantizers at once, the { 2 , 3 } model learned the smallest total number of word detectors of any model. Additionally, we were unable to successfully train a cold-start { 2 , 3 , 4 } model; these experiments suggest that training quantizers one by one may be easier in general. For all models beginning from a “ ∅ ” or “ { 2 } ” initial model, we do not observe any word detectors at either VQ2 or VQ3.

Original: [Figure]
Revised: However, a third quantizer at the VQ4 position in the “ { 2 } → { 2 , 3 } → { 2 , 3 , 4 } ” model was able to capture words. We hypothesize that in all models not trained from a “ { 3 } ” initialization, word recognition is implicitly learned by the Res4 layer, and adding a quantizer to the output of this layer serves to make the categorical nature of those representations explicit.

Original: A.VQ3 W ORD D ETECTOR T ABLES FOR V ARIOUS M ODELS
Revised: A. W ORD D ETECTOR T ABLES FOR V ARIOUS M ODELS

Original: In Table [Table] we show a sampling of 50 word-detecting codebook entries from the VQ layer of the “ { 3 } → { 2 , 3 } ” model.
Revised: In Table 7, we show a sampling of 50 word-detecting codebook entries from the VQ layer of the “ { 3 } → { 2 , 3 } ” model (many word detectors learned).

Original: Analagous results for the “ { 2 } → { 2 , 3 } ” model  are shown in Table 7.
Revised: Analagous results for the “ { 2 } → { 2 , 3 } ” model (few word detectors learned) are shown in Table 8.

Revised: A. U NIT V ISUALIZATION FOR I NDIVIDUAL C APTION S PECTROGRAMS

Original: [Table]
Revised: To provide a better intuitive understanding of what the units learned by our models look like, in Figures 5, 6, and 7, we display speech spectrograms for several Places caption fragments. Along with each spectrogram we display the time-aligned, ground-truth, word-level text (top transcription), the inferred unit sequence for the VQ4 layer (middle transcription), and the unit sequence for the VQ3 (bottom transcription) layer.

Revised: All VQ unit alignments in these ﬁgures are derived from the “ { 2 } → { 2 , 3 } → { 2 , 3 , 4 } ” model.

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: auditorium

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: 6 highway

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: highlights

Revised: [Table]

Revised: [Table]

Revised: church

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: religious

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: 13 distance

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: lounge

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: distance

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: farmland

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: 5 gallery

Original: country 0.
Revised: [Table] [Table] [Table] [Table]

Revised: [Table]

Original: 13 airport 86.
Revised: [Table] [Table] 9 large [Table]

Revised: [Table]

Original: escalator 2.
Revised: [Table] very [Table] [Table]

Original: 23 background 86.
Revised: 94 cathedral [Table] [Table] [Table]

Revised: feed

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: 1 purple

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: proportion

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: 1 trees

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: tree

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: 171 boy

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: boys

Revised: [Table]

Revised: [Table]

Revised: [Table]

Original: back 8.
Revised: 18 ground [Table] [Table] [Table]

Revised: playground

Revised: [Table]

Revised: [Table]

Original: 1233 subway 86.
Revised: [Table] 27 courtyard 69. [Table] plaza

Revised: [Table]

Original: station 1.
Revised: [Table] [Table] 9 wooden 69. [Table]

Original: 99 patio 85.
Revised: [Table] wood [Table]

Original: patios 2.
Revised: [Table]

Original: 28 rocky 85.
Revised: 153 lighthouse 69.

Original: rock 3.
Revised: lighthouses 11.

Original: 266 driveway 85.
Revised: 5 house 69.

Original: sidewalk 1.
Revised: houses 18.

Original: 47 hospital 84.
Revised: 88 dark 69.

Original: hot 0.
Revised: darker 3.

Original: 17 palm 83.
Revised: 6 building 69.

Original: concrete 1.
Revised: buildings 25.

Original: 56 bamboo 83.
Revised: 281 grass 69.

Original: abandoned 5.
Revised: grassy 21.

Original: 109 walking 83.
Revised: 109 lake 68.

Original: walk 7.
Revised: late 5.

Original: 519 stage 83. concert 1. 51 body 83. large 3.
Revised: 5 trash 68.

Original: 1324 stadium 83.

Original: boardwalk 10.
Revised: boulders 10.

Original: 275 pantry 83. country 2. 29 courtyard [Table]
Revised: 4 photograph 68.

Original: [Table] graveyard 2. 45 volcano 82. volcanoes 5.
Revised: photographs 30.

Original: 13 yellow

Original: [Table]

Original: [Table] yellowish [Table] 158 standing 82.
Revised: 181 lobby 68.

Original: stand 5.
Revised: waiting 9.

Original: 534 trees 82. tree 14. 3436 course 82. golf 6.
Revised: 14 shirt 68.

Original: 195 distance 82.

Original: background 1.
Revised: shirts 18.

Original: 423 church 82.
Revised: 43 grass 68.

Original: religious 5.
Revised: grassy 15.

Original: 263 highway 82. highways 3. 22 small 82. smaller 2.
Revised: [Table]

Original: 263 shower 81. showers 3. 25 river 81. rivers 1.
Revised: 491 large 39.

Original: 25 station 68.

Original: gas 28. 1773 shirt 68. shirts 15. 757 night 68.
Revised: cars 6.

Original: nighttime 13. 241 computer 68. computers 22. 254 empty 67.
Revised: 224 red 39.

Original: terminal 4.
Revised: bed 9.

Original: 176 ruins 67. ruin 21. 142 coffee 67.
Revised: 168 woman 38.

Original: cream 7.
Revised: women 7.

Original: 357 man 67. men 13.
Revised: 126 people 37.

Original: 1390 baby 67. baby’s 24. 249 train 67. trains 10.
Revised: table 22.

Original: 306 lake 67.
Revised: 656 water 37.

Original: late 2.
Revised: river 5.

Original: 53 house 67.
Revised: 215 sky 35.

Original: houses 18.
Revised: skies 11.

Original: 1868 alleyway 67. alley 37. 863 orange 67.
Revised: 141 has 34.

Original: oranges 3.
Revised: house 9.

Original: 63 pink 67.
Revised: 330 yellow 34.

Original: paint 5. [Table]
Revised: ﬂowers 13. 237 picture 34. pictures 16.

Original: 13439 yellow 32.
Revised: 698 trees 33.

Original: ﬂowers 14.
Revised: tree 10.

Revised: 293 man 33.

Original: 5623 with 32.
Revised: standing 9. 532 black 33. background 21. 759 with 33.

Original: white 7.
Revised: white 8.

Original: 17037 yellow 32.
Revised: 966 large 33. bridge 19. 371 trees 32.

Original: area 14.
Revised: train 12.

Original: 6956 small 32.
Revised: 353 yellow 32.

Original: large 5.
Revised: area 15.

Original: 4664 large 31.
Revised: 354 front 32.

Original: bridge 18.
Revised: from 14.

Original: 7066 black 31.
Revised: 358 black 31.

Original: glass 8.
Revised: glass 8.

Revised: 223 small 31.

Original: 4787 picture 30.
Revised: large 5. 244 picture 31.

Original: pictures 13.
Revised: pictures 13.

Original: 11276 man 29.
Revised: 591 white 29.

Original: many 17.
Revised: black 16.

Original: 10692 white 29.
Revised: 900 white 29.

Original: black 16.
Revised: like 9.

Original: 18818 trees 29.
Revised: 388 picture 29. like 9. 400 trees 29.

Original: station 3.
Revised: showcasing 3.

Original: 3144 white 29.
Revised: 155 area 6.

Original: like 9.
Revised: snow 6.

Original: 7730 there’s 6.
Revised: 257 picture 6. pictures 4. 463 there’s 6.

Original: that 4.
Revised: that 5.

Original: 11745 old 6.
Revised: 614 front 6.

Original: all 5.
Revised: some 4.

Revised: 345 multi 6.

Original: 5490 structure
Revised: towers 5. 1 photo 6. by 5. 229 ﬁeld 6.

Original: [Table]
Revised: photo 6.

Original: [Table] kids [Table] [Table]
Revised: 289 man 5.

Original: [Table] [Table] that
Revised: middle 3.

Original: [Table] [Table] [Table]
Revised: 314 sitting 5.

Original: with 
Revised: with 5.

Original: [Table] [Table] [Table] there’s
Revised: 447 trees 5.

Original: [Table] [Table] [Table] [Table]
Revised: with 5.

Original: [Table] [Table] ﬁeld [Table]
Revised: 718 with 5.

Original: [Table] [Table] [Table] [Table]
Revised: that 3.

Original: [Table] with [Table] [Table]
Revised: 447 purple 5.

Original: [Table] [Table] [Table] [Table]
Revised: parked 4.

Original: by

Original: [Table] [Table] [Table] [Table]
Revised: 120 structure 5.

Original: [Table] [Table] [Table] [Table]
Revised: graph 4.

Original: [Table] [Table] [Table] woman
Revised: 2 parking 5.

Original: [Table] [Table] [Table] [Table]
Revised: park 5.

Original: table [Table] [Table] [Table]
Revised: 126 standing 5.

Original: see 4.
Revised: woman 3.

Original: [Table]