How transferable are features in convolutional neural network acoustic models across languages?
Abstract
Characterization of the representations learned in intermediate layers of deep networks can provide valuable insight into the nature of a task and can guide the development of well-tailored learning strategies.
Here we study convolutional neural network-based acoustic models in the context of automatic speech recognition.
Adapting a method proposed by Yosinski et al.
[2014], we measure the transferability of each layer between German and English to assess their languagespeciﬁty.
We observe three distinct regions of transferability: (1) the ﬁrst two layers are entirely transferable between languages, (2) layers 2–8 are also highly transferable
but we ﬁnd evidence of some language speciﬁcity, (3) the subsequent fully connected layers are more language speciﬁc but can be successfully ﬁnetuned to the target language.
To further probe the effect of weight freezing, we performed follow-up experiments using freeze-training [Raghu et al., 2017]. Our results are consistent with the observation that CNNs converge ‘bottom up’ during training and demonstrate the beneﬁt of freeze training, especially for transfer learning.
Introduction
The acoustic properties of speech vary across languages.
This is evidenced by the fact that monolingual acoustic models (AMs) are the de facto standard in automatic speech recognition (ASR), while multi-lingual AMs are an active area of development [Heigold et al., 2013, Tuske et al., 2013, Sercu et al., 2016, Watanabe et al., 2017].
Requiring large amounts of training data to build separate AMs for every language is a barrier to successful ASR systems for low-resource languages.
Ideally, AMs would be designed to strategically leverage off-task data as much as possible.
AMs often take the form of a deep network which learns to map from acoustic features to context-dependent phones in a language-speciﬁc phone set.
It is not clear how exactly this transformation is performed or what is represented in the intermediate layers of such networks.
Better characterization of the intermediate representations of AMs may help to guide data-efﬁcient training procedures.
Similar characterizations of networks trained on visual tasks have inspired new transfer learning procedures.
For example,
Yosinski et al.
[2014] characterized the task speciﬁcity at each layer of a network trained on ImageNet using transferability as a proxy for task-speciﬁcity.
This characterization motivated Adaptive Transfer
Networks [Long et al., 2015] where parts of a network are trained on the source domain while other parts of the network are ﬁnetuned, or adapted, to the target domain, preserving the limited target data for learning highly task-speciﬁc parameters.
Similar adaptive transfer learning procedures may also prove to be useful for building AMs for data-poor languages.
However, the exact shape of the transition from task-general to task-speciﬁc representations in deep network-based AMs is unknown.
Much of the previous work on characterizing intermediate layers of deep networks has focused on relatively solvable tasks in the visual domain (e.g. hand written digit recognition, visual object recognition).
Few studies have characterized the intermediate representations of networks trained on acoustic tasks [Lee et al., 2009, Golik et al., 2015, Nagamine et al., 2015], which, in practice, are not always trained long enough to converge completely (test error still slowly decreasing at the end of training) due to the long training time required.
It is not clear to what extent existing methods developed to probe networks trained on visual tasks will be applicable and useful to study networks that may be underﬁtting on difﬁcult acoustic tasks.
Here we studied convolutional neural networks (CNNs) used for acoustic modeling in ASR systems.
We characterized the language-speciﬁcity of each layer across languages using an approach inspired by Yosinski et al.
Subsets of a network trained on one language were “implanted” into another network which was trained on a second language.
The effect of the implant on performance indicated the language-speciﬁcity of the features in the implant.
Our main contribution is the characterization of the language-speciﬁcity of intermediate layers of CNN-based acoustic models.
Additionally, we demonstrate the adaptation of an analysis method originally designed to probe visual networks to study networks in an underﬁtting regime on a phone classiﬁcation task.
Experiments
The datasets for this experiment consisted of 68 hours and 83 hours of German (GER) and American
English (ENU) speech respectively, recorded in comparable environments, with corresponding text transcriptions.
We chose these languages because we expected a large degree of transferability based on their phonetic similarity.
Logarithmic Mel ﬁlter bank features were calculated, creating a 45-dimensional feature vector for every 10ms of audio (spectrograms).
Each observation was associated with one of 9000 target context-dependent phone classes.
Phonesets consisted of 54 unique phones for English and 47 unique phones for German.
A CNN consisting of nine convolutional layers followed by three fully connected layers was trained to recognize context-dependent phones from each language. The architecture was as follows, where the triplets specify the ﬁlter size and number of feature maps in each conv layer and the singletons specify how many units in each of the fully connected layers: (7, 7, 1024), (3, 3, 256), (3, 3, 256), (3, 3, 128), (3, 3, 128), (3, 3, 128), (3, 3, 64), (3, 3, 64), (3, 3, 64), (600), (190), (9000). This resulted in a total of approximately 7.2 million parameters.
Both networks were trained using the ADAM optimizer [Kingma and Ba, 2015] as implemented in Tensorﬂow [Abadi et al., 2016] with a minibatch size of 256, a starting learning rate of 10 e − 5 and rectiﬁed linear units.
Approximately 98% of the data was used for training and the remaining 2% for testing.
All model parameters were replicated on four GPUs.
Different minibatches were given to each GPU, their gradients averaged to calculate updates.
As a balance between training time and accuracy, each network was trained for a ﬁxed period of 100 epochs (which took approximately two weeks).
The subsequent experimental setup was similar to that described in Yosinski et al.
Several ‘network surgeries’ were performed.
The ﬁrst n layers of a network trained on Language A were implanted into a new network of identical architecture where the layers after layer n were randomly initialized.
This ‘chimera’ network was further trained in four different ways.
It was either trained on language A (self-transfer or ‘selfer’ network) or language B (transfer network) and the implanted parameters were either ﬁxed or allowed to be ﬁnetuned during training.
This process was repeated ∀ 1 ≤ n ≤ 11 and for both English and German resulting in 88 networks total (see Figure 1 in Yosinski et al.
[2014] for a graphical depiction of a similar experimental setup).
The selfer networks served as a control to capture any changes in performance associated with the surgery but unrelated to the transfer.
All networks were trained for 100 epochs.
Training parameters were identical to those of the baseline models.
Results
We found representations throughout the networks to be highly transferable between English and
German.
Top-1 test phone classiﬁcation accuracy for each network is plotted as a function of the layer at which the surgery was performed in Figure 1.
Phone classiﬁcation accuracy is measured with respect to per frame phone-labels established in a forced alignment.
The only models that [Figure] performed considerably worse than the monolingual baseline models were the transfer networks without ﬁnetuning whose surgery occurred at one of the fully connected layers (the penultimate two layers).
Transfer networks cut at any of the convolutional layers performed as well as the monolingual baseline model, regardless of whether the implanted layers were ﬁnetuned or not.
We observed a slight improvement over the monolingual baseline (~1.5 percentage points (pp)) for transfer networks with ﬁnetuning chopped at one of the fully connected layers.
All selfer networks with ﬁnetuning performed at the same level as the mono-lingual baseline.
Somewhat unexpectedly, the selfer networks without ﬁnetuning performed best overall among the chimera networks.
Selfer networks chopped at late layers whose implants were not ﬁnetuned showed an improvement of 3+ pp.
Previous work has shown that random, untrained weights can often perform remarkably well in certain scenarios [Jarrett et al., 2009, Rahimi and Recht, 2007].
Figure ?? shows accuracy as a function of the layer at which training began, meaning that layers below layer n were randomly initialized and never updated.
Subsequent layers (layer n and above) were trained for 100 epochs with the same training parameters as the baseline models.
We observed a gradual drop in performance as a function of depth.
Random weights in early layers did not have a large impact on performance.
Using random weights for all but the last layer resulted in near-chance performance.
This veriﬁes the non-triviality of the success of our transfer networks without ﬁnetuning.
The training of our selfer networks without ﬁnetuning somewhat resembles the freeze training procedure proposed by Raghu et al.
According to this procedure, layers are successively frozen over the course of training, gradually reducing the number of parameters to be updated until, by the end of training, only the last layer is being updated.
We hypothesized that weight freezing partly explained the success of our selfer networks without ﬁnetuning, so we created freeze trained versions of both our selfer and transfer networks.
Starting with a pre-trained network, layers 1–11 (excluding layer 0) were trained for 5 epochs.
Then, for the next 5 epochs, only layers 2–11 were trained.
From then on, another layer was removed from the trainable parameters every 10 epochs for a total of 100 training epochs.
The freeze trained models are represented by the coloured dashed lines in Figure 1.
The freeze training led to better overall performance, especially among the transfer networks.
Discussion
Our results suggest that, despite a large degree of transferability of intermediate acoustic features between languages, naive approaches to transfer (e.g. initializing with parameters from another language) are not the most efﬁcient.
In particular, early layers need not be ﬁnetuned on the target language at all.
Subsequent layers beneﬁt greatly from freeze training on the target language.
These freeze trained transfer networks outperform networks trained solely on the target language, which demonstrates the improved generalization that can be achieved when incorporating data from multiple sources.
The performance of the networks with ﬁnetuning is largely consistent with Yosinski et al.
However, the performance of networks without ﬁnetuning deviates considerably.
The transfer networks without ﬁnetuning in Yosinski et al.
[2014] show a gradual drop in performance, starting at the 4th convolutional layer and eventually dropping nearly 8 pp by the penultimate layer (see Figure 2 from Yosinski et al.
Our transfer networks without ﬁnetuning, on the other hand, show a sharp drop in performance that starts only at the ﬁrst fully connected layer (layer 9).
For the selfer networks without ﬁnetuning, we did not observe a performance drop when networks were chopped at middle layers, as was reported in Yosinski et al.
Instead, our selfer networks without ﬁnetuning outperformed all other models, with accuracy increasing nearly monotonically with the depth at which the network was chopped.
Yosinski et al.
[2014]’s experiments with random weights quickly drop to near-chance performance by layer 3, whereas our networks with random weights decline gradually with depth, only approaching near-chance performance when all but the last layer are random.
The success of our selfer networks without ﬁnetuning is at least partly explained by the fact that we are in an underﬁtting regime.
Unlike in Yosinski et al.
[2014], our baseline model has not converged completely and we would expect continued training to improve performance.
However, if that were the only factor at play, then we would expect our selfer networks with ﬁnetuning to also improve but they do not.
Something about freezing all but the last layer(s) facilitates a ~3 pp improvement over baseline in the selfer but not the transfer networks.
This suggests that there is some important language-speciﬁc information in the layers that show a difference between the selfer and transfer networks without ﬁnetuning (layer 3+).
Layers 10 and 11 show worse than baseline performance for the transfer network without ﬁnetuning, indicating a larger degree of language-speciﬁcity in these representations.
Our freeze training results corroborate the interpretation that weight freezing is responsible for the success of our selfer networks without ﬁnetuning.
Furthermore, our freeze-trained transfer networks performed best overall, demonstrating that freeze training can actually recover the language-speciﬁc information lacking in our transfer networks without ﬁnetuning, yielding improved generalization.
This likely reﬂects the observation from Raghu et al.
[2017] that CNNs converge ‘bottom-up’ during training, with early layers stabilizing earlier in training.
Relatedly, Alain and Bengio [2016] state the proposition that no intermediate layer of a multi-layer neural network will contain more target-related information than the raw input, which requires a ‘bottom-up’ ﬂow of information; intermediate layers cannot pass on target-related information that they do not receive.
Thus we conclude that freezing the weights of a given layer can improve performance iff that layer already passes on the target-related information in a representation that can be disentangled by subsequent layers.
This was not generally the case in our transfer chimera networks because important language-speciﬁc information was not being conveyed.
The progressive freeze training regime, proposed by Raghu et al.
[2017], allowed this important language-speciﬁc information to be learned, whereas generic ﬁne-tuning did not.
In this way, making fewer parameter updates actually led to signiﬁcant performance gains.
This may be partly explained by the fact that smaller networks train faster [Saxe et al., 2015].
Perhaps generic ﬁne-tuning would eventually achieve the same accuracy, but after many more iterations.