How transferable are features in convolutional neural network acoustic models across languages?
Anonymous Author(s)
AfﬁliationAddress email
Abstract
Characterization of the representations learned in intermediate layers of deep networks can provide valuable insight into the nature of a task and can guide thedevelopment of well-tailored learning strategies.
Here we study convolutionalneural network-based acoustic models in the context of automatic speech recognition.
Adapting a method proposed by Yosinski et al.
[2014], we measure thetransferability of each layer between German and English to assess their languagespeciﬁty.
We observe three distinct regions of transferability: (1) the ﬁrst twolayers are entirely transferable between languages, (2) layers 2–8 are also highlytransferable
but we ﬁnd evidence of some language speciﬁcity, (3) the subsequentfully connected layers are more language speciﬁc but can be successfully ﬁnetunedto the target language.
Our results are consistent with the observation that CCNsconverge ’bottom up’ during training and demonstrate the beneﬁt of freeze training
[Raghu et al., 2017], especially for transfer learning.
Introduction
The acoustic properties of speech vary across languages.
This is evidenced by the fact that monolingual acoustic models (AMs) are the defacto standard in automatic speech recognition (ASR), whilemulti-lingual AMs are an active area of development [Heigold et al., 2013, Tuske et al., 2013, Sercuet al., 2016, Watanabe et al., 2017].
Requiring large amounts of training data to build separate AMsfor every language is a barrier to successful ASR systems for low-resource languages.
Ideally, AMswould be designed to strategically leverage off-task data as much as possible.
AMs often take theform of a deep network which learns to map from acoustic features to context-dependent phones in alanguage-speciﬁc phone set.
It is not clear how exactly this transformation is performed or what isrepresented in the intermediate layers of such networks.
Better characterization of the intermediaterepresentations of AMs may help to guide data-efﬁcient training procedures.
Similar characterizationsof networks trained on visual tasks have inspired new transfer learning procedures.
For example, Yosinski et al.
[2014] characterized the task speciﬁcity at each layer of a network trained on ImageNetusing transferability as a proxy for task-speciﬁcity.
This characterization motivated Adaptive Transfer Networks [Long et al., 2015] where parts of a network are trained on the source domain while otherparts of the network are ﬁnetuned, or adapted, to the target domain, preserving the limited target datafor learning highly task-speciﬁc parameters.
Similar adaptive transfer learning procedures may alsoprove to be useful for building AMs for low-resource (data-poor) languages.
However, the exactshape of the transition from task-general to task-speciﬁc representations in deep network-based AMsis unknown.
Much of the previous work on characterizing intermediate layers of deep networks has focusedon relatively solvable tasks in the visual domain (e.g. hand written digit recognition, visual objectrecognition).
Few studies have characterized the intermediate representations of networks trained on acoustic tasks [Lee et al., 2009, Golik et al., 2015], which are often not trained long enough toconverge completely (test error still slowly decreasing at the end of training).
It is not clear to whatextent existing methods developed to probe networks trained on visual tasks will be applicable anduseful to study networks that may be underﬁtting on difﬁcult acoustic tasks.
Here we studied convolutional neural networks (CNNs) used for acoustic modeling in (ASR) systems.
We characterized the language-speciﬁcity of each layer across languages using an approach inspiredby Yosinski et al.
Subsets of a network trained on one language were “implanted” into anothernetwork which was trained on a second language.
The effect of the implant on performance indicatedthe language-speciﬁcity of the features in the implant.
Our main contribution is the characterizationof the language-speciﬁcity of intermediate layers of CCN-based acoustic models.
Additionally, wedemonstrate the adaptation of an analysis method originally designed to probe visual networks tostudy networks in an underﬁtting regime on a phone classiﬁcation task.
Experiments
The datasets for this experiment consisted of 68 hours and 83 hours of German (GER) and American
English (ENU) speech respectively, recorded in comparable environments, with corresponding texttranscriptions.
We chose these languages because we expected a large degree of transferabilitybased on their phonetic similarity.
Logarithmic Mel ﬁlter bank features were calculated, creatinga 45-dimensional feature vector for every 10ms of audio (spectrograms).
Each observation wasassociated with one of 9000 target context-dependent phone classes.
Phonesets were different foreach language.
A CNN consisting of nine convolutional layers followed by three fully connected layers with thefollowing architecture was trained to recognize context-dependent phones from each language,resulting in a total of approximately 7.2 million parameters: (7, 7, 1024), (3, 3, 256), (3, 3, 256), (3,3, 128), (3, 3, 128), (3, 3, 128), (3, 3, 64), (3, 3, 64), (3, 3, 64), (600), (190), (9000).
Both networkswere trained using the ADAM optimizer [Kingma and Ba, 2015] as implemented in Tensorﬂow[Abadi et al., 2016] with a minibatch size of 256, a starting learning rate of 10 e − 5 and rectiﬁed linearunits.
Approximately 98% of the data was used for training and the remaining 2% for testing.
Allmodel parameters were replicated on four GPUs.
Different minibatches were given to each GPU,their gradients averaged to calculate updates.
As a balance between training time and accuracy, eachnetwork was trained for a ﬁxed period of 100 epochs (which took approximately two weeks).
The subsequent experimental setup was similar to that described in Yosinski et al.
Several“network surgeries“ were performed.
The ﬁrst n layers of a network trained on Language A wereimplanted into a new network of identical architecture where the layers after layer n were randomlyinitialized.
This ’chimera’ network was further trained in four different ways.
It was either trained onlanguage A (self-transfer or “selfer“ network) or language B (transfer network) and the implantedparameters were either ﬁxed or allowed to be ﬁnetuned during training.
This process was repeated ∀1 ≤ n ≤ 11 and for both English and German resulting in 88 networks total (see Figure 1 in Yosinskiet al.
[2014] for a graphical depiction of a similar experimental setup).
The selfer networks servedas a control to capture any changes in performance associated with the surgery but unrelated to thetransfer.
All networks were trained for 100 epochs.
Training parameters were identical to those of thebaseline models.
Results
We found representations throughout the networks to be highly transferable between English and
German.
Top-1 test phone classiﬁcation accuracy for each network is plotted as a function of thelayer at which the surgery was performed in Figure 1.
Phone classiﬁcation accuracy is measuredwith respect to per frame phone-labels established in a forced alignment.
The only models thatperformed considerably worse than the monolingual baseline models were the transfer networkswithout ﬁnetuning whose surgery occurred at one of the fully connected layers (the penultimatetwo layers).
Transfer networks cut at any of the convolutional layers performed as well as themonolingual baseline model, regardless of whether the implanted layers were ﬁnetuned or not.
Weobserved a slight improvement over the monolingual baseline (~1.5 percentage points (pp)) fortransfer networks with ﬁnetuning chopped at one of the fully connected layers. All selfer networks
[Figure]
with ﬁnetuning performed at the same level as the mono-lingual baseline.
Somewhat unexpectedly,the selfer networks without ﬁnetuning performed best overall among the chimera networks.
Selfernetworks chopped at late layers whose implants were not ﬁnetuned showed an improvement of 3+pp.
Previous work has shown that random, untrained weights can often perform remarkably wellin certain scenarios [Jarrett et al., 2009, Rahimi and Recht, 2007].
Figure 2 shows accuracy as afunction of the layer at which training began, meaning that layers below layer n were randomlyinitialized and never updated.
Subsequent layers (layer n and above) were trained for 100 epochswith the same training parameters as the baseline models.
We observed a gradual drop in performanceas a function of depth.
Random weights in early layers did not have a large impact on performance.
Using random weights for all but the last layer resulted in near-chance performance.
This veriﬁes thenon-triviality of the success of our transfer networks without ﬁnetuning.
The training of our selfer
99 networks without ﬁnetuning somewhat resembles the freeze training procedure proposed by Raghuet al.
According to this procedure, layers are successively frozen over the course of training,gradually reducing the number of parameters to be updated until, by the end of training, only thelast layer is being updated.
We hypothesized that weight freezing partly explained the success of our selfer networks without ﬁnetuning, so we created freeze trained versions of both our selfer andtransfer networks.
Starting with a pre-trained network, layers 1–11 (excluding layer 0) were trainedfor 5 epochs.
Then, for the next 5 epochs, only layers 2–11 were trained.
From then on, another layerwas removed from the trainable parameters every 10 epochs for a total of 100 training epochs.
Thefreeze trained models are represented by the coloured dashed lines in Figure 1.
The freeze trainingled to better overall performance, especially among the transfer networks.
Discussion
Our results suggest that, despite a large degree of transferability of intermediate acoustic featuresbetween languages, naive approaches to transfer (e.g. initializing with parameters from anotherlanguage) are not the most efﬁcient.
In particular, early layers need not be ﬁnetuned on the targetlanguage at all.
Subsequent layers beneﬁt greatly from freeze training on the target language.
Thesefreeze trained transfer networks outperform networks trained solely on the target language, whichdemonstrates the improved generalization that can be achieved when incorporating data from multiplesources.
The performance of the networks with ﬁnetuning is largely consistent with Yosinski et al.
However, the performance of networks without ﬁnetuning deviates considerably.
The transfernetworks without ﬁnetuning in Yosinski et al.
[2014] show a gradual drop in performance, starting atthe 4th convolutional layer and eventually dropping nearly 8 pp by the penultimate layer (see Figure2 from Yosinski et al.
Our transfer networks without ﬁnetuning, on the other hand, show asharp drop in performance that starts only at the ﬁrst fully connected layer (layer 9).
For the selfernetworks without ﬁnetuning, we did not observe a performance drop when networks were choppedat middle layers, as was reported in Yosinski et al.
Instead, our selfer networks withoutﬁnetuning outperformed all other models, with accuracy increasing nearly monotonically with thedepth at which the network was chopped.
Yosinski et al.
[2014]’s experiments with random weightsquickly drop to near-chance performance by layer 3, whereas our networks with random weightsdecline gradually with depth, only approaching near-chance performance when all but the last layerare random.
The success of our selfer networks without ﬁnetuning is at least partly explained by the fact that weare in an underﬁtting regime.
Unlike in Yosinski et al.
[2014], our baseline model has not convergedcompletely and we would expect continued training to improve performance.
However, if that werethe only factor at play, then we would expect our selfer networks with ﬁnetuning to also improvebut they do not.
Something about freezing all but the last layer(s) facilitates a ~3 pp improvementover baseline in the selfer but not the transfer networks.
This suggests that there is some importantlanguage-speciﬁc information in the layers that show a difference between the selfer and transfernetworks without ﬁnetuning (layer 3+).
Layers 10 and 11 show worse than baseline performance forthe transfer network without ﬁnetuning, indicating a larger degree of language-speciﬁcity in theserepresentations.
Our freeze training results corroborate the interpretation that weight freezing is responsible for thesuccess of our selfer networks without ﬁnetuning.
Furthermore, our freeze-trained transfer networksperformed best overall, demonstrating that freeze training can actually recover the language-speciﬁcinformation lacking in our transfer networks without ﬁnetuning, yielding improved generalization.
This likely reﬂects the observation from Raghu et al.
[2017] that CNNs converge ’bottom-up’ duringtraining, with early layers stabilizing earlier in training.
Relatedly, Alain and Bengio [2016] state theproposition that no intermediate layer of a multi-layer neural network will contain more target-relatedinformation than the raw input, which requires a ’bottom-up’ ﬂow of information; intermediate layerscannot pass on target-related information that they do not receive.
Thus we conclude that freezing theweights of a given layer can improve performance iff that layer already passes on the target-relatedinformation in a representation that can be disentangled by subsequent layers.
This was not generallythe case in our transfer chimera networks because important language-speciﬁc information was notbeing conveyed.
The progressive freeze training regime, proposed by Raghu et al.
[2017], allowedthis important language-speciﬁc information to be learned, whereas generic ﬁne-tuning did not.
Inthis way, making fewer parameter updates actually led to signiﬁcant performance gains.
This may bepartly explained by the fact that smaller networks train faster [Saxe et al., 2015].
Perhaps genericﬁne-tuning would eventually achieve the same accuracy, but after many more iterations.