Variable-rate hierarchical CPC leads to acoustic unit discovery in speech
Anonymous Author(s)
AffiliationAddress
Abstract
The success of deep learning comes from its ability to capture the hierarchicalstructure of data by learning high-level representations defined in terms of low-levelones.
In this paper we explore self-supervised learning of hierarchical representations of speech by applying multiple levels of Contrastive Predictive Coding(CPC).
We observe that simply stacking two CPC models does not yield significantimprovements over single-level architectures.
Inspired by the fact that speech isoften described as a sequence of discrete units unevenly distributed in time, wepropose a model in which the output of a low-level CPC module is non-uniformlydownsampled to directly minimize the loss of a high-level CPC module.
Thelatter is designed to also enforce a prior of separability and discreteness in itsrepresentations by enforcing dissimilarity of successive high-level representationsthrough focused negative sampling, and by quantization of the prediction targets.
Accounting for the structure of the speech signal improves upon single-level CPCfeatures and enhances the disentanglement of the learned representations, as measured by downstream speech recognition tasks, while resulting in a meaningfulsegmentation of the signal that closely resembles phone boundaries.
Introduction
Perceptual signals, such as speech, images, or natural language are hierarchical, and exhibit a nonuniform information density.
We believe that by caring for these two aspects of the data we canlearn improved representations of such signals.
Modern self-supervised representation learners doso only partially: while they are implemented using deep neural networks which internally learn ahierarchy of concepts, the usual training criteria are only applied at the top layer and do not fullybenefit from the hierarchical nature of the signals.
Moreover, the models often assume a uniforminformation density and process the data at fixed input-driven rates (e.g., by striding or uniformpooling over time or pixels on speech and images, respectively).
We demonstrate that the quality oflearned representations can be improved by explicitly modeling the data at two different, non-uniformsampling rates and with different training criteria applied at each level of data modeling.
We demonstrate our results on speech, which is a hierarchical signal with a well understood structure.
We extend a frame-wise feature extraction model with a boundary predictor, followed by anotherfeature transformation applied to variable length contiguous segments of frames.
We show thatthrough a careful design of the second level training criterion we can improve on the quality ofthe learned representations and perform unit discover by detecting boundaries which overlap with ground-truth segmentation of the data.
Therefore, we show the benefits of coupling variable-rate datarepresentations with several levels of contrastive training.
1.1 Background: unsupervised representation learning
The deep learning revolution was started by layerwise stacking of pre-trained shallow models such as
Restricted Boltzman Machines [Hinton and Salakhutdinov, 2006] or denoising autoencoders [Vincentet al., 2008].
However, it was soon observed that deep models can be trained from scratch withoutthe need for unsupervised pretraining [Krizhevsky et al., 2012].
Consequently, unsupervised featuremodeling was limited to learning shallow word representations [Mikolov et al., 2013].
Recently, novelself-supervised training criteria such as unmasking [Devlin et al., 2018], or contrastive predictions[van den Oord et al., 2018, Baevski et al., 2020, Chen et al., 2020] have facilitated pre-training deepnetworks on unlabeled datasets.
Despite these advances, improving the quality of self-supervised representations of data by modelingthe signal at several levels of hierarchy remains an unsolved problem.
Löwe et al.
[2019] have stackedseveral Contrastive Predictive Coding (CPC, van den
Oord et al.
[2018]) modules, however theirgoal was not to improve the representations, but to avoid a global backpropagation training.
In thespeech domain, their pre-training regime has failed to match the quality attained by training of asingle deep CPC network.
Bhati et al.
[2021b] introduced a hierarchical model which stacked two
CPC modules separated by a differentiable boundary detector.
They demonstrated that adding thesecond level of CPC modeling leads to improved phone boundary detection, but at the expense of thepredictive capability of the learned features.
This tension between boundary detection and featureinformativeness was further confirmed by Cuervo et al.
We demonstrate that it is possible to improve the predictive quality of features by introducing a second
CPC level which is trained on variable length segments extracted to directly optimize the performanceof the second-level CPC criterion.
This result has several implications.
First, it demonstrates thebenefits of more closely matching the variable information rate of speech.
Second, it paves the wayfor unsupervised discovery of structure in data, for instance acoustic unit discovery in speech.
1.2 Unit discovery in speech
Speech is a well-suited benchmark modality for structure discovery using predictive coding methods.
Patterns at the level of acoustic frames change in a predictable way due to the physical constraintsof the vocal apparatus.
Yet these evolve depending on higher level latent variables, such as phonesor prosody.
Due to the structure of language, speech is also predictable at the level of phones: in
English we would expect the phone " IH " to be followed by the phone " T " to form the word "it".
Atword level, we can predict that the pronoun "it" will be followed by a verb like "is" or "does".
Thesame rationale can be applied to other language-related signals such as handwritten text.
Furthermore acoustic units have some easy to characterise properties.
For instance, phone durationsare limited by the rate at which they can be articulated and perceived, averaging around 100 msin English.
Useful units are also contrastive, for easier distinguishing between subsequent phonesby a listener, and inherently redundant, which makes them robust to external noise, but also morepredictable.
Existing approaches to acoustic unit discovery impose those properties on the featuresextracted with a frame-level, self-supervised model.
The main idea behind our method is to enforcesuch priors of average duration, contrastiveness and predictability in a differentiable, self-supervisedobjective function, which a two-level formulation permits.
By optimizing this function we learn toextract information bearing units from unlabeled continuous sequential data.
Model
We describe the operation of our model in the speech domain, noting that the design considerationsinvolved could apply to other kinds of sequential data with an inherent discrete structure, such as handwriting encoded as long lines of written characters.
The major parts of the model (Figure 1) aretwo CPC modules, separated by a trained variable-rate downsampling segment extractor.
Low-level feature extractor
The feature extractor is a typical CPC network for audio data [van den
Oord et al., 2018, Rivière et al., 2020] that models the regularities of the signal at the acoustic level.
It takes as input the raw waveform x and maps it to sequences of encodings z ∈ R T × d and contextvectors c ∈ R T × d , both d -dimensional vectors calculated for all T time steps.
The encodings areproduced by a strided convolutional encoder g enc and contain local information on the receptive fieldof the encoder at time t : z t = g enc ( x t ) .
The context vectors are the output of a recurrent neuralnetwork g ar which summarizes observations up to time t : c t = g ar ( z <t ) .
Downsampling module This module aims to downsample the frame representations depending ontheir information content so as to improve the performance of the higher level feature extractor.
Itcontains a boundary predictor and an average pooling layer.
The boundary predictor is a transformernetwork with bidirectional attention [Vaswani et al., 2017] that predicts a sequence of unit boundaryindicators b ∈ { 0 , 1 } T conditioned on c .
For two consecutive non-zero elements in b : b t and and b t + l ,the pooling layer compresses the sub-sequence of encodings between boundaries z t , . . .
, z t + l − 1 intoa single segment representation s = 1 l (cid:80) [-l − 1 i =
-][Table] =
[Table] i .
High-level feature extractor After downsampling, the resulting sequence of segment representationsis processed by a second CPC network aiming to model the signal at the level of discrete units.
It mapsthe sequence s ∈ R T ′ ×
d ′ to sequences of pseudo-unit representations u ∈ R T ′ ×
d ′ and unit-levelcontext representations h ∈ R T ′ ×
d ′ .
The pseudo-unit representations are produced by a fullyconnected network s enc :
u k = s enc ( s t ) .
The context vectors are the output of a recurrent neuralnetwork s ar which summarizes the history of pseudo-units up to the k -th (high-level) time-step:
h k
= s ar ( u <k ) .
For the CPC task at the segment level we quantize the prediction targets using anonline k-means vector quantizer [van den Oord et al., 2017] q , which transforms the sequence ofcandidate units into a sequence of quantized targets u q 1 , . . .
, u qT ′ :
u qk = q ( u k ) .
Quantizing the targetshas been shown to improve the quality of speech representations in contrastive prediction-basedself-supervised learning [Baevski et al., 2020, Nguyen et al., 2022], perhaps by contributing to theprior of discreteness we wish to impose on the high-level features.
2.1 Training objective
[Figure] The objective function capturing our strategy of modeling the signal simultaneously at the level ofhigh resolution low-level continuous features and variable-rate high-level discrete units can be simplydefined as the sum of CPC losses L L and L H operating at both levels.
Because the hard decisions ofthe boundary predictor in the downsampling module are not differentiable, we model the boundarydetector as a stochastic reinforcement learning policy.
Its loss L π minimizes the expected value of thehigh-level loss L H , whose gradient can be estimated through policy gradient algorithms [Williams,1992, Sutton et al., 1999].
Finally, the model also learns to optimize the loss L Q of the k-meansquantizer in the high-level module, and a regularization term L ¯ l for the boundary predictor, whichreduces the search space of the policy by imposing a prior on the expected average sampling rate ofdiscrete units.
The total loss is the sum of all described terms L =
L L + L H + L Q + L π + L ¯ l .
Low-level CPC
The low-level feature extractor is trained to minimize the standard CPC loss
[Equation]
where p 1 , . .
. , p N are predictions made conditioned on c t for the N upcoming encodings, and N isthe set of negative samples drawn randomly from the set of encodings out of the prediction window.
High-level CPC
In contrast to the frame-level features in the low-level CPC loss, where the acousticframes change smoothly over time, at the level of discrete units we expect successive elements to bedistinguishable from each other.
Therefore, at the high-level we modify the standard CPC loss toincorporate this prior knowledge of contiguous heterogeneity.
We do so by sampling the distractorsin the contrastive task from representations adjacent to the prediction target
[Equation]
p 1 , . .
. , p M being predictions conditioned on h k for the M upcoming pseudo-unit representations.
Quantizer To quantize a pseudo-unit representation u k we choose the i -th embedding e i from afinite size learnable codebook that minimizes the distance to the candidate unit representation
[Equation]
The codebook is trained through the online k-means loss from van den Oord et al.
[Equation]
where sg is the stop gradient operator and λ is an hyperparameter.
Boundary predictor We model the boundary detector as a θ -parameterized stochastic policy π θ thattakes as input the sequence of context vectors c and defines for each time-step the probability of aunit boundary conditioned on c : π θ ( b t | c ) = p ( b t | c ) .
The policy is trained to minimize the expectedvalue of L H
[Equation]
We use REINFORCE
[Williams, 1992] to estimate the gradient of L π with respect to θ as
[Equation]
(6) where the expectation can be approximated through Monte-Carlo methods from experience rollouts.
Average sampling rate regularization The model of the boundary detector as a stochastic policyallows us to promote an average pseudo-unit length (or sampling rate) in a differentiable way.
Let ¯ l denote the desired average pseudo-unit length, we define an additional regularization term as
[Equation]
with the right hand side resulting from assuming b t as a Bernoulli variable.
During training wecompute equation 7 on randomly sampled segments from all the possible segments of length ¯ l in abatch.
This prior reduces the difficulty of the policy optimization as it restricts the search space topolicies that produce segmentations matching the average expected unit length.
Experimental setup
The experiments were performed on the train-clean-100 subset of LibriSpeech (Panayotov et al.[2015], CC BY 4.0) using the aligned phone labels provided by van den Oord et al.
Toevaluate low-level frame representations, we measured phone prediction accuracy of a linear classifiertrained on frozen features.
Additionally, we report phone accuracy of the transcriptions obtainedfrom the sequence of representations.
It was computed with a 1-layer bidirectional LSTM networkwith 256 hidden units, followed by a 1D convolution with 256 filters, kernel width 8 and stride of 4,and a CTC loss [Graves et al., 2006].
We modified the loss to label each frame or segment with aphone label, and forbade emitting the blank token.
Since high-level representations have a higherresolution, we evaluated them only using the CTC transcription task.
The model is trained to extract segments which on average match the duration of phones, and we alsoevaluated unit discovery capabilities.
We report the agreement between segment boundaries learnedby the downsampling strategy and the boundaries of human-defined information-bearing units, suchas phones.
We evaluate phone segmentation performance through precision, recall, F1-score andover-segmentation robust R-value
[Räsänen et al., 2009] of the predicted pseudo-unit boundarieswith respect to phone boundaries obtained from the aligned frame labels.
We also evaluate boundaryprediction on a processed version of the TIMIT dataset [Garofolo et al., 1993] in which non-speechevents have been trimmed to a maximum of 20 ms.
The setup of our boundary detection evaluationwas chosen for comparability with recent studies on unsupervised phone boundary detection [Kreuket al., 2020, Bhati et al., 2021b, Cuervo et al., 2021, Kamper, 2022].
3.1 Models
Our model reads single channel raw waveforms sampled at 16kHz, chunked into sequences of 20480samples (1.28 sec).
At the low-level model, the encoder g enc applies five 1D convolutions withinternal dimension 256 and filter widths (10; 8; 4; 4; 4).
Convolutions are followed by channel-wisemagnitude normalization and ReLU activations.
Convolutions are strided by (5; 4; 2; 2; 2), resultingin a 160-fold rate reduction, yielding 256-dimensional frame representations extracted every 10ms.
The context-building model g ar is a two-layer LSTM network [Hochreiter and Schmidhuber, 1997]with 256 units.
At the high-level, s enc is a network with two fully connected hidden layers of 256units and ReLU activations.
s ar is a single layer LSTM network with 256 units.
We use N =12and M =2 predictions, and 128 and 1 negative samples, for the CPC losses at low and high levelmodels respectively.
Each prediction head accesses all past contexts through a single auto-regressivetransformer layer [Vaswani et al., 2017] with 8 scaled dot-product attention heads with internaldimension 2048 and dropout [Srivastava et al., 2014] with p = 0 .
The target quantizer q uses acodebook of 512 embeddings each of dimension 256, and we use the standard literature value ofλ = 0 .
25 in equation 4.
The boundary predictor is a single layer bidirectional transformer networkwith 8 scaled dot-product attention heads with internal dimension 2048.
We set the expected averageunit duration ¯ l = 8 in equation 7 to roughly match the average phone duration measured on the 100hour subset of the LibriSpeech dataset (7.58 frames).
We compare our model to several baselines.
Regarding models with single-level feature extraction:the CPC implementation from [Rivière et al., 2020] which we use as our frame-level model, ACPC[Chorowski
et al., 2021] which improved on vanilla CPC by promoting temporally-stable features,and the CPC optimized for phone boundary detection by Kreuk et al. [Table]
For multi-level modeling we compare: a stack of two CPC modules both using the architecture from
Rivière et al.
[2020] in which the output z vectors of the low-level model are used as input for thehigh-level model (two-level CPC) with no downsampling; a stack of two CPC modules in which thehigh-level model downsamples the outputs of the low-level model at a fixed rate (two-level CPC withdownsampling) using a 1D convolution with internal dimension 256, filter width 9 and stride of 9,resulting in a sampling rate close to the average phone sampling rate in LibriSpeech clean 100; SCPC[Bhati et al., 2021b] which uses a boundary predictor based on detecting peaks of feature pairwisedissimilarities; mACPC which improves upon SCPC on representation learning by adding long-termprediction at both levels [Cuervo et al., 2021].
Finally, we include a supervised topline in which theboundary-predictor is forced to return boundaries at their ground-truth locations and otherwise isequal to our two-level CPC model.
For the model of Kreuk et al.
[2020], SCPC, and mACPC, we use the implementations providedby Cuervo et al. [2021] as the original implementations were not available.
We make our code andconfiguration scripts for all experiments available in the supplemental material and on a GitHubrepository at http://github.com/anonymous/repo .
For single-level models we train until convergence for 300 epochs on the train split of LibriSpeechclean 100.
Multi-level models are trained for 50 additional epochs after appending to the pre-trainedframe-level modules the high-level network and downsampler, where applicable.
All models aretrained using a batch size of 64, the Adam optimizer [Kingma and Ba, 2014] with a learning rate of0.0002, and an initial warm-up phase where the learning rate is increased linearly during the first 10epochs.
For the quantization module we follow the setup from Ła´ncucki et al.
Training ourmodel takes around 13 hours on 2 GPU Nvidia RTX3970X. Evaluation models are trained for 10epochs using the same hyperparameters, with the exception of the warm-up learning rate schedule.
Results
Low-level representations In the frame-level evaluation (Table 1), the models which performcontent-dependent downsampling have the highest accuracy.
Conversely, two-level CPC baselineswith constant or no downsampling show little to no improvements over single-level models.
Theproposed model achieves higher frame accuracy than the model with downsampling according tophone labels, hinting at a possible misalignment between the objectives of matching human-definedphone boundaries and disentanglement of representations.
Finally, we observe poor accuracy of
SCPC despite having two levels.
This is consistent with [Bhati et al., 2021a, Cuervo et al., 2021], andcaused by the absence of context modeling in the frame-level module of SCPC, which optimizes forboundary prediction (see below), but cripples the ability to capture phone class information.
Transcription performance measured by phone accuracy shows a smaller difference between unsupervised models, likely due to preservation of similar amount of phonetic information in the [Table] [Table] representation, which a non-linear classifier network is able to access.
Incorporating top-down information about the structure of the signal might promote disentanglement of low-level representations,therefore making them more amenable to linear discrimination.
High-level representations Table 2 shows the results of the evaluations on high-level representations.
Our model performs best among unsupervised models with variable downsampling, bothin terms of compression rate and phone transcription accuracy.
The model with no downsamplingobtains the best performance on phone transcription among unsupervised models, showing that alldownsampling strategies result in lossy compression.
On the other hand, the model which downsamples the signal according to phone labels aligned with supervision exceeds the performanceof not downsampling, proving that a better compression is possible and can be beneficial in suchunsupervised transcription task (c.f. discussion in sec. 5.2).
Matching phone boundaries Boundary detection results are presented in Table 3.
The boundariesof units predicted by our model often match with human annotated phone boundaries.
On the
LibriSpeech dataset our models attain the highest precision among tested models, at the expenseof lower recall.
Through a visual inspection of the boundaries made by other models we learnedthat the remaining models tend to predict false positive boundaries inside non-speech segments,which explains their poor precision.
This is caused by segmentation criteria of detecting peaks ofdissimilarity between adjacent representations.
Non-speech segments of audio show little variation,and noise can easily trigger the peak detector.
We confirm this by performing boundary detection on
TIMIT without non-speech events, in which dissimilarity-based segmentation models give the bestoverall performance.
We conclude that due to a different training objective, our model might be morerobust to the quality of recordings.
[Table]
Discussion5.1 On the relative importance of the priors
Our model is based on several priors about the high-level structure of the speech signal, implementedthrough architectural choices.
This begs to question the relative impact of them on the results.
Table 1 and Table 2 already show the importance of the priors implied by multi-level modeling andvariable-rate downsampling.
To complement these results we provide in Table 4 an ablation studyon the priors imposed to the high-level features, such as average downsampling rate through policyregularization, and the prior of discreteness using adjacent negative sampling and target quantization.
We present the results of the ablation in terms of linear frame-wise phone classification accuracy, inwhich our model improved the most with respect to the baselines.
Adjacent negative sampling, through which we implement the prior of unit distinguishability, appearsessential.
Without it, the boundary predictor converged to predict a roughly constant probability,sampling the boundaries during training roughly every four frames.
In early experiments, which weadd to the supplemental material, we observed that using the standard strategy for negative samplingat the high level to train the boundary detector resulted in rewarding over-segmentation.
The modelturned the high-level task into the easier problem of short term prediction at the level of smoothacoustic features.
When coupled with the average sampling rate constraint, this seemed to result in arandom production of segments, and frame accuracy plunged even below single-level models.
Policy regularization in terms of the expected average segment length is another crucial element.
Without it, frame accuracy dropped close to that of a single-level model.
Segmentations produced atthe early stages of training were unreasonable, therefore not producing any informative experiencerollout for the policy gradient estimator.
Eventually, the policy would collapse to a local minimum ofnot predicting any boundaries at all.
This could be addressed by promoting exploration or through adifferent parametrization of the actions space.
Note that the average segment length prior can be seenboth as a feature and a limitation.
It allows to control the time scale at which the signal is modeled.
However it implies some degree of supervision in an ideally unsupervised method, particularly inour experimental setup we used the training data to estimate it.
In other applications and domainsdefining a characteristic sampling rate might be difficult.
Finally, target quantization seems to contribute the least to the accuracy.
However, we observed thatit tended to accelerate training convergence.
Early experiments available in the supplemental materialalso show that target quantization provides a stronger supervision signal for the reinforcement learningpolicy by more strongly penalizing unreasonable segmentations.
5.2 On human defined units vs. discovered pseudo-units for representation quality
Results at both low and high level display a non-intuitive misalignment between matching humandefined unit (phone) boundaries and improving representations.
At frame-level, it seems that theobjective of maximizing for discrete predictable structure at the high-level, which not necessarilymeans pushing towards phonetic structure, results in more useful top-down feedback than downsampling to match phonetic information content.
At high-level however, our optimization objective results in phonetic information loss, as we were not able to match the phone accuracy of the modelwith downsampling according to phonetic units.
It could be possible that there are patterns betweenn-grams of phones, which map to a single high-level unit, and facilitate the predictive task more thanat the level of individual phones.
Yet, by being many-to-one mappings, they could make it impossibleto retrieve the original constituents after compression.
We consider that an interesting objective forfuture work is characterizing the patterns of the discovered pseudo-units sequences, and perhapsinvestigate their correlation with models of human speech perception.
Related Work
Perhaps the most widely used variable-length feature discovery approach in deep learning is theextraction of text tokens using byte-pair encodings [Sennrich et al., 2015] or unigram pieces [Kudoand Richardson, 2018].
These approaches work on text and rely on exact matches to collect variablelength symbols into categories.
Thus, they serve as an inspiration but are not directly applicable tolearning a non-uniform segmentation of dense representations.
The segmental unit-discovery system by Kamper et al.
[2017] performed word-type and locationdiscovery by chunking the training corpus into segments that were then clustered into a smallvocabulary.
We use similar assumptions in our system: every frame belongs to a segment and thehigh-level representations are trained to match discrete units.
Variable-length unit discovery in a deep learning setting was recently proposed by Chorowskiet al.
[2019] who modified the VQ-VAE quantization objective to promote token stability betweenneighboring frames and Dieleman et al. [2021] who used run-length encoding of VQ-VAE tokens.
These approaches are similar in spirit to the dissimilarity-based peak detector used in SCPC [Bhatiet al., 2021b]: they detect segments by merging frames with similar representations.
In contrast,we learn a more generic boundary detection which is trained using top-down supervision from theupper-level CPC loss operating at the segment level.
Hierarchical modeling of data is sometimes performed in the context of generative models.
VQVAE [-
-]2 Razavi et al.
[2019] trained a language model on discrete tokens and generated output imagesin two steps: it first sampled sequence of discrete tokens form the language model, then sampledthe image conditioned on them.
In a similar spirit Lakhotia et al.
[2021] trained language modelson raw speech.
However, in these approaches the second-level models still operated at a frequencysynchronous with the input one (without content-dependent variable downsampling) and did notinfluence the learning of the low-level representations.
We have demonstrated the benefits of usingthe high-level loss to tune the low-level representation.
Finally, recently proposed systems that train speech recognizers on unpaired speech and transcriptions[Yeh et al., 2018, Baevski et al., 2021] typically require a step which approximately detects segmentsof similar frames and tries to pair transcriptions with speech using these noisy segments.
Thesemethods could benefit from using our proposed segmentation and high-level features.
Conclusions
We have successfully demonstrated the benefits of training a two-level CPC system which extractsframe-synchronous low level features using a first-level CPC loss, and then discovers a variable-ratesegmentation using supervision from a second high level CPC criterion.
A key component of oursystem is a segment boundary predictor trained using REINFORCE to directly optimize the secondlevel CPC objective.
This leads to the discovery of discernible and predictable second-level segmentswhich coincide with a ground-truth phonetic segmentation.
Moreover, the extra supervision from thesecond level yields improvements to the quality of the frame-level representations, nearly matchingthat of a supervised topline using ground-truth phonetic boundaries.
Checklist
For all authors...(
a) Do the main claims made in the abstract and introduction accurately reflect the paper’scontributions and scope?
[Yes] We provide experimental results that back the claims inthe abstract.
(b) Did you describe the limitations of your work?
Yes In
the discussion section wespeak about e.g. the still unresolved tension between better boundary prediction andbetter separability of the features
(c) Did you discuss any potential negative societal impacts of your work?
N/A
] Wepresent a generic technique which improves unsupervised feature extraction.
(d) Have you read the ethics review guidelines and ensured that your paper conforms to
them?
[Yes] We develop a generic feature learning technique on well known datasets.
If you are including theoretical results...
(a) Did you state the full set of assumptions of all theoretical results?
[N/A]
(b) Did you include complete proofs of all theoretical results?
[N/A]
If you ran experiments...
(a) Did you include the code, data, and instructions needed to reproduce the main ex-
perimental results (either in the supplemental material or as a URL)?
[Yes] All codeand experimental instructions are hosted on a public GitHub Repo (anonymised in the
submission)and we include an anonymised snapshot in the supplementary materials
(b) Did you specify all the training details (e.g., data splits, hyperparameters,
how they
were chosen)?
[Yes] We described them in the text and we provide the source code.
Hyperparameters were for baselines were taken from previously published models.
Hyperparameters of our model are given in the text.
(c) Did you report error bars (e.g., with respect to the random seed after running exper- iments multiple times)?
[No] Due to the limitations of the academic budget, most
experiments were not run multiple times
(d) Did you include the total amount of compute and the type of resources used (e.g., type of GPUs, internal cluster, or cloud provider)?
[Yes]
In the section on trainig procedure
If you are using existing assets (e.g., code, data, models) or curating/releasing new assets...
(a) If your work uses existing assets, did you cite the creators?
[Yes] We cite librispeech
dataset and the initial repository form which we have forked our code
(b) Did you mention the license of the assets?
[Yes] We specify it for Librispeech
(c) Did you include any new assets either in the supplemental material or as a URL?
[No]
(d) Did you discuss whether and how consent was obtained from people whose data you’re
using/curating?
[No] We only use widely used and publicly available datasets
(e) Did you discuss whether the data you are using/curating contains personally identifiable
[Table]
information or offensive content?
[No] We only use widely used and publicly available
datasets
If you used crowdsourcing or conducted research with human subjects...
(a) Did you include the full text of instructions given to participants and screenshots, if [Table]
applicable?
[N/A]
(b) Did you describe any potential participant risks, with links to Institutional Review
Board (IRB) approvals, if applicable?
[N/A]
(c) Did you include the estimated hourly wage paid to participants and the total amount
spent on participant compensation?
[N/A]