Losses Can Be Blessings: Routing Self-Supervised Speech Representations Towards Efficient Multilingual and Multitask Speech Processing
Abstract
Self-supervised learning (SSL) for rich speech representations has achieved empirical success in low-resource Automatic Speech Recognition (ASR) and other speechprocessing tasks, which can mitigate the necessity of a large amount of transcribedspeech and thus has driven a growing demand for on-device ASR and other speechprocessing.
However, advanced speech SSL models have become increasinglylarge, which contradicts the limited on-device resources.
This gap could be moresevere in multilingual/multitask scenarios requiring simultaneously recognizingmultiple languages or executing multiple speech processing tasks.
Additionally,strongly overparameterized speech SSL models tend to suffer from overfittingwhen being finetuned on low-resource speech corpus.
This work aims to enhancethe practical usage of speech SSL models towards a win-win in both enhancedefficiency and alleviated overfitting via our proposed S 3 -Router framework, whichfor the first time discovers that simply discarding no more than 10% of modelweights via only finetuning model connections of speech SSL models can achievebetter accuracy over standard weight finetuning on downstream speech processingtasks.
More importantly, S 3 -Router can serve as an all-in-one technique to enable(1) a new finetuning scheme, (2) an efficient multilingual/multitask solution, (3) astate-of-the-art pruning technique, and (4) a new tool to quantitatively analyze thelearned speech representation.
All codes will be released upon acceptance.
Introduction
Deep neural network (DNN) breakthroughs have tremendously advanced the field of Automatic
Speech Recognition (ASR).
However, one major driving force for powerful DNNs, i e., the availabilityof a large amount of training data, is not always possible for ASR.
This is because collecting largescale transcriptions is costly, especially for low-resource spoken languages around the world, limitingthe wide application of deep ASR models.
Fortunately, recent advances in self-supervised learning(SSL) for rich speech representations [1, 2, 3, 4, 5, 6, 7, 8, 9] have achieved empirical success inlow-resource ASR, where SSL models pretrained on raw audio data without transcriptions can befinetuned on low-resource transcribed speech to match the accuracy of their supervised counterparts.
However, there exists a dilemma between the trends of speech SSL models and the growing demandfor speech processing applications on the edge.
While advanced speech SSL models become increasingly larger to learn more generalizable features, it is highly desired to process the captured speechsignals in real time on edge devices, which have limited resources and conflict with the prohibitivecomplexity of existing speech SSL models.
Such an efficiency concern would be more severe in multilingual/multitask scenarios where simultaneously recognizing multiple languages or executingmultiple speech processing tasks is required: if one separate model is finetuned for each targetlanguage/task, the storage and computational cost will be significantly increased, thus prohibitingthe practical deployment of existing speech SSL models.
Additionally, strongly overparameterizedspeech SSL models tend to suffer from overfitting when being finetuned on a low-resource speechcorpus [10, 11, 12, 13, 14], limiting the achievable accuracy improvement brought about by moreparameters and thus the achievable performance-efficiency trade-off.
In this work, we aim to facilitate the practical usage of speech SSL models towards a win-win inenhanced efficiency and alleviated overfitting for boosting task accuracy under low-resource settings.
Excitingly, we develop a framework, dubbed S elf- S upervised S peech Representation Router (S 3Router), that can serve as an all-in-one technique to tackle the aforementioned challenges and largelyenhance the practical usage of speech SSL models, contributing (1) a new finetuning scheme fordownstream speech processing, i.e., finetuning the connections of the model structure via learninga binary mask on top of pretrained model weights, which notably alleviates model overfitting andthus improves the achievable accuracy over standard weight finetuning methods under a low-resourcesetting; (2) a multilingual/multitask technique via learning language-/task-specific binary masks ontop of shared model weights inherited from SSL pretraining; (3) a competitive pruning technique totrim down the complexity of speech SSL models while maintaining task accuracy; and (4) a new toolto quantitatively analyze what is encoded in speech SSL models thanks to the learned masks’ binarynature on top of shared model weights.
We summarize our contributions below:
• We propose a framework dubbed S 3 -Router, which offers an alternative to the mainstream
finetuning of model weights that finetunes the structure of speech SSL models, by learning
a binary mask on top of the pretrained model weights and integrating it with a novel mask
initialization strategy customized for the pretrain-finetune paradigm;
• We are the first to discover that discarding no more than 10% of weights without finetuningpretrained model weights can achieve better task performances as compared to the mainstream method of weight finetuning on downstream speech processing tasks.
Notably, our
method can scale well to even larger models;
• We extend our S 3 -Router framework for enhanced deployment efficiency, contributing (1) anovel multilingual/multitask solution, and (2) a competitive pruning technique achieving better performance-efficiency trade-offs than state-of-the-art (SOTA)
ASR pruning techniques; • We demonstrate the capability of S 3 -Router as a tool to quantitatively understand what is
encoded in speech SSL models thanks to the binary nature of the learned masks.
S 3 -Router has opened up a new perspective for empowering efficient multilingual/multitask speech
processing and enhancing our understanding about what is encoded in speech SSL models
Related Work
Automatic speech recognition.
Early ASR systems [15, 16, 17, 18, 19, 20] were mainly based onthe combinations of hidden Markov models (HMM) with Gaussian mixture models or DNNs, andoften contain multiple modules (e.g., an acoustic model, a language model, and a lexicon model)trained separately.
Recent works process raw audio sequences end-to-end, including CTC [21]-basedmodels [22, 23, 24, 25, 26], recurrent neural network(RNN)-transducers [27, 28, 29, 30], sequenceto-sequence models [31, 32, 33, 34, 35, 36], and transformer-based models [37, 38, 39].
Specifically,transformer-based models have been widely adopted as speech SSL models [6, 9, 7].
Self-supervised learning for speech representation.
Considering the high cost of collecting largescale transcriptions, learning rich speech representations via SSL has become crucial and promisingfor empowering low-resource ASR.
Early works [40, 41, 42, 43, 44, 45, 46, 47] build generativemodels for speech with latent variables.
Recently, prediction-based SSL methods have becomeincreasingly popular.
We refer the readers to a recent survey [48] for more details.
Among prior arts,[49] is a pioneering work relevant to our method, and adopts masking as an alternative to weightfinetuning on pretrained language models for natural language processing (NLP).
Nevertheless, our
Header
Header Header
[Figure]
English
Spanish
Mandarin
Hello World 你好世界
Hola Mundo
Our -Router Framework S S 3 -Router is non-trivially different from [49] in that (1) we target SSL models in the speech domainand the learned speech representation is required to be generalizable across different spoken languagesunder a cross-lingual transfer setting, where the speech SSL models suffer from a higher risk ofoverfitting on downstream low-resource tasks as compared to NLP, making our findings non-trivialcontributions; (2) S 3 -Router features a win-win in both efficiency and accuracy thanks to our proposedmask initialization strategy, which plays a crucial role in maintaining task accuracy under a highsparsity and enables the extension of S 3 -Router towards a competitive pruning technique with SOTAaccuracy and 3) we further develop S 3 -Router for multilingual/multitask speech processing and as asimple yet effective tool for analyzing language- similarities.
Multitask learning for speech processing.
There exists a growing demand for DNNs to simultaneously process multiple tasks [50, 51].
In the speech domain, previous works attempt to train a singlemodel to solve multiple tasks [52, 53, 54, 55, 56, 57] or use an auxiliary task to enhance the accuracy ofa primary task [58, 59, 60, 61, 62, 63].
Motivated by the success of SSL in low-resource downstreamtasks, multitask/multilingual learning has been adopted in both SSL pretraining [64, 65, 66, 67, 7] toenrich learned speech representations, and finetuning scenarios [68].
For example, a recent relevantwork [67] adopts language-adaptive pretraining on top of [7], where different sparse sub-networks areactivated for different languages during multilingual pretraining and the gradients are accumulatedon the shared super-network to learn better multilingual representations.
In contrast, S 3 -Router isfundamentally different, as it targets the finetuning stage of a given pretrained speech SSL model andonly tunes the connections of the model structure without tuning the SSL pretrained weights.
ASR pruning.
As ASR models become increasingly overparameterized for abstracting generalizablerepresentations, pruning large-scale ASR models has drawn a growing attention.
Early works trimdown either the decoding search spaces [69, 70, 71, 72, 73, 74] or HMM state space [75].
Modern
ASR paradigm has gradually shifted its focus to pruning end-to-end ASR models [76, 77, 78, 79, 80,81, 82, 83, 84, 85, 86, 87].
Recently, [88, 89, 90] prune speech SSL models towards more efficientlow-resource ASR, e.g., PARP [88] proposes a finetuning pipeline to identify the existence of lotterytickets within speech SSL models.
In addition to the boosted pruning efficiency over PARP [88],
S 3 -Router emphasizes the role of sparsity in encoding language-/task-specific information which canempower multitask/multilingual speech processing without tuning the pretrained weights.
The Proposed S 3 -Router Framework
3.1 Drawn Inspirations from Previous Work
Recent works [91, 92, 93] find that sub-networks featuring a decent inborn accuracy and adversarialrobustness are hidden within randomly initialized networks without any weight training.
Specifically, [91, 92] show that sub-networks with a decent accuracy, even matching that of their dense networks, can be identified from randomly initialized networks, and [93] shows an even strongerevidence: merely updating the sparsity patterns of model connections without modifying the randomlyinitiated model weights can produce both accurate and adversarially robust models.
These pioneeringworks imply that tuning the connections of the model structure, which can be characterized by thelearned connection sparsity patterns, can be as effective as training the model weights.
We hypothesize that model sparsity not only can favor model efficiency, but also can serve as a similar role,i.e., another optimization knob, as model weights to encode language-/task-specific information .
3.2 Formulation and Optimization of S 3 -Router
Inspired by the aforementioned intriguing hypothesis, we propose the S 3 -Router framework toempower efficient multilingual and multitask speech processing on top of SSL speech representations.
Overview.
As shown in Fig.
1, given the raw audios of different spoken languages (or differenttasks), S 3 -Router finetunes the connection patterns of the model structure for each target spokenlanguage/task via optimizing language-/task-specific binary masks on top of the shared weights of agiven speech SSL model, instead of finetuning the model weights as adopted in the common pretrainfinetune paradigm.
Specifically, the learned binary masks of each language/task are multipliedwith the shared model weights to mask out some connections within the given speech SSL model,where the remaining connections (or the induced connection sparsity) encode language-/task-specificinformation for different spoken languages and downstream tasks.
Note that for each language or task,only one set of binary masks and one lightweight header, e.g., the classification head for ASR whichis naturally non-shareable across languages due to diverse dictionary sizes, need to be independentlytrained, incurring negligible overhead (e.g., ≤ 6.3% storage of the whole backbone model).
Formulation.
Formally, our S 3 -Router framework can be formulated as:
[Equation]
141 where ( x t , y t ) are the input audio and corresponding transcriptions/labels of a spoken language/task tin a downstream dataset D t , and θ SSL is the SSL pretrained weights of the given speech SSL modelf .
Specifically, the mask set m t is applied on top of the model weights θ SSL , and optimized tominimize the loss function l t , e.g., a CTC loss [21] for ASR, subject to an L 0 sparsity constraint,where the number of non-zero elements in m t is limited to k t , which serves as a hyperparameterfor controlling and balancing (1) the amount of language-/task-specific information encoded in theconnection sparsity and (2) model efficiency.
Intuitively, if the sparsity is 0% or 100%, no newinformation is introduced during finetuning on the corresponding new/downstream language/task.
Optimization.
To differentiably optimize m t in Eq.
(1), we binarize m t and activate only its top k telements during forward, while all the elements in m t are updated via straight-through estimation [94]during backward.
More specifically, during forward, we binarize m t to ˆ m t via enforcing its top k telements to 1 and other elements to 0, thus the forward function becomes f ( ˆ m t ⊙ θ SSL , x t ) , whichis the same for the inference process; during backward, we directly propagate the gradients from thebinary mask ˆ m to m , i.e., ∂l∂m t ≈ ∂l∂ ˆ m t , thus m t can be learned in a gradient-based manner.
3.3 How to Initialize the Masks in S 3 -Router?
Importance of mask initialization.
A low learning rate is commonly adopted during finetuning toensure effective inheritance of the SSL speech representations, resulting in merely smaller changesin each set of the masks m t in Eq.
(1) as compared to training from scratch.
Therefore, the maskinitialization strategy plays an important role for the quality of the finally optimized masks in
S 3 -Router.
Next, we discuss the pros and cons of two intuitive mask initialization strategies:① Random initialization (RI).
We empirically find that adopting commonly used random initialization [95] in S 3 -Router can achieve a decent accuracy under a low sparsity.
However, since no priorknowledge of the speech SSL model is utilized, the accuracy can largely drop with a high sparsityunder random mask initialization, limiting extending S 3 -Router to be a practical pruning technique.
② Weight magnitude based initialization (WMI).
As weights magnitude can quantify the importance of weights as commonly used in pruning [96, 97, 98, 99, 100], taking the magnitudes of thegiven SSL pretrained weights || θ SSL || in Eq.
1 as the initial values of masks might help utilize theknowledge learned during SSL pretraining.
However, we empirically find that doing so causes worsetrainability, i.e., the ranking of mask values is then more stable during finetuning than that underrandom initialization, and learned binary masks seldom change and mostly stick to their initial values.
This may inhibit the optimization process and lead to sub-optimal learned masks.③ Proposed Order-Preserving Random Initialization (ORI).
To marry the best of both aboveinitialization strategies, we propose a new one, which can be viewed as a weight rank based initialization.
In ORI, we first acquire mask values of the same dimension as the shared weights viarandom initialization like [95], and then perform magnitude-based sorting to assign a larger maskvalue to weight elements with a larger magnitude.
Thus, the ranking order between the mask valuesof the weight elements is the same as that of their magnitudes.
In this way, the mask trainability ismaintained while the learned speech SSL model knowledge can be exploited (see Sec 4).
3.4 S 3 -Router is Useful in Various Application Scenarios
A new finetuning scheme.
S 3 -Router offers a new and equally effective finetuning scheme as itfinetunes model connections given a speech SSL model.
As large-scale speech SSL models tend tooverfit when finetuning their weights under a low-resource setting, we hypothesize that the binaryoptimization of the mask patterns in Eq.
(1) can serve as regularization and thus alleviate overfitting.
An efficient multilingual and multitask method.
S 3 -Router can naturally enable multilingual andmultitask speech processing by merely switching among its learned language-/task-specific binarymasks.
One advantage is that since the gradients of different spoken languages or tasks can nowbe independently accumulated on their corresponding masks without interfering each other, thecommonly observed gradient conflict issue [101] can be alleviated.
A new pruning technique.
Since S 3 -Router encodes language/task-specific information via binarymasks on top of shared model weights, it naturally introduces sparsity into the given model, and thuscan naturally serve as a pruning technique.
To further improve the achievable accuracy-efficiencytrade-off and more fairly benchmark with SOTA ASR pruning methods, we propose a variant of
S 3 -Router dubbed S 3 -Router-P, which first finetunes the model weights on the downstream audiosand then prunes the model connections based on the learned binary masks in Eq.
An effective and simple tool to analyze what is encoded across language/task-specific speech
SSL models.
Another exciting advantage of S 3 -Router is that it can provide a quantitative metricabout how the pretained SSL speech presentation is utilized by different spoken languages/tasks viamasking out languages/tasks-specific weights.
S 3 -Router: Discarding ≤ 10% Weights is All You Need4.1 Experiment Setup
Here we evaluate S 3 -Router as a new finetuning scheme for downstream speech processing.
Models:
We adopt wav2vec 2.0 base (wav2vec2-base) and large (wav2vec2-large) [6] (with 95M/317M parameters, respectively) pretrained on LibriSpeech 960 hours [102] and xlsr [7] (with 317M parameters)pretrained on 53 languages sampled from CommonVoice [103] as our speech SSL models.
Datasets:
We consider 10 speech processing tasks, including low-resource English ASR on LibriSpeech [7]with only 10min/1h/10h labeled data, following the dataset split in [6, 88], and low-resource phonemerecognition on CommonVoice [103] with 1h labeled data per language, following the dataset splitin [104, 7], as well as 8 speech processing tasks from SUPERB [105].
Finetuning settings: Our codeis built on top of Fairseq [106] and we follow the standard finetuning settings for each task, i.e., thedefault configurations in Fairseq [106] for ASR/phoneme recognition and those in SUPERB [105] forother tasks. In particular, all our experiments on ASR/phoneme recognition adopt an Adam optimizerwith an initial learning rate of 5e-5 plus a tri-stage schedule [6] and we finetune wav2vec2-base/largefor 12k/15k/20k steps on the 10m/1h/10h splits, respectively, and xlsr is finetuned for 12k steps foreach spoken language. It takes about 10/24/24 GPU hours to finetune wav2vec2-base/large/xlsr for
[Figure] [Figure]
12k steps with our S 3 -Router. We do not freeze all the layers except the final linear layer for the first10k steps [6], following [88].
S 3 -Router settings: If not specifically stated, S 3 -Router only tunes theconnections of (i.e., applies learnable masks on) the feed-forward networks (FFNs) of transformerstructures [107] and fixes all other weights, which is empirically found to be optimal based on theablation studies in Sec. 6.
In addition, if not stated, we adopt our ORI mask initialization by defaultand do not apply language models for a fair benchmark of ASR performances, following [88].
4.2 Benchmark on Low-resource English ASR
Finetuning on wav2vec2-base.
We apply our S 3 -Router on wav2vec2-base under different lowresources settings as shown in Fig.
2 (a) ∼ (c).
We can observe that our S 3 -Router can consistentlyoutperform the standard weight finetuning in terms of the achievable WER, i.e., the lowest WER atthe corresponding optimal sparsity.
In particular, our S 3 -Router achieves a 1.72%/2.34% reduction inword error rate (WER) under a sparsity ratio of 8% on the test-clean/test-other set of LibriSpeech,respectively, when being finetuned on 10min labeled data.
This indicates that finetuning connectionsvia our S 3 -Router can be a competitive alternative with reduced WER for finetuning weights underlow-resource settings, which provides a new paradigm for finetuning speech SSL models.
Ablation studies of mask initialization.
We equip our S 3 -Router with different mask initializationschemes in Sec.
3.3 and show their achievable WER on low-resource English ASR in Tab.
We canobserve that (1) the proposed ORI mask initialization consistently outperforms the other two schemesin terms of the achievable WER, and (2) even random mask initialization can match or surpass theperformances of standard weight finetuning.
We also provide their complete sparsity-WER trade-offs in the appendix and find that weight magnitude based initialization favors larger sparsity ratios, whileit is harder to overturn the ranking between masks via gradients, resulting in inferior achievable WER.
[Table]
Scalability to larger speech SSL models.
Weapply S 3 -Router to a larger model wav2veclarge.
As shown in Fig.
2 (e) ∼ (g), we can seethat the achievable WER of S 3 -Router still consistently outperforms the standard weight finetuning across all downstream datasets, e.g., a1.99%/2.37% WER reduction under a 4% sparsity on the test-clean/test-other set, respectively,when being finetuned on the 1h labeled data.
Consistent results on xlsr are provided in the appendix.
Insights.
This set of experiments indicates that ❶ our S 3 -Router features a good scalability to largerspeech SSL models as well as a good generality for different pretraining schemes, and ❷ our methodeffectively reduces the overfitting on more overparameterized speech SSL models according to thelarger performance gains, thanks to the regularization effect of the binary optimization process in
Eq.
(1), making our method an appealing solution for more advanced speech SSL models.
[Table]
Add language models (LMs).
We further apply a 4-gram LM [108] (detailsin the appendix) as the decoder for bothour S 3 -Router at the optimal sparsityratio in Fig.
2 and the weight finetuning baselines.
As shown in Tab. 2, our
S 3 -Router still consistently outperformsweight finetuning with more notable reductions on wav2vec2-large, e.g., a 1.66%/3.22% WERreduction on the test-clean/test-other set of LibriSpeech when being finetuned on 10min labeled data.
4.3 Benchmark on Low-resource Cross-lingual Transfer
We further evaluate our S 3 -Router under two cross-lingual transfer settings, including (1) finetuningthe multilingual pretrained xlsr on different low-resource languages in CommonVoice [103], and (2)a high-to-low resource transfer setting where the monolingual (English) pretrained wav2vec2-base isfinetuned on multiple low-resource languages in CommonVoice.
Cross-lingual transfer on xlsr.
As shown in Fig.
3, our S 3 -Router consistently outperforms standardweight finetuning in terms of the achievable phoneme error rate (PER) across all the 10 spokenlanguages, e.g., a 1.12%/2.01% PER reduction on Dutch/Mandarin, respectively.
[Table]
High-to-low resource transfer on wav2vec2base.
As shown in Tab. 3, on top of the Englishpretrained wav2vec2-base, finetuning the connections with S 3 -Router still wins the lowestachievable PER over the baseline.
Consistentresults on wav2vec2-large are in the appendix.
Insights.
This set of experiments implies that foreach downstream spoken language, there exist decent subnetworks for processing its language-specificinformation even in monolingual pretrained speech SSL models.
In another words, properly learnedsparsity can encode language-specific information with competitive performances.
4.4 Benchmark on More Downstream Speech Processing Tasks
We further evaluate our S 3 -Router via finetuning wav2vec2-base on 8 speech processing tasks from
SUPERB [105], covering different aspects of speech (content/speaker/semantics/paralinguistics).
Weshow the achievable task performances as well as the corresponding optimal sparsity ratios in Tab.
4and find that our method surpasses the task performances over standard weight finetuning across all8 tasks via discarding ≤ 10% weights, indicating that properly learned sparsity can also effectivelyencode task-specific information.
We provide the sparsity-performance trade-offs in the appendix.
[Table]
4.5 Empowering Multilingual and Multitask Speech Processing
Since all the aforementioned results achieved on the same speech SSL model via S 3 -Router sharepretrained weights, it can simultaneously enable multilingual and multitask speech processing, thanksto the independent accumulation of task-specific gradients that avoids gradient conflicts [101], ofsuperior scalability with the number of tasks.
For example, as compared to independent weightfinetuning for each language/task, S 3 -Router can simultaneously support 11 languages in Sec.
4.2/ 4.3and 8 tasks in Sec.
4.4 using one wav2vec2-base while achieving a win-win in both accuracy (asvalidated in Sec.
4.2/ 4.3/ 4.4) and efficiency, i.e., more than 88.5% reductions in model parameters.
[Figure]
S 3 -Router-P: Pruning ASR Models for Enhancing Efficiency
Setup.
For evaluating S 3 -Router-P (see Sec. 3.4), we conduct two modifications: (1) both FFN andself-attention (SA) modules are pruned for a fair comparison, and (2) we adopt weight magnitudebased mask initialization for two reasons: firstly, it features the best scalability to high sparsity amongthe three initialization schemes; secondly, since the weights have been finetuned, their magnitudescan serve to indicate their importance on the downstream speech.
Ablation studies for pruning underdifferent mask initialization schemes are in the appendix.
[Figure]
For our baselines, we benchmark with three ASR pruning methods, i.e., one-shot/iterative magnitude pruning (OMP/IMP),and the SOTA method PARP [88] under the best setting(dubbed PARP-P), and directly adopt their reported performances in [88].
Pruning wav2vec2-base on LibriSpeech.
As shown in Fig.
4,we can observe that (1) our S 3 -Router-P consistently achievesthe most competitive WER-sparsity trade-offs across the twomodels and three datasets, e.g., a 6.46% lower WER over PARPunder a sparsity ratio of 0.7 with 10min labeled data, and (2) our method features a better scalability tomore stringent low-resource scenarios according to the large performance gains on LibriSpeech-10m.
Pruning wav2vec2-base on Mandarin@CommonVoice.
As shown in Fig.
5, consistent observationscan be drawn that our method still wins the WER-sparsity trade-offs.
More pruning results areprovided in the appendix.
S 3 -Router: What is Encoded in Speech SSL Models?
6.1 What is The Roles of
We tune the connections of a subset of the modules in the speech SSL model via our S 3 -Router to study their contributions with other modules fixed in terms of both weights and connections
[Table]
FFN vs.
SA.
We tune the connections of
FFN, SA, or both on top of wav2vec2- and
LibriSpeech-1h as shown in Tab.
We can observe that tuning the connections of FFN only
[Table]
wins the lowest achievable WER, even outperforming tuning both FFN and SA.
This indicatesthat the SSL pretrained SA modules are generalizable enough to capture temporal relationship between tokens for downstream speech thus only FFNneeds to be tuned for encoding new information about downstream tasks.
We provide more analysis
[Table]
about the roles of different blocks in the appendix.
[Table]
6.2 How Much New Information is Learned by Finetuning?
[Table]
We visualize the optimal sparsity ratio forachieving the lowest WER on LibriSpeech10m/1h/10h, which serves as an indicator aboutthe amount of new information learned byfinetuning, across wav2vec2-base/large/xlsr in
Tab. 6.
We can observe that (1) larger speech
SSL models reach their optimal performances under lower sparsity, i.e., relatively less amount oftuning could lead to their decent downstream performances thanks to their overparameterization, and(2) finetuning on more resources leads to higher optimal sparsity, indicating that speech SSL modelshave more confidence to update the SSL pretrained representations given more resources.
6.3 How is The Learned Masks Correlated to Phonetics?
Given the decent performances achieved by the learned masks, we are interested their correlationswith human expertise in phonetics.
In particular, we wonder whether the similarity between thelearned masks of two languages aligns with that between their phoneme inventories.
Setup.
Given the 11 languages adopted in Sec.
4.2 and 4.3, we calculate layer-wise cosine similaritiesof their learned masks, under a sparsity ratio of 0.1 with near-optimal performances, between eachpair of them.
We pick the mask similarity of the first layer as our metric without losing generality.
We further measure the cosine similarity between their corresponding phoneme inventories acquiredfrom Phoible [109], following [110], and more detailed settings are in the appendix.
[Figure]
Results and analysis.
We visualize the two similarity metrics of each language pair in Fig.
6and find that the Pearson Correlation Coefficient [111] and the Spearman correlation coefficient [112] between the two similarity metricsare 0.527 and 0.548, respectively.
This indicatesthat the similarities of our learned masks arenon-trivially correlated with that of phoneticswhile the former also provides new insights inthe eyes of speech SSL models, which couldshed light on future advances in zero-shot crosslingual transfer [110].
We provide more insightful correlation analysis as well as the visualization oflayer-wise mask similarities between different languages in the appendix.
Conclusion
Motivating by the empirical success of SSL speech representations in low-resource speech processing,we propose S 3 -Router to facilitate the practical usage of speech SSL models via finetuning their connections instead of weights and encoding language-/task-specific information via sparsity.
Extensiveexperiments validate that S 3 -Router not only serves as a stronger alternative with alleviated overfittingand enhanced accuracy for standard weight finetuning, but also empowers efficient multilingual andmultitask speech processing.
Our insights could enhance our understandings about what is encodedin speech SSL models and thus shed light on future advances in SSL speech representations.
Checklist
The checklist follows the references.
Please read the checklist guidelines carefully for information onhow to answer these questions.
For each question, change the default [TODO] to [Yes] , [No] , or[N/A] .
You are strongly encouraged to include a justification to your answer , either by referencingthe appropriate section of your paper or providing a brief inline description.
For example:
• Did you include the license to the code and datasets?
[Yes] See Section ?? .•
Did you include the license to the code and datasets?
[No] The code and the data areproprietary.•
Did you include the license to the code and datasets?
[N/A]
Please do not modify the questions and only use the provided macros for your answers.
Note that the
Checklist section does not count towards the page limit.
In your paper, please delete this instructionsblock and only keep the Checklist section heading above along with the questions/answers below.
For all authors.
(a) Do the main claims made in the abstract and introduction accurately reflect the paper’s
contributions and scope?
[Yes]
(b) Did you describe the limitations of your work?
[Yes] In Sec.
6.3, we discuss the
potential of supporting zero-shot cross-lingual transfer as our future work, which is anew problem created by our work.
(c) Did you discuss any potential negative societal impacts of your work?
[No] Our
work proposes a framework to empower efficient multilingual and multitask speech
processing, which does not notably correlate with any negative societal impacts.
(d) Have you read the ethics review guidelines and ensured that your paper conforms to
them?
[Yes]
If you are including theoretical results...
(a) Did you state the full set of assumptions of all theoretical results?
[N/A]
(b) Did you include complete proofs of all theoretical results?
[N/A]
If you ran experiments...
(a) Did you include the code, data, and instructions needed to reproduce the main experi-
mental results (either in the supplemental material or as a URL)?
[No] We will release
our code, data, and instructions upon acceptance so that we can ensure ease of use.
(b) Did you specify all the training details (e
g., data splits, hyperparameters,
how they
were chosen)?
[Yes] See Sec.
4.1 and the appendix.
(c) Did you report error bars (e g., with respect to the random seed after running experi-
ments multiple times)?
[No] We report the average results for clarity of the figures.
(d) Did you include the total amount of compute and the type of resources used (e.g., type
of GPUs, internal cluster, or cloud provider)?
[No] because the experiments are not
computationally intensive.
If you are using existing assets (e
g., code, data, models) or curating/releasing new assets...
(a) If your work uses existing assets, did you cite the creators?
[Yes]
(b) Did you mention the license of the assets?
[No] The used dataset and the license
information can be found via the cited reference.
(c) Did you include any new assets either in the supplemental material or as a URL?
[No]
(d) Did you discuss whether and how consent was obtained from people whose data you’re
using/curating?
[No]
(e) Did you discuss whether the data you are using/curating contains personally identifiable
information or offensive content?
[No]
If you used crowdsourcing or conducted research with human subjects...
(a) Did you include the full text of instructions given to participants and screenshots, if
applicable?
[N/A]
(b) Did you describe any potential participant risks, with links to Institutional Review
Board (IRB) approvals, if applicable?
[N/A]
(c) Did you include the estimated hourly wage paid to participants and the total amount
spent on participant compensation?
[N/A]