Self-supervised Representation Learning for Speech Processing (2022)

Abstract: There is a trend in the machine learning community to adopt self-supervised approaches to pre-train deep networks. Self-supervised representation learning (SSL) utilizes proxy supervised learning tasks, for example, distinguishing parts of the input signal from distractors, or generating masked input segments conditioned on the unmasked ones, to obtain training data from unlabeled corpora. BERT and GPT in NLP and SimCLR and BYOL in CV are famous examples in this direction. These approaches make it possible to use a tremendous amount of unlabeled data available on the web to train large networks and solve complicated tasks. Thus, SSL has the potential to scale up current machine learning technologies, especially for low-resourced, under-represented use cases, and democratize the technologies. Recently self-supervised approaches for speech processing are also gaining popularity. There are several workshops in relevant topics hosted at ICML 2020 ( https://icml-sas.gitlab.io/ ), NeurIPS 2020 ( https://neurips-sas-2020.github.io/ ), and AAAI 2022 ( https://aaai-sas-2022.github.io/ ). However, there is no previous tutorial about a similar topic based on the authors’ best knowledge. Due to the growing popularity of SSL, and the shared mission of the areas in bringing speech and language technologies to more use cases with better quality and scaling the technologies for under-represented languages, we propose this tutorial to systematically survey the latest SSL techniques, tools, datasets, and performance achievement in speech processing. The proposed tutorial is highly relevant to the special theme of ACL about language diversity. One of the main focuses of the tutorial is leveraging SSL to reduce the dependence of speech technologies on labeled data, and to scale up the technologies especially for under-represented languages and use cases.

## introduction there is a trend in the machine learning community to adopt self-supervised approaches to pretrain deep networks. self-supervised representation learning (ssl) utilizes proxy supervised learning tasks, for example, distinguishing parts of the input signal from distractors, or generating masked input segments conditioned on the unmasked ones, to obtain training data from unlabeled corpora. bert and gpt in nlp and simclr and byol in cv are famous examples in this direction. these approaches make it possible to use a tremendous amount of unlabeled data available on the web to train large networks and solve complicated tasks. thus, ssl has the potential to scale up current machine learning technologies, especially for lowresourced, under-represented use cases, and democratize the technologies. recently self-supervised approaches for speech processing are also gaining popularity. there are several workshops in relevant topics hosted at icml 2020 foot_0 , neurips 2020 foot_1 , and aaai 2022 foot_2 foot_3 . we also found ssl for speech starting to be one of the focused topics in special/regular sessions of mainstream speech conferences such as icassp and interspeech foot_4 foot_5 . on the other hand, there is a growing synergy between the speech and computational linguistic community because of the proximity of the two areas. many problems including speech assistant, dialog management, speech translation, and automatic speech recognition attract researchers from both areas. due to the growing popularity of ssl, and the shared mission of the areas in bringing speech and language technologies to more use cases with better quality and scaling the technologies for underrepresented languages, we propose this tutorial in the type of cutting-edge to systematically survey the latest ssl techniques, tools, datasets, and performance achievement in speech processing. there is no previous tutorial about similar topic based on the authors' best knowledge. the tutorial aims to make the researchers in speech and language community aware of existing ssl innovation, and equipped to try out the new techniques. we also hope to bring researchers interested in the topics from both areas connected, catalyze new ideas and collaboration, and drive the ssl research frontier. ## tutorial structure and content this is a three-hour tutorial. in the reference below, the red asterisks ( * ) indicate the papers of the speakers. this tutorial will cover at least 70% of the content not from the authors' papers. ## diversity the proposed tutorial is highly relevant to the special theme of acl about language diversity. one of the main focuses of the tutorial is leveraging ssl to reduce the dependence of speech and language technologies on labeled data, and to scale up the technologies especially for under-represented languages and use cases. we will also discuss the new challenges and ethical consideration brought by ssl to communities, such as heavy memory footprint, expensive computation for pre-training and inference, and carbon emission. these topics aim at stimulating discussion and investment in allowing more use cases, in terms of quantity and diversity, to benefit from the advancement of speech and language technologies with the application of ssl. hence, acl would be preferred because of the alignment of themes. naacl-hlt/emnlp/coling are also acceptable due to the importance and relevance of ssl techniques for speech and language community. in addition to the themes of tutorial, the presenters are also diverse in countries and genders. there are both senior and junior instructors, and come from academia and industry. with the diverse background of presenters, we aim to offer attendees a comprehensive review and encourage diversified discussion. ## attendee prerequisites and reading list we will introduce every speech and language task discussed in the tutorial and require no domain knowledge about these tasks from attendees. instead, the attendees should understand derivatives as found in introductory calculus, possess basic knowledge in machine learning concepts such as classification, model optimization, gradient descent, pre-training, and transformer. we also encourage the audience to read the papers of some well-known ssl techniques before the tutorial, which are listed below: @xcite @xcite . those papers focus on cv or nlp, so the content does not highly overlap with the tutorial, but the audience can learn more from the tutorial if they already have general ideas about ssl. ## tutorial logistics there is no previous tutorial on similar topics. given our experiences from related icml and neurips workshops in 2020 (we observed 13 invited talks, 28 accepted papers, and over 150 participants combined) and the growing interests in ssl from academy, we estimate the number of participants to be between 100 and 200. we do not have special requirements for technical equipment and we will allow the publication of our slides and recording of the tutorial in the acl anthology. shu-wen yang is currently pursuing his ph.d. degree in ntu. his research focuses on self-supervised learning (ssl) in speech. he is dedicated to establishing the benchmark in this field, speech processing universal performance benchmark (superb), which focuses on ssl's generalizability across unseen data domains and tasks. he is also the co-creator of the s3prl toolkit which includes numerous recipes for both pre-training and benchmarking for ssl in speech. katrin kirchhoff is a director of applied science at amazon web services, where she heads several teams in speech and audio processing. prior to joining amazon she was a research professor at the university of washington, seattle, for 17 years, where she co-founded the signal, speech and language interpretation lab. her research interests are in speech processing, conversational ai, and machine learning, including representation learning, continual learning, and low-resource asr. she has previously served on the editorial boards of speech communication and computer, speech, and language, and was a member of the ieee speech technical committee.