FA t N et: Cost-Effective Approach Towards Mitigating the Linguistic Bias in Speaker Verification Systems (2022)

Abstract: Linguistic bias in Deep Neural Network (DNN) based Natural Language Processing (NLP) systems is a critical problem that needs attention. The problem further intensifies in the case of security systems, such as speaker verification, where fairness is essential. Speaker verification systems are intelligent systems that determine if two speech recordings belong to the same speaker. Such human-oriented security systems should be usable by diverse people speaking varied languages. Thus, a speaker verification system trained on speech in one language should generalize when tested for other languages. However, DNN-based models are often language-dependent. Previous works explore domain adaptation to fine-tune the pre-trained model for out-of-domain languages. Fine-tuning the model individually for each existing language is expensive. Hence, it limits the usability of the system. This paper proposes the cost-effective idea of integrating a lightweight embedding with existing speaker verification systems to mitigate linguistic bias without adaptation. This work is motivated by the theoretical hypothesis that attentive-frames could help generate language-agnostic embeddings. For scientific validation of this hypothesis, we propose two frame-attentive networks and investigate the effect of their integration with baselines for twelve languages. Empirical results suggest that frame-attentive embedding can cost-effectively reduce linguistic bias and enhance the usability of baselines.

## introduction mitigating the linguistic bias in deep neural network (dnn) based models is one of the critical challenges in natural language processing (nlp). the linguistic bias, specifically in the security systems, such as speaker verification models, is a far more critical problem requiring much research. speaker verification systems are biometric authentication systems that use speech signals to authenticate a speaker. these systems use the fact that every speaker has unique traits in their voice @xcite . such systems have real-world applications in e-commerce, forensics, law, business, and access control mechanisms @xcite . these systems can be text-dependent or text-independent @xcite ). textindependent speaker verification systems are more user-friendly than text-dependent systems. these systems authenticate a speaker without any constraint on the content of speech. however, speaker verification models often tend to be language-dependent @xcite . the reason is that a robust speaker verification system would require memory to analyze the sequential speech data and capture relevant discriminatory information. memory helps in remembering past information. remembering the past and predicting the future can contribute to linguistic content in the embedding @xcite . therefore, the generated embedding for speaker verification may contain linguistic detail. language-dependent speaker verification models perform relatively well on test sets containing speech recordings in the same language as the training set. however, the performance of these systems degrades on test sets containing speech recordings in different languages. the majority of the publicly available speech datasets are in english. it is a tedious task to get labeled datasets for various lowresource languages. most of the previous works use domain adaptation to improve the performance of speaker verification models only for a limited set of languages @xcite @xcite . it is also costly to fine-tune a pre-trained speaker verification model individually for each existing language. further, studies show that the linguistic content in the embedding increase with the temporal scope of representations @xcite . our proposed work is based on the theoretical hypothesis that frame-level features contain less lin-guistic information due to the low-temporal scope of frames. thus, frame-level features may help generate a language-agnostic embedding. furthermore, an intelligent selection of frame-level features may help in enhancing the model's generalizability to out-of-domain testing. we aim to address the problem of language dependency in text-independent speaker verification systems cost-effectively, without the overhead of domain adaptation. we propose an idea that incorporates a lightweight embedding with existing speaker verification systems which may help in improving the generalizability of these systems to out-of-domain testing. to scientifically validate the theoretical hypothesis, we present and investigate two variants of frame-attentive networks: fatnet-v1 and fatnet-v2. our proposed models accept two speech recordings as input and determine if they belong to the same speaker. the speakers in the trial pair may be unknown. we qualitatively compare the generalization ability of our proposed models with two strong baselines on four publicly available data sets. we perform quantitative experiments on 12 languages to assess the integration of our proposed fatnet embeddings with the baselines on publicly available out-of-domain test sets without domain adaptation. we have released the code foot_0 to encourage more research on this problem. we summarize our main contributions below: ## background and motivation language dependency in speaker verification: the current state-of-the-art explores deep neural networks (dnn) to solve speaker recognition problems @xcite @xcite @xcite @xcite @xcite . however, most dnn-based feature extractors are language-dependent @xcite . language dependency can make the system less usable as users may belong to different geographic locations and speak varied languages. it is challenging to get labeled datasets for various low-resource languages @xcite . moreover, when tested on multilingual datasets and features, the models that show consistent behavior may be helpful in other applications (such as code-switching) through information sharing @xcite . we know that there are approximately 7,000 languages in the world @xcite . one of the critical challenges in natural language processing (nlp) is to develop techniques to overcome this linguistic bias and enhance the usability of the model across the globe @xcite . recent works: transfer learning is a solution to address the problem of domain mismatch. however, it is challenging to get labeled datasets for various low-resource languages @xcite . recent works investigate adversarial domain adaptation techniques for solving cross-lingual speaker verification problems @xcite @xcite . however, most of these approaches can improve the performance of speaker verification models for a limited set of languages as these approaches require an additional overhead of domain adaptation. to the best of our knowledge, @xcite is the closest work related to our problem statement where the authors attempt to reduce linguistic bias in speaker verification without domain adaptation. in @xcite , the authors suggest that training a speaker verification model in multiple languages can increase its generalizability to out-of-domain languages. however, they trained the model on an extensive training set consisting of 1,96,000 speakers and 2,06,18,000 utterances. training on such an extensive dataset requires significant computational requirements, often not feasible in a realistic scenario. in addition to that, their proposed work involves a combination of text-dependent and text-independent speaker verification systems. our proposed method involves integrating a lightweight embedding with the existing text-independent speaker verification models to reduce linguistic bias in those systems. linguistic components in a frame: speech signals are non-stationary, and hence they are divided into frames. a speech signal is assumed to be stationary within a frame @xcite . the tem-poral scope of a frame is usually a few milliseconds. researchers have investigated the role of temporal scope in their study of phonology in neural models @xcite . studies show that representational similarity analysis (rsa) applied to local representations results in lower correlations between phonemes and neural activation patterns @xcite . thus, considering the low temporal scope of frames, it is intuitive that they carry less linguistic information. theoretical hypothesis: our proposed work investigates the theoretical hypothesis that utterancelevel embedding captures more linguistic information than frame-level embedding. therefore, a frame-level embedding can be more languagerobust than an utterance-level embedding. this abstract knowledge of frame-level features can allow the model to learn relevant discriminatory information from frames and ignore the linguistic information from speech. researchers have stated that some frames tend to be more critical than others for the final-encoded representation of speech @xcite . attention mechanisms are popular in state-of-the-art speaker verification models @xcite . we explore the effectiveness of attention for an intelligent selection of features within a frame. ## proposed approach to investigate the theoretical hypothesis, we propose the following two variants of frame-attentive networks: fatnet-v1 and fatnet-v2. as illustrated in figure 1 and figure 2 , the timedelay neural network (tdnn) paths are similar in both the fatnet versions and the details are as follows: the model accepts a pair of mel-frequency cepstral coefficients foot_1 (mfcc) for speaker verification @xcite @xcite . mfccs finetune the features to what human beings hear @xcite . let d be the dimension of input mfccs and l 1 and l 2 be the number of frames in the given pair for speaker verification. the values of l 1 and l 2 may differ due to the duration variability issue. the models were trained on 3-second chunks of speech @xcite . eighty-dimensional mfccs of shape (94,80) generated using these audio clips for training are input to the model. as shown in figure 1 and figure 2, we pass each of the input mfccs to the adap-tiveavgpool2d layer to get features of shape (b, 94, 80), where b is the batch-size @xcite . it allows the model to accept variable duration speech recordings during test time without any special augmentation strategy. it facilitates easy integration of fatnet embedding with other speaker verification models and enhances the usability of the models. the next step is to compute frame-level features for further analysis. the abstract knowledge of frame-level features can reduce the linguistic information in the final embedding. we use four stacked tdnn foot_3 layers to extract the frame-level features @xcite . given two speech recordings as input, the problem is to determine if they belong to the same speaker or not. thus, we have two such tdnn paths for input audio clips. fatnet-v1: we concatenate the frame-level features obtained for both the input speech recordings as shown in figure 1 . we apply batch normalization. we further pass these concatenated features through an eight-head frame-level attention block. in each frame, attention gives more weight to relevant features. fatnet-v2: we pass the frame-level features obtained for the input speech recordings through separate four-head attention blocks. thus, we get embeddings 1 and 2 as shown in figure 2 . in each frame, the attention block gives more weight to relevant features. we then concatenate the outputs to get embedding 3. attention: attention mechanisms are popular in state-of-the-art speaker verification models @xcite . our proposed fat-net attention mechanism is inspired by @xcite . the frame-level features are sent through a multi-head residual self-attention block, as shown in figure 1 and figure 2 , respectively. the input to the attention block can be a tensor of shape (b, l, d) where b is the batch size, l is the number of frames, and d is the number of features or dimensions in each frame. let dv be the dimension of linear space where the input needs to be projected, and nv is the number of heads in the multi-head self-attention block. we pass the same tensor to the attention block as the query tensor, key tensor, and, value tensor. the idea is to use the query tensor and key tensor to generate a weight tensor for the value tensor. the first step involves passing each of the three tensors: query, key, and value through separate fully-connected layers consisting of dv * nv output units and applying the relu activation function to get the modified query, key and value tensors, say q, k, and v , respectively. after adequate reshaping, the dimensions of q, k, and v should be (b, l, nv, dv). for each example i, the following computation is performed within the attention block using q, k, and v : 1. k i permute := k i .permute(0, 2, 1) 2. prob i := q i *k i permute 3. prob i scaled := prod i √ dv 4. weights i attn :=softmax(prod i scaled , @xmath0. rprod i := weights i attn *v i we further include a residual connection that acts as a memory to combine the initial set of framelevel features with rprod i . so, to compute the output of the residual attention block, we add rprod i to the original query tensor and pass it through a fully-connected layer consisting of d out neurons and apply relu . the remaining layers are similar in both fatnet-v1 and fatnet-v2. the details are as follows: the generated attentive-frames are batch normalized. then these are passed through a fully connected layer for fine-grained analysis. we further apply a leaky-relu activation function with l 2 -normalization. we aggregate these frame-level features by computing a mean and then pass them through a fully connected layer consisting of two units for speaker verification. ## discussion this work investigates the cost-effective idea of integrating the lightweight frame-attentive embedding with heavier and stronger baselines to mitigate the linguistic bias in such baselines without adaptation. after comprehensive experimentation on twelve languages and ablation studies, we observed that the proposed method showed significant and consistent improvements in reducing the linguistic bias in the baselines. some final considerations: model complexity: ## conclusions and future work in this paper, we introduced the cost-effective idea of utilizing a lightweight frame-level embedding for reducing linguistic bias in existing speaker verification systems without the overhead of domain adaptation. we also explored applying attention to individual frames to focus on relevant framelevel discriminative information. for an in-depth analysis of our proposed theoretical hypothesis, we proposed two variants of frame-attentive networks: fatnet-v1 and fatnet-v2. we investigated the effect of their integration with the baselines for twelve languages. empirical results showed consistent improvements in the performance of baselines on out-of-domain test sets without domain adaptation after integration with the fatnet embedding. qualitative comparison with baselines suggested that the proposed models are comparatively more generalizable. additionally, we did an ablation study of our proposed networks. it turns out that frame-level embedding captures less linguistic information from speech than utterance-level embedding. an intelligent selection of features from frames can further improve the performance of speaker verification models. our analysis points to vital problems for future work. for instance, it may be worthwhile to explore standalone domain-invariant architectures. this work focuses on the scenario where the input speech recordings within a trial pair are in the same language. however, it may be helpful to explore this idea further for bilingual speakers where the input speech recordings in the trial pair are in different languages.