E co S peak: Cost-Efficient Bias Mitigation for Partially Cross-Lingual Speaker Verification (2024)

Abstract: Linguistic bias is a critical problem concerning the diversity, equity, and inclusiveness of Natural Language Processing tools. The severity of this problem intensifies in security systems, such as speaker verification, where fairness is paramount. Speaker verification systems are biometric systems that determine whether two speech recordings are of the same speaker. Such user-centric systems should be inclusive to bilingual speakers. However, Deep neural network models are linguistically biased. Linguistic bias can be full or partial. Partially cross-lingual bias occurs when one test trial pair recording is in the training set’s language, and the other is in an unseen target language. Such linguistic mismatch influences the speaker verification model’s decision, dissuading bilingual speakers from using the system. Domain adaptation can mitigate this problem. However, adapting to each existing language is expensive. This paper explores cost-efficient bias mitigation techniques for partially cross-lingual speaker verification. We study the behavior of five baselines in five partially cross-lingual scenarios. Using our baseline behavioral insights, we propose EcoSpeak, a low-cost solution to partially cross-lingual speaker verification. EcoSpeak incorporates contrastive linguistic (CL) attention. CL attention utilizes linguistic differences in trial pairs to emphasize relevant speaker verification embedding parts. Experimental results demonstrate EcoSpeak’s robustness to partially cross-lingual testing.

## introduction linguistic bias is a crucial problem that harms the diversity, equity, and inclusiveness of natural language processing (nlp) tools. the severity of this problem further increases in security systems, such as speaker verification, where fairness is critical. speaker verification systems are biometric systems that determine whether two speech recordings are of the same speaker. the two input speech recordings form a trial pair. positive or negative trial pairs indicate whether the recordings are of the same speaker. speaker verification systems have applications in forensics, e-commerce, law, and accesscontrol mechanisms @xcite . these systems can be text-independent or textdependent @xcite ). text-independent systems verify speakers without any constraint on speech content. such systems work by analyzing the acoustic differences in trial pairs, consequently saving users from memorizing passphrases. therefore, text-independent systems offer a better user experience than text-dependent systems. deep neural network (dnn) models have shown outstanding results in text-independent speaker verification @xcite @xcite . however, the embeddings obtained from dnn models often entangle acoustic and linguistic information @xcite . consequently, dnn-based speaker verification models become linguistically biased @xcite . linguistic bias makes the model consider irrelevant language information in embeddings while making decisions for speaker verification, leading to performance degradation on unseen target languages. such a bias can be full or partial. in the fully cross-lingual scenario, both the test trial pair recordings are in the target language t that is different from the source (or the training set) language s. in contrast, partially cross-lingual is another crucial scenario where one of the test trial pair recordings is in s, and the other is in t most of the previous works focus on the fully cross-lingual scenario. however, about 40% of the global population is bilingual @xcite . therefore, addressing the partially cross-lingual challenge is essential to enhance the usability of speaker verification models. a viable solution to this problem is domain adaptation @xcite @xcite @xcite @xcite . however, adapting the model to each of the 7,000 existing languages is expensive. alternatively, we can train models on large-scale crosslingual datasets to enhance their generalizability to unseen languages @xcite . however, this approach incurs enormous computational and storage costs. high computational cost leads to high carbon emissions, which impacts the environment @xcite . in this work, we investigate cost-efficient techniques to mitigate partially cross-lingual bias in text-independent speaker verification. we propose ecospeak, a low-cost solution to mitigate partially cross-lingual bias. the ecospeak architecture incorporates a lightweight residual network, novel contrastive linguistic (cl) attention, and the bias corrector. here, residual connections allow the model to emphasize the low-level acoustic features essential for speaker verification. our proposed cl attention mechanism utilizes linguistic differences in trial pair recordings to generate attention weights for speaker verification. the bias corrector module penalizes the speaker verification probabilities based on linguistic differences in trial pair recordings. we first study the behavior of five baselines on five partially cross-lingual test sets created using speech recordings in four low-resource languages. subsequently, we investigate the effectiveness of ecospeak on these test sets without domain adaptation. furthermore, we explore low-cost fine-tuning techniques to enhance the generalizability of ecospeak for unseen low-resource languages. we summarize our main contributions below: ## proposed approach the remarkable success of the deep convolutional neural network (cnn) based speaker recognition models motivates us to investigate linguistic bias in these models. these models consist of multiple cnn layers @xcite @xcite . lower layers capture low-level speech features, whereas higher layers capture highlevel speech features. low-level speech features contain mostly acoustic information, essential for speaker verification @xcite . on the other hand, high-level speech features contain more linguistic information @xcite . therefore, deeper models can learn more linguistic details during training and become biased. thus, to mitigate linguistic bias in speaker verification, we propose ecospeak. this section describes the architectural details of ecospeak. hypothesis: we know that residual connections add lower layer output features to the higher layer output features in a deep cnn model @xcite . consequently, low-level acoustic features get added to the higher-level advanced representations. this summation allows the model to emphasize the low-level acoustic features essential for speaker verification. accordingly, we hypothesize that residual connections help mitigate linguistic bias by making the model focus more on low-level acoustic information. input: firstly, we preprocess the trial pair recordings to crop silent parts. we then compute 64-dimensional normalized log mel spectrogram features of shape (b, 1, t i , m) as shown in figure 1 . here, b denotes the batch size, 1 indicates monochannel audio, t i denotes the time steps, and m is the number of mel bands (@xmath0. since the duration of input speech recordings may vary during test time, the values of t 1 and t 2 may differ. next, we pass these features through the resnet (lite) and the s-detect model. resnet (lite): resnet (lite) is a lighter variant of the resnet-34 @xcite . 1 the model is pre-trained for speaker identification. speaker identification is a multi-class classification problem where the system accepts a speech recording as input and determines the speaker's identity from the known speakers in the training set. we use the pre-trained resnet (lite) in ecospeak to extract d-dimensional speaker embeddings as shown in figure 1 . first, we get emb 1 and emb 2 for the trial pair recordings from the avgpool layer of resnet (lite). 2 next, we compute the absolute difference between these embeddings: computing the difference of the trial pair embeddings enables ecospeak to focus on the discriminatory information for speaker verification. furthermore, computing absolute difference ensures that the model's output is unaffected by the input order, as absolute difference is a commutative operation. contrastive linguistic (cl) attention: recent works have demonstrated the effectiveness of attention in speaker verification @xcite . we propose the contrastive linguistic (cl) attention mechanism for partially cross-lingual speaker verification. cl attention utilizes the linguistic differences between the trial pair recordings to generate attention weights. the attention block receives x as input and lid 1 and lid 2 as prompt inputs. cl attention works as follows: here w and b linear are the weights and biases of the fully-connected layer. here θ 1 is a learned parameter. bias corrector: to perform speaker verification, we feed x ′ through fully connected layers, as shown in figure 1 . we pass the resulting speaker verification probabilities to the bias corrector. the bias correction process involves two steps: language verification and bias adjustment. ## experimental setup the datasets 3 and baseline models 4 used in this study are publicly available. all sets have an equal number of positive and negative trial pairs. in our experiments, english is the source language s, whereas tamil, telugu, malayalam, and kannada are the low-resource target languages t. ## conclusions and future work this paper investigates the behavior of five baseline speaker verification models on five partially crosslingual test sets. empirical results demonstrate that a high linguistic similarity in negative trial pair recordings and a low linguistic similarity in positive trial pair recordings causes performance degradation. furthermore, residual networks are relatively robust to cross-lingual testing. using these insights, we proposed ecospeak, a low-cost solution to mitigate bias in partially cross-lingual speaker verification. ecospeak incorporates residual connections, contrastive linguistic attention, and the bias corrector. empirical results demonstrate the robustness of our proposed model on partially cross-lingual test sets. ecospeak's performance trend deviates from the baselines. it turns out that utilizing linguistic differences to emphasize and de-emphasize relevant speaker verification embedding parts can mitigate partially cross-lingual bias. our insights can contribute to the development of more robust domain-invariant architectures. furthermore, this work encourages the community to explore greener approaches to expensive speech processing. for instance, based on our empirical results, we recommend leveraging diverse datasets in a weakly related language for bias mitigation in an unseen low-resource target language. additionally, our proposed data balancing schemes can save the cost of training on large-scale datasets. we also recommend a detailed cost analysis to develop environment-friendly and inclusive models.