CAARMA : Class Augmentation with Adversarial Mixup Regularization (2025)

Abstract: Speaker verification is a typical zero-shot learning task, where inference of unseen classes is performed by comparing embeddings of test instances to known examples. The models performing inference must hence naturally generate embeddings that cluster same-class instances compactly, while maintaining separation across classes. In order to learn to do so, they are typically trained on a large number of classes (speakers), often using specialized losses. However real-world speaker datasets often lack the class diversity needed to effectively learn this in a generalizable manner. We introduce CAARMA, a class augmentation framework that addresses this problem by generating synthetic classes through data mixing in the embedding space, expanding the number of training classes. To ensure the authenticity of the synthetic classes we adopt a novel adversarial refinement mechanism that minimizes categorical distinctions between synthetic and real classes. We evaluate CAARMA on multiple speaker verification tasks, as well as other representative zero-shot comparison-based speech analysis tasks and obtain consistent improvements: our framework demonstrates a significant improvement of 8% over all baseline models. Code for CAARMA will be released.

## introduction speaker verification is fundamentally a zero-shot learning (zsl) task, where verification is accomplished by comparing embeddings from enrollment and verification samples without the need for further training @xcite . this process aligns with the principles of zsl, where models are expected to operate effectively on unseen data. therefore, while the following discussion is framed within the broader context of zsl, it is specifically tailored to address the challenges in speaker verification. figure 1 : (a) when trained with fewer classes the model can spread the embeddings of individual classes out while still learning to classify the training data accurately and with large margins. this will not, however translate to compact representations for newer unseen classes. (b) with additional synthetic classes (shaded grey), the model must now learn to compact classes more. this will translate to more compact unseen classes as well. to address the challenge of limited class diversity in training datasets, a common issue in speaker verification, we propose a novel augmentationbased training paradigm. this approach leverages synthetic data augmentation to enhance the robustness and generalization capabilities of speaker verification systems, particularly in low-diversity environments. our method not only stays true to the essence of zsl by facilitating effective generalization to new speakers but also introduces a practical solution to overcome the inherent limitations of traditional training datasets. effective zero-shot learning lies in generating embeddings that cluster same-class (in our case, same-speaker) instances closely while maintaining separation between different classes @xcite . traditional training approaches for zsl models rely on two key components: exposure to a large number of diverse classes and the use of specialized loss functions that promote both inter-class separation and intra-class compactness @xcite . the underlying principle here is that by learning from a sufficiently large number of classes, and through proper encouragement embodied in the losses, the model learns not merely to separate the classes it has seen, but the more general principle that instances from a class must be clustered closely together while begin separated from those from other classes @xcite . however, when the training datasets lack the necessary variety of classes (speakers), this can severely limit the model's ability to develop robust and transferable representations @xcite . indeed, it may be argued that in the highdimensional space of the embeddings, any finite set of training classes is insufficient to cover the space adequately. this limitation leads to models that fail to generalize effectively to unseen categories, resulting in suboptimal zero-shot inference performance @xcite . popular approaches to address training data limitations often rely on data augmentation techniques. methods such as autoaugment @xcite and @xcite generate new samples by modifying existing ones through transformations like geometric distortions, time warping, and frequency masking. however, while these techniques increase intra-class diversity, they do not introduce new classes, limiting their effectiveness in zero-shot learning scenarios. of most relevance to our paper are mixup-based data augmentation techniques, e.g. @xcite , that aim to enhance training by generating new samples through interpolation @xcite of both the features and their labels. regardless of the interpolation, yet these methods also do not generate new classes; they merely improve the generalization of the model by mapping mixed data to mixed-class labels. still other methods use generative models such as variational autoencoders (vaes), generative adversarial networks (gans), diffusion models etc. to generate authentically novel data to enhance the training @xcite ; however these too are generally restricted to generating novel instances for known classes, limiting their effectiveness in zero-shot scenarios @xcite . thus, while these approaches are generally very successful in improving generalization in classification problems, they fail at addressing the problem zsl learning faces, that of increasing the number of classes themselves, leading to inconsistent general-ization to unseen classes @xcite . in this paper, we introduce class augmentation with adversarial mixup regulariaztion (caarma), a data augmentation framework to introduce synthetic classes (speakers) to enhance zsl training for speaker verification. caarma utilizes a mixup-like strategy to generate data from fictitious speakers. however, unlike conventional mixup which mixes data in the input space, which would arguably be meaningless in our setting (a straight-forward mix of two speech recordings will merely result in a mixed signal, and not a new speaker), the mixup is performed in the embedding space in a manner that permits assignment of new class identities to the mixed-up data. critically, we must now ensure that the mixed-up embeddings resemble those from actual speakers. we do so through a discriminator that is used to minimize categorical distinctions between synthetic and authentic data through adversarial training. we demonstrate caarma's effectiveness through extensive evaluation on speaker verification, where it achieves substantial improvements in generalizing to diverse speaker distributions. additional experiments on other zsl speech tasks further validate our approach's broad applicability. our main contributions are as follows: ## results & analysis to validate the effectiveness of our approach, we conduct comprehensive experiments across speaker verification, emotion classification, and speaker distance estimation tasks. our analysis demonstrates significant improvements through adversarial refinement on model generalization. ## conclusion in this work, we introduce caarma, a novel class augmentation framework designed to tackle the challenge of limited class diversity in zeroshot inference tasks. our approach synthesizes strategic data mixing with an adversarial refinement mechanism to align real and synthetic classes effectively within the embedding space. we validate caarma's effectiveness in speaker verification, achieving an 8% improvement over baseline models, and extend its application to emotion classification and speaker distance estimation, where it also shows significant gains. these results underscore caarma's capability to enhance embedding structures in various zero-shot inference scenarios. our framework offers a scalable solution to the class diversity problem, facilitating integration into existing systems without the need for new real-world data collection. with our code released publicly, we anticipate that caarma will aid both research and practical applications in zero-shot learning. future work will focus on expanding caarma's utility to larger datasets and other domains, such as computer vision.