GenAID

GenAID is a Chinese accent encoder based on facebook/wav2vec2-large-xlsr-53. It produces a 64-dimensional accent embedding designed to reduce speaker information. The model recognizes nine labels: north, Sichuan, Guangdong, south, Henan, Shanghai, Wuhan, Tianjin, and Singapore.

GenAID and WhisAID Medium side by side

The comparison excludes Whisper's text decoder because accent identification uses only its audio encoder.

Component	GenAID (this model)	WhisAID Medium
Model-native input	Raw 16 kHz waveform [B, T]	80-bin Whisper log-Mel [B, 80, 3000]
Backbone	Wav2Vec2 XLSR-53 Large	Whisper Medium audio encoder
Audio frontend	7 convolution layers, combined stride 320	2 convolution layers, second layer stride 2
Frontend output	[B, T′, 512]	[B, 1500, 1024]
Feature projection	512 → 1024	None after the CNN
Transformer blocks	24	24
Transformer width	1024	1024
Attention heads	16 × 64 dimensions	16 × 64 dimensions
FFN per block	1024 → 4096 → 1024	1024 → 4096 → 1024
Transformer output	[B, T′, 1024]	[B, 1500, 1024]
Pooling	Attention-mask-aware temporal mean	Temporal mean
Accent embedding	1024 → 64 → 64	1024 → 256
Accent output	64 → 9	256 → 9
Speaker disentanglement	64 → 336; uniform-distribution adversarial MSE	GRL + speaker-token attention, 256 → 336
Task-relevant parameters	315,530,560	306,473,897, excluding text decoder
Frozen parameters	4,210,176 (7-layer XLSR CNN)	Audio encoder is evaluated under torch.no_grad() in the compared recipe
Effectively updated parameters	311,320,384	444,249

T′ is the downsampled XLSR sequence length. For a 10-second, 16 kHz waveform, T=160,000 and T′≈499.

GenAID module details

Module	Layers	Input → output	Parameters	Training behavior
XLSR feature extractor	7 CNN layers	[B,T] → [B,T′,512]	4,210,176	Frozen
Feature projection + XLSR encoder	Projection + 24 Transformer blocks	[B,T′,512] → [B,T′,1024]	311,228,544	Updated
Masked mean pooling	1	[B,T′,1024] → [B,1024]	0	No parameters
Information bottleneck	2 × Linear + GELU	1024 → 64 → 64	69,760	Updated
Accent classifier	Linear, no bias	64 → 9	576	Updated
Speaker adversarial classifier	Linear, no bias	64 → 336	21,504	Updated during training
Total			315,530,560

The 64-dimensional output of the second GELU is the reusable accent embedding. The speaker classifier is a training-only probe: its predicted speaker distribution is optimized toward a uniform distribution, discouraging the bottleneck from retaining speaker identity. The speaker branch is not required when extracting embeddings.

Usage

import librosa
import torch
from transformers import AutoFeatureExtractor, AutoModel

repo = "walston/GenAID"
processor = AutoFeatureExtractor.from_pretrained(repo)
model = AutoModel.from_pretrained(repo, trust_remote_code=True).cuda().eval()

wav, _ = librosa.load("audio.wav", sr=16000, mono=True)
inputs = processor(wav, sampling_rate=16000, return_tensors="pt")
inputs = {key: value.cuda() for key, value in inputs.items()}

with torch.inference_mode():
    output = model(**inputs)

accent_embedding = output.embedding       # [batch, 64]
accent_logits = output.accent_logits       # [batch, 9]

Loading this repository requires trust_remote_code=True because it includes the small GenAID bottleneck and classification heads around the standard XLS-R encoder.