utter-project

/

mHuBERT-147

like 46

Hi! Thanks for the interest. Please note this checkpoint is not an ASR model: this is a speech representation model.
In order to use it as you want, you need to first fine-tune it using ASR data. You can check huggingface tutorials for this, such as this one: https://huggingface.co/blog/fine-tune-xlsr-wav2vec2 (just replace the wav2vec2 classes by hubert).
Best,

Hi! Thanks for the interest. Please note this checkpoint is not an ASR model: this is a speech representation model.
In order to use it as you want, you need to first fine-tune it using ASR data. You can check huggingface tutorials for this, such as this one: https://huggingface.co/blog/fine-tune-xlsr-wav2vec2 (just replace the wav2vec2 classes by hubert).
Best,

Hi, Thank you for your quick response!
I know that it is a speech representation model. What I am doing here is to extract audio feature. I would use this feature in a downstream task. What I would like to do is extract its feature and build a text to speech system with the feature it extract. I think during the training, the mHubert-147 parameters are freezed. In this case, am I doing the right step to extract the feature of an audio?

Best

Hi again,

I see! Sorry for the confusion.
Feature extraction should work as for any other HuBERT model.

mHuBERT-147 is a base architecture, so your output features will be of dimension 768.
If you face any issues, let me know. :)

@mzboito It'd be really helpful if you can provide a colab for the ASR training. I tried to follow the tutorial (https://huggingface.co/blog/fine-tune-xlsr-wav2vec2) but I got eval_wer=1.0 for hours of training. I was able to successfully train my dataset with wav2vec2-xls-r-300m. Thanks!

tokenizer = Wav2Vec2CTCTokenizer(
  "tokenizer/vocab.json", unk_token="[UNK]", pad_token="[PAD]", word_delimiter_token="|"
)

model_id = "utter-project/mHuBERT-147"

feature_extractor = Wav2Vec2FeatureExtractor.from_pretrained(model_id)
processor = Wav2Vec2Processor(feature_extractor=feature_extractor, tokenizer=tokenizer)

model = HubertForCTC.from_pretrained(
  model_id,
  pad_token_id=processor.tokenizer.pad_token_id,
  vocab_size=len(processor.tokenizer),
)
)
model.freeze_feature_encoder()
data_collator = DataCollatorCTCWithPadding(processor=processor, padding=True)


def prepare_dataset(batch):
  audio = batch["audio"]

  # batched output is "un-batched"
  batch["input_values"] = processor(
    audio["array"], sampling_rate=audio["sampling_rate"]
  ).input_values[0]
  batch["input_length"] = len(batch["input_values"])

  with processor.as_target_processor():
    batch["labels"] = processor(batch["transcription"]).input_ids
  return batch


# ...

training_args = TrainingArguments(
  output_dir="training",
  report_to="tensorboard",
  per_device_train_batch_size=32,
  per_device_eval_batch_size=8,
  eval_strategy="steps",
  num_train_epochs=50,
  fp16=True,
  save_steps=400,
  eval_steps=400,
  logging_steps=100,
  learning_rate=3e-4,
  warmup_steps=500,
  save_total_limit=5,
  push_to_hub=False,
  load_best_model_at_end=True,
)

Hi. Which dataset/language are you fine-tuning on?
I recommend you tune your hyper-parameters for mHuBERT, specially because XLS-R is much larger.
If you are using the same datasets we use in the paper, you can check our hyper-parameters there (https://arxiv.org/abs/2406.06371).

In my experience, the optimal training hyper-parameters tend to vary quite a lot across different SSL backbones.
Modifying things like learning rate, warm-up ratio, and increasing dropout (if dataset is small) will help you better converge.

@mzboito I used OpenSLR 42 dataset (https://www.openslr.org/42/) or (https://huggingface.co/datasets/openslr/openslr). I honestly don't know what to do next. The train loss and eval loss doesn't seem to go down.

Hyper-parameters are very important. They can be the difference between a good ASR model and 100% WER/no convergence.
It's not because an ASR fine-tuning recipe with given hyper-parameters work for a given SSL backbone, that they will automatically work on a different model. On my previous message I mentioned important hyper-parameters you could explore for better convergence.

I unfortunately do not have the time to provide you with a recipe for this particular dataset, but I did train few-shot ASR models on Khmer using fleurs-102.
The hyper-parameters are very probably not optimal for Khmer (I optimized them using other languages), but they worked. They were:

evaluation_strategy: "steps"
num_train_epochs: 100
fp16: False
gradient_checkpointing: True
eval_steps: 500
save_steps: 500
logging_steps: 500
learning_rate: 1e-5
adam_beta1: 0.9
adam_beta2: 0.98
adam_epsilon: 1e-08
warmup_ratio: 0.1
save_total_limit: 2
per_device_train_batch_size: 2
per_device_eval_batch_size: 2 
push_to_hub: False
load_best_model_at_end: True
metric_for_best_model: "loss"
greater_is_better: False
}

Moreover, for those experiments, the mHuBERT model.final_dropout was set to 0.3.

@mzboito Thank you! I will try to train again with the provided parameters.

This is my colab fine tuning mHuBERT-147 OpenSLR 42
https://colab.research.google.com/drive/1m9Wdbnv8S7G4UzIu1QDYho6bCyPFeJWP?usp=sharing

After 10 hours of training, I managed to get to 44.02 WER and 0.57 eval loss. I think the previous issue was because I use fp16 instead of fp32.

After 10 hours of training, I managed to get to 44.02 WER and 0.57 eval loss. I think the previous issue was because I use fp16 instead of fp32.

Hi, in comparison, what is the result of xls-r?

After 10 hours of training, I managed to get to 44.02 WER and 0.57 eval loss. I think the previous issue was because I use fp16 instead of fp32.

Hi, in comparison, what is the result of xls-r?

Still in training. Currently, 27% WER. xls-r got around ~16% WER