Whisper Large FineTuning with 8 16 GB GPUs, running into OOM error
Hi, I was wondering if anyone has tried to fine tune the whisper large v2 and was facing an OOM error. I would appreciate some help on this issue.
I have 8 GPUs 16 GB each and I am fine tuning whisper large using deep speed, yet I am facing OOM error. Here is what my training args look like :
training_args = Seq2SeqTrainingArguments(
output_dir = "/home/ec2-user/SageMaker/whisper-large-finetuned-1.5M-ATM-CS-100-1000-8000Steps",
per_device_train_batch_size = 2, # if you get cuda out-of-memory issue, try to decrease batch size by 2x
gradient_accumulation_steps = 4 , # increase by 2x for every 2x decrease in batch size
learning_rate = 1e-5,
warmup_steps = 10,
max_steps = 4000,
gradient_checkpointing = True,
fp16 = True,
evaluation_strategy = "steps",
per_device_eval_batch_size = 2,
eval_accumulation_steps = 2,
predict_with_generate = True,
generation_max_length = 225,
save_steps = 1000,
eval_steps = 1000,
logging_steps = 50,
load_best_model_at_end = True,
metric_for_best_model = "wer",
greater_is_better = False,
push_to_hub = False,
optim="adafactor",
deepspeed="/home/ec2-user/SageMaker/packages/WhisperFinetune/src/whisper_fine_tune/ds_config.json"
)
Serious firepower @nehacho ! What's your DS config looking like? And are you launching with multiple CUDA devices as per this guide?
Hey, I could make it run at last. Just tweaked the DS config. Thank you for the response though. Appreciate it!
