luisotorres/hubert_gtzan

Description

This model is a specialized version of the distilhubert model fine-tuned on the gtzan dataset for the task of Music Genre Classification.

Development

• Kaggle Notebook: Audio Data: Music Genre Classification

Training Parameters

evaluation_strategy = 'epoch',
save_strategy = 'epoch',
load_best_model_at_end = True,
metric_for_best_model = 'accuracy',
learning_rate = 5e-5,
seed = 42,
per_device_train_batch_size = 8,
per_device_eval_batch_size = 8,
gradient_accumulation_steps = 1,
num_train_epochs = 15,
warmup_ratio = 0.1,
fp16 = True,
save_total_limit = 2,
report_to = 'none'

Training and Validation Results

Epoch	Training Loss	Validation Loss	Accuracy
1	      No log	        2.050576	0.395000
2	      No log	        1.387915	0.565000
3	      No log	        1.141497	0.665000
4	      No log	        1.052763	0.675000
5	      1.354600	        0.846402	0.745000
6	      1.354600	        0.858698	0.750000
7	      1.354600	        0.864531	0.730000
8	      1.354600	        0.765039	0.775000
9	      1.354600	        0.790847	0.785000
10	      0.250100	        0.873926	0.785000
11	      0.250100	        0.928275	0.770000
12	      0.250100	        0.851429	0.780000
13	      0.250100	        0.922214	0.770000
14	      0.250100	        0.916481	0.780000
15	      0.028000	        0.946075	0.770000
TrainOutput(global_step=1500, training_loss=0.5442592652638754,
metrics={'train_runtime': 12274.2966, 'train_samples_per_second': 0.976,
'train_steps_per_second': 0.122, 'total_flos': 8.177513845536e+17, 'train_loss': 0.5442592652638754, 'epoch': 15.0})

Reference

This model is based on the original HuBERT architecture, as detailed in:

Hsu et al. (2021). HuBERT: Self-Supervised Speech Representation Learning by Masked Prediction of Hidden Units. arXiv:2106.07447