wav2vec2-large-mms-1b-DZ

This model is a fine-tuned version of facebook/mms-1b-all on an unknown dataset. It achieves the following results on the evaluation set:

• Loss: 0.1983
• Wer: 0.1456
• Bleu: {'bleu': 0.7492044741492486, 'precisions': [0.8730063030967389, 0.7883760255527025, 0.7152661166678835, 0.6510430304617264], 'brevity_penalty': 0.9957339815437508, 'length_ratio': 0.995743055176554, 'translation_length': 18245, 'reference_length': 18323}
• Rouge: {'rouge1': 0.8748364322650914, 'rouge2': 0.7928723605220098, 'rougeL': 0.8745015805780032, 'rougeLsum': 0.8743113489400196}

Model description

More information needed

Intended uses & limitations

More information needed

Training and evaluation data

More information needed

Training procedure

Training hyperparameters

The following hyperparameters were used during training:

• learning_rate: 0.0001
• train_batch_size: 8
• eval_batch_size: 16
• seed: 42
• gradient_accumulation_steps: 4
• total_train_batch_size: 32
• optimizer: Use OptimizerNames.ADAMW_TORCH with betas=(0.9,0.999) and epsilon=1e-08 and optimizer_args=No additional optimizer arguments
• lr_scheduler_type: linear
• lr_scheduler_warmup_steps: 500
• num_epochs: 40
• mixed_precision_training: Native AMP

Training results

Training Loss	Epoch	Step	Validation Loss	Wer	Bleu	Rouge
5.9588	1.0	285	2.7873	0.9827	{'bleu': 0.0, 'precisions': [0.059154929577464786, 0.0, 0.0, 0.0], 'brevity_penalty': 0.11433624320676729, 'length_ratio': 0.3155956311912624, 'translation_length': 2485, 'reference_length': 7874}	{'rouge1': 0.03875997597517414, 'rouge2': 0.0, 'rougeL': 0.03892048827301332, 'rougeLsum': 0.038793322006122906}
0.4602	2.0	570	0.3555	0.2311	{'bleu': 0.6283054199052752, 'precisions': [0.7993283417749394, 0.6805363212891854, 0.5854142185663925, 0.5068155835018908], 'brevity_penalty': 0.9912846225149247, 'length_ratio': 0.9913223817060525, 'translation_length': 18164, 'reference_length': 18323}	{'rouge1': 0.7917720369498722, 'rouge2': 0.6755927340957304, 'rougeL': 0.7912377266650397, 'rougeLsum': 0.7913035186420103}
0.3842	3.0	855	0.3058	0.2090	{'bleu': 0.6605337548855493, 'precisions': [0.8175150519978106, 0.7074787393696849, 0.616208731142045, 0.5403591352859135], 'brevity_penalty': 0.99710327314591, 'length_ratio': 0.9971074605686842, 'translation_length': 18270, 'reference_length': 18323}	{'rouge1': 0.8146357732958056, 'rouge2': 0.7071194587560558, 'rougeL': 0.814052758845498, 'rougeLsum': 0.8138241815295024}
0.3719	4.0	1140	0.2868	0.2010	{'bleu': 0.6712219662583173, 'precisions': [0.8250068549492734, 0.7188067932568779, 0.6293752283522105, 0.5544580419580419], 'brevity_penalty': 0.9951857415819062, 'length_ratio': 0.9951972930197021, 'translation_length': 18235, 'reference_length': 18323}	{'rouge1': 0.8222029713967768, 'rouge2': 0.7197657000623603, 'rougeL': 0.8217923739918656, 'rougeLsum': 0.821719465931791}
0.3336	5.0	1425	0.2742	0.1959	{'bleu': 0.6785372107003715, 'precisions': [0.8300621254604431, 0.7259128904531457, 0.6393695014662757, 0.5666900666900667], 'brevity_penalty': 0.9926599804903764, 'length_ratio': 0.9926867870981826, 'translation_length': 18189, 'reference_length': 18323}	{'rouge1': 0.8265645999560629, 'rouge2': 0.7247965384723267, 'rougeL': 0.8259582425920233, 'rougeLsum': 0.8259017198295564}
0.3442	6.0	1710	0.2642	0.1903	{'bleu': 0.6873788245362181, 'precisions': [0.832968476357268, 0.7314617981743153, 0.6461886022445708, 0.5733948950256991], 'brevity_penalty': 0.9972127357870568, 'length_ratio': 0.9972166130000546, 'translation_length': 18272, 'reference_length': 18323}	{'rouge1': 0.8325100938158259, 'rouge2': 0.7340864170622121, 'rougeL': 0.83191579042451, 'rougeLsum': 0.8318541463934617}
0.3341	7.0	1995	0.2582	0.1843	{'bleu': 0.6963666431657629, 'precisions': [0.839673317254988, 0.7409495177251659, 0.6574662285505659, 0.5849270806043141], 'brevity_penalty': 0.9956791710107048, 'length_ratio': 0.9956884789608689, 'translation_length': 18244, 'reference_length': 18323}	{'rouge1': 0.838537901118194, 'rouge2': 0.7427526814996277, 'rougeL': 0.8379557459558569, 'rougeLsum': 0.8379007055417265}
0.3172	8.0	2280	0.2494	0.1804	{'bleu': 0.7001144932835086, 'precisions': [0.843205001645278, 0.7451742291301078, 0.6613327487943884, 0.5895822408669813], 'brevity_penalty': 0.9951309011278089, 'length_ratio': 0.9951427168040168, 'translation_length': 18234, 'reference_length': 18323}	{'rouge1': 0.8428458068475617, 'rouge2': 0.7473487117700717, 'rougeL': 0.8422200154587571, 'rougeLsum': 0.8420994714247132}
0.294	9.0	2565	0.2444	0.1784	{'bleu': 0.7015679264162828, 'precisions': [0.8458917284968398, 0.7474398441917447, 0.6644683813292299, 0.5931088900578643], 'brevity_penalty': 0.9929897872850882, 'length_ratio': 0.9930142443922938, 'translation_length': 18195, 'reference_length': 18323}	{'rouge1': 0.8447332085359476, 'rouge2': 0.7490673320409132, 'rougeL': 0.8441531451631874, 'rougeLsum': 0.843987486458086}
0.2927	10.0	2850	0.2395	0.1734	{'bleu': 0.7094676103718882, 'precisions': [0.8488569705608245, 0.7529912923635909, 0.6714379610019718, 0.6010480349344978], 'brevity_penalty': 0.9955147214626567, 'length_ratio': 0.9955247503138133, 'translation_length': 18241, 'reference_length': 18323}	{'rouge1': 0.8488302350839227, 'rouge2': 0.7554473763418516, 'rougeL': 0.848226337259566, 'rougeLsum': 0.8481517731134592}
0.2974	11.0	3135	0.2361	0.1757	{'bleu': 0.7073575226963291, 'precisions': [0.8489256198347107, 0.7540322580645161, 0.6735038964858109, 0.6032737833318665], 'brevity_penalty': 0.9905136020383278, 'length_ratio': 0.9905583146864596, 'translation_length': 18150, 'reference_length': 18323}	{'rouge1': 0.8469188343317979, 'rouge2': 0.753826185237243, 'rougeL': 0.8462106498895938, 'rougeLsum': 0.8462690600047347}
0.2929	12.0	3420	0.2323	0.1710	{'bleu': 0.7104887814499381, 'precisions': [0.8529363110008271, 0.7578987198082866, 0.6768970339294914, 0.6069980609906575], 'brevity_penalty': 0.9896868547353578, 'length_ratio': 0.9897396714511816, 'translation_length': 18135, 'reference_length': 18323}	{'rouge1': 0.851535800343566, 'rouge2': 0.7594371464363054, 'rougeL': 0.8507125679902645, 'rougeLsum': 0.8505546552290876}
0.2922	13.0	3705	0.2277	0.1690	{'bleu': 0.7147023714998528, 'precisions': [0.8544513001322168, 0.7611188106337408, 0.6811967068509261, 0.6115804294262583], 'brevity_penalty': 0.9906237838889417, 'length_ratio': 0.99066746711783, 'translation_length': 18152, 'reference_length': 18323}	{'rouge1': 0.8532183741925291, 'rouge2': 0.7629739149453718, 'rougeL': 0.8526967418105296, 'rougeLsum': 0.8525344290554377}
0.2723	14.0	3990	0.2255	0.1665	{'bleu': 0.7185286342228578, 'precisions': [0.8572294801830108, 0.7651768265775705, 0.6859412933127345, 0.6166784078901022], 'brevity_penalty': 0.9900176348680043, 'length_ratio': 0.9900671287452928, 'translation_length': 18141, 'reference_length': 18323}	{'rouge1': 0.8554844372155834, 'rouge2': 0.7665585766259185, 'rougeL': 0.8550678894072193, 'rougeLsum': 0.8549249072117735}
0.2618	15.0	4275	0.2216	0.1647	{'bleu': 0.7205920025415866, 'precisions': [0.8575904144223371, 0.7653304850464941, 0.6855488787923201, 0.6164635749978084], 'brevity_penalty': 0.9929348269851479, 'length_ratio': 0.9929596681766086, 'translation_length': 18194, 'reference_length': 18323}	{'rouge1': 0.8572991962911986, 'rouge2': 0.7687939439030189, 'rougeL': 0.8569135698404339, 'rougeLsum': 0.8567069159809071}
0.2824	16.0	4560	0.2194	0.1625	{'bleu': 0.7248245517316763, 'precisions': [0.859585463741822, 0.7690277166739992, 0.6911516750971336, 0.6221929824561403], 'brevity_penalty': 0.9926599804903764, 'length_ratio': 0.9926867870981826, 'translation_length': 18189, 'reference_length': 18323}	{'rouge1': 0.860740052677103, 'rouge2': 0.7738344427906813, 'rougeL': 0.8602549832284943, 'rougeLsum': 0.8601224055557424}
0.2584	17.0	4845	0.2184	0.1624	{'bleu': 0.7233957743502292, 'precisions': [0.8591843464878531, 0.7673410404624278, 0.689066393082222, 0.6201332865661171], 'brevity_penalty': 0.9929348269851479, 'length_ratio': 0.9929596681766086, 'translation_length': 18194, 'reference_length': 18323}	{'rouge1': 0.8596520621812659, 'rouge2': 0.7716232895345339, 'rougeL': 0.8592810276384565, 'rougeLsum': 0.85911704018123}
0.2626	18.0	5130	0.2162	0.1575	{'bleu': 0.7321328546559496, 'precisions': [0.8624205218153914, 0.7733308280095202, 0.6966267523364486, 0.6292125021826436], 'brevity_penalty': 0.9956791710107048, 'length_ratio': 0.9956884789608689, 'translation_length': 18244, 'reference_length': 18323}	{'rouge1': 0.8630838420026828, 'rouge2': 0.7773667756363078, 'rougeL': 0.8628126973557786, 'rougeLsum': 0.8627419660979692}
0.2518	19.0	5415	0.2132	0.1577	{'bleu': 0.7305988372304458, 'precisions': [0.8624547548535703, 0.7728127350213086, 0.6953821423352331, 0.6268461067901774], 'brevity_penalty': 0.9951309011278089, 'length_ratio': 0.9951427168040168, 'translation_length': 18234, 'reference_length': 18323}	{'rouge1': 0.8638253584058928, 'rouge2': 0.7773261199114829, 'rougeL': 0.8632427380502744, 'rougeLsum': 0.8631565081641605}
0.2624	20.0	5700	0.2114	0.1561	{'bleu': 0.7336678313741997, 'precisions': [0.8624801706690006, 0.7734799725051553, 0.6966431224058837, 0.6291880602210426], 'brevity_penalty': 0.9977051698422599, 'length_ratio': 0.9977077989412214, 'translation_length': 18281, 'reference_length': 18323}	{'rouge1': 0.8643711864157274, 'rouge2': 0.7789576656096648, 'rougeL': 0.864045843841154, 'rougeLsum': 0.8639432546192249}
0.274	21.0	5985	0.2125	0.1572	{'bleu': 0.7327413298256028, 'precisions': [0.8631249656989188, 0.7743210186288654, 0.6986762232136327, 0.6313303594856993], 'brevity_penalty': 0.9944177028031576, 'length_ratio': 0.9944332260001092, 'translation_length': 18221, 'reference_length': 18323}	{'rouge1': 0.8637561560608497, 'rouge2': 0.7775626243197682, 'rougeL': 0.8633528770604967, 'rougeLsum': 0.8632085396165139}
0.2484	22.0	6270	0.2091	0.1552	{'bleu': 0.7357839881765593, 'precisions': [0.8648885962023927, 0.7774711490215755, 0.7019160450489981, 0.634892872759073], 'brevity_penalty': 0.9944725821787992, 'length_ratio': 0.9944878022157944, 'translation_length': 18222, 'reference_length': 18323}	{'rouge1': 0.8659727708387619, 'rouge2': 0.7816164515395694, 'rougeL': 0.8657951715895931, 'rougeLsum': 0.8655840959367318}
0.2473	23.0	6555	0.2090	0.1554	{'bleu': 0.7356730614040227, 'precisions': [0.8642543859649123, 0.7763438165643403, 0.7012123867952089, 0.6340143181421337], 'brevity_penalty': 0.9954598989631315, 'length_ratio': 0.995470174098128, 'translation_length': 18240, 'reference_length': 18323}	{'rouge1': 0.865456270958754, 'rouge2': 0.7804951440612873, 'rougeL': 0.8652647943705735, 'rougeLsum': 0.8649670956520807}
0.2418	24.0	6840	0.2055	0.1553	{'bleu': 0.7347926831245415, 'precisions': [0.8653296703296703, 0.7771008667252858, 0.7016554351010841, 0.634770417104802], 'brevity_penalty': 0.9932645437985937, 'length_ratio': 0.9932871254707198, 'translation_length': 18200, 'reference_length': 18323}	{'rouge1': 0.8654752140097928, 'rouge2': 0.7807218611868961, 'rougeL': 0.8652326088546789, 'rougeLsum': 0.8650739915057691}
0.2577	25.0	7125	0.2040	0.1515	{'bleu': 0.7395907984479824, 'precisions': [0.8676938260774207, 0.7802005012531328, 0.7051132213294375, 0.6386026200873363], 'brevity_penalty': 0.9953502449878465, 'length_ratio': 0.9953610216667577, 'translation_length': 18238, 'reference_length': 18323}	{'rouge1': 0.8685153389774867, 'rouge2': 0.7838568977155047, 'rougeL': 0.8683321242058704, 'rougeLsum': 0.8680531173155077}
0.2478	26.0	7410	0.2035	0.1523	{'bleu': 0.7389574395410246, 'precisions': [0.8673581385138843, 0.7797917711991972, 0.7048413046657891, 0.639496239286339], 'brevity_penalty': 0.9944725821787992, 'length_ratio': 0.9944878022157944, 'translation_length': 18222, 'reference_length': 18323}	{'rouge1': 0.8684026564775361, 'rouge2': 0.7837147217327451, 'rougeL': 0.8680473992168228, 'rougeLsum': 0.86781442567829}
0.2345	27.0	7695	0.2036	0.1525	{'bleu': 0.7394120868432645, 'precisions': [0.8659697898423818, 0.7782293360010004, 0.7032206353832702, 0.6378124183575721], 'brevity_penalty': 0.9972127357870568, 'length_ratio': 0.9972166130000546, 'translation_length': 18272, 'reference_length': 18323}	{'rouge1': 0.8680049443409963, 'rouge2': 0.7830505254725744, 'rougeL': 0.8677384137559521, 'rougeLsum': 0.8675358853961506}
0.2398	28.0	7980	0.2025	0.1513	{'bleu': 0.7410617179914926, 'precisions': [0.8681987713909609, 0.7813087626927416, 0.7069570301081555, 0.6415836392239119], 'brevity_penalty': 0.9950212112404683, 'length_ratio': 0.9950335643726465, 'translation_length': 18232, 'reference_length': 18323}	{'rouge1': 0.8691516758267378, 'rouge2': 0.7855889417660622, 'rougeL': 0.8688010484175208, 'rougeLsum': 0.8685495828117757}
0.2595	29.0	8265	0.2005	0.1506	{'bleu': 0.7408702837830967, 'precisions': [0.8690567695179532, 0.7818775100401606, 0.7071564466559345, 0.6421881838074398], 'brevity_penalty': 0.9940334633019136, 'length_ratio': 0.9940511924903127, 'translation_length': 18214, 'reference_length': 18323}	{'rouge1': 0.8705678413238229, 'rouge2': 0.7865174894976992, 'rougeL': 0.8703173243712404, 'rougeLsum': 0.8700254528394962}
0.224	30.0	8550	0.2009	0.1487	{'bleu': 0.7450849356742056, 'precisions': [0.8704333516182118, 0.7844784353059178, 0.7112995176143838, 0.6476140534871526], 'brevity_penalty': 0.9949115093798545, 'length_ratio': 0.994924411941276, 'translation_length': 18230, 'reference_length': 18323}	{'rouge1': 0.8713511668656566, 'rouge2': 0.7878011114559871, 'rougeL': 0.8710130200505186, 'rougeLsum': 0.8707877973869348}
0.2218	31.0	8835	0.2005	0.1492	{'bleu': 0.7446394749315683, 'precisions': [0.8707180500658761, 0.7851047810264776, 0.7116622768510389, 0.6470073503675183], 'brevity_penalty': 0.9941432609952691, 'length_ratio': 0.9941603449216831, 'translation_length': 18216, 'reference_length': 18323}	{'rouge1': 0.871600980620826, 'rouge2': 0.7896333858185615, 'rougeL': 0.8713249999389157, 'rougeLsum': 0.8710035822779216}
0.2249	32.0	9120	0.2002	0.1496	{'bleu': 0.7428408162780507, 'precisions': [0.8707584007039543, 0.7850361521534108, 0.7109644297763109, 0.646134947793279], 'brevity_penalty': 0.9923300656862052, 'length_ratio': 0.9923593298040714, 'translation_length': 18183, 'reference_length': 18323}	{'rouge1': 0.871521799540266, 'rouge2': 0.7886835586110674, 'rougeL': 0.8711811073148175, 'rougeLsum': 0.8711031407486098}
0.2258	33.0	9405	0.2008	0.1489	{'bleu': 0.7436253569604943, 'precisions': [0.8706385780118499, 0.7836990595611285, 0.7097222222222223, 0.6447552447552447], 'brevity_penalty': 0.9948017955446742, 'length_ratio': 0.9948152595099056, 'translation_length': 18228, 'reference_length': 18323}	{'rouge1': 0.8721419286921401, 'rouge2': 0.7882963455000708, 'rougeL': 0.8717333624704182, 'rougeLsum': 0.8716064100420517}
0.2301	34.0	9690	0.1997	0.1469	{'bleu': 0.7477380046910622, 'precisions': [0.871839868384974, 0.78724070940653, 0.7143274640169504, 0.6500393116100288], 'brevity_penalty': 0.9951857415819062, 'length_ratio': 0.9951972930197021, 'translation_length': 18235, 'reference_length': 18323}	{'rouge1': 0.8733291718803134, 'rouge2': 0.7912828262461826, 'rougeL': 0.8730615834810505, 'rougeLsum': 0.8729508479014352}
0.2352	35.0	9975	0.1989	0.1474	{'bleu': 0.746786327866007, 'precisions': [0.8721523851347642, 0.7870004391743523, 0.7138049601287585, 0.6497506343512118], 'brevity_penalty': 0.9941981553480093, 'length_ratio': 0.9942149211373683, 'translation_length': 18217, 'reference_length': 18323}	{'rouge1': 0.873008906192879, 'rouge2': 0.790428630395299, 'rougeL': 0.8727160101378854, 'rougeLsum': 0.872513685033558}
0.2368	36.0	10260	0.1987	0.1481	{'bleu': 0.7450258680082317, 'precisions': [0.8723720418271876, 0.7871255977850491, 0.7142123036264866, 0.6497408416059035], 'brevity_penalty': 0.9916148795787761, 'length_ratio': 0.9916498390001637, 'translation_length': 18170, 'reference_length': 18323}	{'rouge1': 0.872557108952186, 'rouge2': 0.7899616699266603, 'rougeL': 0.8723067110327652, 'rougeLsum': 0.8722357990611533}
0.2387	37.0	10545	0.1987	0.1472	{'bleu': 0.7461895367214819, 'precisions': [0.871914426769062, 0.7861083249749248, 0.7125420260195878, 0.6478762454116413], 'brevity_penalty': 0.9949115093798545, 'length_ratio': 0.994924411941276, 'translation_length': 18230, 'reference_length': 18323}	{'rouge1': 0.8734065376381056, 'rouge2': 0.7908717014566278, 'rougeL': 0.8730576045837826, 'rougeLsum': 0.8729880391818262}
0.2198	38.0	10830	0.1984	0.1460	{'bleu': 0.7480290753832786, 'precisions': [0.8731273665148439, 0.7880840388836626, 0.7149542961608775, 0.6505465675557499], 'brevity_penalty': 0.9945274585595061, 'length_ratio': 0.9945423784314795, 'translation_length': 18223, 'reference_length': 18323}	{'rouge1': 0.8746790367634547, 'rouge2': 0.7924947185694857, 'rougeL': 0.8743742574864739, 'rougeLsum': 0.8741497830000428}
0.2261	39.0	11115	0.1981	0.1461	{'bleu': 0.7482856703011707, 'precisions': [0.8729975861312267, 0.7880877742946708, 0.7149853801169591, 0.6507867132867133], 'brevity_penalty': 0.9948017955446742, 'length_ratio': 0.9948152595099056, 'translation_length': 18228, 'reference_length': 18323}	{'rouge1': 0.8746130363164816, 'rouge2': 0.7924162570451201, 'rougeL': 0.8742493430241225, 'rougeLsum': 0.8740282212067312}
0.2274	39.8604	11360	0.1983	0.1456	{'bleu': 0.7492044741492486, 'precisions': [0.8730063030967389, 0.7883760255527025, 0.7152661166678835, 0.6510430304617264], 'brevity_penalty': 0.9957339815437508, 'length_ratio': 0.995743055176554, 'translation_length': 18245, 'reference_length': 18323}	{'rouge1': 0.8748364322650914, 'rouge2': 0.7928723605220098, 'rougeL': 0.8745015805780032, 'rougeLsum': 0.8743113489400196}

Framework versions

• Transformers 4.49.0
• Pytorch 2.6.0+cu124
• Datasets 3.2.0
• Tokenizers 0.21.0