--- license: apache-2.0 base_model: google/byt5-base tags: - generated_from_trainer metrics: - wer model-index: - name: byt5-finetuned-indocollex-informal-to-formal results: [] --- # Byt5-base Finetuned IndoCollex Informal to Formal This model is a fine-tuned version of [google/byt5-base](https://huggingface.co/google/byt5-base) on on [IndoCollex dataset](https://github.com/haryoa/indo-collex) on informal-formal transformation. It achieves the following results on the evaluation set: - Loss: 0.2191 - Cer: 0.208 - Wer: 0.5286 - Word Acc: 0.4714 - Gen Len: 7.7486 On test set, it achieves following results : - CER: 0.2589 - WER: 0.575 - Word Accuracy: 0.425 ## Model description More information needed ## Intended uses & limitations More information needed ## Training and evaluation data More information needed ## Training procedure ### Training hyperparameters The following hyperparameters were used during training: - learning_rate: 2e-05 - train_batch_size: 16 - eval_batch_size: 16 - seed: 42 - optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-08 - lr_scheduler_type: linear - num_epochs: 100 ### Training results | Training Loss | Epoch | Step | Validation Loss | Cer | Wer | Word Acc | Gen Len | |:-------------:|:-----:|:----:|:---------------:|:------:|:------:|:--------:|:-------:| | No log | 1.0 | 93 | 15.6714 | 2.1749 | 2.0857 | -1.0857 | 19.0 | | No log | 2.0 | 186 | 12.9426 | 2.1265 | 1.6619 | -0.6619 | 19.0 | | No log | 3.0 | 279 | 9.8664 | 1.9525 | 1.0524 | -0.0524 | 17.8343 | | No log | 4.0 | 372 | 4.9424 | 1.0051 | 1.0 | 0.0 | 0.1086 | | No log | 5.0 | 465 | 3.8691 | 0.5688 | 0.9667 | 0.0333 | 6.8171 | | 11.8639 | 6.0 | 558 | 3.0433 | 0.5161 | 0.9619 | 0.0381 | 6.1486 | | 11.8639 | 7.0 | 651 | 1.1874 | 0.5374 | 0.9619 | 0.0381 | 5.9143 | | 11.8639 | 8.0 | 744 | 0.5482 | 0.5756 | 0.9714 | 0.0286 | 7.2457 | | 11.8639 | 9.0 | 837 | 0.4749 | 0.5195 | 0.9476 | 0.0524 | 7.1771 | | 11.8639 | 10.0 | 930 | 0.3678 | 0.3591 | 0.8952 | 0.1048 | 7.0286 | | 1.4703 | 11.0 | 1023 | 0.3154 | 0.2988 | 0.8524 | 0.1476 | 7.0343 | | 1.4703 | 12.0 | 1116 | 0.2753 | 0.2895 | 0.819 | 0.181 | 7.3314 | | 1.4703 | 13.0 | 1209 | 0.2561 | 0.2674 | 0.7667 | 0.2333 | 7.1543 | | 1.4703 | 14.0 | 1302 | 0.2386 | 0.2581 | 0.7667 | 0.2333 | 7.3657 | | 1.4703 | 15.0 | 1395 | 0.2249 | 0.2453 | 0.7429 | 0.2571 | 7.4629 | | 1.4703 | 16.0 | 1488 | 0.2163 | 0.2428 | 0.7286 | 0.2714 | 7.4514 | | 0.3131 | 17.0 | 1581 | 0.2093 | 0.2538 | 0.7286 | 0.2714 | 7.4743 | | 0.3131 | 18.0 | 1674 | 0.2027 | 0.2436 | 0.7143 | 0.2857 | 7.6057 | | 0.3131 | 19.0 | 1767 | 0.1986 | 0.2453 | 0.7 | 0.3 | 7.6229 | | 0.3131 | 20.0 | 1860 | 0.1923 | 0.2453 | 0.6952 | 0.3048 | 7.6571 | | 0.3131 | 21.0 | 1953 | 0.1907 | 0.2411 | 0.6667 | 0.3333 | 7.5829 | | 0.2093 | 22.0 | 2046 | 0.1876 | 0.2368 | 0.6571 | 0.3429 | 7.6171 | | 0.2093 | 23.0 | 2139 | 0.1889 | 0.2351 | 0.6476 | 0.3524 | 7.5543 | | 0.2093 | 24.0 | 2232 | 0.1864 | 0.2334 | 0.6429 | 0.3571 | 7.6 | | 0.2093 | 25.0 | 2325 | 0.1802 | 0.2284 | 0.6238 | 0.3762 | 7.6 | | 0.2093 | 26.0 | 2418 | 0.1786 | 0.2292 | 0.6381 | 0.3619 | 7.6 | | 0.1641 | 27.0 | 2511 | 0.1760 | 0.2182 | 0.6286 | 0.3714 | 7.56 | | 0.1641 | 28.0 | 2604 | 0.1769 | 0.2199 | 0.6143 | 0.3857 | 7.6057 | | 0.1641 | 29.0 | 2697 | 0.1735 | 0.225 | 0.619 | 0.381 | 7.6114 | | 0.1641 | 30.0 | 2790 | 0.1764 | 0.2207 | 0.6 | 0.4 | 7.6057 | | 0.1641 | 31.0 | 2883 | 0.1727 | 0.208 | 0.5952 | 0.4048 | 7.5943 | | 0.1641 | 32.0 | 2976 | 0.1735 | 0.208 | 0.5905 | 0.4095 | 7.64 | | 0.1336 | 33.0 | 3069 | 0.1694 | 0.2063 | 0.581 | 0.419 | 7.6743 | | 0.1336 | 34.0 | 3162 | 0.1728 | 0.2114 | 0.5857 | 0.4143 | 7.6914 | | 0.1336 | 35.0 | 3255 | 0.1722 | 0.2207 | 0.5857 | 0.4143 | 7.64 | | 0.1336 | 36.0 | 3348 | 0.1703 | 0.2224 | 0.6 | 0.4 | 7.6857 | | 0.1336 | 37.0 | 3441 | 0.1715 | 0.2173 | 0.581 | 0.419 | 7.6629 | | 0.1115 | 38.0 | 3534 | 0.1735 | 0.2148 | 0.5762 | 0.4238 | 7.68 | | 0.1115 | 39.0 | 3627 | 0.1715 | 0.2088 | 0.5762 | 0.4238 | 7.7143 | | 0.1115 | 40.0 | 3720 | 0.1754 | 0.2131 | 0.5714 | 0.4286 | 7.72 | | 0.1115 | 41.0 | 3813 | 0.1757 | 0.2122 | 0.5524 | 0.4476 | 7.76 | | 0.1115 | 42.0 | 3906 | 0.1725 | 0.2122 | 0.5571 | 0.4429 | 7.7714 | | 0.1115 | 43.0 | 3999 | 0.1724 | 0.2173 | 0.5619 | 0.4381 | 7.76 | | 0.0954 | 44.0 | 4092 | 0.1799 | 0.2071 | 0.5429 | 0.4571 | 7.7371 | | 0.0954 | 45.0 | 4185 | 0.1771 | 0.2097 | 0.5524 | 0.4476 | 7.7657 | | 0.0954 | 46.0 | 4278 | 0.1780 | 0.2063 | 0.5476 | 0.4524 | 7.72 | | 0.0954 | 47.0 | 4371 | 0.1791 | 0.2088 | 0.5381 | 0.4619 | 7.7714 | | 0.0954 | 48.0 | 4464 | 0.1799 | 0.2105 | 0.5429 | 0.4571 | 7.7486 | | 0.0814 | 49.0 | 4557 | 0.1799 | 0.2054 | 0.5333 | 0.4667 | 7.7657 | | 0.0814 | 50.0 | 4650 | 0.1830 | 0.2037 | 0.5381 | 0.4619 | 7.6971 | | 0.0814 | 51.0 | 4743 | 0.1824 | 0.2088 | 0.5429 | 0.4571 | 7.76 | | 0.0814 | 52.0 | 4836 | 0.1846 | 0.2037 | 0.5286 | 0.4714 | 7.7771 | | 0.0814 | 53.0 | 4929 | 0.1837 | 0.2046 | 0.5286 | 0.4714 | 7.7429 | | 0.073 | 54.0 | 5022 | 0.1816 | 0.2054 | 0.5333 | 0.4667 | 7.7943 | | 0.073 | 55.0 | 5115 | 0.1825 | 0.2029 | 0.519 | 0.481 | 7.7771 | | 0.073 | 56.0 | 5208 | 0.1870 | 0.208 | 0.5286 | 0.4714 | 7.7829 | | 0.073 | 57.0 | 5301 | 0.1870 | 0.2105 | 0.5381 | 0.4619 | 7.7829 | | 0.073 | 58.0 | 5394 | 0.1932 | 0.2054 | 0.5286 | 0.4714 | 7.7543 | | 0.073 | 59.0 | 5487 | 0.1880 | 0.2046 | 0.5143 | 0.4857 | 7.7886 | | 0.0653 | 60.0 | 5580 | 0.1890 | 0.2071 | 0.519 | 0.481 | 7.7714 | | 0.0653 | 61.0 | 5673 | 0.1952 | 0.2105 | 0.5286 | 0.4714 | 7.7886 | | 0.0653 | 62.0 | 5766 | 0.1940 | 0.2054 | 0.5238 | 0.4762 | 7.8 | | 0.0653 | 63.0 | 5859 | 0.1948 | 0.2063 | 0.5143 | 0.4857 | 7.7829 | | 0.0653 | 64.0 | 5952 | 0.1972 | 0.208 | 0.5238 | 0.4762 | 7.7943 | | 0.0582 | 65.0 | 6045 | 0.1965 | 0.2046 | 0.5238 | 0.4762 | 7.7543 | | 0.0582 | 66.0 | 6138 | 0.1968 | 0.2046 | 0.5095 | 0.4905 | 7.7657 | | 0.0582 | 67.0 | 6231 | 0.1981 | 0.2071 | 0.519 | 0.481 | 7.7886 | | 0.0582 | 68.0 | 6324 | 0.1977 | 0.2063 | 0.519 | 0.481 | 7.7771 | | 0.0582 | 69.0 | 6417 | 0.2018 | 0.2054 | 0.5238 | 0.4762 | 7.7657 | | 0.0522 | 70.0 | 6510 | 0.1992 | 0.2088 | 0.5286 | 0.4714 | 7.7657 | | 0.0522 | 71.0 | 6603 | 0.1999 | 0.2131 | 0.5381 | 0.4619 | 7.7714 | | 0.0522 | 72.0 | 6696 | 0.1998 | 0.2173 | 0.5429 | 0.4571 | 7.7943 | | 0.0522 | 73.0 | 6789 | 0.1991 | 0.2156 | 0.5381 | 0.4619 | 7.7829 | | 0.0522 | 74.0 | 6882 | 0.2024 | 0.2088 | 0.5333 | 0.4667 | 7.72 | | 0.0522 | 75.0 | 6975 | 0.2053 | 0.2046 | 0.5286 | 0.4714 | 7.7257 | | 0.0494 | 76.0 | 7068 | 0.2055 | 0.2054 | 0.5333 | 0.4667 | 7.7429 | | 0.0494 | 77.0 | 7161 | 0.2064 | 0.208 | 0.5333 | 0.4667 | 7.7029 | | 0.0494 | 78.0 | 7254 | 0.2057 | 0.208 | 0.5286 | 0.4714 | 7.7257 | | 0.0494 | 79.0 | 7347 | 0.2087 | 0.2097 | 0.5381 | 0.4619 | 7.68 | | 0.0494 | 80.0 | 7440 | 0.2085 | 0.2131 | 0.5476 | 0.4524 | 7.6971 | | 0.0462 | 81.0 | 7533 | 0.2099 | 0.2122 | 0.5476 | 0.4524 | 7.6914 | | 0.0462 | 82.0 | 7626 | 0.2090 | 0.2071 | 0.5286 | 0.4714 | 7.7429 | | 0.0462 | 83.0 | 7719 | 0.2127 | 0.2088 | 0.5286 | 0.4714 | 7.7086 | | 0.0462 | 84.0 | 7812 | 0.2135 | 0.2012 | 0.519 | 0.481 | 7.7371 | | 0.0462 | 85.0 | 7905 | 0.2148 | 0.2029 | 0.519 | 0.481 | 7.7486 | | 0.0462 | 86.0 | 7998 | 0.2148 | 0.2046 | 0.5238 | 0.4762 | 7.7657 | | 0.0434 | 87.0 | 8091 | 0.2148 | 0.2029 | 0.519 | 0.481 | 7.7543 | | 0.0434 | 88.0 | 8184 | 0.2150 | 0.2037 | 0.519 | 0.481 | 7.7657 | | 0.0434 | 89.0 | 8277 | 0.2160 | 0.2063 | 0.5238 | 0.4762 | 7.7543 | | 0.0434 | 90.0 | 8370 | 0.2167 | 0.2054 | 0.5238 | 0.4762 | 7.7486 | | 0.0434 | 91.0 | 8463 | 0.2168 | 0.2037 | 0.519 | 0.481 | 7.7657 | | 0.0419 | 92.0 | 8556 | 0.2172 | 0.2037 | 0.5238 | 0.4762 | 7.7543 | | 0.0419 | 93.0 | 8649 | 0.2183 | 0.2037 | 0.5238 | 0.4762 | 7.7486 | | 0.0419 | 94.0 | 8742 | 0.2190 | 0.2063 | 0.5286 | 0.4714 | 7.7371 | | 0.0419 | 95.0 | 8835 | 0.2185 | 0.2054 | 0.5238 | 0.4762 | 7.7543 | | 0.0419 | 96.0 | 8928 | 0.2184 | 0.2054 | 0.5238 | 0.4762 | 7.7543 | | 0.0402 | 97.0 | 9021 | 0.2190 | 0.208 | 0.5286 | 0.4714 | 7.7429 | | 0.0402 | 98.0 | 9114 | 0.2189 | 0.208 | 0.5286 | 0.4714 | 7.7486 | | 0.0402 | 99.0 | 9207 | 0.2190 | 0.208 | 0.5286 | 0.4714 | 7.7486 | | 0.0402 | 100.0 | 9300 | 0.2191 | 0.208 | 0.5286 | 0.4714 | 7.7486 | ### Framework versions - Transformers 4.33.0 - Pytorch 2.0.0 - Datasets 2.1.0 - Tokenizers 0.13.3