--- license: apache-2.0 base_model: google/byt5-small tags: - generated_from_trainer metrics: - wer model-index: - name: byt5-finetuned-indocollex-informal-to-formal results: [] --- # ByT5 Finetuned IndoCollex Informal to Formal with Word Formation Tag This model is a fine-tuned version of [google/byt5-small](https://huggingface.co/google/byt5-small) on [IndoCollex dataset](https://github.com/haryoa/indo-collex) on informal-formal transformation. It achieves the following results on the evaluation set: - Loss: 0.1665 - Cer: 0.1952 - Wer: 0.481 - Word Acc: 0.519 - Gen Len: 7.6914 On test set, it achieves following results : - CER: 0.2152 - WER: 0.5125 - Word Accuracy: 0.4875 ## Model description Inputs are constructed like this `tag transformasi kata: %s. kata: %s` For example : `tag transformasi kata: sound-alter. kata: sampe` ## Intended uses & limitations More information needed ## Training and evaluation data More information needed ## Training procedure ### Training hyperparameters The following hyperparameters were used during training: - learning_rate: 2e-05 - train_batch_size: 16 - eval_batch_size: 16 - seed: 42 - optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-08 - lr_scheduler_type: linear - num_epochs: 100 ### Training results | Training Loss | Epoch | Step | Validation Loss | Cer | Wer | Word Acc | Gen Len | |:-------------:|:-----:|:----:|:---------------:|:------:|:------:|:--------:|:-------:| | No log | 1.0 | 93 | 33.2385 | 2.2445 | 2.4 | -1.4 | 19.0 | | No log | 2.0 | 186 | 16.9556 | 2.3667 | 1.081 | -0.081 | 19.0 | | No log | 3.0 | 279 | 5.1125 | 1.3005 | 1.0 | 0.0 | 6.1886 | | No log | 4.0 | 372 | 3.0517 | 0.8676 | 0.9857 | 0.0143 | 8.5029 | | No log | 5.0 | 465 | 1.8607 | 0.4058 | 0.981 | 0.019 | 6.5486 | | 17.3258 | 6.0 | 558 | 0.7701 | 0.3769 | 0.9762 | 0.0238 | 6.3486 | | 17.3258 | 7.0 | 651 | 0.4911 | 0.3328 | 0.9619 | 0.0381 | 6.48 | | 17.3258 | 8.0 | 744 | 0.4172 | 0.3183 | 0.9476 | 0.0524 | 6.6971 | | 17.3258 | 9.0 | 837 | 0.3590 | 0.3014 | 0.9095 | 0.0905 | 6.8114 | | 17.3258 | 10.0 | 930 | 0.3303 | 0.3039 | 0.8762 | 0.1238 | 7.2686 | | 0.696 | 11.0 | 1023 | 0.3030 | 0.2912 | 0.8286 | 0.1714 | 7.2971 | | 0.696 | 12.0 | 1116 | 0.2969 | 0.3048 | 0.8429 | 0.1571 | 7.4514 | | 0.696 | 13.0 | 1209 | 0.2799 | 0.298 | 0.8238 | 0.1762 | 7.4286 | | 0.696 | 14.0 | 1302 | 0.2656 | 0.2946 | 0.8 | 0.2 | 7.4743 | | 0.696 | 15.0 | 1395 | 0.2524 | 0.2555 | 0.7619 | 0.2381 | 7.2457 | | 0.696 | 16.0 | 1488 | 0.2427 | 0.2564 | 0.7286 | 0.2714 | 7.4 | | 0.3225 | 17.0 | 1581 | 0.2317 | 0.2309 | 0.7095 | 0.2905 | 7.2343 | | 0.3225 | 18.0 | 1674 | 0.2196 | 0.2258 | 0.6857 | 0.3143 | 7.2971 | | 0.3225 | 19.0 | 1767 | 0.2162 | 0.2334 | 0.7095 | 0.2905 | 7.24 | | 0.3225 | 20.0 | 1860 | 0.2094 | 0.2224 | 0.7 | 0.3 | 7.2571 | | 0.3225 | 21.0 | 1953 | 0.2050 | 0.219 | 0.6714 | 0.3286 | 7.28 | | 0.2482 | 22.0 | 2046 | 0.2006 | 0.2148 | 0.6571 | 0.3429 | 7.3314 | | 0.2482 | 23.0 | 2139 | 0.1985 | 0.225 | 0.6619 | 0.3381 | 7.3543 | | 0.2482 | 24.0 | 2232 | 0.1962 | 0.2156 | 0.6429 | 0.3571 | 7.4114 | | 0.2482 | 25.0 | 2325 | 0.1927 | 0.2173 | 0.6381 | 0.3619 | 7.3429 | | 0.2482 | 26.0 | 2418 | 0.1943 | 0.2199 | 0.6524 | 0.3476 | 7.3943 | | 0.2055 | 27.0 | 2511 | 0.1940 | 0.2122 | 0.6381 | 0.3619 | 7.2571 | | 0.2055 | 28.0 | 2604 | 0.1869 | 0.2046 | 0.6143 | 0.3857 | 7.3314 | | 0.2055 | 29.0 | 2697 | 0.1849 | 0.1995 | 0.6 | 0.4 | 7.3543 | | 0.2055 | 30.0 | 2790 | 0.1833 | 0.2114 | 0.6048 | 0.3952 | 7.3543 | | 0.2055 | 31.0 | 2883 | 0.1812 | 0.2054 | 0.5952 | 0.4048 | 7.4457 | | 0.2055 | 32.0 | 2976 | 0.1772 | 0.208 | 0.5905 | 0.4095 | 7.52 | | 0.1792 | 33.0 | 3069 | 0.1768 | 0.2046 | 0.5905 | 0.4095 | 7.4743 | | 0.1792 | 34.0 | 3162 | 0.1756 | 0.2114 | 0.581 | 0.419 | 7.4857 | | 0.1792 | 35.0 | 3255 | 0.1735 | 0.2165 | 0.5714 | 0.4286 | 7.52 | | 0.1792 | 36.0 | 3348 | 0.1713 | 0.2165 | 0.5714 | 0.4286 | 7.6114 | | 0.1792 | 37.0 | 3441 | 0.1726 | 0.2037 | 0.5619 | 0.4381 | 7.4914 | | 0.1591 | 38.0 | 3534 | 0.1663 | 0.2063 | 0.5619 | 0.4381 | 7.4629 | | 0.1591 | 39.0 | 3627 | 0.1664 | 0.1995 | 0.5524 | 0.4476 | 7.44 | | 0.1591 | 40.0 | 3720 | 0.1661 | 0.1986 | 0.5381 | 0.4619 | 7.4457 | | 0.1591 | 41.0 | 3813 | 0.1658 | 0.1995 | 0.5333 | 0.4667 | 7.5429 | | 0.1591 | 42.0 | 3906 | 0.1646 | 0.191 | 0.519 | 0.481 | 7.48 | | 0.1591 | 43.0 | 3999 | 0.1619 | 0.1995 | 0.5381 | 0.4619 | 7.5543 | | 0.1427 | 44.0 | 4092 | 0.1641 | 0.1969 | 0.5333 | 0.4667 | 7.4229 | | 0.1427 | 45.0 | 4185 | 0.1672 | 0.1944 | 0.5286 | 0.4714 | 7.4743 | | 0.1427 | 46.0 | 4278 | 0.1645 | 0.1952 | 0.5381 | 0.4619 | 7.5143 | | 0.1427 | 47.0 | 4371 | 0.1667 | 0.1952 | 0.5381 | 0.4619 | 7.4686 | | 0.1427 | 48.0 | 4464 | 0.1663 | 0.1961 | 0.5143 | 0.4857 | 7.5543 | | 0.1322 | 49.0 | 4557 | 0.1640 | 0.1986 | 0.5333 | 0.4667 | 7.44 | | 0.1322 | 50.0 | 4650 | 0.1646 | 0.1935 | 0.4905 | 0.5095 | 7.4857 | | 0.1322 | 51.0 | 4743 | 0.1644 | 0.1927 | 0.5143 | 0.4857 | 7.4971 | | 0.1322 | 52.0 | 4836 | 0.1637 | 0.2148 | 0.5381 | 0.4619 | 7.5257 | | 0.1322 | 53.0 | 4929 | 0.1668 | 0.1978 | 0.5 | 0.5 | 7.5371 | | 0.1227 | 54.0 | 5022 | 0.1650 | 0.1995 | 0.519 | 0.481 | 7.5257 | | 0.1227 | 55.0 | 5115 | 0.1661 | 0.1952 | 0.4952 | 0.5048 | 7.6 | | 0.1227 | 56.0 | 5208 | 0.1642 | 0.2012 | 0.5095 | 0.4905 | 7.6057 | | 0.1227 | 57.0 | 5301 | 0.1667 | 0.2037 | 0.5048 | 0.4952 | 7.64 | | 0.1227 | 58.0 | 5394 | 0.1650 | 0.1893 | 0.4857 | 0.5143 | 7.52 | | 0.1227 | 59.0 | 5487 | 0.1665 | 0.1944 | 0.481 | 0.519 | 7.5657 | | 0.1165 | 60.0 | 5580 | 0.1652 | 0.1902 | 0.4905 | 0.5095 | 7.5429 | | 0.1165 | 61.0 | 5673 | 0.1649 | 0.1885 | 0.4857 | 0.5143 | 7.5543 | | 0.1165 | 62.0 | 5766 | 0.1679 | 0.1893 | 0.4905 | 0.5095 | 7.5371 | | 0.1165 | 63.0 | 5859 | 0.1670 | 0.1935 | 0.4905 | 0.5095 | 7.56 | | 0.1165 | 64.0 | 5952 | 0.1667 | 0.1944 | 0.4905 | 0.5095 | 7.5714 | | 0.1074 | 65.0 | 6045 | 0.1676 | 0.1978 | 0.4952 | 0.5048 | 7.5886 | | 0.1074 | 66.0 | 6138 | 0.1653 | 0.2012 | 0.481 | 0.519 | 7.5771 | | 0.1074 | 67.0 | 6231 | 0.1667 | 0.1961 | 0.4857 | 0.5143 | 7.5943 | | 0.1074 | 68.0 | 6324 | 0.1666 | 0.1927 | 0.4762 | 0.5238 | 7.5886 | | 0.1074 | 69.0 | 6417 | 0.1671 | 0.2003 | 0.4952 | 0.5048 | 7.52 | | 0.1038 | 70.0 | 6510 | 0.1648 | 0.2046 | 0.4857 | 0.5143 | 7.6 | | 0.1038 | 71.0 | 6603 | 0.1653 | 0.1935 | 0.481 | 0.519 | 7.6514 | | 0.1038 | 72.0 | 6696 | 0.1663 | 0.1952 | 0.4762 | 0.5238 | 7.6171 | | 0.1038 | 73.0 | 6789 | 0.1655 | 0.1995 | 0.481 | 0.519 | 7.6971 | | 0.1038 | 74.0 | 6882 | 0.1653 | 0.1969 | 0.4762 | 0.5238 | 7.6857 | | 0.1038 | 75.0 | 6975 | 0.1661 | 0.1995 | 0.4762 | 0.5238 | 7.7143 | | 0.1004 | 76.0 | 7068 | 0.1649 | 0.2003 | 0.4762 | 0.5238 | 7.7143 | | 0.1004 | 77.0 | 7161 | 0.1657 | 0.1969 | 0.4762 | 0.5238 | 7.6971 | | 0.1004 | 78.0 | 7254 | 0.1652 | 0.1986 | 0.481 | 0.519 | 7.7029 | | 0.1004 | 79.0 | 7347 | 0.1669 | 0.1969 | 0.481 | 0.519 | 7.68 | | 0.1004 | 80.0 | 7440 | 0.1665 | 0.2003 | 0.4762 | 0.5238 | 7.68 | | 0.0966 | 81.0 | 7533 | 0.1656 | 0.2012 | 0.481 | 0.519 | 7.7143 | | 0.0966 | 82.0 | 7626 | 0.1660 | 0.1995 | 0.481 | 0.519 | 7.7143 | | 0.0966 | 83.0 | 7719 | 0.1639 | 0.1978 | 0.4762 | 0.5238 | 7.7029 | | 0.0966 | 84.0 | 7812 | 0.1654 | 0.1986 | 0.481 | 0.519 | 7.7086 | | 0.0966 | 85.0 | 7905 | 0.1661 | 0.1995 | 0.481 | 0.519 | 7.7143 | | 0.0966 | 86.0 | 7998 | 0.1662 | 0.1986 | 0.481 | 0.519 | 7.7143 | | 0.0958 | 87.0 | 8091 | 0.1660 | 0.1969 | 0.4762 | 0.5238 | 7.7143 | | 0.0958 | 88.0 | 8184 | 0.1659 | 0.1944 | 0.481 | 0.519 | 7.6914 | | 0.0958 | 89.0 | 8277 | 0.1656 | 0.1952 | 0.481 | 0.519 | 7.6914 | | 0.0958 | 90.0 | 8370 | 0.1658 | 0.1952 | 0.481 | 0.519 | 7.6914 | | 0.0958 | 91.0 | 8463 | 0.1661 | 0.1952 | 0.481 | 0.519 | 7.6914 | | 0.0944 | 92.0 | 8556 | 0.1661 | 0.1961 | 0.481 | 0.519 | 7.6971 | | 0.0944 | 93.0 | 8649 | 0.1662 | 0.1944 | 0.481 | 0.519 | 7.6914 | | 0.0944 | 94.0 | 8742 | 0.1657 | 0.1961 | 0.481 | 0.519 | 7.7029 | | 0.0944 | 95.0 | 8835 | 0.1663 | 0.1944 | 0.481 | 0.519 | 7.6914 | | 0.0944 | 96.0 | 8928 | 0.1664 | 0.1944 | 0.481 | 0.519 | 7.6914 | | 0.0923 | 97.0 | 9021 | 0.1663 | 0.1952 | 0.481 | 0.519 | 7.6914 | | 0.0923 | 98.0 | 9114 | 0.1666 | 0.1952 | 0.481 | 0.519 | 7.6914 | | 0.0923 | 99.0 | 9207 | 0.1664 | 0.1952 | 0.481 | 0.519 | 7.6914 | | 0.0923 | 100.0 | 9300 | 0.1665 | 0.1952 | 0.481 | 0.519 | 7.6914 | ### Framework versions - Transformers 4.34.1 - Pytorch 2.1.0+cu118 - Datasets 2.14.6 - Tokenizers 0.14.1