--- license: mit language: - am - ar - hy - eu - bn - bs - bg - my - hr - ca - cs - da - nl - en - et - fi - fr - ka - de - el - gu - ht - iw - hi - hu - is - in - it - ja - kn - km - ko - lo - lv - lt - ml - mr - ne - no - or - pa - ps - fa - pl - pt - ro - ru - sr - zh - sd - si - sk - sl - es - sv - tl - ta - te - th - tr - uk - ur - ug - vi - cy tags: - generated_from_trainer model-index: - name: verdict-classifier-en results: - task: type: text-classification name: Verdict Classification widget: - "One might think that this is true, but it's taken out of context." --- # Multilingual Verdict Classifier This model is a fine-tuned version of [xlm-roberta-base](https://huggingface.co/xlm-roberta-base) on 1,500 deduplicated multilingual verdicts from [Google Fact Check Tools API](https://developers.google.com/fact-check/tools/api/reference/rest/v1alpha1/claims/search), translated into 65 languages with the [Google Cloud Translation API](https://cloud.google.com/translate/docs/reference/rest/). It achieves the following results on the evaluation set, being 1,000 such verdicts, but here including duplicates to represent the true distribution: - Loss: 0.2245 - F1 Macro: 0.8818 - F1 Misinformation: 0.9842 - F1 Factual: 0.9688 - F1 Other: 0.6923 - Prec Macro: 0.8668 - Prec Misinformation: 0.9887 - Prec Factual: 0.9688 - Prec Other: 0.6429 ## Training procedure ### Training hyperparameters The following hyperparameters were used during training: - learning_rate: 2e-05 - train_batch_size: 4 - eval_batch_size: 4 - seed: 42 - gradient_accumulation_steps: 8 - total_train_batch_size: 32 - optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-08 - lr_scheduler_type: linear - lr_scheduler_warmup_steps: 30066 - num_epochs: 1000 ### Training results | Training Loss | Epoch | Step | Validation Loss | F1 Macro | F1 Misinformation | F1 Factual | F1 Other | Prec Macro | Prec Misinformation | Prec Factual | Prec Other | |:-------------:|:-----:|:-----:|:---------------:|:--------:|:-----------------:|:----------:|:--------:|:----------:|:-------------------:|:------------:|:----------:| | 1.0833 | 0.13 | 500 | 0.9331 | 0.3136 | 0.9408 | 0.0 | 0.0 | 0.2961 | 0.8882 | 0.0 | 0.0 | | 0.9697 | 0.27 | 1000 | 0.8839 | 0.3136 | 0.9408 | 0.0 | 0.0 | 0.2961 | 0.8882 | 0.0 | 0.0 | | 0.9406 | 0.4 | 1500 | 0.7238 | 0.3591 | 0.9394 | 0.0 | 0.1379 | 0.4304 | 0.8911 | 0.0 | 0.4 | | 0.9183 | 0.53 | 2000 | 0.6494 | 0.3482 | 0.9336 | 0.0 | 0.1111 | 0.3528 | 0.8916 | 0.0 | 0.1667 | | 0.9103 | 0.67 | 2500 | 0.5686 | 0.3621 | 0.9323 | 0.0 | 0.1538 | 0.3643 | 0.8930 | 0.0 | 0.2 | | 0.8637 | 0.8 | 3000 | 0.5720 | 0.4233 | 0.9367 | 0.0 | 0.3333 | 0.4036 | 0.9108 | 0.0 | 0.3 | | 0.7905 | 0.93 | 3500 | 0.3918 | 0.4721 | 0.9418 | 0.0 | 0.4746 | 0.4402 | 0.9206 | 0.0 | 0.4 | | 0.6625 | 1.06 | 4000 | 0.2771 | 0.8225 | 0.9666 | 0.9375 | 0.5634 | 0.7853 | 0.9929 | 0.9375 | 0.4255 | | 0.5403 | 1.2 | 4500 | 0.2187 | 0.8083 | 0.9703 | 0.9062 | 0.5484 | 0.7799 | 0.9861 | 0.9062 | 0.4474 | | 0.4837 | 1.33 | 5000 | 0.1708 | 0.8138 | 0.9728 | 0.9231 | 0.5455 | 0.7916 | 0.9817 | 0.9091 | 0.4839 | | 0.4411 | 1.46 | 5500 | 0.1734 | 0.7873 | 0.9707 | 0.9206 | 0.4706 | 0.7843 | 0.9729 | 0.9355 | 0.4444 | | 0.3991 | 1.6 | 6000 | 0.1671 | 0.8131 | 0.9705 | 0.9355 | 0.5333 | 0.7976 | 0.9816 | 0.9667 | 0.4444 | | 0.344 | 1.73 | 6500 | 0.1719 | 0.7989 | 0.9749 | 0.8955 | 0.5263 | 0.7667 | 0.9885 | 0.8571 | 0.4545 | | 0.3005 | 1.86 | 7000 | 0.1855 | 0.8052 | 0.9704 | 0.9206 | 0.5246 | 0.7839 | 0.9838 | 0.9355 | 0.4324 | | 0.2638 | 2.0 | 7500 | 0.1802 | 0.7896 | 0.9752 | 0.9231 | 0.4706 | 0.7777 | 0.9796 | 0.9091 | 0.4444 | | 0.2362 | 2.13 | 8000 | 0.1752 | 0.7762 | 0.9718 | 0.8986 | 0.4583 | 0.7578 | 0.9773 | 0.8378 | 0.4583 | | 0.2077 | 2.26 | 8500 | 0.1739 | 0.8101 | 0.9740 | 0.9206 | 0.5357 | 0.7953 | 0.9817 | 0.9355 | 0.4688 | | 0.1858 | 2.39 | 9000 | 0.1986 | 0.8073 | 0.9716 | 0.9412 | 0.5091 | 0.7748 | 0.9839 | 0.8889 | 0.4516 | | 0.1755 | 2.53 | 9500 | 0.1945 | 0.7872 | 0.9754 | 0.9180 | 0.4681 | 0.8049 | 0.9710 | 0.9655 | 0.4783 | | 0.1591 | 2.66 | 10000 | 0.2366 | 0.7880 | 0.9692 | 0.8857 | 0.5091 | 0.7504 | 0.9838 | 0.8158 | 0.4516 | | 0.1457 | 2.79 | 10500 | 0.2346 | 0.7614 | 0.9671 | 0.8857 | 0.4314 | 0.7334 | 0.9771 | 0.8158 | 0.4074 | | 0.1376 | 2.93 | 11000 | 0.2361 | 0.8015 | 0.9729 | 0.9118 | 0.52 | 0.7802 | 0.9795 | 0.8611 | 0.5 | | 0.126 | 3.06 | 11500 | 0.2276 | 0.8331 | 0.9751 | 0.9688 | 0.5556 | 0.8168 | 0.9818 | 0.9688 | 0.5 | | 0.1133 | 3.19 | 12000 | 0.2972 | 0.8014 | 0.9727 | 0.9231 | 0.5085 | 0.7746 | 0.9861 | 0.9091 | 0.4286 | | 0.1114 | 3.33 | 12500 | 0.2600 | 0.8038 | 0.9705 | 0.8955 | 0.5455 | 0.7742 | 0.9816 | 0.8571 | 0.4839 | | 0.1099 | 3.46 | 13000 | 0.3221 | 0.8273 | 0.9738 | 0.9118 | 0.5965 | 0.7882 | 0.9884 | 0.8611 | 0.5152 | | 0.1116 | 3.59 | 13500 | 0.2277 | 0.8376 | 0.9775 | 0.9231 | 0.6122 | 0.8296 | 0.9797 | 0.9091 | 0.6 | | 0.106 | 3.73 | 14000 | 0.2347 | 0.8148 | 0.9774 | 0.8955 | 0.5714 | 0.7997 | 0.9819 | 0.8571 | 0.56 | | 0.098 | 3.86 | 14500 | 0.2337 | 0.8487 | 0.9775 | 0.9688 | 0.6 | 0.8418 | 0.9797 | 0.9688 | 0.5769 | | 0.0899 | 3.99 | 15000 | 0.2072 | 0.8636 | 0.9820 | 0.9688 | 0.64 | 0.8561 | 0.9842 | 0.9688 | 0.6154 | | 0.0855 | 4.12 | 15500 | 0.2385 | 0.8409 | 0.9762 | 0.9538 | 0.5926 | 0.8189 | 0.9840 | 0.9394 | 0.5333 | | 0.0864 | 4.26 | 16000 | 0.2780 | 0.8462 | 0.9774 | 0.9688 | 0.5926 | 0.8287 | 0.9841 | 0.9688 | 0.5333 | | 0.0784 | 4.39 | 16500 | 0.2668 | 0.8277 | 0.9776 | 0.9524 | 0.5532 | 0.8361 | 0.9754 | 0.9677 | 0.5652 | | 0.0923 | 4.52 | 17000 | 0.2893 | 0.8399 | 0.9738 | 0.9254 | 0.6207 | 0.8012 | 0.9884 | 0.8857 | 0.5294 | | 0.0794 | 4.66 | 17500 | 0.3101 | 0.8556 | 0.9773 | 0.9688 | 0.6207 | 0.8289 | 0.9885 | 0.9688 | 0.5294 | | 0.082 | 4.79 | 18000 | 0.2245 | 0.8818 | 0.9842 | 0.9688 | 0.6923 | 0.8668 | 0.9887 | 0.9688 | 0.6429 | | 0.084 | 4.92 | 18500 | 0.2771 | 0.8247 | 0.9797 | 0.8986 | 0.5957 | 0.8102 | 0.9841 | 0.8378 | 0.6087 | | 0.0757 | 5.06 | 19000 | 0.2971 | 0.8594 | 0.9773 | 0.9677 | 0.6333 | 0.8388 | 0.9885 | 1.0 | 0.5278 | | 0.0709 | 5.19 | 19500 | 0.3601 | 0.8410 | 0.9774 | 0.9688 | 0.5769 | 0.8288 | 0.9819 | 0.9688 | 0.5357 | | 0.0698 | 5.32 | 20000 | 0.2772 | 0.8333 | 0.9762 | 0.9524 | 0.5714 | 0.8173 | 0.9840 | 0.9677 | 0.5 | | 0.0652 | 5.45 | 20500 | 0.3397 | 0.8186 | 0.9752 | 0.9524 | 0.5283 | 0.8100 | 0.9796 | 0.9677 | 0.4828 | | 0.0735 | 5.59 | 21000 | 0.3027 | 0.8412 | 0.9785 | 0.9524 | 0.5926 | 0.8284 | 0.9841 | 0.9677 | 0.5333 | | 0.0746 | 5.72 | 21500 | 0.3122 | 0.8384 | 0.9751 | 0.9688 | 0.5714 | 0.8176 | 0.9840 | 0.9688 | 0.5 | | 0.0714 | 5.85 | 22000 | 0.2683 | 0.8381 | 0.9787 | 0.9524 | 0.5833 | 0.8429 | 0.9776 | 0.9677 | 0.5833 | | 0.073 | 5.99 | 22500 | 0.2436 | 0.8676 | 0.9786 | 0.9841 | 0.64 | 0.8650 | 0.9797 | 1.0 | 0.6154 | | 0.0653 | 6.12 | 23000 | 0.3380 | 0.8559 | 0.9761 | 0.9688 | 0.6230 | 0.8243 | 0.9907 | 0.9688 | 0.5135 | ### Framework versions - Transformers 4.11.3 - Pytorch 1.9.0+cu102 - Datasets 1.9.0 - Tokenizers 0.10.2