---
license: mit
language:
- am
- ar
- hy
- eu
- bn
- bs
- bg
- my
- hr
- ca
- cs
- da
- nl
- en
- et
- fi
- fr
- ka
- de
- el
- gu
- ht
- iw
- hi
- hu
- is
- in
- it
- ja
- kn
- km
- ko
- lo
- lv
- lt
- ml
- mr
- ne
- no
- or
- pa
- ps
- fa
- pl
- pt
- ro
- ru
- sr
- zh
- sd
- si
- sk
- sl
- es
- sv
- tl
- ta
- te
- th
- tr
- uk
- ur
- ug
- vi
- cy
tags:
- generated_from_trainer
model-index:
- name: verdict-classifier-en
  results:
  - task:
      type: text-classification
      name: Verdict Classification
widget:
- "One might think that this is true, but it's taken out of context."
---

# Multilingual Verdict Classifier

This model is a fine-tuned version of [xlm-roberta-base](https://huggingface.co/xlm-roberta-base) on 1,500 deduplicated multilingual verdicts from [Google Fact Check Tools API](https://developers.google.com/fact-check/tools/api/reference/rest/v1alpha1/claims/search), translated into 65 languages with the [Google Cloud Translation API](https://cloud.google.com/translate/docs/reference/rest/).
It achieves the following results on the evaluation set, being 1,000 such verdicts, but here including duplicates to represent the true distribution:
- Loss: 0.1856
- F1 Macro: 0.8148
- F1 Misinformation: 0.9764
- F1 Factual: 0.9375
- F1 Other: 0.5306
- Precision Macro: 0.8117
- Precision Misinformation: 0.9775
- Precision Factual: 0.9375
- Precision Other: 0.52

## Training procedure

### Training hyperparameters

The following hyperparameters were used during training:
- learning_rate: 2e-05
- train_batch_size: 4
- eval_batch_size: 4
- seed: 42
- gradient_accumulation_steps: 8
- total_train_batch_size: 32
- optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-08
- lr_scheduler_type: linear
- lr_scheduler_warmup_steps: 30066
- num_epochs: 1000

### Training results

| Training Loss | Epoch | Step  | Validation Loss | F1 Macro | F1 Misinformation | F1 Factual | F1 Other | Precision Macro | Precision Misinformation | Precision Factual | Precision Other |
|:-------------:|:-----:|:-----:|:---------------:|:--------:|:-----------------:|:----------:|:--------:|:----------:|:-------------------:|:------------:|:----------:|
| 0.8707        | 1.0   | 3758  | 0.2414          | 0.7832   | 0.9639            | 0.7857     | 0.6      | 0.7950     | 0.9683              | 0.9167       | 0.5        |
| 0.3918        | 2.0   | 7516  | 0.1856          | 0.8148   | 0.9764            | 0.9375     | 0.5306   | 0.8117     | 0.9775              | 0.9375       | 0.52       |
| 0.1766        | 3.0   | 11274 | 0.1942          | 0.8394   | 0.9809            | 0.9538     | 0.5833   | 0.8349     | 0.9820              | 0.9394       | 0.5833     |
| 0.1071        | 4.0   | 15032 | 0.2078          | 0.8676   | 0.9786            | 0.9841     | 0.64     | 0.8650     | 0.9797              | 1.0          | 0.6154     |


### Framework versions

- Transformers 4.11.3
- Pytorch 1.9.0+cu102
- Datasets 1.9.0
- Tokenizers 0.10.2