|
--- |
|
language: |
|
- en |
|
- cs |
|
license: cc-by-4.0 |
|
metrics: |
|
- bleurt |
|
- bleu |
|
- bertscore |
|
pipeline_tag: text-classification |
|
--- |
|
# AlignScoreCS |
|
MultiTask multilingual model for assessing facticity in various NLU tasks in Czech and English language. We followed the initial paper AlignScore https://arxiv.org/abs/2305.16739. |
|
We trained a model using a shared architecture of checkpoint xlm-roberta-large https://huggingface.co/FacebookAI/xlm-roberta-large with three linear layers for regression, |
|
binary classification and ternary classification. |
|
|
|
|
|
# Usage |
|
```python |
|
# Assuming you copied the attached Files_and_versions/AlignScore.py file for ease of use in transformers. |
|
from AlignScoreCS import AlignScoreCS |
|
alignScoreCS = AlignScoreCS.from_pretrained("krotima1/AlignScoreCS") |
|
# put the model to cuda to accelerate |
|
print(alignScoreCS.score(context="This is context", claim="This is claim")) |
|
|
|
``` |
|
|
|
# Results |
|
|
|
|
|
|
|
# Training datasets |
|
The following table shows datasets that has been utilized for training the model. We translated these english datasets to Czech using seamLessM4t. |
|
|
|
| NLP Task | Dataset | Training Task | Context (n words) | Claim (n words) | Sample Count | |
|
|-----------------------|-------------------|---------------|-------------------|-----------------|--------------| |
|
| NLI | SNLI | 3-way | 10 | 13 | Cs: 500k | |
|
| | | | | | En: 550k | |
|
| | MultiNLI | 3-way | 16 | 20 | Cs: 393k | |
|
| | | | | | En: 393k | |
|
| | Adversarial NLI | 3-way | 48 | 54 | Cs: 163k | |
|
| | | | | | En: 163k | |
|
| | DocNLI | 2-way | 97 | 285 | Cs: 200k | |
|
| | | | | | En: 942k | |
|
| Fact Verification | NLI-style FEVER | 3-way | 48 | 50 | Cs: 208k | |
|
| | | | | | En: 208k | |
|
| | Vitamin C | 3-way | 23 | 25 | Cs: 371k | |
|
| | | | | | En: 371k | |
|
| Paraphrase | QQP | 2-way | 9 | 11 | Cs: 162k | |
|
| | | | | | En: 364k | |
|
| | PAWS | 2-way | - | 18 | Cs: - | |
|
| | | | | | En: 707k | |
|
| | PAWS labeled | 2-way | 18 | - | Cs: 49k | |
|
| | | | | | En: - | |
|
| | PAWS unlabeled | 2-way | 18 | - | Cs: 487k | |
|
| | | | | | En: - | |
|
| STS | SICK | reg | - | 10 | Cs: - | |
|
| | | | | | En: 4k | |
|
| | STS Benchmark | reg | - | 10 | Cs: - | |
|
| | | | | | En: 6k | |
|
| | Free-N1 | reg | 18 | - | Cs: 20k | |
|
| | | | | | En: - | |
|
| QA | SQuAD v2 | 2-way | 105 | 119 | Cs: 130k | |
|
| | | | | | En: 130k | |
|
| | RACE | 2-way | 266 | 273 | Cs: 200k | |
|
| | | | | | En: 351k | |
|
| Information Retrieval| MS MARCO | 2-way | 49 | 56 | Cs: 200k | |
|
| | | | | | En: 5M | |
|
| Summarization | WikiHow | 2-way | 434 | 508 | Cs: 157k | |
|
| | | | | | En: 157k | |
|
| | SumAug | 2-way | - | - | Cs: - | |
|
| | | | | | En: - | |
|
|
|
|
|
|
|
|
