File size: 3,641 Bytes
90f60b4 456945f 90f60b4 |
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 |
---
tags:
- generation
language:
- multilingual
- cs
- en
---
# Mt5-base for Czech+English Generative Question Answering
This is the [mt5-base](https://huggingface.co/google/mt5-base) model with an LM head for a generation of extractive answers. In contrary to our [mt5-base-priming](https://huggingface.co/gaussalgo/mt5-base-priming-QA_en-cs/edit/main/README.md), this is a traditional sequence2sequence model without priming, though can also be used on other Text extraction tasks, such as Named Entity Recognition in zero-shot settings (with a significant decay in quality, compared to priming).
## Intended uses & limitations
This model is purposed to *generate* a segment of a given context that contains an answer to a given question (Extractive Question Answering) in English and Czech.
Given the fine-tuning on two languages and a good reported zero-shot cross-lingual applicability of other fine-tuned multilingual large language models, the model will likely also work on other languages as well, with a specific decay in quality.
Note that despite its size, English SQuAD has a variety of reported biases,
conditioned by the relative position or type of the answer in the context that can affect the model's performance on new data
(see, e.g. [L. Mikula (2022)](https://is.muni.cz/th/adh58/?lang=en), Chap. 4.1).
## Usage
Here is how to use this model to answer the question on a given context using 🤗 Transformers in PyTorch:
```python
from transformers import AutoTokenizer, AutoModelForSeq2SeqLM
tokenizer = AutoTokenizer.from_pretrained("gaussalgo/mt5-base-generative-QA_en-cs")
model = AutoModelForSeq2SeqLM.from_pretrained("gaussalgo/mt5-base-generative-QA_en-cs")
context = """
Podle slovenského lidového podání byl Juro Jánošík obdařen magickými předměty (kouzelná valaška, čarovný opasek),
které mu dodávaly nadpřirozené schopnosti. Okrádal především šlechtice,
trestal panské dráby a ze svého lupu vyděloval část pro chudé, tedy bohatým bral a chudým dával.
"""
question = "Jaké schopnosti daly magické předměty Juro Jánošíkovi?"
inputs = tokenizer(question, context, return_tensors="pt")
outputs = model.generate(**inputs)
print("Answer:")
print(tokenizer.decode(outputs))
```
## Training
The model has been trained using [Adaptor library](https://github.com/gaussalgo/adaptor) v0.1.5, in parallel on both Czech and English data, with the following parameters:
```python
training_arguments = AdaptationArguments(output_dir="train_dir",
learning_rate=5e-5,
stopping_strategy=StoppingStrategy.ALL_OBJECTIVES_CONVERGED,
do_train=True,
do_eval=True,
warmup_steps=1000,
max_steps=100000,
gradient_accumulation_steps=4,
eval_steps=100,
logging_steps=10,
save_steps=1000,
num_train_epochs=50,
evaluation_strategy="steps",
remove_unused_columns=False)
```
You can find the full training script in [train_mt5_qa_en+cs.py](https://huggingface.co/gaussalgo/mt5-base-generative-QA_en-cs/blob/main/train_mt5_qa_en%2Bcs.py), reproducible after a specific data preprocessing for Czech SQAD in [parse_czech_squad.py](parse_czech_squad.py) |