shahrukhx01/roberta-base-squad2-boolq-baseline

Multiple Prediction Heads

• ExtractiveQA Head
• Three Class Classification Head, classes => (yes, no, extra_qa) to answer binary questions or direct to ExtractiveQA Head

BoolQ Validation dataset Evaluation:

support => 3270
accuracy => 0.73
macro f1 => 0.71

SQuAD Validation dataset Evaluation:

eval_HasAns_exact = 78.0196
eval_HasAns_f1 = 84.0327
eval_HasAns_total = 5928
eval_NoAns_exact = 81.8167
eval_NoAns_f1 = 81.8167
eval_NoAns_total = 5945
eval_best_exact = 79.9208
eval_best_f1 = 82.9231
eval_exact = 79.9208
eval_f1 = 82.9231
eval_samples = 12165
eval_total = 11873

Uasge in transformers

Import the script from here

from multitask_model import RobertaForMultitaskQA
from transformers import RobertaTokenizerFast
device = torch.device("cuda" if torch.cuda.is_available() else "cpu")
model = RobertaForMultitaskQA.from_pretrained(
        "shahrukhx01/roberta-base-squad2-boolq-baseline",
        task_labels_map={"squad_v2": 2, "boolq": 3},
    ).to(device)
tokenizer = RobertaTokenizerFast.from_pretrained("shahrukhx01/roberta-base-squad2-boolq-baseline")