Multilingual BERT fine-tuned on SQuADv1.1

WandB run link

GPU: Tesla P100-PCIE-16GB

Training Arguments

max_seq_length              = 512
doc_stride                  = 256
max_answer_length           = 64
bacth_size                  = 16
gradient_accumulation_steps = 2
learning_rate               = 5e-5
weight_decay                = 3e-7
num_train_epochs            = 3
warmup_ratio                = 0.1
fp16                        = True
fp16_opt_level              = "O1"
seed                        = 0

Results

EM F1
81.731 89.009

Zero-shot performance

on ARCD

EM F1
20.655 48.051

on XQuAD

Language EM F1
Arabic 42.185 57.803
English 73.529 85.01
German 55.882 72.555
Greek 45.21 62.207
Spanish 58.067 76.406
Hindi 40.588 55.29
Russian 55.126 71.617
Thai 26.891 39.965
Turkish 34.874 51.138
Vietnamese 47.983 68.125
Chinese 47.395 58.928
New: fine-tune this model in a few clicks by selecting AutoNLP in the "Train" menu!
Downloads last month
1,032
Hosted inference API
Question Answering
This model can be loaded on the Inference API on-demand.