1 --- 2 datasets: 3 - squad 4 --- 5 6 # BART-LARGE finetuned on SQuADv1 7 8 This is bart-large model finetuned on SQuADv1 dataset for question answering task 9 10 ## Model details 11 BART was propsed in the [paper](https://arxiv.org/abs/1910.13461) **BART: Denoising Sequence-to-Sequence Pre-training for Natural Language Generation, Translation, and Comprehension**. 12 BART is a seq2seq model intended for both NLG and NLU tasks.  13 14 To use BART for question answering tasks, we feed the complete document into the encoder and decoder, and use the top 15 hidden state of the decoder as a representation for each 16 word. This representation is used to classify the token. As given in the paper bart-large achives comparable to ROBERTa on SQuAD. 17 Another notable thing about BART is that it can handle sequences with upto 1024 tokens. 18 19 | Param | #Value | 20 |---------------------|--------| 21 | encoder layers | 12 | 22 | decoder layers | 12 | 23 | hidden size | 4096 | 24 | num attetion heads | 16 | 25 | on disk size | 1.63GB | 26 27 28 ## Model training 29 This model was trained on google colab v100 GPU.  30 You can find the fine-tuning colab here 31 [![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/drive/1I5cK1M_0dLaf5xoewh6swcm5nAInfwHy?usp=sharing). 32 33 34 ## Results 35 The results are actually slightly worse than given in the paper.  36 In the paper the authors mentioned that bart-large achieves 88.8 EM and 94.6 F1 37 38 | Metric | #Value | 39 |--------|--------| 40 | EM | 86.8022| 41 | F1 | 92.7342| 42 43 44 ## Model in Action 🚀 45 python3 46 from transformers import BartTokenizer, BartForQuestionAnswering 47 import torch 48 49 tokenizer = BartTokenizer.from_pretrained('valhalla/bart-large-finetuned-squadv1') 50 model = BartForQuestionAnswering.from_pretrained('valhalla/bart-large-finetuned-squadv1') 51 52 question, text = "Who was Jim Henson?", "Jim Henson was a nice puppet" 53 encoding = tokenizer(question, text, return_tensors='pt') 54 input_ids = encoding['input_ids'] 55 attention_mask = encoding['attention_mask'] 56 57 start_scores, end_scores = model(input_ids, attention_mask=attention_mask, output_attentions=False)[:2] 58 59 all_tokens = tokenizer.convert_ids_to_tokens(input_ids[0]) 60 answer = ' '.join(all_tokens[torch.argmax(start_scores) : torch.argmax(end_scores)+1]) 61 answer = tokenizer.convert_tokens_to_ids(answer.split()) 62 answer = tokenizer.decode(answer) 63 #answer => 'a nice puppet'  64  65 66 > Created with ❤️ by Suraj Patil [![Github icon](https://cdn0.iconfinder.com/data/icons/octicons/1024/mark-github-32.png)](https://github.com/patil-suraj/) 67 [![Twitter icon](https://cdn0.iconfinder.com/data/icons/shift-logotypes/32/Twitter-32.png)](https://twitter.com/psuraj28) 68