julien-c HF staff commited on
Commit
ee38c86
β€’
1 Parent(s): 0d631bf

Migrate model card from transformers-repo

Browse files

Read announcement at https://discuss.huggingface.co/t/announcement-all-model-cards-will-be-migrated-to-hf-co-model-repos/2755
Original file history: https://github.com/huggingface/transformers/commits/master/model_cards/mrm8488/spanbert-large-finetuned-squadv2/README.md

Files changed (1) hide show
  1. README.md +82 -0
README.md ADDED
@@ -0,0 +1,82 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ ---
2
+ language: en
3
+ thumbnail:
4
+ ---
5
+
6
+ # SpanBERT large fine-tuned on SQuAD v2
7
+
8
+ [SpanBERT](https://github.com/facebookresearch/SpanBERT) created by [Facebook Research](https://github.com/facebookresearch) and fine-tuned on [SQuAD 2.0](https://rajpurkar.github.io/SQuAD-explorer/) for **Q&A** downstream task ([by them](https://github.com/facebookresearch/SpanBERT#finetuned-models-squad-1120-relation-extraction-coreference-resolution)).
9
+
10
+ ## Details of SpanBERT
11
+
12
+ [SpanBERT: Improving Pre-training by Representing and Predicting Spans](https://arxiv.org/abs/1907.10529)
13
+
14
+ ## Details of the downstream task (Q&A) - Dataset πŸ“š 🧐 ❓
15
+
16
+ [SQuAD2.0](https://rajpurkar.github.io/SQuAD-explorer/) combines the 100,000 questions in SQuAD1.1 with over 50,000 unanswerable questions written adversarially by crowdworkers to look similar to answerable ones. To do well on SQuAD2.0, systems must not only answer questions when possible, but also determine when no answer is supported by the paragraph and abstain from answering.
17
+
18
+ | Dataset | Split | # samples |
19
+ | -------- | ----- | --------- |
20
+ | SQuAD2.0 | train | 130k |
21
+ | SQuAD2.0 | eval | 12.3k |
22
+
23
+ ## Model fine-tuning πŸ‹οΈβ€
24
+
25
+ You can get the fine-tuning script [here](https://github.com/facebookresearch/SpanBERT)
26
+
27
+ ```bash
28
+ python code/run_squad.py \
29
+ --do_train \
30
+ --do_eval \
31
+ --model spanbert-large-cased \
32
+ --train_file train-v2.0.json \
33
+ --dev_file dev-v2.0.json \
34
+ --train_batch_size 32 \
35
+ --eval_batch_size 32 \
36
+ --learning_rate 2e-5 \
37
+ --num_train_epochs 4 \
38
+ --max_seq_length 512 \
39
+ --doc_stride 128 \
40
+ --eval_metric best_f1 \
41
+ --output_dir squad2_output \
42
+ --version_2_with_negative \
43
+ --fp16
44
+ ```
45
+
46
+ ## Results Comparison πŸ“
47
+
48
+ | | SQuAD 1.1 | SQuAD 2.0 | Coref | TACRED |
49
+ | ---------------------- | ------------- | --------- | ------- | ------ |
50
+ | | F1 | F1 | avg. F1 | F1 |
51
+ | BERT (base) | 88.5* | 76.5* | 73.1 | 67.7 |
52
+ | SpanBERT (base) | [92.4*](https://huggingface.co/mrm8488/spanbert-base-finetuned-squadv1) | [83.6*](https://huggingface.co/mrm8488/spanbert-base-finetuned-squadv2) | 77.4 | [68.2](https://huggingface.co/mrm8488/spanbert-base-finetuned-tacred) |
53
+ | BERT (large) | 91.3 | 83.3 | 77.1 | 66.4 |
54
+ | SpanBERT (large) | [94.6](https://huggingface.co/mrm8488/spanbert-large-finetuned-squadv1) | **88.7** (this) | 79.6 | [70.8](https://huggingface.co/mrm8488/spanbert-large-finetuned-tacred) |
55
+
56
+
57
+ Note: The numbers marked as * are evaluated on the development sets because those models were not submitted to the official SQuAD leaderboard. All the other numbers are test numbers.
58
+
59
+ ## Model in action
60
+
61
+ Fast usage with **pipelines**:
62
+
63
+ ```python
64
+ from transformers import pipeline
65
+
66
+ qa_pipeline = pipeline(
67
+ "question-answering",
68
+ model="mrm8488/spanbert-large-finetuned-squadv2",
69
+ tokenizer="SpanBERT/spanbert-large-cased"
70
+ )
71
+
72
+ qa_pipeline({
73
+ 'context': "Manuel Romero has been working very hard in the repository hugginface/transformers lately",
74
+ 'question': "How has been working Manuel Romero lately?"
75
+
76
+ })
77
+ # Output: {'answer': 'very hard', 'end': 40, 'score': 0.9052708846768347, 'start': 31}
78
+ ```
79
+
80
+ > Created by [Manuel Romero/@mrm8488](https://twitter.com/mrm8488)
81
+
82
+ > Made with <span style="color: #e25555;">&hearts;</span> in Spain