system commited on
Commit
5483f35
1 Parent(s): 1d1e44a

Update README.md

Browse files
Files changed (1) hide show
  1. README.md +60 -0
README.md ADDED
@@ -0,0 +1,60 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ ### Model
2
+ **[`allenai/scibert_scivocab_uncased`](https://huggingface.co/allenai/scibert_scivocab_uncased)** fine-tuned on **[`SQuAD V2`](https://rajpurkar.github.io/SQuAD-explorer/)** using **[`run_squad.py`](https://github.com/huggingface/transformers/blob/master/examples/run_squad.py)**
3
+
4
+ ### Training Parameters
5
+ Trained on 4 NVIDIA GeForce RTX 2080 Ti 11Gb
6
+ ```bash
7
+ BASE_MODEL=allenai/scibert_scivocab_uncased
8
+ python run_squad.py \
9
+ --version_2_with_negative \
10
+ --model_type albert \
11
+ --model_name_or_path $BASE_MODEL \
12
+ --output_dir $OUTPUT_MODEL \
13
+ --do_eval \
14
+ --do_lower_case \
15
+ --train_file $SQUAD_DIR/train-v2.0.json \
16
+ --predict_file $SQUAD_DIR/dev-v2.0.json \
17
+ --per_gpu_train_batch_size 18 \
18
+ --per_gpu_eval_batch_size 64 \
19
+ --learning_rate 3e-5 \
20
+ --num_train_epochs 3.0 \
21
+ --max_seq_length 384 \
22
+ --doc_stride 128 \
23
+ --save_steps 2000 \
24
+ --threads 24 \
25
+ --warmup_steps 550 \
26
+ --gradient_accumulation_steps 1 \
27
+ --fp16 \
28
+ --logging_steps 50 \
29
+ --do_train
30
+ ```
31
+
32
+ ### Evaluation
33
+
34
+ Evaluation on the dev set. I did not sweep for best threshold.
35
+
36
+ | | val |
37
+ |-------------------|-------------------|
38
+ | exact | 75.07790785816559 |
39
+ | f1 | 78.47735207283013 |
40
+ | total | 11873.0 |
41
+ | HasAns_exact | 70.76585695006747 |
42
+ | HasAns_f1 | 77.57449412292718 |
43
+ | HasAns_total | 5928.0 |
44
+ | NoAns_exact | 79.37762825904122 |
45
+ | NoAns_f1 | 79.37762825904122 |
46
+ | NoAns_total | 5945.0 |
47
+ | best_exact | 75.08633032931863 |
48
+ | best_exact_thresh | 0.0 |
49
+ | best_f1 | 78.48577454398324 |
50
+ | best_f1_thresh | 0.0 |
51
+
52
+ ### Usage
53
+
54
+ See [huggingface documentation](https://huggingface.co/transformers/model_doc/bert.html#bertforquestionanswering). Training on `SQuAD V2` allows the model to score if a paragraph contains an answer:
55
+ ```python
56
+ start_scores, end_scores = model(input_ids)
57
+ span_scores = start_scores.softmax(dim=1).log()[:,:,None] + end_scores.softmax(dim=1).log()[:,None,:]
58
+ ignore_score = span_scores[:,0,0] #no answer scores
59
+
60
+ ```