Chua, Vui Seng
commited on
Commit
•
96c9894
1
Parent(s):
c25c3d2
Add documentation
Browse files
readme.md
ADDED
@@ -0,0 +1,24 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
1 |
+
* A set of unstructured sparse bert-base-uncased models fine-tuned for SQuADv1.
|
2 |
+
* Tensorflow models are created using ```TFAutoModelForQuestionAnswering.from_pretrained(..., from_pt=True)``` and ```model.save_pretrained(tf_pth)```.
|
3 |
+
* Observed issue - loss in model translation, discrepancy observed in evaluation between pytorch and tensorflow models.
|
4 |
+
* Table below is evaluated in HF's transformers v4.9.2.
|
5 |
+
* Evaluation cli:
|
6 |
+
```bash
|
7 |
+
python run_qa.py \
|
8 |
+
--model_name_or_path <model identifier> \
|
9 |
+
--dataset_name squad \
|
10 |
+
--do_eval \
|
11 |
+
--per_device_eval_batch_size 384 \
|
12 |
+
--max_seq_length 68 \
|
13 |
+
--doc_stride 26 \
|
14 |
+
--output_dir /tmp/eval-squad
|
15 |
+
```
|
16 |
+
|
17 |
+
|
18 |
+
| | HF Model Hub Identifier | sparsity | em (pytorch) | em (tf) | f1 (pytorch) | f1 (tf) |
|
19 |
+
|---:|:------------------------------------------------------------------------------------------------------------------------|-----------:|---------------:|----------:|---------------:|----------:|
|
20 |
+
| 0 | [vuiseng9/bert-base-uncased-squadv1-85.4-sparse](https://huggingface.co/vuiseng9/bert-base-uncased-squadv1-85.4-sparse) | 85.4 | 69.9338 | 14.2573 | 77.6861 | 23.4917 |
|
21 |
+
| 1 | [vuiseng9/bert-base-uncased-squadv1-72.9-sparse](https://huggingface.co/vuiseng9/bert-base-uncased-squadv1-72.9-sparse) | 72.9 | 74.6358 | 31.0596 | 82.2555 | 39.8446 |
|
22 |
+
| 2 | [vuiseng9/bert-base-uncased-squadv1-65.1-sparse](https://huggingface.co/vuiseng9/bert-base-uncased-squadv1-65.1-sparse) | 65.1 | 76.1306 | 43.0274 | 83.4117 | 51.4300 |
|
23 |
+
| 3 | [vuiseng9/bert-base-uncased-squadv1-59.6-sparse](https://huggingface.co/vuiseng9/bert-base-uncased-squadv1-59.6-sparse) | 59.6 | 76.8590 | 50.4920 | 84.1267 | 59.0881 |
|
24 |
+
| 4 | [vuiseng9/bert-base-uncased-squadv1-52.0-sparse](https://huggingface.co/vuiseng9/bert-base-uncased-squadv1-52.0-sparse) | 52.0 | 78.0038 | 54.2857 | 85.2000 | 62.2914 |
|