patrickvonplaten commited on
Commit
891192d
1 Parent(s): bc5d071

Update README.md

Browse files
Files changed (1) hide show
  1. README.md +12 -13
README.md CHANGED
@@ -100,22 +100,21 @@ The following table summarizes the results for [fnet-base](https://huggingface.c
100
  | WNLI | [00:02:37](https://huggingface.co/gchhablani/fnet-base-finetuned-wnli) | [00:03:23](https://huggingface.co/gchhablani/bert-base-cased-finetuned-wnli)|
101
  | SUM | 16:30:45 | 24:23:56 |
102
 
103
-
104
- | Task | MNLI-(m/mm) | QQP | QNLI | SST-2 | CoLA | STS-B | MRPC | RTE | WNLI | SUM |
105
- |:----:|:-----------:|:----:|:----:|:-----:|:----:|:-----:|:----:|:----:|:----:|:-------:|
106
- |FNet-base (PyTorch)| [06:40:55](https://huggingface.co/gchhablani/fnet-base-finetuned-mnli)| [06:21:16](https://huggingface.co/gchhablani/fnet-base-finetuned-qqp) | [01:48:22](https://huggingface.co/gchhablani/fnet-base-finetuned-qnli) | [01:09:27](https://huggingface.co/gchhablani/fnet-base-finetuned-sst2) | [00:09:47](https://huggingface.co/gchhablani/fnet-base-finetuned-cola) | [00:07:09](https://huggingface.co/gchhablani/fnet-base-finetuned-stsb) | [00:07:48](https://huggingface.co/gchhablani/fnet-base-finetuned-mrpc) | [00:03:24](https://huggingface.co/gchhablani/fnet-base-finetuned-rte) | [00:02:37](https://huggingface.co/gchhablani/fnet-base-finetuned-wnli) | 16:30:45 |
107
- |Bert-base (PyTorch)| [09:52:33](https://huggingface.co/gchhablani/bert-base-cased-finetuned-mnli)| [09:25:01](https://huggingface.co/gchhablani/bert-base-cased-finetuned-qqp) | [02:40:22](https://huggingface.co/gchhablani/bert-base-cased-finetuned-qnli) | [01:42:17](https://huggingface.co/gchhablani/bert-base-cased-finetuned-sst2) | [00:14:20](https://huggingface.co/gchhablani/bert-base-cased-finetuned-cola) | [00:10:24](https://huggingface.co/gchhablani/bert-base-cased-finetuned-stsb) | [00:11:12](https://huggingface.co/gchhablani/bert-base-cased-finetuned-mrpc) | [00:04:51](https://huggingface.co/gchhablani/bert-base-cased-finetuned-rte) | [00:03:23](https://huggingface.co/gchhablani/bert-base-cased-finetuned-wnli) | 24:23:56 |
108
-
109
- On average the PyTorch version of FNet-base requires *ca.* 30% less time for GLUE fine-tuning on GPU.
110
 
111
  The following table summarizes the results for [fnet-base](https://huggingface.co/google/fnet-base) (called *FNet (PyTorch) - Reproduced*) and [bert-base-cased](https://hf.co/models/bert-base-cased) (called *Bert (PyTorch) - Reproduced*) in terms of performance and compares it to the reported performance of the official FNet-base model (called *FNet (Flax) - Official*).
112
 
113
- | Task | MNLI-(m/mm) | QQP | QNLI | SST-2 | CoLA | STS-B | MRPC | RTE | WNLI | Avg |
114
- |:----:|:-----------:|:----:|:----:|:-----:|:----:|:-----:|:----:|:----:|:----:|:-------:|
115
- | Metric | Accuracy or Match/Mismatch | mean(Accuracy,F1) | Accuracy | Accuracy | Matthews corr or Accuracy | Spearman corr. | mean(F1/Accuracy) | Accuracy | Accuracy | - |
116
- |FNet-base (PyTorch)| [76.75](https://huggingface.co/gchhablani/fnet-base-finetuned-mnli)| [86.5](https://huggingface.co/gchhablani/fnet-base-finetuned-qqp) | [84.39](https://huggingface.co/gchhablani/fnet-base-finetuned-qnli) | [89.45](https://huggingface.co/gchhablani/fnet-base-finetuned-sst2) | [35.94](https://huggingface.co/gchhablani/fnet-base-finetuned-cola) | [82.19](https://huggingface.co/gchhablani/fnet-base-finetuned-stsb) | [81.15](https://huggingface.co/gchhablani/fnet-base-finetuned-mrpc) | [62.82](https://huggingface.co/gchhablani/fnet-base-finetuned-rte) | [54.93](https://huggingface.co/gchhablani/fnet-base-finetuned-wnli) | - |
117
- |Bert-base (PyTorch)| [84.10](https://huggingface.co/gchhablani/bert-base-cased-finetuned-mnli)| [89.26](https://huggingface.co/gchhablani/bert-base-cased-finetuned-qqp) | [90.99](https://huggingface.co/gchhablani/bert-base-cased-finetuned-qnli) | [92.32](https://huggingface.co/gchhablani/bert-base-cased-finetuned-sst2) | [59.57](https://huggingface.co/gchhablani/bert-base-cased-finetuned-cola) | [88.98](https://huggingface.co/gchhablani/bert-base-cased-finetuned-stsb) | [88.15](https://huggingface.co/gchhablani/bert-base-cased-finetuned-mrpc) | [67.15](https://huggingface.co/gchhablani/bert-base-cased-finetuned-rte) | [46.48](https://huggingface.co/gchhablani/bert-base-cased-finetuned-wnli) | - |
118
- | FNet-Base (Flax - official) | 72/73 | 83 | 80 | 95 | 69 | 79 | 76 | 63 | - | 76.7 |
 
 
 
 
 
119
 
120
  We can see that FNet-base achieves around 93% of BERT-base's performance on average.
121
 
 
100
  | WNLI | [00:02:37](https://huggingface.co/gchhablani/fnet-base-finetuned-wnli) | [00:03:23](https://huggingface.co/gchhablani/bert-base-cased-finetuned-wnli)|
101
  | SUM | 16:30:45 | 24:23:56 |
102
 
103
+ On average the PyTorch version of FNet-base requires *ca.* 32% less time for GLUE fine-tuning on GPU.
 
 
 
 
 
 
104
 
105
  The following table summarizes the results for [fnet-base](https://huggingface.co/google/fnet-base) (called *FNet (PyTorch) - Reproduced*) and [bert-base-cased](https://hf.co/models/bert-base-cased) (called *Bert (PyTorch) - Reproduced*) in terms of performance and compares it to the reported performance of the official FNet-base model (called *FNet (Flax) - Official*).
106
 
107
+ | Task/Model | Metric | FNet-base (PyTorch) | Bert-base (PyTorch) | FNet-Base (Flax - official) |
108
+ | MNLI-(m/mm) | Accuracy or Match/Mismatch | [76.75](https://huggingface.co/gchhablani/fnet-base-finetuned-mnli) | [84.10](https://huggingface.co/gchhablani/bert-base-cased-finetuned-mnli) | 72/73 |
109
+ | QQP | mean(Accuracy,F1) | [86.5](https://huggingface.co/gchhablani/fnet-base-finetuned-qqp) | [89.26](https://huggingface.co/gchhablani/bert-base-cased-finetuned-qqp) | 83 |
110
+ | QNLI | Accuracy | [84.39](https://huggingface.co/gchhablani/fnet-base-finetuned-qnli) | [90.99](https://huggingface.co/gchhablani/bert-base-cased-finetuned-qnli) | 80 |
111
+ | SST-2 | Accuracy | [89.45](https://huggingface.co/gchhablani/fnet-base-finetuned-sst2) | [92.32](https://huggingface.co/gchhablani/bert-base-cased-finetuned-sst2) | 95 |
112
+ | CoLA | Matthews corr or Accuracy | [35.94](https://huggingface.co/gchhablani/fnet-base-finetuned-cola) | [59.57](https://huggingface.co/gchhablani/bert-base-cased-finetuned-cola) | 69 |
113
+ | STS-B | Spearman corr. | [82.19](https://huggingface.co/gchhablani/fnet-base-finetuned-stsb) | [88.98](https://huggingface.co/gchhablani/bert-base-cased-finetuned-stsb) | 79 |
114
+ | MRPC | mean(F1/Accuracy) | [81.15](https://huggingface.co/gchhablani/fnet-base-finetuned-mrpc) | [88.15](https://huggingface.co/gchhablani/bert-base-cased-finetuned-mrpc) | 76 |
115
+ | RTE | Accuracy | [62.82](https://huggingface.co/gchhablani/fnet-base-finetuned-rte) | [67.15](https://huggingface.co/gchhablani/bert-base-cased-finetuned-rte) | 63 |
116
+ | WNLI | Accuracy | [54.93](https://huggingface.co/gchhablani/fnet-base-finetuned-wnli) | [46.48](https://huggingface.co/gchhablani/bert-base-cased-finetuned-wnli) | - |
117
+ | Avg | - | - | - | 76.7 |
118
 
119
  We can see that FNet-base achieves around 93% of BERT-base's performance on average.
120