nielsr
HF staff
commited on
Commit
56e7cb9
1 Parent(s): 823de39

First draft of model card

Browse files
Files changed (1) hide show
  1. README.md +103 -0
README.md ADDED
@@ -0,0 +1,103 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ ---
2
+ language: en
3
+ tags:
4
+ - tapas
5
+ - sequence-classification
6
+ license: apache-2.0
7
+ datasets:
8
+ - tab_fact
9
+
10
+ ---
11
+
12
+ # TAPAS tiny model fine-tuned on Tabular Fact Checking (TabFact)
13
+
14
+ This model has 2 versions which can be used. The latest version, which is the default one, corresponds to the `tapas_tabfact_inter_masklm_tiny_reset` checkpoint of the [original Github repository](https://github.com/google-research/tapas).
15
+ This model was pre-trained on MLM and an additional step which the authors call intermediate pre-training, and then fine-tuned on [TabFact](https://github.com/wenhuchen/Table-Fact-Checking). It uses relative position embeddings by default (i.e. resetting the position index at every cell of the table).
16
+
17
+ The other (non-default) version which can be used is the one with absolute position embeddings:
18
+ - `no_reset`, which corresponds to `tapas_tabfact_inter_masklm_tiny`
19
+
20
+ Disclaimer: The team releasing TAPAS did not write a model card for this model so this model card has been written by
21
+ the Hugging Face team and contributors.
22
+
23
+ ## Model description
24
+
25
+ TAPAS is a BERT-like transformers model pretrained on a large corpus of English data from Wikipedia in a self-supervised fashion.
26
+ This means it was pretrained on the raw tables and associated texts only, with no humans labelling them in any way (which is why it
27
+ can use lots of publicly available data) with an automatic process to generate inputs and labels from those texts. More precisely, it
28
+ was pretrained with two objectives:
29
+
30
+ - Masked language modeling (MLM): taking a (flattened) table and associated context, the model randomly masks 15% of the words in
31
+ the input, then runs the entire (partially masked) sequence through the model. The model then has to predict the masked words.
32
+ This is different from traditional recurrent neural networks (RNNs) that usually see the words one after the other,
33
+ or from autoregressive models like GPT which internally mask the future tokens. It allows the model to learn a bidirectional
34
+ representation of a table and associated text.
35
+ - Intermediate pre-training: to encourage numerical reasoning on tables, the authors additionally pre-trained the model by creating
36
+ a balanced dataset of millions of syntactically created training examples. Here, the model must predict (classify) whether a sentence
37
+ is supported or refuted by the contents of a table. The training examples are created based on synthetic as well as counterfactual statements.
38
+
39
+ This way, the model learns an inner representation of the English language used in tables and associated texts, which can then be used
40
+ to extract features useful for downstream tasks such as answering questions about a table, or determining whether a sentence is entailed
41
+ or refuted by the contents of a table. Fine-tuning is done by adding a classification head on top of the pre-trained model, and then
42
+ jointly train this randomly initialized classification head with the base model on TabFact.
43
+
44
+
45
+ ## Intended uses & limitations
46
+
47
+ You can use this model for classifying whether a sentence is supported or refuted by the contents of a table.
48
+
49
+ For code examples, we refer to the documentation of TAPAS on the HuggingFace website.
50
+
51
+
52
+ ## Training procedure
53
+
54
+ ### Preprocessing
55
+
56
+ The texts are lowercased and tokenized using WordPiece and a vocabulary size of 30,000. The inputs of the model are
57
+ then of the form:
58
+
59
+ ```
60
+ [CLS] Sentence [SEP] Flattened table [SEP]
61
+ ```
62
+
63
+ ### Fine-tuning
64
+
65
+ The model was fine-tuned on 32 Cloud TPU v3 cores for 80,000 steps with maximum sequence length 512 and batch size of 512.
66
+ In this setup, fine-tuning takes around 14 hours. The optimizer used is Adam with a learning rate of 2e-5, and a warmup
67
+ ratio of 0.05. See the [paper](https://arxiv.org/abs/2010.00571) for more details (appendix A2).
68
+
69
+
70
+ ### BibTeX entry and citation info
71
+
72
+ ```bibtex
73
+ @misc{herzig2020tapas,
74
+ title={TAPAS: Weakly Supervised Table Parsing via Pre-training},
75
+ author={Jonathan Herzig and Paweł Krzysztof Nowak and Thomas Müller and Francesco Piccinno and Julian Martin Eisenschlos},
76
+ year={2020},
77
+ eprint={2004.02349},
78
+ archivePrefix={arXiv},
79
+ primaryClass={cs.IR}
80
+ }
81
+ ```
82
+
83
+ ```bibtex
84
+ @misc{eisenschlos2020understanding,
85
+ title={Understanding tables with intermediate pre-training},
86
+ author={Julian Martin Eisenschlos and Syrine Krichene and Thomas Müller},
87
+ year={2020},
88
+ eprint={2010.00571},
89
+ archivePrefix={arXiv},
90
+ primaryClass={cs.CL}
91
+ }
92
+ ```
93
+
94
+ ```bibtex
95
+ @inproceedings{2019TabFactA,
96
+ title={TabFact : A Large-scale Dataset for Table-based Fact Verification},
97
+ author={Wenhu Chen, Hongmin Wang, Jianshu Chen, Yunkai Zhang, Hong Wang, Shiyang Li, Xiyou Zhou and William Yang Wang},
98
+ booktitle = {International Conference on Learning Representations (ICLR)},
99
+ address = {Addis Ababa, Ethiopia},
100
+ month = {April},
101
+ year = {2020}
102
+ }
103
+ ```