nielsr
HF staff
commited on
Commit
91b5760
1 Parent(s): 886bc77

First draft of model card

Browse files
Files changed (1) hide show
  1. README.md +111 -0
README.md ADDED
@@ -0,0 +1,111 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ ---
2
+ language: en
3
+ tags:
4
+ - tapas
5
+ - question-answering
6
+ license: apache-2.0
7
+ datasets:
8
+ - wtq
9
+ ---
10
+
11
+ # TAPAS base model fine-tuned on WikiTable Questions (WTQ)
12
+
13
+ This model has 2 versions which can be used. The default version corresponds to the `tapas_wtq_wikisql_sqa_inter_masklm_base_reset` checkpoint of the [original Github repository](https://github.com/google-research/tapas).
14
+ This model was pre-trained on MLM and an additional step which the authors call intermediate pre-training, and then fine-tuned in a chain on [SQA](https://www.microsoft.com/en-us/download/details.aspx?id=54253), [WikiSQL](https://github.com/salesforce/WikiSQL) and finally [WTQ](https://github.com/ppasupat/WikiTableQuestions). It uses relative position embeddings (i.e. resetting the position index at every cell of the table).
15
+
16
+ The other (non-default) version which can be used is:
17
+ - `no_reset`, which corresponds to `tapas_wtq_wikisql_sqa_inter_masklm_base` (intermediate pre-training, absolute position embeddings).
18
+
19
+ Disclaimer: The team releasing TAPAS did not write a model card for this model so this model card has been written by
20
+ the Hugging Face team and contributors.
21
+
22
+ ## Model description
23
+
24
+ TAPAS is a BERT-like transformers model pretrained on a large corpus of English data from Wikipedia in a self-supervised fashion.
25
+ This means it was pretrained on the raw tables and associated texts only, with no humans labelling them in any way (which is why it
26
+ can use lots of publicly available data) with an automatic process to generate inputs and labels from those texts. More precisely, it
27
+ was pretrained with two objectives:
28
+
29
+ - Masked language modeling (MLM): taking a (flattened) table and associated context, the model randomly masks 15% of the words in
30
+ the input, then runs the entire (partially masked) sequence through the model. The model then has to predict the masked words.
31
+ This is different from traditional recurrent neural networks (RNNs) that usually see the words one after the other,
32
+ or from autoregressive models like GPT which internally mask the future tokens. It allows the model to learn a bidirectional
33
+ representation of a table and associated text.
34
+ - Intermediate pre-training: to encourage numerical reasoning on tables, the authors additionally pre-trained the model by creating
35
+ a balanced dataset of millions of syntactically created training examples. Here, the model must predict (classify) whether a sentence
36
+ is supported or refuted by the contents of a table. The training examples are created based on synthetic as well as counterfactual statements.
37
+
38
+ This way, the model learns an inner representation of the English language used in tables and associated texts, which can then be used
39
+ to extract features useful for downstream tasks such as answering questions about a table, or determining whether a sentence is entailed
40
+ or refuted by the contents of a table. Fine-tuning is done by adding a cell selection head and aggregation head on top of the pre-trained model, and then jointly train these randomly initialized classification heads with the base model on SQa, WikiSQL and finally WTQ.
41
+
42
+
43
+ ## Intended uses & limitations
44
+
45
+ You can use this model for answering questions related to a table.
46
+
47
+ For code examples, we refer to the documentation of TAPAS on the HuggingFace website.
48
+
49
+
50
+ ## Training procedure
51
+
52
+ ### Preprocessing
53
+
54
+ The texts are lowercased and tokenized using WordPiece and a vocabulary size of 30,000. The inputs of the model are
55
+ then of the form:
56
+
57
+ ```
58
+ [CLS] Question [SEP] Flattened table [SEP]
59
+ ```
60
+
61
+ The authors did first convert the WTQ dataset into the format of SQA using automatic conversion scripts.
62
+
63
+ ### Fine-tuning
64
+
65
+ The model was fine-tuned on 32 Cloud TPU v3 cores for 50,000 steps with maximum sequence length 512 and batch size of 512.
66
+ In this setup, fine-tuning takes around 10 hours. The optimizer used is Adam with a learning rate of 1.93581e-5, and a warmup
67
+ ratio of 0.128960. An inductive bias is added such that the model only selects cells of the same column. This is reflected by the
68
+ `select_one_column` parameter of `TapasConfig`. See the [paper](https://arxiv.org/abs/2004.02349) for more details (tables 11 and
69
+ 12).
70
+
71
+
72
+ ### BibTeX entry and citation info
73
+
74
+ ```bibtex
75
+ @misc{herzig2020tapas,
76
+ title={TAPAS: Weakly Supervised Table Parsing via Pre-training},
77
+ author={Jonathan Herzig and Paweł Krzysztof Nowak and Thomas Müller and Francesco Piccinno and Julian Martin Eisenschlos},
78
+ year={2020},
79
+ eprint={2004.02349},
80
+ archivePrefix={arXiv},
81
+ primaryClass={cs.IR}
82
+ }
83
+ ```
84
+
85
+ ```bibtex
86
+ @misc{eisenschlos2020understanding,
87
+ title={Understanding tables with intermediate pre-training},
88
+ author={Julian Martin Eisenschlos and Syrine Krichene and Thomas Müller},
89
+ year={2020},
90
+ eprint={2010.00571},
91
+ archivePrefix={arXiv},
92
+ primaryClass={cs.CL}
93
+ }
94
+ ```
95
+
96
+ ```bibtex
97
+ @article{DBLP:journals/corr/PasupatL15,
98
+ author = {Panupong Pasupat and
99
+ Percy Liang},
100
+ title = {Compositional Semantic Parsing on Semi-Structured Tables},
101
+ journal = {CoRR},
102
+ volume = {abs/1508.00305},
103
+ year = {2015},
104
+ url = {http://arxiv.org/abs/1508.00305},
105
+ archivePrefix = {arXiv},
106
+ eprint = {1508.00305},
107
+ timestamp = {Mon, 13 Aug 2018 16:47:37 +0200},
108
+ biburl = {https://dblp.org/rec/journals/corr/PasupatL15.bib},
109
+ bibsource = {dblp computer science bibliography, https://dblp.org}
110
+ }
111
+ ```