nielsr commited on
Commit
4dee185
1 Parent(s): bc48b07

First draft of model card

Browse files
Files changed (1) hide show
  1. README.md +88 -0
README.md ADDED
@@ -0,0 +1,88 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ ---
2
+ language: en
3
+ tags:
4
+ - tapas
5
+ - sequence-classification
6
+ license: apache-2.0
7
+ ---
8
+
9
+ # TAPAS base model
10
+
11
+ This model has 2 versions which can be used. The latest version, which is the default one, corresponds to the `tapas_inter_masklm_base_reset` checkpoint of the [original Github repository](https://github.com/google-research/tapas).
12
+ This model was pre-trained on MLM and an additional step which the authors call intermediate pre-training. It uses relative position embeddings by default (i.e. resetting the position index at every cell of the table).
13
+
14
+ The other (non-default) version which can be used is the one with absolute position embeddings:
15
+ - `revision="no_reset"`, which corresponds to `tapas_inter_masklm_base`
16
+
17
+ Disclaimer: The team releasing TAPAS did not write a model card for this model so this model card has been written by
18
+ the Hugging Face team and contributors.
19
+
20
+ ## Model description
21
+
22
+ TAPAS is a BERT-like transformers model pretrained on a large corpus of English data from Wikipedia in a self-supervised fashion.
23
+ This means it was pretrained on the raw tables and associated texts only, with no humans labelling them in any way (which is why it
24
+ can use lots of publicly available data) with an automatic process to generate inputs and labels from those texts. More precisely, it
25
+ was pretrained with two objectives:
26
+
27
+ - Masked language modeling (MLM): taking a (flattened) table and associated context, the model randomly masks 15% of the words in
28
+ the input, then runs the entire (partially masked) sequence through the model. The model then has to predict the masked words.
29
+ This is different from traditional recurrent neural networks (RNNs) that usually see the words one after the other,
30
+ or from autoregressive models like GPT which internally mask the future tokens. It allows the model to learn a bidirectional
31
+ representation of a table and associated text.
32
+ - Intermediate pre-training: to encourage numerical reasoning on tables, the authors additionally pre-trained the model by creating
33
+ a balanced dataset of millions of syntactically created training examples. Here, the model must predict (classify) whether a sentence
34
+ is supported or refuted by the contents of a table. The training examples are created based on synthetic as well as counterfactual statements.
35
+
36
+ This way, the model learns an inner representation of the English language used in tables and associated texts, which can then be used
37
+ to extract features useful for downstream tasks such as answering questions about a table, or determining whether a sentence is entailed
38
+ or refuted by the contents of a table. Fine-tuning is done by adding one or more classification heads on top of the pre-trained model, and then
39
+ jointly train these randomly initialized classification heads with the base model on a downstream task.
40
+
41
+
42
+ ## Intended uses & limitations
43
+
44
+ You can use the raw model for getting hidden representatons about table-question pairs, but it's mostly intended to be fine-tuned on a downstream task such as question answering or sequence classification. See the [model hub](https://huggingface.co/models?filter=tapas) to look for fine-tuned versions on a task that interests you.
45
+
46
+
47
+ ## Training procedure
48
+
49
+ ### Preprocessing
50
+
51
+ The texts are lowercased and tokenized using WordPiece and a vocabulary size of 30,000. The inputs of the model are
52
+ then of the form:
53
+
54
+ ```
55
+ [CLS] Sentence [SEP] Flattened table [SEP]
56
+ ```
57
+
58
+ ### Pre-training
59
+
60
+ The model was pre-trained on 32 Cloud TPU v3 cores for 1,000,000 steps with maximum sequence length 512 and batch size of 512.
61
+ In this setup, pre-training on MLM only takes around 3 days. Aditionally, the model has been further pre-trained on a second task (table entailment). See the original TAPAS [paper](https://www.aclweb.org/anthology/2020.acl-main.398/) and the [follow-up paper](https://www.aclweb.org/anthology/2020.findings-emnlp.27/) for more details.
62
+
63
+ The optimizer used is Adam with a learning rate of 5e-5, and a warmup
64
+ ratio of 0.01.
65
+
66
+ ### BibTeX entry and citation info
67
+
68
+ ```bibtex
69
+ @misc{herzig2020tapas,
70
+ title={TAPAS: Weakly Supervised Table Parsing via Pre-training},
71
+ author={Jonathan Herzig and Paweł Krzysztof Nowak and Thomas Müller and Francesco Piccinno and Julian Martin Eisenschlos},
72
+ year={2020},
73
+ eprint={2004.02349},
74
+ archivePrefix={arXiv},
75
+ primaryClass={cs.IR}
76
+ }
77
+ ```
78
+
79
+ ```bibtex
80
+ @misc{eisenschlos2020understanding,
81
+ title={Understanding tables with intermediate pre-training},
82
+ author={Julian Martin Eisenschlos and Syrine Krichene and Thomas Müller},
83
+ year={2020},
84
+ eprint={2010.00571},
85
+ archivePrefix={arXiv},
86
+ primaryClass={cs.CL}
87
+ }
88
+ ```