satyaalmasian commited on
Commit
ca4804f
1 Parent(s): c4cc163

Create README.md

Browse files
Files changed (1) hide show
  1. README.md +54 -0
README.md ADDED
@@ -0,0 +1,54 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ # BERT based temporal tagged
2
+
3
+ Token classifier for temporal tagging of plain text using BERT language model and CRFs. The model is introduced in the paper BERT got a Date: Introducing Transformers to Temporal Tagging and release in this [repository](https://github.com/satya77/Transformer_Temporal_Tagger).
4
+
5
+ # Model description
6
+ BERT is a transformers model pretrained on a large corpus of English data in a self-supervised fashion. We use BERT for token classification to tag the tokens in text with classes:
7
+ ```
8
+ O -- outside of a tag
9
+ I-TIME -- inside tag of time
10
+ B-TIME -- beginning tag of time
11
+ I-DATE -- inside tag of date
12
+ B-DATE -- beginning tag of date
13
+ I-DURATION -- inside tag of duration
14
+ B-DURATION -- beginning tag of duration
15
+ I-SET -- inside tag of the set
16
+ B-SET -- beginning tag of the set
17
+ ```
18
+ On top of the BERT classification layer, we add a custom CRF layer. This is a variant of `satyaalmasian/temporal_tagger_BERT_tokenclassifier` with slightly better
19
+ performance but can not be used out of the box with huggingface models and needs the code from the accompanying [repository](https://github.com/satya77/Transformer_Temporal_Tagger).
20
+
21
+ # Intended uses & limitations
22
+ This model is best used accompanied with code from the [repository](https://github.com/satya77/Transformer_Temporal_Tagger). Especially for inference, the direct output might be noisy and hard to decipher, in the repository we provide alignment functions and voting strategies for the final output.
23
+
24
+ # How to use
25
+ you can load the model as follows:
26
+ ```
27
+ tokenizer = AutoTokenizer.from_pretrained("satyaalmasian/temporal_tagger_BERTCRF_tokenclassifier", use_fast=False)
28
+ model = BertForTokenClassification.from_pretrained("satyaalmasian/temporal_tagger_BERTCRF_tokenclassifier")
29
+
30
+ ```
31
+ for inference use:
32
+ ```
33
+ processed_text = tokenizer(input_text, return_tensors="pt")
34
+ processed_text["inference_mode"]=True
35
+ result = model(**processed_text)
36
+ classification= result[0]
37
+
38
+ ```
39
+ for an example with post-processing, refer to the [repository](https://github.com/satya77/Transformer_Temporal_Tagger).
40
+ We provide a function `merge_tokens` to decipher the output.
41
+ to further fine-tune, use the `Trainer` from hugginface. An example of a similar fine-tuning can be found [here](https://github.com/satya77/Transformer_Temporal_Tagger/blob/master/run_token_classifier.py).
42
+
43
+ #Training data
44
+ We use 3 data sources:
45
+ [Tempeval-3](https://www.cs.york.ac.uk/semeval-2013/task1/index.php%3Fid=data.html), Wikiwars, Tweets datasets. For the correct data versions please refer to our [repository](https://github.com/satya77/Transformer_Temporal_Tagger).
46
+
47
+ #Training procedure
48
+ The model is trained from publicly available checkpoints on huggingface (`bert-base-uncased`), with a batch size of 34. We use a learning rate of 5e-05 with an Adam optimizer and linear weight decay.
49
+ We fine-tune with 5 different random seeds, this version of the model is the only seed=19.
50
+ For training, we use 2 NVIDIA A100 GPUs with 40GB of memory.
51
+
52
+
53
+
54
+