satyaalmasian commited on
Commit
eb10738
1 Parent(s): a9d2f6b

Update README.md

Browse files
Files changed (1) hide show
  1. README.md +4 -4
README.md CHANGED
@@ -1,12 +1,12 @@
1
  # RoBERTa2RoBERTa temporal tagger
2
 
3
- Seq2seq model for temporal tagging of plain text using RoBERTa language model. The model is introduced in the paper BERT got a Date: Introducing Transformers to Temporal Tagging and release in this [repository] (https://github.com/satya77/Transformer_Temporal_Tagger).
4
 
5
  # Model description
6
  RoBERTa is a transformers model pretrained on a large corpus of English data in a self-supervised fashion. We use RoBERTa in an encoder-decoder architecture for text generation, where the input is raw text and the output is the temporally annotated text. The model is pre-trained on a weakly annotated dataset from a rule-based system (HeidelTime) and fine-tuned on the temporal benchmark datasets (Wikiwars, Tweets, Tempeval-3).
7
 
8
  # Intended uses & limitations
9
- This model is best used accompanied with code from the [repository] (https://github.com/satya77/Transformer_Temporal_Tagger). Especially for inference, the direct output might be noisy and hard to decipher, in the repository we provide cleaning functions for the output and insert the temporal tags from the generated text in the input text. If you have temporally annotated data you can fine-tune this model.
10
 
11
  # How to use
12
  you can load the model as follows:
@@ -22,7 +22,7 @@ out = model.generate(**model_inputs)
22
  decoded_preds = tokenizer.batch_decode(out, skip_special_tokens=True)
23
 
24
  ```
25
- for an example with post-processing, refer to the [repository] (https://github.com/satya77/Transformer_Temporal_Tagger).
26
  to further fine-tune, use the `Seq2SeqTrainer` from hugginface. An example of a similar fine-tuning can be found [here](https://github.com/satya77/Transformer_Temporal_Tagger/blob/master/run_seq2seq_bert_roberta.py).
27
  ```
28
  trainer = Seq2SeqTrainer(
@@ -40,7 +40,7 @@ where the `training_args` is an instance of `Seq2SeqTrainingArguments`.
40
  #Training data
41
  We use four data sources:
42
  For Pretraining :1 million weakly annotated samples from heideltime. The samples are from news articles between the 1st January 2019 and the 30th July.
43
- Fine-tunning: [Tempeval-3](https://www.cs.york.ac.uk/semeval-2013/task1/index.php%3Fid=data.html), Wikiwars, Tweets datasets. For the correct data versions please refer to our [repository] (https://github.com/satya77/Transformer_Temporal_Tagger).
44
 
45
  #Training procedure
46
  The model is pre-trained on the weakly labeled data for $3$ epochs on the train set, from publicly available checkpoints on huggingface (`roberta-base`), with a batch size of 12. We use a learning rate of 5e-05 with an Adam optimizer and linear weight decay.
1
  # RoBERTa2RoBERTa temporal tagger
2
 
3
+ Seq2seq model for temporal tagging of plain text using RoBERTa language model. The model is introduced in the paper BERT got a Date: Introducing Transformers to Temporal Tagging and release in this [repository](https://github.com/satya77/Transformer_Temporal_Tagger).
4
 
5
  # Model description
6
  RoBERTa is a transformers model pretrained on a large corpus of English data in a self-supervised fashion. We use RoBERTa in an encoder-decoder architecture for text generation, where the input is raw text and the output is the temporally annotated text. The model is pre-trained on a weakly annotated dataset from a rule-based system (HeidelTime) and fine-tuned on the temporal benchmark datasets (Wikiwars, Tweets, Tempeval-3).
7
 
8
  # Intended uses & limitations
9
+ This model is best used accompanied with code from the [repository](https://github.com/satya77/Transformer_Temporal_Tagger). Especially for inference, the direct output might be noisy and hard to decipher, in the repository we provide cleaning functions for the output and insert the temporal tags from the generated text in the input text. If you have temporally annotated data you can fine-tune this model.
10
 
11
  # How to use
12
  you can load the model as follows:
22
  decoded_preds = tokenizer.batch_decode(out, skip_special_tokens=True)
23
 
24
  ```
25
+ for an example with post-processing, refer to the [repository](https://github.com/satya77/Transformer_Temporal_Tagger).
26
  to further fine-tune, use the `Seq2SeqTrainer` from hugginface. An example of a similar fine-tuning can be found [here](https://github.com/satya77/Transformer_Temporal_Tagger/blob/master/run_seq2seq_bert_roberta.py).
27
  ```
28
  trainer = Seq2SeqTrainer(
40
  #Training data
41
  We use four data sources:
42
  For Pretraining :1 million weakly annotated samples from heideltime. The samples are from news articles between the 1st January 2019 and the 30th July.
43
+ Fine-tunning: [Tempeval-3](https://www.cs.york.ac.uk/semeval-2013/task1/index.php%3Fid=data.html), Wikiwars, Tweets datasets. For the correct data versions please refer to our [repository](https://github.com/satya77/Transformer_Temporal_Tagger).
44
 
45
  #Training procedure
46
  The model is pre-trained on the weakly labeled data for $3$ epochs on the train set, from publicly available checkpoints on huggingface (`roberta-base`), with a batch size of 12. We use a learning rate of 5e-05 with an Adam optimizer and linear weight decay.