1 ---
2 language: "en"
3 tags:
4 - twitter
5 - stance-detection
6 - election2020
7 license: "gpl-3.0"
8 ---
9
10 # Pre-trained BERT on Twitter US Election 2020 for Stance Detection towards Joe Biden (f-BERT)
11
12 Pre-trained weights for **f-BERT** in [Knowledge Enhance Masked Language Model for Stance Detection](https://www.aclweb.org/anthology/2021.naacl-main.376), NAACL 2021.
13
14 # Training Data
15
16 This model is pre-trained on over 5 million English tweets about the 2020 US Presidential Election. Then fine-tuned using our [stance-labeled data](https://github.com/GU-DataLab/stance-detection-KE-MLM) for stance detection towards Joe Biden.
17
18 # Training Objective
19
20 This model is initialized with BERT-base and trained with normal MLM objective with classification layer fine-tuned for stance detection towards Joe Biden.
21
22 # Usage
23
24 This pre-trained language model is fine-tuned to the stance detection task specifically for Joe Biden.
25
26 Please see the [official repository](https://github.com/GU-DataLab/stance-detection-KE-MLM) for more detail.
27
28 ```python
29 from transformers import AutoTokenizer, AutoModelForSequenceClassification
30 import torch
31 import numpy as np
32
33 # choose GPU if available
34 device = torch.device("cuda" if torch.cuda.is_available() else "cpu")
35
36 # select mode path here
37 pretrained_LM_path = "kornosk/bert-election2020-twitter-stance-biden"
38
39 # load model
40 tokenizer = AutoTokenizer.from_pretrained(pretrained_LM_path)
41 model = AutoModelForSequenceClassification.from_pretrained(pretrained_LM_path)
42
43 id2label = {
44 0: "AGAINST",
45 1: "FAVOR",
46 2: "NONE"
47 }
48
49 ##### Prediction Neutral #####
50 sentence = "Hello World."
51 inputs = tokenizer(sentence.lower(), return_tensors="pt")
52 outputs = model(**inputs)
53 predicted_probability = torch.softmax(outputs[0], dim=1)[0].tolist()
54
55 print("Sentence:", sentence)
56 print("Prediction:", id2label[np.argmax(predicted_probability)])
57 print("Against:", predicted_probability[0])
58 print("Favor:", predicted_probability[1])
59 print("Neutral:", predicted_probability[2])
60
61 ##### Prediction Favor #####
62 sentence = "Go Go Biden!!!"
63 inputs = tokenizer(sentence.lower(), return_tensors="pt")
64 outputs = model(**inputs)
65 predicted_probability = torch.softmax(outputs[0], dim=1)[0].tolist()
66
67 print("Sentence:", sentence)
68 print("Prediction:", id2label[np.argmax(predicted_probability)])
69 print("Against:", predicted_probability[0])
70 print("Favor:", predicted_probability[1])
71 print("Neutral:", predicted_probability[2])
72
73 ##### Prediction Against #####
74 sentence = "Biden is the worst."
75 inputs = tokenizer(sentence.lower(), return_tensors="pt")
76 outputs = model(**inputs)
77 predicted_probability = torch.softmax(outputs[0], dim=1)[0].tolist()
78
79 print("Sentence:", sentence)
80 print("Prediction:", id2label[np.argmax(predicted_probability)])
81 print("Against:", predicted_probability[0])
82 print("Favor:", predicted_probability[1])
83 print("Neutral:", predicted_probability[2])
84
85 # please consider citing our paper if you feel this is useful :)
86 ```
87
88 # Reference
89
90 - [Knowledge Enhance Masked Language Model for Stance Detection](https://www.aclweb.org/anthology/2021.naacl-main.376), NAACL 2021.
91
92 # Citation
93 ```bibtex
94 @inproceedings{kawintiranon2021knowledge,
95 title={Knowledge Enhanced Masked Language Model for Stance Detection},
96 author={Kawintiranon, Kornraphop and Singh, Lisa},
97 booktitle={Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies},
98 year={2021},
99 publisher={Association for Computational Linguistics},
100 url={https://www.aclweb.org/anthology/2021.naacl-main.376}
101 }
102 ```