crisis_emotion_roberta

This emotion classification model is a fine-tuned version of finiteautomata/bertweet-base-sentiment-analysis on a dataset of 9,300 tweets in the Flint Water Crisis (Wu, Wong, Zhao, & Liu, 2021). It achieves the following results on the testing set: 0.75 accuracy, 0.74 weighted accuracy, and 0.68 macro accuracy.

Classify the primary emotion of a crisis tweet into one of the following 6 categories (The F-1 score for each emotion):

0. Anger (0.83)
1. Sadness (0.67)
2. Joy (0.69)
3. Sympathy (0.80)
4. Sarcasm (0.44)
5. Neutral (0.64)

To cite our work: Wu, J., Wong, C.-W., Zhao, X., & Liu, X. (2021). Toward effective automated content analysis via crowdsourcing. Paper presented at the IEEE International Conference on Multimedia and Expo (ICME). https://doi.org/10.1109/ICME51207.2021.9428220

Intended uses & limitations

For classifying the emotion of English tweets during crises & disasters

Training and evaluation data

Dataset: 9,300 tweets in the Flint water crisis. Each tweet was labeled by trained & qualified crowdsourcing workers for 3-5 times. For detail, see our IEEE ICME paper - Wu, Wong, Zhao, & Liu, 2021. (https://arxiv.org/pdf/2101.04615.pdf)

Training procedure

Training hyperparameters

The following hyperparameters were used during training:

• learning_rate: 2e-05
• train_batch_size: 16
• eval_batch_size: 16
• seed: 42
• optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-08
• lr_scheduler_type: linear
• num_epochs: 15

Training results

Training Loss	Epoch	Step	Validation Loss	Accuracy
0.7558	1.0	349	0.8849	0.6839
0.7716	2.0	698	0.8137	0.7306
1.0935	3.0	1047	0.8435	0.7333
0.4497	4.0	1396	0.9084	0.7371
0.3247	5.0	1745	1.0200	0.7355
0.0225	6.0	2094	1.1517	0.7344
0.2034	7.0	2443	1.2812	0.7333
0.0224	8.0	2792	1.4054	0.7258
0.008	9.0	3141	1.4090	0.7242
0.0067	10.0	3490	1.4884	0.7204
0.4066	11.0	3839	1.5450	0.7220
0.0033	12.0	4188	1.6056	0.7247
0.003	13.0	4537	1.6327	0.7247
0.0037	14.0	4886	1.6871	0.7285
0.0025	15.0	5235	1.6898	0.7274

Framework versions

• Transformers 4.23.0.dev0
• Pytorch 1.13.0.dev20220917+cu117
• Datasets 2.4.0
• Tokenizers 0.12.1