eventdata-utd

Model Name

ConfliBERT: A Pre-trained Language Model for Political Conflict and Violence

Developers

Yibo Hu, MohammadSaleh Hosseini, Erick Skorupa Parolin, Javier Osorio, Latifur Khan, Patrick Brandt, Vito D’Orazio

Released

2022, NAACL 2022 conference

Repository

GitHub Repository

Paper

ConfliBERT: A Pre-trained Language Model for Political Conflict and Violence

Model Description

ConfliBERT is a transformer model pretrained on a vast corpus of texts related to political conflict and violence. This model is based on the BERT architecture and is specialized for analyzing texts within its domain, using masked language modeling (MLM) and next sentence prediction (NSP) as its main pretraining objectives. It is designed to improve performance in tasks like event extraction, and entity recognition for texts dealing with political subjects.

Analyzing conflicts and political violence around the world is a persistent challenge in the political science and policy communities due in large part to the vast volumes of specialized text needed to monitor conflict and violence on a global scale. To help advance research in political science, we introduce ConfliBERT, a domain-specific pre-trained language model for conflict and political violence. We first gather a large domain-specific text corpus for language modeling from various sources. We then build ConfliBERT using two approaches: pre-training from scratch and continual pre-training. To evaluate ConfliBERT, we collect 12 datasets and implement 18 tasks to assess the models’ practical application in conflict research. Finally, we evaluate several versions of ConfliBERT in multiple experiments. Results consistently show that ConfliBERT outperforms BERT when analyzing political violence and conflict.

Model Variants

ConfliBERT has several variants, each fine-tuned on specific datasets to cater to different use cases within the domain of political conflict and violence:

• ConfliBERT-scr-uncased-BBC_News (Binary Classification)
• ConfliBERT-cont-cased-BBC_News (Binary Classification)
• ConfliBERT-scr-uncased-20news (Binary Classification)
• ConfliBERT-cont-cased-20news (Binary Classification)
• ConfliBERT-re3d-ner (Named Entity Recognition)
• ConfliBERT-indiapolice-events-multilabel (Text Classification)
• ConfliBERT-named-entity-recognition (Named Entity Recognition)
• ConfliBERT-insight-crime-multilabel (Text Classification)

These models are fine-tuned versions intended for specific text classification and named entity recognition tasks, enhancing their effectiveness in practical applications.

Intended Uses & Limitations

ConfliBERT is intended for use in tasks related to its training domain (political conflict and violence). It can be used for masked language modeling or next sentence prediction and is particularly useful when fine-tuned on downstream tasks such as classification or information extraction in political contexts.

How to Use

To load and use a specific ConfliBERT model variant, you can follow these steps using the transformers library:

from transformers import AutoTokenizer, AutoModelForMaskedLM

# Example for using the ConfliBERT-scr-uncased-BBC_News model
tokenizer = AutoTokenizer.from_pretrained("eventdata-utd/ConfliBERT-scr-uncased-BBC_News")
model = AutoModelForMaskedLM.from_pretrained("eventdata-utd/ConfliBERT-scr-uncased-BBC_News")

# Example of usage for masking task
text = "The government of [MASK] was overthrown in a coup."
input_ids = tokenizer.encode(text, return_tensors='pt')
outputs = model(input_ids)

# Example for using the ConfliBERT-cont-cased-20news model
tokenizer = AutoTokenizer.from_pretrained("eventdata-utd/ConfliBERT-cont-cased-20news")
model = AutoModelForMaskedLM.from_pretrained("eventdata-utd/ConfliBERT-cont-cased-20news")

# Example of usage for binary classification task
text = "The Prime Minister of Brunei asked for protestors to remain peaceful during the upcoming Independence Day holiday."
input_ids = tokenizer.encode(text, return_tensors='pt')
outputs = model(input_ids)

Limitations and Bias

While ConfliBERT is pretrained on data related to political conflicts, it may inherit biases present in its training corpus or exhibit limitations in understanding contexts outside its trained domain. As with any model, users should evaluate its fairness and suitability for their specific applications.

Training Data

ConfliBERT was trained on a specialized corpus of 33 GB of texts about politics and conflict, curated to provide comprehensive coverage of its intended application domain. This corpus includes diverse sources such as news articles, reports, and books related to global political events and conflicts.

Training Procedure

The model was pretrained using masked language modeling and next sentence prediction tasks, following procedures similar to those used for BERT. Specific training details, including configurations and scripts, are available in the model's GitHub repository.

Evaluation Results

ConfliBERT has shown improved performance on several benchmarks relevant to its domain compared to general-purpose language models like BERT, especially in tasks that require understanding of political contexts.

Citation

If you use ConfliBERT in your research, please cite the following paper:

@inproceedings{hu2022conflibert,
  title={Conflibert: A pre-trained language model for political conflict and violence},
  author={Hu, Yibo and Hosseini, MohammadSaleh and Skorupa Parolin, Erick and Osorio, Javier and Khan, Latifur and Brandt, Patrick and D’Orazio, Vito},
  year={2022},
  organization={Association for Computational Linguistics},
  doi={10.18653/v1/2022.naacl-main.400}
}