CASE 2021 Task 2: Zero-Shot Classification of Fine-Grained Sociopolitical Events with Transformer Models (2021)

Abstract: We introduce a method for the classification of texts into fine-grained categories of sociopolitical events. This particular method is responsive to all three Subtasks of Task 2, Fine-Grained Classification of Socio-Political Events, introduced at the CASE workshop of ACL-IJCNLP 2021. We frame Task 2 as textual entailment: given an input text and a candidate event class (“query”), the model predicts whether the text describes an event of the given type. The model is able to correctly classify in-sample event types with an average F1-score of 0.74 but struggles with some out-of-sample event types. Despite this, the model shows promise for the zero-shot identification of certain sociopolitical events by achieving an F1-score of 0.52 on one wholly out-of-sample event class.

## introduction we introduce a method for the classification of text excerpts into fine-grained categories of sociopolitical events. this particular method is responsive to all three subtasks of task 2, fine-grained classification of socio-political events, introduced at the challenges and applications of automated extraction of socio-political events from text (case) workshop of acl-ijcnlp 2021 @xcite . we frame task 2 as textual entailment: given an input text and a candidate event class ("query"), the model predicts whether the text describes an event of the given type. because the query is given as an arbitrary sequence of character tokens, the model is open-ended with respect to query and can, theoretically, predict classes completely out-of-sample. three shared task challenges were introduced at case: (1) multilingual protest news detection, (2) fine-grained classification of socio-political events, and (3) discovering black lives matter events in united states. the second of these is further divided into three subtasks: (1) supervised text classification of 25 event types, (2) unsupervised text classification of three additional event types, and ( 3 ) unsupervised text classification of a further two additional event types. no training data were provided by the shared task organizers; teams were given only the 25 initial event type descriptions. these event types were drawn from the armed conflict location and event data project (acled) event ontology @xcite . the subsequent five event types introduced by subtasks 2 and 3 were provided by the shared task organizers immediately prior to the response submission deadline. ## data we downloaded all acled events and the corresponding source texts within which those events were discovered. source texts are short excerpts from news articles and are typically no more than a few sentences in length. we use this event-text corpus as training data for our model. these events and sentences represent only the 25 event types of subtask 1. event types by subtask are given in the second column of to generate training data for the model, we pair every text excerpt with all 24 of the subtask 1 event types that are not described by the excerpt and assign these artificial pairs a value of zero. we then assign all observed pairs, text excerpts paired with the correct event type, a value of one. the model's job is to take a text-query pair and predict whether it is a correct pair or an incorrect pair. an example text-query pair from an acled event is given in figure 1 . ## model we select a pre-trained roberta model as the base of our solution. foot_1 roberta is a transformerbased language model that is initially trained on a very large english language corpus and can then be fine-tuned to specific tasks with fewer training examples @xcite . we take the final layer hidden states for each token and apply global max pooling (i.e., find the element-wise maximum for each dimension of the hidden states). we add a fully-connected dense layer with a single neuron and sigmoid activation function to this pooled value. we use adam to minimize the binary cross-entropy loss of our model @xcite . we train the model for a single epoch with a learning rate of 5 × 10 -5 and use a variable batch size to manage memory usage. during the inference stage, the model must select a single class to best represent each text. all possible queries are appended to an input text and every pair is passed to the model independently. the model produces a prediction between 0.0 and 1.0 for each pair and the event class associated with the text-query pair that receives the highest predicted value is chosen. however, the other category may result in misclassifications: can the model distinguish an out-of-sample class, like those from subtasks 2 and 3, from the in-sample class other? a second aggregation rule is therefore applied: if the greatest predicted value is associated with the other class, the next highest probability class is inspected. if this runner-up class is out-ofsample (i.e., not present in subtask 1), then it is chosen. if the runner-up class is present in subtask 1, then the class other is chosen. results presented for subtasks 2 and 3 are derived from this second aggregation method. ## results comparison of predictions against target classes reveals that class overlap may to be blame for some of the poor out-of-sample performance. for example, the model correctly identifies organized crime only 10% of the time and often misclassifies it as arrest (21%), mob violence (21%), and looting property destruction (17%). one example of this, drawn from the test set, is given in ## conclusion failure to account for ambiguity between event classes is likely to be an issue for the next generation of automated fine-grained event classification models. in the case of the model presented here, predictions are not necessarily calibrated properly: the model has no ability to specify that a text does not describe one and only one event type. this is enforced by the fact that a final classification is chosen by identifying the maximum value among all text-query pair predictions. were the model calibrated by class, we would hope that predicted values greater than 0.5 denote a positive class membership and values below this threshold denote nonmembership. in that case, multiple classes (or no class) could be indicated by the model for a single text. however, given the zero-shot nature of subtasks 2 and 3, we were unable to calibrate those particular classes. furthermore, the organizers have specified that all texts should be assigned one and only one label. however, it seems clear from inspection of the errors that the given ontology does not describe a mutually exclusive set of classes. accounting for hierarchical or complementary classes within the ontology may help to produce more useful or consistent event coding models. doing so will require a novel technique for selecting predicted classes in which each class prediction is not made independently of the other classes (as is the case here). one solution may be to pose all queries to the model simultaneously. a single input example would comprise the source text concatenated with every possible event class: <@xmath1... 
. the model would then output a vector of probabilities the same length (in tokens) as the input sequence. classes for the source text would be chosen by inspecting this probability vector and selecting categories corresponding to relatively high probability-valued sub-sequences. when appropriate, the model may weight multiple (or no) class tokens very highly. queries could be shuffled per source text to prevent the model from learning offset values for common classes rather than attending to the query texts themselves. despite the poor out-of-sample performance of this particular model on certain zero-shot event categories, the model's performance in-sample and on natural disaster and diplomatic event suggests that transformers will play a major role in future event coding systems. with additional time and resources, it is likely that substantial improvements are possible to the model described here. in fact, the performance of this model, given zero hyper-parameter tuning or model search, suggests that the upper limit for transformer performance on this task is likely very high. 2