Zero-Shot Ranking Socio-Political Texts with Transformer Language Models to Reduce Close Reading Time (2022)

Abstract: We approach the classification problem as an entailment problem and apply zero-shot ranking to socio-political texts. Documents that are ranked at the top can be considered positively classified documents and this reduces the close reading time for the information extraction process. We use Transformer Language Models to get the entailment probabilities and investigate different types of queries. We find that DeBERTa achieves higher mean average precision scores than RoBERTa and when declarative form of the class label is used as a query, it outperforms dictionary definition of the class label. We show that one can reduce the close reading time by taking some percentage of the ranked documents that the percentage depends on how much recall they want to achieve. However, our findings also show that percentage of the documents that should be read increases as the topic gets broader.

## introduction for the information retrieval process positively labeled documents in a dataset are important and should not be missed, therefore achieving high recall is extremely important. however, there is generally a large number of documents that are relevant or not to the concerned topic and doing close reading for all documents and annotating them requires lots of time and resources @xcite . therefore, ranking documents according to relevance to the investigated class may help to reduce close reading time and decrease the likelihood of missing critical information. baeza @xcite propose ranking documents in decreasing order of being relevant to a given query to accelerate the information retrieval process. @xcite apply this method with natural language understanding (nlu) models for binary classification problems using the entailment probabilities of a document and a declarative form of the label. therefore, to catch a high percentage of positively labeled documents, reading some percentage of documents would be enough since documents that are relevant would be at the top with a high probability. however, their dataset india police events focuses on a relatively specific task in information retrieval that is police actions like killing, arresting, failing to intervene, etc. besides, they apply this method at the sentence level and as they also stated their model suffers from understanding multi-sentence context that increases the false negative rate. we apply this approach to protestnews dataset @xcite along with the india police events dataset @xcite and investigate whether sentence level evaluation or document level evaluation ranks positive documents at the higher level measured by different evaluation metrics. we further investigate whether using the dictionary definition of a class or the declarative form of a class for the query performs this task better. we compare two nlu models deberta-large-mnli @xcite and roberta-large-mnli @xcite in terms of recall and mean average precision. we present the related work in section 2. next, we introduce two datasets we used in our experiments in section 3. then we explain our methodology and list all queries used in this work in section 4. we detail our experiments for both datasets and present results in section 5. finally, we conclude this work in section 6 and state what can be done as future work in section 7. ## related work protest event detection protest event extraction holds an important place in political social sciences and detection of protest events is generally the first step of the extraction. due to the cost of manual event extraction, besides the presence of digital news articles and enhancing machine learning methods; automated event extraction comes into play. @xcite presents mpeds, an automated system for protest event extraction that contains an ensemble of shallow machine learning classifiers (svm, sgd and logistic regression) to detect protest-related documents. @xcite proposes domain adaptive retraining for transformer language models and shows that further training bert with domain-specific dataset improves the performance. they present protest-er by retraining pre-trained bert with protest related data from trec washington post corpus. @xcite classifies protest related documents in german local news using pretrained language models. they attempt to improve performance and generalizability by eliminating protest-unrelated sentences with keyword search and also by masking named entities with the idea of models may overfit on data by recognizing actors, organizatons and places. @xcite focuses on both protest events and police actions i.e. protest repression events in twitter with machine learning models with the claim of news articles suffer from bias, censorship and duplication. @xcite detects and analyze protest events in geotagged tweets and associated images with convolutional neural networks. @xcite presents a comprehensive survey of how bert @xcite works, ranking documents with bert, retrieve and rerank approach with monobert, ranking metrics, etc. one of the most remarkable works in the survey is monobert and duobert, a multistage ranking approach with transformer language models proposed by @xcite . the first stage retrieves the candidate documents with bm25 by treating the query as a bag of words and later, documents are reranked with their relevance score with bert. duobert also takes into account one document being more relevant than the other at a third stage. however, we rank the documents with a language model at one stage. @xcite rank documents with roberta-large-mnli @xcite on sentence level by being relevant to a police activity. yet sentence level evaluation does not take into consideration the relationship between the sentences. moreover, the task of extracting police events is a relatively specific topic in political event extraction. ## data we carried out the experiments on two different datasets: india police events dataset foot_0 @xcite and the protestnews dataset of the workshop case @ acl-ijcnlp 2021 foot_1 @xcite . ## method first, the probability of entailment for each document and a query is calculated with nlu models from huggingface foot_2 , and documents are ranked by the decreasing probability of being relevant to the query. thus we expect the documents that are more relevant are ranked at the top. entailment probabilities are evaluated on both sentence and document levels. at sentence level evaluation, entailment probabilities of sentences in a document with the given query are calculated and the largest probability among the sentences is considered as the probability of the document being relevant. for the document level evaluation since rlm is limited to 512 tokens, we divided documents into parts such that each part does not exceed 512 tokens. similar to the sentence-level approach, probabilities of each part are calculated and the one with the largest probability is considered as the probability of the document. after getting the probabilities for all documents, they are ranked in the decreasing probability. we compare the results by checking how much recall is achieved when a specified proportion of data is read from the ranked documents following @xcite and also by calculating the mean average precision. we release our code publicly foot_3 . ## experiments & results protestnews dataset is tested with declarative queries, definitional queries and extended definitions on models dlm and rlm and results are presented in figure 1a for the sentence level evalu- for document level evaluation, figure 1b illustrates the comparison of the models. dlm achieves higher recall scores than rlm, however, the query type does not affect the performance of the model at the document level significantly. we compare the extended and annotation manual definitions at document level using the dlm model since the dlm achieves higher recall compared to rlm at the document level as in figure 1b . however, from figure 1c we see that extending the protest definition performs slightly worse than using the only dictionary definition. also, annotation manual definitions do not perform better than the dictionary definition as we see from figure 1d . india police events dataset is tested with declarative and definitional queries on rlm and dlm as in protestnews dataset. for all event types, we see from figure 2 and figure 3 that dlm with declarative query gives the best result that is positive documents are accumulated at more top-level, whereas rlm with a definitional query stays behind other combinations of model and queries. ## conclusion we investigate the performances of two transformer language models (dlm and rlm), different query types (declarative and definitional) in different document lengths (document and sen-tence level). our experiments that conclude dlm achieves higher map scores than rlm are consistent with the findings of @xcite and @xcite . in general, we find that the combination of dlm with a declarative query in sentence level outperforms other combinations in map score. however, scores decrease as the topic or event type gets broader where protest events can be considered broader than specific police actions.