pwc-archive/papers-with-abstracts huggingface dataset:
SU-NLP at CASE 2021 Task 1: Protest News Detection for English
Authors: Furkan Çelik, Tuğberk Dalkılıç, Fatih Beyhan, Reyyan Yeniterzi
Date: 2021-08-01 00:00:00
Proceeding: ACL (CASE) 2021 8
Tasks: Event Extraction

This paper summarizes our group’s efforts in the multilingual protest news detection shared task, which is organized as a part of the Challenges and Applications of Automated Extraction of Socio-political Events from Text (CASE) Workshop. We participated in all four subtasks in English. Especially in the identification of event containing sentences task, our proposed ensemble approach using RoBERTa and multichannel CNN-LexStem model yields higher performance. Similarly in the event extraction task, our transformer-LSTM-CRF architecture outperforms regular transformers significantly.
IBM MNLP IE at CASE 2021 Task 1: Multigranular and Multilingual Event Detection on Protest News
Authors: Parul Awasthy, Jian Ni, Ken Barker, Radu Florian
Date: 2021-08-01 00:00:00
Proceeding: ACL (CASE) 2021 8
Tasks: Event Detection, Language Modeling, Language Modelling, XLM-R

In this paper, we present the event detection models and systems we have developed for Multilingual Protest News Detection - Shared Task 1 at CASE 2021. The shared task has 4 subtasks which cover event detection at different granularity levels (from document level to token level) and across multiple languages (English, Hindi, Portuguese and Spanish). To handle data from multiple languages, we use a multilingual transformer-based language model (XLM-R) as the input text encoder. We apply a variety of techniques and build several transformer-based models that perform consistently well across all the subtasks and languages. Our systems achieve an average F_1 score of 81.2. Out of thirteen subtask-language tracks, our submissions rank 1st in nine and 2nd in four tracks.
ALEM at CASE 2021 Task 1: Multilingual Text Classification on News Articles
Authors: Alaeddin Gürel, Emre Emin
Date: 2021-08-01 00:00:00
Proceeding: ACL (CASE) 2021 8
Tasks: Articles, Data Augmentation, Multilingual text classification, text-classification, Text Classification

We participated CASE shared task in ACL-IJCNLP 2021. This paper is a summary of our experiments and ideas about this shared task. For each subtask we shared our approach, successful and failed methods and our thoughts about them. We submit our results once for every subtask, except for subtask3, in task submission system and present scores based on our validation set formed from given training samples in this paper. Techniques and models we mentioned includes BERT, Multilingual BERT, oversampling, undersampling, data augmentation and their implications with each other. Most of the experiments we came up with were not completed, as time did not permit, but we share them here as we plan to do them as suggested in the future work part of document.
Team “NoConflict” at CASE 2021 Task 1: Pretraining for Sentence-Level Protest Event Detection
Authors: Tiancheng Hu, Niklas Stoehr
Date: 2021-08-01 00:00:00
Proceeding: ACL (CASE) 2021 8
Tasks: Articles, Data Augmentation, Event Detection, Language Modeling, Language Modelling, Sentence, Sentence Classification, Translation

An ever-increasing amount of text, in the form of social media posts and news articles, gives rise to new challenges and opportunities for the automatic extraction of socio-political events. In this paper, we present our submission to the Shared Tasks on Socio-Political and Crisis Events Detection, Task 1, Multilingual Protest News Detection, Subtask 2, Event Sentence Classification, of CASE @ ACL-IJCNLP 2021. In our submission, we utilize the RoBERTa model with additional pretraining, and achieve the best F1 score of 0.8532 in event sentence classification in English and the second-best F1 score of 0.8700 in Portuguese via simple translation. We analyze the failure cases of our model. We also conduct an ablation study to show the effect of choosing the right pretrained language model, adding additional training data and data augmentation.
Team “DaDeFrNi” at CASE 2021 Task 1: Document and Sentence Classification for Protest Event Detection
Authors: Francesco Re, Daniel Vegh, Dennis Atzenhofer, Niklas Stoehr
Date: 2021-08-01 00:00:00
Proceeding: ACL (CASE) 2021 8
Tasks: Articles, Binary Classification, Event Detection, Sentence, Sentence Classification, XLM-R

This paper accompanies our top-performing submission to the CASE 2021 shared task, which is hosted at the workshop on Challenges and Applications of Automated Extraction of Socio-political Events from Text. Subtasks 1 and 2 of Task 1 concern the classification of newspaper articles and sentences into “conflict” versus “not conflict”-related in four different languages. Our model performs competitively in both subtasks (up to 0.8662 macro F1), obtaining the highest score of all contributions for subtask 1 on Hindi articles (0.7877 macro F1). We describe all experiments conducted with the XLM-RoBERTa (XLM-R) model and report results obtained in each binary classification task. We propose supplementing the original training data with additional data on political conflict events. In addition, we provide an analysis of unigram probability estimates and geospatial references contained within the original training corpus.
Fine-grained Event Classification in News-like Text Snippets - Shared Task 2, CASE 2021
Authors: Jacek Haneczok, Guillaume Jacquet, Jakub Piskorski, Nicolas Stefanovitch
Date: 2021-08-01 00:00:00
Proceeding: ACL (CASE) 2021 8
Tasks: Classification, Generalized Zero-Shot Learning, Task 2, Zero-Shot Learning

This paper describes the Shared Task on Fine-grained Event Classification in News-like Text Snippets. The Shared Task is divided into three sub-tasks: (a) classification of text snippets reporting socio-political events (25 classes) for which vast amount of training data exists, although exhibiting different structure and style vis-a-vis test data, (b) enhancement to a generalized zero-shot learning problem, where 3 additional event types were introduced in advance, but without any training data (‘unseen’ classes), and (c) further extension, which introduced 2 additional event types, announced shortly prior to the evaluation phase. The reported Shared Task focuses on classification of events in English texts and is organized as part of the Workshop on Challenges and Applications of Automated Extraction of Socio-political Events from Text (CASE 2021), co-located with the ACL-IJCNLP 2021 Conference. Four teams participated in the task. Best performing systems for the three aforementioned sub-tasks achieved 83.9%, 79.7% and 77.1% weighted F1 scores respectively.
IBM MNLP IE at CASE 2021 Task 2: NLI Reranking for Zero-Shot Text Classification
Authors: Ken Barker, Parul Awasthy, Jian Ni, Radu Florian
Date: 2021-08-01 00:00:00
Proceeding: ACL (CASE) 2021 8
Tasks: Classification, Natural Language Inference, Reranking, Task 2, text-classification, Text Classification, Zero-Shot Text Classification

Supervised models can achieve very high accuracy for fine-grained text classification. In practice, however, training data may be abundant for some types but scarce or even non-existent for others. We propose a hybrid architecture that uses as much labeled data as available for fine-tuning classification models, while also allowing for types with little (few-shot) or no (zero-shot) labeled data. In particular, we pair a supervised text classification model with a Natural Language Inference (NLI) reranking model. The NLI reranker uses a textual representation of target types that allows it to score the strength with which a type is implied by a text, without requiring training data for the types. Experiments show that the NLI model is very sensitive to the choice of textual representation, but can be effective for classifying unseen types. It can also improve classification accuracy for the known types of an already highly accurate supervised model.
CASE 2021 Task 2: Zero-Shot Classification of Fine-Grained Sociopolitical Events with Transformer Models
Authors: Benjamin J. Radford
Date: 2021-08-01 00:00:00
Proceeding: ACL (CASE) 2021 8
Tasks: Natural Language Inference, Task 2, zero-shot-classification, Zero-Shot Learning

We introduce a method for the classification of texts into fine-grained categories of sociopolitical events. This particular method is responsive to all three Subtasks of Task 2, Fine-Grained Classification of Socio-Political Events, introduced at the CASE workshop of ACL-IJCNLP 2021. We frame Task 2 as textual entailment: given an input text and a candidate event class (“query”), the model predicts whether the text describes an event of the given type. The model is able to correctly classify in-sample event types with an average F1-score of 0.74 but struggles with some out-of-sample event types. Despite this, the model shows promise for the zero-shot identification of certain sociopolitical events by achieving an F1-score of 0.52 on one wholly out-of-sample event class.
CASE 2021 Task 2 Socio-political Fine-grained Event Classification using Fine-tuned RoBERTa Document Embeddings
Authors: Samantha Kent, Theresa Krumbiegel
Date: 2021-08-01 00:00:00
Proceeding: ACL (CASE) 2021 8
Tasks: Task 2

We present our submission to Task 2 of the Socio-political and Crisis Events Detection Shared Task at the CASE @ ACL-IJCNLP 2021 workshop. The task at hand aims at the fine-grained classification of socio-political events. Our best model was a fine-tuned RoBERTa transformer model using document embeddings. The corpus consisted of a balanced selection of sub-events extracted from the ACLED event dataset. We achieved a macro F-score of 0.923 and a micro F-score of 0.932 during our preliminary experiments on a held-out test set. The same model also performed best on the shared task test data (weighted F-score = 0.83). To analyze the results we calculated the topic compactness of the commonly misclassified events and conducted an error analysis.
Discovering Black Lives Matter Events in the United States: Shared Task 3, CASE 2021
Authors: Salvatore Giorgi, Vanni Zavarella, Hristo Tanev, Nicolas Stefanovitch, Sy Hwang, Hansi Hettiarachchi, Tharindu Ranasinghe, Vivek Kalyan, Paul Tan, Shaun Tan, Martin Andrews, Tiancheng Hu, Niklas Stoehr, Francesco Ignazio Re, Daniel Vegh, Dennis Atzenhofer, Brenda Curtis, Ali Hürriyetoğlu
Date: 2021-08-01 00:00:00
Proceeding: ACL (CASE) 2021 8
Tasks: Articles, Event Detection

Evaluating the state-of-the-art event detection systems on determining spatio-temporal distribution of the events on the ground is performed unfrequently. But, the ability to both (1) extract events “in the wild” from text and (2) properly evaluate event detection systems has potential to support a wide variety of tasks such as monitoring the activity of socio-political movements, examining media coverage and public support of these movements, and informing policy decisions. Therefore, we study performance of the best event detection systems on detecting Black Lives Matter (BLM) events from tweets and news articles. The murder of George Floyd, an unarmed Black man, at the hands of police officers received global attention throughout the second half of 2020. Protests against police violence emerged worldwide and the BLM movement, which was once mostly regulated to the United States, was now seeing activity globally. This shared task asks participants to identify BLM related events from large unstructured data sources, using systems pretrained to extract socio-political events from text. We evaluate several metrics, accessing each system’s ability to identify protest events both temporally and spatially. Results show that identifying daily protest counts is an easier task than classifying spatial and temporal protest trends simultaneously, with maximum performance of 0.745 and 0.210 (Pearson r), respectively. Additionally, all baselines and participant systems suffered from low recall, with a maximum recall of 5.08.
Can Recurrent Neural Networks Learn Nested Recursion?
Authors: Jean-Phillipe Bernardy
Date: 2018-07-01 00:00:00
Proceeding: LILT 2018 7

Context-free grammars (CFG) were one of the first formal tools used to model natural languages, and they remain relevant today as the basis of several frameworks. A key ingredient of CFG is the presence of nested recursion. In this paper, we investigate experimentally the capability of several recurrent neural networks (RNNs) to learn nested recursion. More precisely, we measure an upper bound of their capability to do so, by simplifying the task to learning a generalized Dyck language, namely one composed of matching parentheses of various kinds. To do so, we present the RNNs with a set of random strings having a given maximum nesting depth and test its ability to predict the kind of closing parenthesis when facing deeper nested strings. We report mixed results: when generalizing to deeper nesting levels, the accuracy of standard RNNs is significantly higher than random, but still far from perfect. Additionally, we propose some non-standard stack-based models which can approach perfect accuracy, at the cost of robustness.
Two Heads are Better than One? Verification of Ensemble Effect in Neural Machine Translation
Authors: Chanjun Park, Sungjin Park, Seolhwa Lee, Taesun Whang, Heuiseok Lim
Date: 2021-11-01 00:00:00
Proceeding: EMNLP (insights) 2021 11
Tasks: Machine Translation, NMT, Translation

In the field of natural language processing, ensembles are broadly known to be effective in improving performance. This paper analyzes how ensemble of neural machine translation (NMT) models affect performance improvement by designing various experimental setups (i.e., intra-, inter-ensemble, and non-convergence ensemble). To an in-depth examination, we analyze each ensemble method with respect to several aspects such as different attention models and vocab strategies. Experimental results show that ensembling is not always resulting in performance increases and give noteworthy negative findings.
Are BERTs Sensitive to Native Interference in L2 Production?
Authors: Zixin Tang, Prasenjit Mitra, David Reitter
Date: 2021-11-01 00:00:00
Proceeding: EMNLP (insights) 2021 11

With the essays part from The International Corpus Network of Asian Learners of English (ICNALE) and the TOEFL11 corpus, we fine-tuned neural language models based on BERT to predict English learners’ native languages. Results showed neural models can learn to represent and detect such native language impacts, but multilingually trained models have no advantage in doing so.
Zero-Shot Cross-Lingual Transfer is a Hard Baseline to Beat in German Fine-Grained Entity Typing
Authors: Sabine Weber, Mark Steedman
Date: 2021-11-01 00:00:00
Proceeding: EMNLP (insights) 2021 11
Tasks: Cross-Lingual Transfer, Entity Typing, Multilingual Word Embeddings, named-entity-recognition, Named Entity Recognition, Named Entity Recognition (NER), Word Embeddings, Zero-Shot Cross-Lingual Transfer

The training of NLP models often requires large amounts of labelled training data, which makes it difficult to expand existing models to new languages. While zero-shot cross-lingual transfer relies on multilingual word embeddings to apply a model trained on one language to another, Yarowski and Ngai (2001) propose the method of annotation projection to generate training data without manual annotation. This method was successfully used for the tasks of named entity recognition and coarse-grained entity typing, but we show that it is outperformed by zero-shot cross-lingual transfer when applied to the similar task of fine-grained entity typing. In our study of fine-grained entity typing with the FIGER type ontology for German, we show that annotation projection amplifies the English model’s tendency to underpredict level 2 labels and is beaten by zero-shot cross-lingual transfer on three novel test sets.
When does Further Pre-training MLM Help? An Empirical Study on Task-Oriented Dialog Pre-training
Authors: Qi Zhu, Yuxian Gu, Lingxiao Luo, Bing Li, Cheng Li, Wei Peng, Minlie Huang, Xiaoyan Zhu
Date: 2021-11-01 00:00:00
Proceeding: EMNLP (insights) 2021 11

Further pre-training language models on in-domain data (domain-adaptive pre-training, DAPT) or task-relevant data (task-adaptive pre-training, TAPT) before fine-tuning has been shown to improve downstream tasks’ performances. However, in task-oriented dialog modeling, we observe that further pre-training MLM does not always boost the performance on a downstream task. We find that DAPT is beneficial in the low-resource setting, but as the fine-tuning data size grows, DAPT becomes less beneficial or even useless, and scaling the size of DAPT data does not help. Through Representational Similarity Analysis, we conclude that more data for fine-tuning yields greater change of the model’s representations and thus reduces the influence of initialization.
Recurrent Attention for the Transformer
Authors: Jan Rosendahl, Christian Herold, Frithjof Petrick, Hermann Ney
Date: 2021-11-01 00:00:00
Proceeding: EMNLP (insights) 2021 11
Tasks: Decoder, Machine Translation, Translation

In this work, we conduct a comprehensive investigation on one of the centerpieces of modern machine translation systems: the encoder-decoder attention mechanism. Motivated by the concept of first-order alignments, we extend the (cross-)attention mechanism by a recurrent connection, allowing direct access to previous attention/alignment decisions. We propose several ways to include such a recurrency into the attention mechanism. Verifying their performance across different translation tasks we conclude that these extensions and dependencies are not beneficial for the translation performance of the Transformer architecture.
Backtranslation in Neural Morphological Inflection
Authors: Ling Liu, Mans Hulden
Date: 2021-11-01 00:00:00
Proceeding: EMNLP (insights) 2021 11
Tasks: Machine Translation, Morphological Inflection, Translation

Backtranslation is a common technique for leveraging unlabeled data in low-resource scenarios in machine translation. The method is directly applicable to morphological inflection generation if unlabeled word forms are available. This paper evaluates the potential of backtranslation for morphological inflection using data from six languages with labeled data drawn from the SIGMORPHON shared task resource and unlabeled data from different sources. Our core finding is that backtranslation can offer modest improvements in low-resource scenarios, but only if the unlabeled data is very clean and has been filtered by the same annotation standards as the labeled data.
Learning Data Augmentation Schedules for Natural Language Processing
Authors: Daphné Chopard, Matthias S. Treder, Irena Spasić
Date: 2021-11-01 00:00:00
Proceeding: EMNLP (insights) 2021 11
Tasks: Data Augmentation, Scheduling

Despite its proven efficiency in other fields, data augmentation is less popular in the context of natural language processing (NLP) due to its complexity and limited results. A recent study (Longpre et al., 2020) showed for example that task-agnostic data augmentations fail to consistently boost the performance of pretrained transformers even in low data regimes. In this paper, we investigate whether data-driven augmentation scheduling and the integration of a wider set of transformations can lead to improved performance where fixed and limited policies were unsuccessful. Our results suggest that, while this approach can help the training process in some settings, the improvements are unsubstantial. This negative result is meant to help researchers better understand the limitations of data augmentation for NLP.
An Investigation into the Contribution of Locally Aggregated Descriptors to Figurative Language Identification
Authors: Sina Mahdipour Saravani, Ritwik Banerjee, Indrakshi Ray
Date: 2021-11-01 00:00:00
Proceeding: EMNLP (insights) 2021 11
Tasks: Language Identification, Natural Language Understanding, Sarcasm Detection

In natural language understanding, topics that touch upon figurative language and pragmatics are notably difficult. We probe a novel use of locally aggregated descriptors – specifically, an architecture called NeXtVLAD – motivated by its accomplishments in computer vision, achieve tremendous success in the FigLang2020 sarcasm detection task. The reported F1 score of 93.1% is 14% higher than the next best result. We specifically investigate the extent to which the novel architecture is responsible for this boost, and find that it does not provide statistically significant benefits. Deep learning approaches are expensive, and we hope our insights highlighting the lack of benefits from introducing a resource-intensive component will aid future research to distill the effective elements from long and complex pipelines, thereby providing a boost to the wider research community.
Flesch-Kincaid is Not a Text Simplification Evaluation Metric
Authors: Teerapaun Tanprasert, David Kauchak
Date: 2021-08-01 00:00:00
Proceeding: ACL (GEM) 2021 8
Tasks: Sentence, Text Simplification

Sentence-level text simplification is currently evaluated using both automated metrics and human evaluation. For automatic evaluation, a combination of metrics is usually employed to evaluate different aspects of the simplification. Flesch-Kincaid Grade Level (FKGL) is one metric that has been regularly used to measure the readability of system output. In this paper, we argue that FKGL should not be used to evaluate text simplification systems. We provide experimental analyses on recent system output showing that the FKGL score can easily be manipulated to improve the score dramatically with only minor impact on other automated metrics (BLEU and SARI). Instead of using FKGL, we suggest that the component statistics, along with others, be used for posthoc analysis to understand system behavior.
Human Perception in Natural Language Generation
Authors: Lorenzo De Mattei, Huiyuan Lai, Felice Dell’Orletta, Malvina Nissim
Date: 2021-08-01 00:00:00
Proceeding: ACL (GEM) 2021 8
Tasks: Text Generation

We ask subjects whether they perceive as human-produced a bunch of texts, some of which are actually human-written, while others are automatically generated. We use this data to fine-tune a GPT-2 model to push it to generate more human-like texts, and observe that this fine-tuned model produces texts that are indeed perceived more human-like than the original model. Contextually, we show that our automatic evaluation strategy well correlates with human judgements. We also run a linguistic analysis to unveil the characteristics of human- vs machine-perceived language.
Semantic Similarity Based Evaluation for Abstractive News Summarization
Authors: Figen Beken Fikri, Kemal Oflazer, Berrin Yanikoglu
Date: 2021-08-01 00:00:00
Proceeding: ACL (GEM) 2021 8
Tasks: Abstractive Text Summarization, News Summarization, Semantic Similarity, Semantic Textual Similarity, Text Summarization

ROUGE is a widely used evaluation metric in text summarization. However, it is not suitable for the evaluation of abstractive summarization systems as it relies on lexical overlap between the gold standard and the generated summaries. This limitation becomes more apparent for agglutinative languages with very large vocabularies and high type/token ratios. In this paper, we present semantic similarity models for Turkish and apply them as evaluation metrics for an abstractive summarization task. To achieve this, we translated the English STSb dataset into Turkish and presented the first semantic textual similarity dataset for Turkish as well. We showed that our best similarity models have better alignment with average human judgments compared to ROUGE in both Pearson and Spearman correlations.