pwc-archive/papers-with-abstracts huggingface dataset:
Detecting Trending Terms in Cybersecurity Forum Discussions
Authors: Jack Hughes, Seth Aycock, Andrew Caines, Paula Buttery, Alice Hutchings
Date: 2020-11-01 00:00:00
Proceeding: EMNLP (WNUT) 2020 11
Tasks: Information Retrieval, Retrieval

We present a lightweight method for identifying currently trending terms in relation to a known prior of terms, using a weighted log-odds ratio with an informative prior. We apply this method to a dataset of posts from an English-language underground hacking forum, spanning over ten years of activity, with posts containing misspellings, orthographic variation, acronyms, and slang. Our statistical approach supports analysis of linguistic change and discussion topics over time, without a requirement to train a topic model for each time interval for analysis. We evaluate the approach by comparing the results to TF-IDF using the discounted cumulative gain metric with human annotations, finding our method outperforms TF-IDF on information retrieval.
Service registration chatbot: collecting and comparing dialogues from AMT workers and service’s users
Authors: Luca Molteni, Mittul Singh, Juho Leinonen, Katri Leino, Mikko Kurimo, Emanuele Della Valle
Date: 2020-11-01 00:00:00
Proceeding: EMNLP (WNUT) 2020 11
Tasks: Chatbot, Text Generation

Crowdsourcing is the go-to solution for data collection and annotation in the context of NLP tasks. Nevertheless, crowdsourced data is noisy by nature; the source is often unknown and additional validation work is performed to guarantee the dataset’s quality. In this article, we compare two crowdsourcing sources on a dialogue paraphrasing task revolving around a chatbot service. We observe that workers hired on crowdsourcing platforms produce lexically poorer and less diverse rewrites than service users engaged voluntarily. Notably enough, on dialogue clarity and optimality, the two paraphrase sources’ human-perceived quality does not differ significantly. Furthermore, for the chatbot service, the combined crowdsourced data is enough to train a transformer-based Natural Language Generation (NLG) system. To enable similar services, we also release tools for collecting data and training the dialogue-act-based transformer-based NLG module.
Automated Assessment of Noisy Crowdsourced Free-text Answers for Hindi in Low Resource Setting
Authors: Dolly Agarwal, Somya Gupta, Nishant Baghel
Date: 2020-11-01 00:00:00
Proceeding: EMNLP (WNUT) 2020 11

The requirement of performing assessments continually on a larger scale necessitates the implementation of automated systems for evaluation of the learners’ responses to free-text questions. We target children of age group 8-14 years and use an ASR integrated assessment app to crowdsource learners’ responses to free text questions in Hindi. The app helped collect 39641 user answers to 35 different questions of Science topics. Since the users are young children from rural India and may not be well-equipped with technology, it brings in various noise types in the answers. We describe these noise types and propose a preprocessing pipeline to denoise user’s answers. We showcase the performance of different similarity metrics on the noisy and denoised versions of user and model answers. Our findings have large-scale applications for automated answer assessment for school children in India in low resource settings.
Punctuation Restoration using Transformer Models for High-and Low-Resource Languages
Authors: Tanvirul Alam, Akib Khan, Firoj Alam
Date: 2020-11-01 00:00:00
Proceeding: EMNLP (WNUT) 2020 11
Tasks: Automatic Speech Recognition, Automatic Speech Recognition (ASR), Punctuation Restoration, speech-recognition, Speech Recognition, Vocal Bursts Intensity Prediction

Punctuation restoration is a common post-processing problem for Automatic Speech Recognition (ASR) systems. It is important to improve the readability of the transcribed text for the human reader and facilitate NLP tasks. Current state-of-art address this problem using different deep learning models. Recently, transformer models have proven their success in downstream NLP tasks, and these models have been explored very little for the punctuation restoration problem. In this work, we explore different transformer based models and propose an augmentation strategy for this task, focusing on high-resource (English) and low-resource (Bangla) languages. For English, we obtain comparable state-of-the-art results, while for Bangla, it is the first reported work, which can serve as a strong baseline for future work. We have made our developed Bangla dataset publicly available for the research community.
Truecasing German user-generated conversational text
Authors: Yulia Grishina, Thomas Gueudre, Ralf Winkler
Date: 2020-11-01 00:00:00
Proceeding: EMNLP (WNUT) 2020 11
Tasks: Language Modeling, Language Modelling

True-casing, the task of restoring proper case to (generally) lower case input, is important in downstream tasks and for screen display. In this paper, we investigate truecasing as an in- trinsic task and present several experiments on noisy user queries to a voice-controlled dia- log system. In particular, we compare a rule- based, an n-gram language model (LM) and a recurrent neural network (RNN) approaches, evaluating the results on a German Q&A cor- pus and reporting accuracy for different case categories. We show that while RNNs reach higher accuracy especially on large datasets, character n-gram models with interpolation are still competitive, in particular on mixed- case words where their fall-back mechanisms come into play.
Fine-Tuning MT systems for Robustness to Second-Language Speaker Variations
Authors: Md Mahfuz ibn Alam, Antonios Anastasopoulos
Date: 2020-11-01 00:00:00
Proceeding: EMNLP (WNUT) 2020 11
Tasks: Machine Translation, NMT, Translation

The performance of neural machine translation (NMT) systems only trained on a single language variant degrades when confronted with even slightly different language variations. With this work, we build upon previous work to explore how to mitigate this issue. We show that fine-tuning using naturally occurring noise along with pseudo-references (i.e. “corrected” non-native inputs translated using the baseline NMT system) is a promising solution towards systems robust to such type of input variations. We focus on four translation pairs, from English to Spanish, Italian, French, and Portuguese, with our system achieving improvements of up to 3.1 BLEU points compared to the baselines, establishing a new state-of-the-art on the JFLEG-ES dataset. All datasets and code are publicly available here: https://github.com/mahfuzibnalam/finetuning_for_robustness .
Impact of ASR on Alzheimer’s Disease Detection: All Errors are Equal, but Deletions are More Equal than Others
Authors: Aparna Balagopalan, Ksenia Shkaruta, Jekaterina Novikova
Date: 2020-11-01 00:00:00
Proceeding: EMNLP (WNUT) 2020 11
Tasks: All, Automatic Speech Recognition, Automatic Speech Recognition (ASR), speech-recognition, Speech Recognition

Automatic Speech Recognition (ASR) is a critical component of any fully-automated speech-based dementia detection model. However, despite years of speech recognition research, little is known about the impact of ASR accuracy on dementia detection. In this paper, we experiment with controlled amounts of artificially generated ASR errors and investigate their influence on dementia detection. We find that deletion errors affect detection performance the most, due to their impact on the features of syntactic complexity and discourse representation in speech. We show the trend to be generalisable across two different datasets for cognitive impairment detection. As a conclusion, we propose optimising the ASR to reflect a higher penalty for deletion errors in order to improve dementia detection performance.
Detecting Entailment in Code-Mixed Hindi-English Conversations
Authors: Sharanya Chakravarthy, Anjana Umapathy, Alan W Black
Date: 2020-11-01 00:00:00
Proceeding: EMNLP (WNUT) 2020 11
Tasks: Data Augmentation, Language Modeling, Language Modelling, Natural Language Inference, Natural Language Understanding, Translation

The presence of large-scale corpora for Natural Language Inference (NLI) has spurred deep learning research in this area, though much of this research has focused solely on monolingual data. Code-mixing is the intertwined usage of multiple languages, and is commonly seen in informal conversations among polyglots. Given the rising importance of dialogue agents, it is imperative that they understand code-mixing, but the scarcity of code-mixed Natural Language Understanding (NLU) datasets has precluded research in this area. The dataset by Khanuja et. al. for detecting conversational entailment in code-mixed Hindi-English text is the first of its kind. We investigate the effectiveness of language modeling, data augmentation, translation, and architectural approaches to address the code-mixed, conversational, and low-resource aspects of this dataset. We obtain an 8.09% increase in test set accuracy over the current state of the art.
Annotation Efficient Language Identification from Weak Labels
Authors: Shriphani Palakodety, Ashiqur KhudaBukhsh
Date: 2020-11-01 00:00:00
Proceeding: EMNLP (WNUT) 2020 11
Tasks: Language Identification

India is home to several languages with more than 30m speakers. These languages exhibit significant presence on social media platforms. However, several of these widely-used languages are under-addressed by current Natural Language Processing (NLP) models and resources. User generated social media content in these languages is also typically authored in the Roman script as opposed to the traditional native script further contributing to resource scarcity. In this paper, we leverage a minimally supervised NLP technique to obtain weak language labels from a large-scale Indian social media corpus leading to a robust and annotation-efficient language-identification technique spanning nine Romanized Indian languages. In fast-spreading pandemic situations such as the current COVID-19 situation, information processing objectives might be heavily tilted towards under-served languages in densely populated regions. We release our models to facilitate downstream analyses in these low-resource languages. Experiments across multiple social media corpora demonstrate the model’s robustness and provide several interesting insights on Indian language usage patterns on social media. We release an annotated data set of 1,000 comments in ten Romanized languages as a social media evaluation benchmark.
Quantifying the Evaluation of Heuristic Methods for Textual Data Augmentation
Authors: Omid Kashefi, Rebecca Hwa
Date: 2020-11-01 00:00:00
Proceeding: EMNLP (WNUT) 2020 11
Tasks: Data Augmentation

Data augmentation has been shown to be effective in providing more training data for machine learning and resulting in more robust classifiers. However, for some problems, there may be multiple augmentation heuristics, and the choices of which one to use may significantly impact the success of the training. In this work, we propose a metric for evaluating augmentation heuristics; specifically, we quantify the extent to which an example is “hard to distinguish” by considering the difference between the distribution of the augmented samples of different classes. Experimenting with multiple heuristics in two prediction tasks (positive/negative sentiment and verbosity/conciseness) validates our claims by revealing the connection between the distribution difference of different classes and the classification accuracy.
Civil Unrest on Twitter (CUT): A Dataset of Tweets to Support Research on Civil Unrest
Authors: Justin Sech, Alexandra DeLucia, Anna L. Buczak, Mark Dredze
Date: 2020-11-01 00:00:00
Proceeding: EMNLP (WNUT) 2020 11

We present CUT, a dataset for studying Civil Unrest on Twitter. Our dataset includes 4,381 tweets related to civil unrest, hand-annotated with information related to the study of civil unrest discussion and events. Our dataset is drawn from 42 countries from 2014 to 2019. We present baseline systems trained on this data for the identification of tweets related to civil unrest. We include a discussion of ethical issues related to research on this topic.
Tweeki: Linking Named Entities on Twitter to a Knowledge Graph
Authors: Bahareh Harandizadeh, Sameer Singh
Date: 2020-11-01 00:00:00
Proceeding: EMNLP (WNUT) 2020 11
Tasks: Entity Linking

To identify what entities are being talked about in tweets, we need to automatically link named entities that appear in tweets to structured KBs like WikiData. Existing approaches often struggle with such short, noisy texts, or their complex design and reliance on supervision make them brittle, difficult to use and maintain, and lose significance over time. Further, there is a lack of a large, linked corpus of tweets to aid researchers, along with lack of gold dataset to evaluate the accuracy of entity linking. In this paper, we introduce (1) Tweeki, an unsupervised, modular entity linking system for Twitter, (2) TweekiData, a large, automatically-annotated corpus of Tweets linked to entities in WikiData, and (3) TweekiGold, a gold dataset for entity linking evaluation. Through comprehensive analysis, we show that Tweeki is comparable to the performance of recent state-of-the-art entity linkers models, the dataset is of high quality, and a use case of how the dataset can be used to improve downstream tasks in social media analysis (geolocation prediction).
Representation learning of writing style
Authors: Julien Hay, Bich-Lien Doan, Fabrice Popineau, Ouassim Ait Elhara
Date: 2020-11-01 00:00:00
Proceeding: EMNLP (WNUT) 2020 11
Tasks: Articles, Authorship Attribution, Clustering, Feature Engineering, Representation Learning

In this paper, we introduce a new method of representation learning that aims to embed documents in a stylometric space. Previous studies in the field of authorship analysis focused on feature engineering techniques in order to represent document styles and to enhance model performance in specific tasks. Instead, we directly embed documents in a stylometric space by relying on a reference set of authors and the intra-author consistency property which is one of two components in our definition of writing style. The main intuition of this paper is that we can define a general stylometric space from a set of reference authors such that, in this space, the coordinates of different documents will be close when the documents are by the same author, and spread away when they are by different authors, even for documents by authors who are not in the set of reference authors. The method we propose allows for the clustering of documents based on stylistic clues reflecting the authorship of documents. For the empirical validation of the method, we train a deep neural network model to predict authors of a large reference dataset consisting of news and blog articles. Albeit the learning process is supervised, it does not require a dedicated labeling of the data but it relies only on the metadata of the articles which are available in huge amounts. We evaluate the model on multiple datasets, on both the authorship clustering and the authorship attribution tasks.
“A Little Birdie Told Me ... ” - Inductive Biases for Rumour Stance Detection on Social Media
Authors: Karthik Radhakrishnan, Tushar Kanakagiri, Sharanya Chakravarthy, Vidhisha Balachandran
Date: 2020-11-01 00:00:00
Proceeding: EMNLP (WNUT) 2020 11
Tasks: Misinformation, Position, Rumour Detection, Stance Detection

The rise in the usage of social media has placed it in a central position for news dissemination and consumption. This greatly increases the potential for proliferation of rumours and misinformation. In an effort to mitigate the spread of rumours, we tackle the related task of identifying the stance (Support, Deny, Query, Comment) of a social media post. Unlike previous works, we impose inductive biases that capture platform specific user behavior. These biases, coupled with social media fine-tuning of BERT allow for better language understanding, thus yielding an F1 score of 58.7 on the SemEval 2019 task on rumour stance detection.
Paraphrase Generation via Adversarial Penalizations
Authors: Gerson Vizcarra, Jose Ochoa-Luna
Date: 2020-11-01 00:00:00
Proceeding: EMNLP (WNUT) 2020 11
Tasks: Paraphrase Generation

Paraphrase generation is an important problem in Natural Language Processing that has been addressed with neural network-based approaches recently. This paper presents an adversarial framework to address the paraphrase generation problem in English. Unlike previous methods, we employ the discriminator output as penalization instead of using policy gradients, and we propose a global discriminator to avoid the Monte-Carlo search. In addition, this work use and compare different settings of input representation. We compare our methods to some baselines in the Quora question pairs dataset. The results show that our framework is competitive against the previous benchmarks.
IITKGP at W-NUT 2020 Shared Task-1: Domain specific BERT representation for Named Entity Recognition of lab protocol
Authors: Tejas Vaidhya, Ayush Kaushal
Date: 2020-11-01 00:00:00
Proceeding: EMNLP (WNUT) 2020 11
Tasks: named-entity-recognition, Named Entity Recognition, Named Entity Recognition (NER), NER

Supervised models trained to predict properties from representations have been achieving high accuracy on a variety of tasks.For in-stance, the BERT family seems to work exceptionally well on the downstream task from NER tagging to the range of other linguistictasks. But the vocabulary used in the medical field contains a lot of different tokens used only in the medical industry such as the name of different diseases, devices, organisms,medicines, etc. that makes it difficult for traditional BERT model to create contextualized embedding. In this paper, we are going to illustrate the System for Named Entity Tagging based on Bio-Bert. Experimental results show that our model gives substantial improvements over the baseline and stood the fourth runner up in terms of F1 score, and first runner up in terms of Recall with just 2.21 F1 score behind the best one.
WNUT 2020 Shared Task-1: Conditional Random Field(CRF) based Named Entity Recognition(NER) for Wet Lab Protocols
Authors: Kaushik Acharya
Date: 2020-11-01 00:00:00
Proceeding: EMNLP (WNUT) 2020 11
Tasks: named-entity-recognition, Named Entity Recognition, Named Entity Recognition (NER), NER

The paper describes how classifier model built using Conditional Random Field detects named entities in wet lab protocols.
Fancy Man Launches Zippo at WNUT 2020 Shared Task-1: A Bert Case Model for Wet Lab Entity Extraction
Authors: Qingcheng Zeng, Xiaoyang Fang, Zhexin Liang, Haoding Meng
Date: 2020-11-01 00:00:00
Proceeding: EMNLP (WNUT) 2020 11
Tasks: Sensitivity

Automatic or semi-automatic conversion of protocols specifying steps in performing a lab procedure into machine-readable format benefits biological research a lot. These noisy, dense, and domain-specific lab protocols processing draws more and more interests with the development of deep learning. This paper presents our teamwork on WNUT 2020 shared task-1: wet lab entity extract, that we conducted studies in several models, including a BiLSTM CRF model and a Bert case model which can be used to complete wet lab entity extraction. And we mainly discussed the performance differences of Bert case under different situations such as transformers versions, case sensitivity that may don’t get enough attention before.
BiTeM at WNUT 2020 Shared Task-1: Named Entity Recognition over Wet Lab Protocols using an Ensemble of Contextual Language Models
Authors: Julien Knafou, Nona Naderi, Jenny Copara, Douglas Teodoro, Patrick Ruch
Date: 2020-11-01 00:00:00
Proceeding: EMNLP (WNUT) 2020 11
Tasks: named-entity-recognition, Named Entity Recognition, Named Entity Recognition (NER), NER, Reading Comprehension

Recent improvements in machine-reading technologies attracted much attention to automation problems and their possibilities. In this context, WNUT 2020 introduces a Name Entity Recognition (NER) task based on wet laboratory procedures. In this paper, we present a 3-step method based on deep neural language models that reported the best overall exact match F1-score (77.99%) of the competition. By fine-tuning 10 times, 10 different pretrained language models, this work shows the advantage of having more models in an ensemble based on a majority of votes strategy. On top of that, having 100 different models allowed us to analyse the combinations of ensemble that demonstrated the impact of having multiple pretrained models versus fine-tuning a pretrained model multiple times.
TATL at WNUT-2020 Task 2: A Transformer-based Baseline System for Identification of Informative COVID-19 English Tweets
Authors: Anh Tuan Nguyen
Date: 2020-11-01 00:00:00
Proceeding: EMNLP (WNUT) 2020 11
Tasks: Task 2

As the COVID-19 outbreak continues to spread throughout the world, more and more information about the pandemic has been shared publicly on social media. For example, there are a huge number of COVID-19 English Tweets daily on Twitter. However, the majority of those Tweets are uninformative, and hence it is important to be able to automatically select only the informative ones for downstream applications. In this short paper, we present our participation in the W-NUT 2020 Shared Task 2: Identification of Informative COVID-19 English Tweets. Inspired by the recent advances in pretrained Transformer language models, we propose a simple yet effective baseline for the task. Despite its simplicity, our proposed approach shows very competitive results in the leaderboard as we ranked 8 over 56 teams participated in total.
NHK_STRL at WNUT-2020 Task 2: GATs with Syntactic Dependencies as Edges and CTC-based Loss for Text Classification
Authors: Yuki Yasuda, Taichi Ishiwatari, Taro Miyazaki, Jun Goto
Date: 2020-11-01 00:00:00
Proceeding: EMNLP (WNUT) 2020 11
Tasks: Graph Attention, Sentence, Task 2, text-classification, Text Classification

The outbreak of COVID-19 has greatly impacted our daily lives. In these circumstances, it is important to grasp the latest information to avoid causing too much fear and panic. To help grasp new information, extracting information from social networking sites is one of the effective ways. In this paper, we describe a method to identify whether a tweet related to COVID-19 is informative or not, which can help to grasp new information. The key features of our method are its use of graph attention networks to encode syntactic dependencies and word positions in the sentence, and a loss function based on connectionist temporal classification (CTC) that can learn a label for each token without reference data for each token. Experimental results show that the proposed method achieved an F1 score of 0.9175, out- performing baseline methods.
NLP North at WNUT-2020 Task 2: Pre-training versus Ensembling for Detection of Informative COVID-19 English Tweets
Authors: Anders Giovanni Møller, Rob van der Goot, Barbara Plank
Date: 2020-11-01 00:00:00
Proceeding: EMNLP (WNUT) 2020 11
Tasks: Task 2

With the COVID-19 pandemic raging world-wide since the beginning of the 2020 decade, the need for monitoring systems to track relevant information on social media is vitally important. This paper describes our submission to the WNUT-2020 Task 2: Identification of informative COVID-19 English Tweets. We investigate the effectiveness for a variety of classification models, and found that domain-specific pre-trained BERT models lead to the best performance. On top of this, we attempt a variety of ensembling strategies, but these attempts did not lead to further improvements. Our final best model, the standalone CT-BERT model, proved to be highly competitive, leading to a shared first place in the shared task. Our results emphasize the importance of domain and task-related pre-training.