pwc-archive/papers-with-abstracts huggingface dataset:
Detect Profane Language in Streaming Services to Protect Young Audiences
Authors: Jingxiang Chen, Kai Wei, Xiang Hao
Date: 2021-08-01 00:00:00
Proceeding: ACL (ECNLP) 2021 8

With the rapid growth of online video streaming, recent years have seen increasing concerns about profane language in their content. Detecting profane language in streaming services is challenging due to the long sentences appeared in a video. While recent research on handling long sentences has focused on developing deep learning modeling techniques, little work has focused on techniques on improving data pipelines. In this work, we develop a data collection pipeline to address long sequence of texts and integrate this pipeline with a multi-head self-attention model. With this pipeline, our experiments show the self-attention model offers 12.5% relative accuracy improvement over state-of-the-art distilBERT model on profane language detection while requiring only 3% of parameters. This research designs a better system for informing users of profane language in video streaming services.
Exploring Inspiration Sets in a Data Programming Pipeline for Product Moderation
Authors: Justine Winkler, Simon Brugman, Bas van Berkel, Martha Larson
Date: 2021-08-01 00:00:00
Proceeding: ACL (ECNLP) 2021 8

We carry out a case study on the use of data programming to create data to train classifiers used for product moderation on a large e-commerce platform. Data programming is a recently-introduced technique that uses human-defined rules to generate training data sets without tedious item-by-item hand labeling. Our study investigates methods for allowing product moderators to quickly modify the rules given their knowledge of the domain and, especially, of textual item descriptions. Our results show promise that moderators can use this approach to steer the training data, making possible fast and close control of classifiers that detect policy violations.
Enhancing Aspect Extraction for Hindi
Authors: Arghya Bhattacharya, Alok Debnath, Manish Shrivastava
Date: 2021-08-01 00:00:00
Proceeding: ACL (ECNLP) 2021 8
Tasks: Aspect-Based Sentiment Analysis, Aspect-Based Sentiment Analysis (ABSA), Aspect Extraction, Sentiment Analysis

Aspect extraction is not a well-explored topic in Hindi, with only one corpus having been developed for the task. In this paper, we discuss the merits of the existing corpus in terms of quality, size, sparsity, and performance in aspect extraction tasks using established models. To provide a better baseline corpus for aspect extraction, we translate the SemEval 2014 aspect-based sentiment analysis dataset and annotate the aspects in that data. We provide rigorous guidelines and a replicable methodology for this task. We quantitatively evaluate the translations and annotations using inter-annotator agreement scores. We also evaluate our dataset using state-of-the-art neural aspect extraction models in both monolingual and multilingual settings and show that the models perform far better on our corpus than on the existing Hindi dataset. With this, we establish our corpus as the gold-standard aspect extraction dataset in Hindi.
Combining semantic search and twin product classification for recognition of purchasable items in voice shopping
Authors: Dieu-Thu Le, Verena Weber, Melanie Bradford
Date: 2021-08-01 00:00:00
Proceeding: ACL (ECNLP) 2021 8
Tasks: Retrieval, Spoken Language Understanding

The accuracy of an online shopping system via voice commands is particularly important and may have a great impact on customer trust. This paper focuses on the problem of detecting if an utterance contains actual and purchasable products, thus referring to a shopping-related intent in a typical Spoken Language Understanding architecture consist- ing of an intent classifier and a slot detec- tor. Searching through billions of products to check if a detected slot is a purchasable item is prohibitively expensive. To overcome this problem, we present a framework that (1) uses a retrieval module that returns the most rele- vant products with respect to the detected slot, and (2) combines it with a twin network that decides if the detected slot is indeed a pur- chasable item or not. Through various exper- iments, we show that this architecture outper- forms a typical slot detector approach, with a gain of +81% in accuracy and +41% in F1 score.
SupportNet: Neural Networks for Summary Generation and Key Segment Extraction from Technical Support Tickets
Authors: Vinayshekhar Bannihatti Kumar, Mohan Yarramsetty, Sharon Sun, Anukul Goel
Date: 2021-08-01 00:00:00
Proceeding: ACL (ECNLP) 2021 8
Tasks: Friction

We improve customer experience and gain their trust when their issues are resolved rapidly with less friction. Existing work has focused on reducing the overall case resolution time by binning a case into predefined categories and routing it to the desired support engineer. However, the actions taken by the engineer during case analysis and resolution are altogether ignored, even though it forms the bulk of the case resolution time. In this work, we propose two systems that enable support engineers to resolve cases faster. The first, a guidance extraction model, mines historical cases and provides technical guidance phrases to the support engineers. The phrases can then be used to educate the customer or to obtain critical information needed to resolve the case and thus minimize the number of correspondences between the engineer and customer. The second, a summarization model, creates an abstractive summary of the case to provide better context to the support engineer. Through quantitative evaluation we obtain an F1 score of 0.64 on the guidance extraction model and a BertScore (F1) of 0.55 on the summarization model.
Product Review Translation: Parallel Corpus Creation and Robustness towards User-generated Noisy Text
Authors: Kamal Kumar Gupta, Soumya Chennabasavaraj, Nikesh Garera, Asif Ekbal
Date: 2021-08-01 00:00:00
Proceeding: ACL (ECNLP) 2021 8
Tasks: Language Modelling, Machine Translation, NMT, Translation

Reviews written by the users for a particular product or service play an influencing role for the customers to make an informative decision. Although online e-commerce portals have immensely impacted our lives, available contents predominantly are in English language- often limiting its widespread usage. There is an exponential growth in the number of e-commerce users who are not proficient in English. Hence, there is a necessity to make these services available in non-English languages, especially in a multilingual country like India. This can be achieved by an in-domain robust machine translation (MT) system. However, the reviews written by the users pose unique challenges to MT, such as misspelled words, ungrammatical constructions, presence of colloquial terms, lack of resources such as in-domain parallel corpus etc. We address the above challenges by presenting an English–Hindi review domain parallel corpus. We train an English–to–Hindi neural machine translation (NMT) system to translate the product reviews available on e-commerce websites. By training the Transformer based NMT model over the generated data, we achieve a score of 33.26 BLEU points for English–to–Hindi translation. In order to make our NMT model robust enough to handle the noisy tokens in the reviews, we integrate a character based language model to generate word vectors and map the noisy tokens with their correct forms. Experiments on four language pairs, viz. English-Hindi, English-German, English-French, and English-Czech show the BLUE scores of 35.09, 28.91, 34.68 and 14.52 which are the improvements of 1.61, 1.05, 1.63 and 1.94, respectively, over the baseline.
ConTest: A Unit Test Completion Benchmark featuring Context
Authors: Johannes Villmow, Jonas Depoix, Adrian Ulges
Date: 2021-08-01 00:00:00
Proceeding: ACL (NLP4Prog) 2021 8

We introduce CONTEST, a benchmark for NLP-based unit test completion, the task of predicting a test’s assert statements given its setup and focal method, i.e. the method to be tested. ConTest is large-scale (with 365k datapoints). Besides the test code and tested code, it also features context code called by either. We found context to be crucial for accurately predicting assertions. We also introduce baselines based on transformer encoder-decoders, and study the effects of including syntactic information and context. Overall, our models achieve a BLEU score of 38.2, while only generating unparsable code in 1.92% of cases.
CommitBERT: Commit Message Generation Using Pre-Trained Programming Language Model
Authors: Tae Hwan Jung
Date: 2021-08-01 00:00:00
Proceeding: ACL (NLP4Prog) 2021 8
Tasks: Decoder, Language Modeling, Language Modelling, Machine Translation, NMT

Commit message is a document that summarizes source code changes in natural language. A good commit message clearly shows the source code changes, so this enhances collaboration between developers. Therefore, our work is to develop a model that automatically writes the commit message. To this end, we release 345K datasets consisting of code modification and commit messages in six programming languages (Python, PHP, Go, Java, JavaScript, and Ruby). Similar to the neural machine translation (NMT) model, using our dataset, we feed the code modification to the encoder input and the commit message to the decoder input and measure the result of the generated commit message with BLEU-4. Also, we propose the following two training methods to improve the result of generating the commit message: (1) A method of preprocessing the input to feed the code modification to the encoder input. (2) A method that uses an initial weight suitable for the code domain to reduce the gap in contextual representation between programming language (PL) and natural language (NL).
Time-Efficient Code Completion Model for the R Programming Language
Authors: Artem Popov, Dmitrii Orekhov, Denis Litvinov, Nikolay Korolev, Gleb Morgachev
Date: 2021-08-01 00:00:00
Proceeding: ACL (NLP4Prog) 2021 8
Tasks: Code Completion, Language Modeling, Language Modelling

In this paper we present a deep learning code completion model for the R language. We introduce several techniques to utilize language modeling based architecture in the code completion task. With these techniques, the model requires low resources, but still achieves high quality. We also present an evaluation dataset for the R language completion task. Our dataset contains multiple autocompletion usage contexts that provides robust validation results. The dataset is publicly available.
DIRECT : A Transformer-based Model for Decompiled Identifier Renaming
Authors: Vikram Nitin, Anthony Saieva, Baishakhi Ray, Gail Kaiser
Date: 2021-08-01 00:00:00
Proceeding: ACL (NLP4Prog) 2021 8
Tasks: Malware Analysis

Decompiling binary executables to high-level code is an important step in reverse engineering scenarios, such as malware analysis and legacy code maintenance. However, the generated high-level code is difficult to understand since the original variable names are lost. In this paper, we leverage transformer models to reconstruct the original variable names from decompiled code. Inherent differences between code and natural language present certain challenges in applying conventional transformer-based architectures to variable name recovery. We propose DIRECT, a novel transformer-based architecture customized specifically for the task at hand. We evaluate our model on a dataset of decompiled functions and find that DIRECT outperforms the previous state-of-the-art model by up to 20%. We also present ablation studies evaluating the impact of each of our modifications. We make the source code of DIRECT available to encourage reproducible research.
Bag-of-Words Baselines for Semantic Code Search
Authors: Xinyu Zhang, Ji Xin, Andrew Yates, Jimmy Lin
Date: 2021-08-01 00:00:00
Proceeding: ACL (NLP4Prog) 2021 8
Tasks: Code Search, Information Retrieval, Natural Language Queries, Retrieval

The task of semantic code search is to retrieve code snippets from a source code corpus based on an information need expressed in natural language. The semantic gap between natural language and programming languages has for long been regarded as one of the most significant obstacles to the effectiveness of keyword-based information retrieval (IR) methods. It is a common assumption that “traditional” bag-of-words IR methods are poorly suited for semantic code search: our work empirically investigates this assumption. Specifically, we examine the effectiveness of two traditional IR methods, namely BM25 and RM3, on the CodeSearchNet Corpus, which consists of natural language queries paired with relevant code snippets. We find that the two keyword-based methods outperform several pre-BERT neural models. We also compare several code-specific data pre-processing strategies and find that specialized tokenization improves effectiveness.
Domain adaptation challenges of BERT in tokenization and sub-word representations of Out-of-Vocabulary words
Authors: Anmol Nayak, Hariprasad Timmapathini, Karthikeyan Ponnalagu, Vijendran Gopalan Venkoparao
Date: 2020-11-01 00:00:00
Proceeding: EMNLP (insights) 2020 11
Tasks: Domain Adaptation, Word Embeddings

BERT model (Devlin et al., 2019) has achieved significant progress in several Natural Language Processing (NLP) tasks by leveraging the multi-head self-attention mechanism (Vaswani et al., 2017) in its architecture. However, it still has several research challenges which are not tackled well for domain specific corpus found in industries. In this paper, we have highlighted these problems through detailed experiments involving analysis of the attention scores and dynamic word embeddings with the BERT-Base-Uncased model. Our experiments have lead to interesting findings that showed: 1) Largest substring from the left that is found in the vocabulary (in-vocab) is always chosen at every sub-word unit that can lead to suboptimal tokenization choices, 2) Semantic meaning of a vocabulary word deteriorates when found as a substring in an Out-Of-Vocabulary (OOV) word, and 3) Minor misspellings in words are inadequately handled. We believe that if these challenges are tackled, it will significantly help the domain adaptation aspect of BERT.
Q. Can Knowledge Graphs be used to Answer Boolean Questions? A. It’s complicated!
Authors: Daria Dzendzik, Carl Vogel, Jennifer Foster
Date: 2020-11-01 00:00:00
Proceeding: EMNLP (insights) 2020 11
Tasks: Entity Linking, Graph Neural Network, Knowledge Graphs, Machine Reading Comprehension, Reading Comprehension

In this paper we explore the problem of machine reading comprehension, focusing on the BoolQ dataset of Yes/No questions. We carry out an error analysis of a BERT-based machine reading comprehension model on this dataset, revealing issues such as unstable model behaviour and some noise within the dataset itself. We then experiment with two approaches for integrating information from knowledge graphs: (i) concatenating knowledge graph triples to text passages and (ii) encoding knowledge with a Graph Neural Network. Neither of these approaches show a clear improvement and we hypothesize that this may be due to a combination of inaccuracies in the knowledge graph, imprecision in entity linking, and the models’ inability to capture additional information from knowledge graphs.
How Far Can We Go with Data Selection? A Case Study on Semantic Sequence Tagging Tasks
Authors: Samuel Louvan, Bernardo Magnini
Date: 2020-11-01 00:00:00
Proceeding: EMNLP (insights) 2020 11
Tasks: Multi-Task Learning, Transfer Learning

Although several works have addressed the role of data selection to improve transfer learning for various NLP tasks, there is no consensus about its real benefits and, more generally, there is a lack of shared practices on how it can be best applied. We propose a systematic approach aimed at evaluating data selection in scenarios of increasing complexity. Specifically, we compare the case in which source and target tasks are the same while source and target domains are different, against the more challenging scenario where both tasks and domains are different. We run a number of experiments on semantic sequence tagging tasks, which are relatively less investigated in data selection, and conclude that data selection has more benefit on the scenario when the tasks are the same, while in case of different (although related) tasks from distant domains, a combination of data selection and multi-task learning is ineffective for most cases.
Which Matters Most? Comparing the Impact of Concept and Document Relationships in Topic Models
Authors: Silvia Terragni, Debora Nozza, Elisabetta Fersini, Messina Enza
Date: 2020-11-01 00:00:00
Proceeding: EMNLP (insights) 2020 11
Tasks: Topic Models

Topic models have been widely used to discover hidden topics in a collection of documents. In this paper, we propose to investigate the role of two different types of relational information, i.e. document relationships and concept relationships. While exploiting the document network significantly improves topic coherence, the introduction of concepts and their relationships does not influence the results both quantitatively and qualitatively.
How Effectively Can Machines Defend Against Machine-Generated Fake News? An Empirical Study
Authors: Meghana Moorthy Bhat, Srinivasan Parthasarathy
Date: 2020-11-01 00:00:00
Proceeding: EMNLP (insights) 2020 11
Tasks: Diagnostic, Sensitivity

We empirically study the effectiveness of machine-generated fake news detectors by understanding the model’s sensitivity to different synthetic perturbations during test time. The current machine-generated fake news detectors rely on provenance to determine the veracity of news. Our experiments find that the success of these detectors can be limited since they are rarely sensitive to semantic perturbations and are very sensitive to syntactic perturbations. Also, we would like to open-source our code and believe it could be a useful diagnostic tool for evaluating models aimed at fighting machine-generated fake news.
Label Propagation-Based Semi-Supervised Learning for Hate Speech Classification
Authors: Ashwin Geet D’Sa, Irina Illina, Dominique Fohr, Dietrich Klakow, Dana Ruiter
Date: 2020-11-01 00:00:00
Proceeding: EMNLP (insights) 2020 11
Tasks: Classification

Research on hate speech classification has received increased attention. In real-life scenarios, a small amount of labeled hate speech data is available to train a reliable classifier. Semi-supervised learning takes advantage of a small amount of labeled data and a large amount of unlabeled data. In this paper, label propagation-based semi-supervised learning is explored for the task of hate speech classification. The quality of labeling the unlabeled set depends on the input representations. In this work, we show that pre-trained representations are label agnostic, and when used with label propagation yield poor results. Neural network-based fine-tuning can be adopted to learn task-specific representations using a small amount of labeled data. We show that fully fine-tuned representations may not always be the best representations for the label propagation and intermediate representations may perform better in a semi-supervised setup.
Layout-Aware Text Representations Harm Clustering Documents by Type
Authors: Catherine Finegan-Dollak, Ashish Verma
Date: 2020-11-01 00:00:00
Proceeding: EMNLP (insights) 2020 11
Tasks: Articles, Clustering, Vocal Bursts Type Prediction

Clustering documents by type—grouping invoices with invoices and articles with articles—is a desirable first step for organizing large collections of document scans. Humans approaching this task use both the semantics of the text and the document layout to assist in grouping like documents. LayoutLM (Xu et al., 2019), a layout-aware transformer built on top of BERT with state-of-the-art performance on document-type classification, could reasonably be expected to outperform regular BERT (Devlin et al., 2018) for document-type clustering. However, we find experimentally that BERT significantly outperforms LayoutLM on this task (p <0.001). We analyze clusters to show where layout awareness is an asset and where it is a liability.
An Analysis of Capsule Networks for Part of Speech Tagging in High- and Low-resource Scenarios
Authors: Andrew Zupon, Faiz Rafique, Mihai Surdeanu
Date: 2020-11-01 00:00:00
Proceeding: EMNLP (insights) 2020 11
Tasks: Part-Of-Speech Tagging

Neural networks are a common tool in NLP, but it is not always clear which architecture to use for a given task. Different tasks, different languages, and different training conditions can all affect how a neural network will perform. Capsule Networks (CapsNets) are a relatively new architecture in NLP. Due to their novelty, CapsNets are being used more and more in NLP tasks. However, their usefulness is still mostly untested.In this paper, we compare three neural network architectures—LSTM, CNN, and CapsNet—on a part of speech tagging task. We compare these architectures in both high- and low-resource training conditions and find that no architecture consistently performs the best. Our analysis shows that our CapsNet performs nearly as well as a more complex LSTM under certain training conditions, but not others, and that our CapsNet almost always outperforms our CNN. We also find that our CapsNet implementation shows faster prediction times than the LSTM for Scottish Gaelic but not for Spanish, highlighting the effect that the choice of languages can have on the models.
Can Knowledge Graph Embeddings Tell Us What Fact-checked Claims Are About?
Authors: Valentina Beretta, Sébastien Harispe, Katarina Boland, Luke Lo Seen, Konstantin Todorov, Andon Tchechmedjiev
Date: 2020-11-01 00:00:00
Proceeding: EMNLP (insights) 2020 11
Tasks: Fact Checking, Graph Embedding, Knowledge Graph Embeddings, Misinformation

The web offers a wealth of discourse data that help researchers from various fields analyze debates about current societal issues and gauge the effects on society of important phenomena such as misinformation spread. Such analyses often revolve around claims made by people about a given topic of interest. Fact-checking portals offer partially structured information that can assist such analysis. However, exploiting the network structure of such online discourse data is as of yet under-explored. We study the effectiveness of using neural-graph embedding features for claim topic prediction and their complementarity with text embeddings. We show that graph embeddings are modestly complementary with text embeddings, but the low performance of graph embedding features alone indicate that the model fails to capture topological features pertinent of the topic prediction task.
Do Transformers Dream of Inference, or Can Pretrained Generative Models Learn Implicit Inferential Rules?
Authors: Zhengzhong Liang, Mihai Surdeanu
Date: 2020-11-01 00:00:00
Proceeding: EMNLP (insights) 2020 11
Tasks: Multi-hop Question Answering, Question Answering

Large pretrained language models (LM) have been used successfully for multi-hop question answering. However, most of these directions are not interpretable, as they do not make the inference hops necessary to explain a candidate answer explicitly. In this work, we investigate the capability of a state-of-the-art transformer LM to generate explicit inference hops, i.e., to infer a new statement necessary to answer a question given some premise input statements. Our analysis shows that such LMs can generate new statements for some simple inference types, but performance remains poor for complex, real-world inference types such as those that require monotonicity, composition, and commonsense knowledge.