pwc-archive/papers-with-abstracts huggingface dataset:
How Java Programmers Test Exceptional Behavior
Authors: Diego Marcilio, Carlo A. Furia
Date: 2020-12-01 00:00:00

Exceptions often signal faulty or undesired behavior; hence, high-quality test suites should also target exceptional behavior. This paper is a large-scale study of exceptional tests---which exercise exceptional behavior---in 1157 open-source Java projects hosted on GitHub. We analyzed JUnit exceptional tests to understand what kinds of exceptions are more frequently tested, what coding patterns are used, and how features of a project, such as its size and number of contributors, correlate to the characteristics of its exceptional tests. We found that exceptional tests are only 13% of all tests, but tend to be larger than other tests on average; unchecked exceptions are tested twice as frequently as checked ones; 42% of all exceptional tests use try/catch blocks and usually are larger than those using other idioms; and bigger projects with more contributors tend to have more exceptional tests written using different styles. The paper also zeroes in on several detailed examples involving  some of the largest analyzed projects, which refine the empirical results with qualitative evidence. The study's findings, and the capabilities of the tool we developed to analyze exceptional tests,  suggest several implications for the practice of software development and for follow-up empirical studies.
Cross-Lingual Pretraining Methods for Spoken Dialog
Authors: Anonymous
Date: 2021-03-17 00:00:00
Tasks: Spoken Dialogue Systems

There has been an increasing interest among NLP researchers towards learning generic representations. However, in the field of multilingual spoken dialogue systems, this problem remains overlooked. Indeed most of the pre-training methods focus on learning representations for written and non-conversational data or are restricted to the monolingual setting. In this work we (1) generalise existing losses to the multilingual setting, (2) develop a new set of losses to leverage parallel conversations when available.  These losses improve the learning of representations by fostering the deep encoder to better learn contextual dependencies. The pre-training relies on \texttt{OpenSubtitles}, a huge multilingual corpus that is composed of $24.3$G tokens; a by-product of the pre-processing includes multilingual aligned conversations. We also introduce two new multilingual tasks and a new benchmark on multilingual dialogue act labels called \texttt{MIAM}. We validate our pre-training on the three aforementioned tasks and show that our model using our newly designed losses achieves better performances than existing models. Our implementation will be available on \url{github.com} and preprocessed data will be available in Datasets
Characterizing Misclassifications of Deep NLP Models
Authors: Anonymous
Date: 2021-03-12 00:00:00
Tasks: named-entity-recognition, Named Entity Recognition, Named Entity Recognition (NER), Question Answering, Visual Question Answering, Visual Question Answering (VQA)

Understanding the reasons for misclassification is a critical point to improve a black-box classifier's performance. We propose a method to characterize these classification errors while considering the challenges that NLP applications pose, such as sparse and high dimensional discrete input spaces. Our approach discovers patterns over the input of a model that strongly correlate with the correctness of the classification. This allows identifying the systematic errors made by the models. We formalize the problem in terms of the Minimum Description Length principle to obtain non-redundant and easily interpretable results, and we propose the Premise algorithm to find good patterns in practice. The discovered patterns allow the user to take action and improve the model, e.g. through changes to the training data or model definition. On synthetic data and two real-world NLP tasks, we show that Premise performs well in practice. For two Visual Question Answering classifiers, we discover that they struggle with aspects like counting, location and reading, and for a Named Entity Recognition model, we leverage the found patterns to improve the F1 performance by almost 10% through targeted fine-tuning.
Open-domain question classification and completion in conversational information search
Authors: Anonymous
Date: 2021-03-11 00:00:00
Tasks: Classification

Searching for new information requires talking to the system. In this research, an Open-domain Conversational information search system has been developed. This system has been implemented using the TREC CAsT 2019 track, which is one of the first attempts to build a framework in this area. According to the user's previous questions, the system firstly completes the question (using the first and the previous question in each turn) and then classifies it (based on the question words). This system extracts the related answers according to the rules of each question. In this research, a simple yet effective method with high performance has been used, which on average, extracts 20% more relevant results than the baseline.
White Paper - Creating a Repository of Objectionable Online Content: Addressing Undesirable Biases and Ethical Considerations
Authors: Thamar Solorio, Mahsa Shafaei, Christos Smailis, Isabelle Augenstein, Margaret Mitchell, Ingrid Stapf, Ioannis Kakadiaris
Date: 2021-02-23 00:00:00

This white paper summarizes the authors' structured brainstorming regarding ethical considerations for creating an extensive repository of online content labeled with tags that describe potentially questionable content for young viewers. The workshop focused on four topics: 1) identifying risks for unintended biases in the data and labels, 2) how to reduce risks for unintended biases; 3) identifying ethical considerations of the annotation task, and 4) reducing the risks for the annotators.
Efficient Text-based Reinforcement Learning by Jointly Leveraging State and Commonsense Graph Representations
Authors: Anonymous
Date: 2021-02-22 00:00:00
Tasks: Decision Making, Graph Attention, reinforcement-learning, Reinforcement Learning (RL), text-based games

Text-based games (TBGs) have emerged as useful benchmarks for evaluating progress at the intersection of grounded language understanding and reinforcement learning (RL). Recent work has proposed the use of external knowledge -- like commonsense knowledge -- to improve the efficiency of RL agents for TBGs; and to resemble human-like decision-making skills. 
In this paper, we posit that to act efficiently in TBGs, an agent must be able to track the state of the game while retrieving and using relevant commonsense knowledge. Thus, we propose an agent for TBGs that induces a graph representation of the game state and jointly grounds it with a graph of commonsense knowledge from ConceptNet. This combination is achieved through bidirectional knowledge graph attention between the two symbolic representations. We show that agents that incorporate commonsense into the game state graph outperform baseline agents.
Did they direct the violence or admonish it? A cautionary tale on contronomy, androcentrism and back-translation foibles
Authors: Anonymous
Date: 2021-02-05 00:00:00
Tasks: Machine Translation, Translation

The recent raft of high-profile gaffes involving neural machine translation technology has brought to light the unreliability and brittleness of this fledgling technology. These revelations have worryingly coincided with  two other developments: The rise of back-translated text being increasingly used to augment training data in so termed low-resource natural language processing (NLP) scenarios and the emergence of 'AI-enhanced legal-tech' as a panacea that promises 'disruptive democratization' of access to legal services. In the backdrop of these quandaries, we present this cautionary tale where we shed light on the specifics of the risks surrounding cavalier deployment of this technology by exploring two specific failings: Androcentrism and Enantiosemy.
In this regard, we empirically investigate the fate of the pronouns and a list of contronyms when subjected to back-translation using the state-of-the-art Google translate API. Through this, we seek to highlight the extent of prevalence of the defaulting-to-the-masculine phenomenon in the context of engendered profession-related translations and also empirically demonstrate the scale and nature of threats pertaining to contronymous phrases covering both current-affairs and legal issues. All of the code and datasets generated in this paper have been open-sourced for the community to build on here: \url{https://bit.ly/3cw42gq}.
Does Putting a Linguist in the Loop Improve NLU Data Collection?
Authors: Anonymous
Date: 2021-02-02 00:00:00
Tasks: Natural Language Inference

Many crowdsourced NLP datasets contain systematic gaps and biases that are often identiﬁed only after data collection is complete. We take natural language inference as a test case to ask whether it is beneﬁcial to put a linguist ‘in the loop’ during data collection to dynamically identify and address these gaps by introducing novel constraints on the task. We directly compare three data collection protocols: (i) a baseline protocol, (ii) a linguist-in-the-loop intervention with iteratively-updated constraints, and (iii) an extension of linguist-in-the-loop that provides direct interaction between linguists and other crowdworkers via a chatroom. We ﬁnd that the datasets collected with linguist involvement are more reliably challenging than the baseline, though we do not see evidence that this beneﬁt extends to better out-of-domain model performance. The addition of a chat platform does not lead to any measurable differences in the resulting dataset. We suggest that using dynamic, expert-guided interventions improves data collection, but that continued one-on-one interaction between experts and crowdworkers is unnecessary.
The characteristic equation of the exceptional Jordan algebra: its eigenvalues, and their possible connection with the mass ratios of quarks and leptons
Authors: Anonymous
Date: 2021-01-30 00:00:00

The exceptional Jordan algebra [also known as the Albert algebra]  is the finite dimensional algebra of 3x3 Hermitean matrices with octonionic entries. Its automorphism group is the exceptional Lie group $F_4$. These matrices admit  a cubic characteristic equation whose eigenvalues are real and depend on the invariant trace, determinant, and an inner product made from the Jordan matrix. Also, there is some evidence in the literature that the group $F_4$ could play a role in the unification of the standard model symmetries, including the Lorentz symmetry. The octonion algebra is known to correctly yield the electric charge values (0, 1/3, 2/3, 1) for standard model fermions, via the eigenvalues of a $U(1)$ number operator, identified with $U(1)_{em}$. In the present article, we use the same octonionic representation of the fermions to compute the eigenvalues of the characteristic equation of  the Albert algebra, and compare the resulting eigenvalues with the known mass ratios for quarks and leptons. We find that the ratios of the eigenvalues correctly reproduce the [square root of the] known  mass ratios for up, charm and top quark. We also propose a diagrammatic representation of the standard model bosons, Higgs and three fermion generations, in terms of the octonions, exhibiting  an $F_4$ symmetry. We motivate from our Lagrangian as to why the eigenvalues computed in this work could bear a relation with mass ratios of quarks and leptons. In conjunction with the trace dynamics Lagrangian, the Jordan eigenvalues also provide  a first principles theoretical derivation of the low energy value of the  fine structure constant, yielding the value $1/137.04006$. The Karolyhazy correction to this value gives an exact match with the measured value of the constant, after assuming a specific value for the electro-weak symmetry breaking energy scale.
MERMAID: Metaphor Generation with Symbolism and Discriminative Decoding
Authors: Anonymous
Date: 2021-01-23 00:00:00
Tasks: Language Modeling, Language Modelling, Masked Language Modeling

Generating metaphors is a challenging task as it requires a proper understanding of abstract concepts, making connections between unrelated concepts, and deviating from the literal meaning. Based on a theoretically-grounded connection between metaphors and symbols, we propose a method to automatically construct a parallel corpus by transforming a large number of metaphorical sentences from the Gutenberg Poetry corpus \cite{jacobs2018gutenberg} to their literal counterpart using recent advances in masked language modeling coupled with commonsense inference. For the generation task, we incorporate a metaphor discriminator to guide the decoding of a sequence to sequence model fine-tuned on our parallel data to generate high quality metaphors. Human evaluation on an independent test set of literal statements shows that our best model generates metaphors better than three well-crafted baselines 66\% of the time on average. A task-based evaluation shows that human-written poems enhanced with metaphors proposed by our model are preferred 68\% of the time compared to poems without metaphors.
CLPLM: Character Level Pretrained Language Model for Extracting Support Phrases for Sentiment Labels
Authors: Raj Ratn Pranesh, Ambesh Shekhar, Sumit Kumar
Date: 2020-10-24 00:00:00
Tasks: Language Modeling, Language Modelling

In this paper, we have designed a character-level pre-trained language model for extracting support phrases from tweets based on the sentiment label. We also propose a character-level ensemble model designed by properly blending Pre-trained Contextual Embeddings (PCE) models- RoBERTa, BERT, and ALBERT along with Neural network models- RNN, CNN and WaveNet at different stages of the model. For a given tweet and associated sentiment label, our model predicts the span of phrases in a tweet that prompts the particular sentiment in the tweet. In our experiments, we have explored various model architectures and configuration for both single as well as ensemble models. We performed a systematic comparative analysis of all the model's performance based on the Jaccard score obtained. The best performing ensemble model obtained the highest Jaccard scores of 73.5, giving it a relative improvement of 2.4\% over the best performing single RoBERTa based character-level model, at 71.5(Jaccard score).
Flow Forecast: A deep learning for time series forecasting, classification, and anomaly detection framework built in PyTorch
Authors: Anonymous
Date: 2021-01-18 00:00:00
Tasks: Anomaly Detection, Time Series, Time Series Analysis, Time Series Forecasting

 Flow Forecast (FF) is a multi-purpose deep learning for time series forecasting, classification, and anomaly detection framework that contains state of the art time series models. Flow Forecast utilizes modular code design, unit/integration tests, model/prediction visualizations, and native cloud provider integration in order to allow researchers to rapidly experiment with new model architectures, benchmark their results on popular datasets and reproduce their results. Simultaneously it aids industry data scientists to deploy models to production, periodically retrain models, and explain model decisions to stakeholders.
Safe Coupled Deep Q-Learning for Recommendation Systems
Authors: Runsheng Yu, Yu Gong, Rundong Wang, Bo An, Qingwen Liu, Wenwu Ou
Date: 2021-01-08 00:00:00
Tasks: Q-Learning, Recommendation Systems, Reinforcement Learning (RL)

Reinforcement Learning (RL) is one of the prevailing approaches to optimize long-term user engagement in Recommendation Systems (RS). However, the well-known exploration strategies of RL (e.g., the $\epsilon$-greedy strategy) encourage agents to interact and explore the environment freely, which may recommend unpleasant items to the users frequently, violating their preferences and making them lose confidence in the RS platform. To avoid such irrelevant and unpleasant recommendations, we propose a novel safe RL approach to maximize accumulated long-term reward under the safety guarantee. Our contributions are three-fold. Firstly, we introduce a novel training scheme with two value functions to maximize the accumulated long-term reward under the safety constraint. Secondly, we theoretically show that our methods are able to converge and maintain safety with a high probability during the training process. Thirdly, we implement two practical methods, including a Simhash-based method as well as a relaxation method for large-scale environments. Experiments on immediate recommendation, sequential recommendations, as well as safe gridworld reveal that our methods outperform the state-of-the-arts dramatically.
COVID-19 Misinformation on Twitter: Multilingual Analysis
Authors: Raj Ratn Pranesh, Mehrdad Farokhenajd, Ambesh Shekhar, Genoveva Vargas-Solar
Date: 2021-01-06 00:00:00
Tasks: Misinformation, Rumour Detection

In the current scenario of the coronavirus disease pandemic (COVID-19), the Internet has become an important source of health information for users worldwide. During pandemic situations, myths, sensationalism, rumours and misinformation, generated intentionally or unintentionally, spread rapidly through social networks. Twitter is one of these popular social networks people use to share COVID-19related news, information, and thoughts that reflect their perception and opinion about the pandemic. Analysis of tweets for identifying misinformation can generate valuable insight to evaluate the quality and readability of online information about the COVID-19. This paper presents a multilingual COVID-19 related tweet analysis method, CMTA, that usesBERT, a deep learning model for multilingual tweet misinformation detection and classification.CMTA extracts features from multilingual textual data, which is then categorized into specific information classes. Classification is done by a Dense-CNN model trained on tweets manually annotated into information classes (i.e., ’false’, ’partly false’, ’misleading’). The paper assesses CMTA experimenting an analysis of multilingual tweets from February to June, showing the distribution type of information spread across different languages.
TextDecepter: Hard Label Black Box Attack on Text Classification
Authors: Anonymous
Date: 2020-10-02 00:00:00
Tasks: Classification, Decision Making, Sentiment Analysis, text-classification, Text Classification

Machine learning has been proven to be susceptible to carefully crafted samples, known as adversarial examples. The generation of these adversarial examples helps to make the models more robust and gives us an insight into the underlying decision-making of these models. Over the years, researchers have successfully attacked image classifiers in both, white and black-box settings. However, these methods are not directly applicable to texts as text data is discrete. In recent years, research on crafting adversarial examples against textual applications has been on the rise. In this paper, we present a novel approach for hard-label black-box attacks against Natural Language Processing (NLP) classifiers, where no model information is disclosed, and an attacker can only query the model to get a final decision of the classifier, without confidence scores of the classes involved. Such an attack scenario applies to real-world black-box models being used for security-sensitive applications such as sentiment analysis and toxic content detection.
Trust Region Policy Optimization Regularized via PID Control
Authors: Anonymous
Date: 2020-12-18 00:00:00

A large portion of popular RL methods can be categorized into on-policy learning, where TRPO is a classical method achieving success in wide range of applications. However, TRPO still has several problems on the computation and exploration. In order to make sure the updated policy lying in the trust region, it has to compute the inverse of Fisher information matrix, incurring large computational overhead. And since every action is drawn from the output distribution of itself, its exploration capacity is not satisfactory in some complex environments. In this work, we propose to formulate the TRPO in a KL-regularized formula and tune the Lagrangian multiplier by PID control method. To augment its exploration, we introduce a framework of multiple actors learning in parallel. The empirical experiments show the benefits of our methods over previous work.
A Structural Transformer with Relative Positions in Trees for Code-to-Sequence Tasks
Authors: Anonymous
Date: 2020-06-04 00:00:00
Tasks: Code Summarization, Machine Translation, Position, Source Code Summarization, Translation

We suggest two approaches to incorporate syntactic information into transformer models encoding trees (e.g. abstract syntax trees) and generating sequences. First, we use self-attention with relative position representations to consider structural relationships between nodes using a representation that encodes movements between any pair of nodes in the tree, and demonstrate how those movements can be computed efficiently on the fly. Second, we suggest an auxiliary loss enforcing the network to predict the lowest common ancestor of node pairs. We apply both methods to source code summarization tasks, where we outperform the state-of-the-art by up to 6% F1. On natural language machine translation, our models yield competitive results, while substantially faster than other. We also consistently outperform sequence-based transformers, and demonstrate that our method yields representations that are more closely aligned with the tree's structure.
Multilingual Multimodal Pretraining for Zero-Shot Cross-Lingual Transfer of Vision-Language Models
Authors: Anonymous
Date: 2020-12-07 00:00:00
Tasks: Cross-Lingual Transfer, Image Retrieval, Text-to-video search, Zero-Shot Cross-Lingual Transfer

This paper studies zero-shot cross-lingual transfer of vision-language models. Specifically, we focus on multilingual text-to-video search and propose a Transformer-based model that learns contextualized multilingual multimodal embeddings. Under a zero-shot setting, we empirically demonstrate that performance degrades significantly when we query the multilingual text-video model with non-English sentences. To address this problem, we introduce a multilingual multimodal pretraining strategy, and collect a new multilingual instructional video dataset (Multi-HowTo100M) for pretraining. Experiments on VTT show that our method significantly improves video search in non-English languages without additional annotations. Furthermore, when multilingual annotations are available, our method outperforms recent baselines by a large margin in multilingual text-to-video search on VTT and VATEX; as well as in multilingual text-to-image search on Multi30K. Our model and Multi-HowTo100M will be made available.
Looking Beyond Short-Premise Natural Language Inference for Downstream Tasks
Authors: Anonymous
Date: 2020-12-04 00:00:00
Tasks: Multiple-choice, Natural Language Inference, Reading Comprehension

Natural Language Inference (NLI) has garnered significant attention in recent years; however, the promise of applying NLI breakthroughs to other downstream NLP tasks has remained unfulfilled. In this work, we use the multiple-choice reading comprehension (MCRC) and checking factual correctness of textual summarization (CFCS) tasks to investigate potential reasons for this. Our findings show that: (1) the relatively shorter length of premises in traditional NLI datasets is the primary challenge prohibiting usage in downstream applications (which do better with longer contexts); (2) this challenge can be addressed by automatically converting resource-rich reading comprehension datasets into longer-premise NLI datasets; and (3) models trained on the converted, longer-premise  datasets outperform those trained using short-premise traditional NLI datasets on downstream tasks primarily due to the difference in premise lengths.