pwc-archive/papers-with-abstracts huggingface dataset:
Distilling BERT for low complexity network training
Authors: Bansidhar Mangalwedhekar
Date: 2021-05-13 00:00:00
Tasks: Sentiment Analysis, SST-2

This paper studies the efficiency of transferring BERT learnings to low complexity models like BiLSTM, BiLSTM with attention and shallow CNNs using sentiment analysis on SST-2 dataset. It also compares the complexity of inference of the BERT model with these lower complexity models and underlines the importance of these techniques in enabling high performance NLP models on edge devices like mobiles, tablets and MCU development boards like Raspberry Pi etc. and enabling exciting new applications.
Stochastic-Shield: A Probabilistic Approach Towards Training-Free Adversarial Defense in Quantized CNNs
Authors: Lorena Qendro, Sangwon Ha, René de Jong, Partha Maji
Date: 2021-05-13 00:00:00
Tasks: Adversarial Defense, Deep Learning, input filtering, Probabilistic Deep Learning

Quantized neural networks (NN) are the common standard to efficiently deploy deep learning models on tiny hardware platforms. However, we notice that quantized NNs are as vulnerable to adversarial attacks as the full-precision models. With the proliferation of neural networks on small devices that we carry or surround us, there is a need for efficient models without sacrificing trust in the prediction in presence of malign perturbations. Current mitigation approaches often need adversarial training or are bypassed when the strength of adversarial examples is increased. In this work, we investigate how a probabilistic framework would assist in overcoming the aforementioned limitations for quantized deep learning models. We explore Stochastic-Shield: a flexible defense mechanism that leverages input filtering and a probabilistic deep learning approach materialized via Monte Carlo Dropout. We show that it is possible to jointly achieve efficiency and robustness by accurately enabling each module without the burden of re-retraining or ad hoc fine-tuning.
Sanity Simulations for Saliency Methods
Authors: Joon Sik Kim, Gregory Plumb, Ameet Talwalkar
Date: 2021-05-13 00:00:00
Tasks: Benchmarking

Saliency methods are a popular class of feature attribution explanation methods that aim to capture a model's predictive reasoning by identifying "important" pixels in an input image. However, the development and adoption of these methods are hindered by the lack of access to ground-truth model reasoning, which prevents accurate evaluation. In this work, we design a synthetic benchmarking framework, SMERF, that allows us to perform ground-truth-based evaluation while controlling the complexity of the model's reasoning. Experimentally, SMERF reveals significant limitations in existing saliency methods and, as a result, represents a useful tool for the development of new saliency methods.
Improved Algorithms for Agnostic Pool-based Active Classification
Authors: Julian Katz-Samuels, Jifan Zhang, Lalit Jain, Kevin Jamieson
Date: 2021-05-13 00:00:00
Tasks: Active Learning, Binary Classification, Classification, Experimental Design, image-classification, Image Classification

We consider active learning for binary classification in the agnostic pool-based setting. The vast majority of works in active learning in the agnostic setting are inspired by the CAL algorithm where each query is uniformly sampled from the disagreement region of the current version space. The sample complexity of such algorithms is described by a quantity known as the disagreement coefficient which captures both the geometry of the hypothesis space as well as the underlying probability space. To date, the disagreement coefficient has been justified by minimax lower bounds only, leaving the door open for superior instance dependent sample complexities. In this work we propose an algorithm that, in contrast to uniform sampling over the disagreement region, solves an experimental design problem to determine a distribution over examples from which to request labels. We show that the new approach achieves sample complexity bounds that are never worse than the best disagreement coefficient-based bounds, but in specific cases can be dramatically smaller. From a practical perspective, the proposed algorithm requires no hyperparameters to tune (e.g., to control the aggressiveness of sampling), and is computationally efficient by means of assuming access to an empirical risk minimization oracle (without any constraints). Empirically, we demonstrate that our algorithm is superior to state of the art agnostic active learning algorithms on image classification datasets.
BERT Busters: Outlier Dimensions that Disrupt Transformers
Authors: Olga Kovaleva, Saurabh Kulshreshtha, Anna Rogers, Anna Rumshisky
Date: 2021-05-14 00:00:00
Proceeding: Findings (ACL) 2021 8
Tasks: Position

Multiple studies have shown that Transformers are remarkably robust to pruning. Contrary to this received wisdom, we demonstrate that pre-trained Transformer encoders are surprisingly fragile to the removal of a very small number of features in the layer outputs (<0.0001% of model weights). In case of BERT and other pre-trained encoder Transformers, the affected component is the scaling factors and biases in the LayerNorm. The outliers are high-magnitude normalization parameters that emerge early in pre-training and show up consistently in the same dimensional position throughout the model. We show that disabling them significantly degrades both the MLM loss and the downstream task performance. This effect is observed across several BERT-family models and other popular pre-trained Transformer architectures, including BART, XLNet and ELECTRA; we also show a similar effect in GPT-2.
EASE: Extractive-Abstractive Summarization with Explanations
Authors: Haoran Li, Arash Einolghozati, Srinivasan Iyer, Bhargavi Paranjape, Yashar Mehdad, Sonal Gupta, Marjan Ghazvininejad
Date: 2021-05-14 00:00:00
Tasks: Abstractive Text Summarization, Document Summarization, Text Generation

Current abstractive summarization systems outperform their extractive counterparts, but their widespread adoption is inhibited by the inherent lack of interpretability. To achieve the best of both worlds, we propose EASE, an extractive-abstractive framework for evidence-based text generation and apply it to document summarization. We present an explainable summarization system based on the Information Bottleneck principle that is jointly trained for extraction and abstraction in an end-to-end fashion. Inspired by previous research that humans use a two-stage framework to summarize long documents (Jing and McKeown, 2000), our framework first extracts a pre-defined amount of evidence spans as explanations and then generates a summary using only the evidence. Using automatic and human evaluations, we show that explanations from our framework are more relevant than simple baselines, without substantially sacrificing the quality of the generated summary.
Counterfactual Interventions Reveal the Causal Effect of Relative Clause Representations on Agreement Prediction
Authors: Shauli Ravfogel, Grusha Prasad, Tal Linzen, Yoav Goldberg
Date: 2021-05-14 00:00:00
Proceeding: CoNLL (EMNLP) 2021 11
Tasks: counterfactual, Sentence

When language models process syntactically complex sentences, do they use their representations of syntax in a manner that is consistent with the grammar of the language? We propose AlterRep, an intervention-based method to address this question. For any linguistic feature of a given sentence, AlterRep generates counterfactual representations by altering how the feature is encoded, while leaving intact all other aspects of the original representation. By measuring the change in a model's word prediction behavior when these counterfactual representations are substituted for the original ones, we can draw conclusions about the causal effect of the linguistic feature in question on the model's behavior. We apply this method to study how BERT models of different sizes process relative clauses (RCs). We find that BERT variants use RC boundary information during word prediction in a manner that is consistent with the rules of English grammar; this RC boundary information generalizes to a considerable extent across different RC types, suggesting that BERT represents RCs as an abstract linguistic category.
Thank you BART! Rewarding Pre-Trained Models Improves Formality Style Transfer
Authors: Huiyuan Lai, Antonio Toral, Malvina Nissim
Date: 2021-05-14 00:00:00
Proceeding: ACL 2021 5
Tasks: Formality Style Transfer, Style Transfer

Scarcity of parallel data causes formality style transfer models to have scarce success in preserving content. We show that fine-tuning pre-trained language (GPT-2) and sequence-to-sequence (BART) models boosts content preservation, and that this is possible even with limited amounts of parallel data. Augmenting these models with rewards that target style and content -- the two core aspects of the task -- we achieve a new state-of-the-art.
Towards Navigation by Reasoning over Spatial Configurations
Authors: Yue Zhang, Quan Guo, Parisa Kordjamshidi
Date: 2021-05-14 00:00:00
Proceeding: ACL (splurobonlp) 2021 8
Tasks: Spatial Reasoning

We deal with the navigation problem where the agent follows natural language instructions while observing the environment. Focusing on language understanding, we show the importance of spatial semantics in grounding navigation instructions into visual perceptions. We propose a neural agent that uses the elements of spatial configurations and investigate their influence on the navigation agent's reasoning ability. Moreover, we model the sequential execution order and align visual objects with spatial configurations in the instruction. Our neural agent improves strong baselines on the seen environments and shows competitive performance on the unseen environments. Additionally, the experimental results demonstrate that explicit modeling of spatial semantic elements in the instructions can improve the grounding and spatial reasoning of the model.
Empathetic Dialog Generation with Fine-Grained Intents
Authors: Yubo Xie, Pearl Pu
Date: 2021-05-14 00:00:00
Proceeding: CoNLL (EMNLP) 2021 11
Tasks: Empathetic Response Generation, Response Generation

Empathetic dialog generation aims at generating coherent responses following previous dialog turns and, more importantly, showing a sense of caring and a desire to help. Existing models either rely on pre-defined emotion labels to guide the response generation, or use deterministic rules to decide the emotion of the response. With the advent of advanced language models, it is possible to learn subtle interactions directly from the dataset, providing that the emotion categories offer sufficient nuances and other non-emotional but emotional regulating intents are included. In this paper, we describe how to incorporate a taxonomy of 32 emotion categories and 8 additional emotion regulating intents to succeed the task of empathetic response generation. To facilitate the training, we also curated a large-scale emotional dialog dataset from movie subtitles. Through a carefully designed crowdsourcing experiment, we evaluated and demonstrated how our model produces more empathetic dialogs compared with its baselines.
Locate and Label: A Two-stage Identifier for Nested Named Entity Recognition
Authors: Yongliang Shen, Xinyin Ma, Zeqi Tan, Shuai Zhang, Wen Wang, Weiming Lu
Date: 2021-05-14 00:00:00
Proceeding: ACL 2021 5
Tasks: Chinese Named Entity Recognition, named-entity-recognition, Named Entity Recognition, Named Entity Recognition (NER), Nested Named Entity Recognition, regression

Named entity recognition (NER) is a well-studied task in natural language processing. Traditional NER research only deals with flat entities and ignores nested entities. The span-based methods treat entity recognition as a span classification task. Although these methods have the innate ability to handle nested NER, they suffer from high computational cost, ignorance of boundary information, under-utilization of the spans that partially match with entities, and difficulties in long entity recognition. To tackle these issues, we propose a two-stage entity identifier. First we generate span proposals by filtering and boundary regression on the seed spans to locate the entities, and then label the boundary-adjusted span proposals with the corresponding categories. Our method effectively utilizes the boundary information of entities and partially matched spans during training. Through boundary regression, entities of any length can be covered theoretically, which improves the ability to recognize long entities. In addition, many low-quality seed spans are filtered out in the first stage, which reduces the time complexity of inference. Experiments on nested NER datasets demonstrate that our proposed method outperforms previous state-of-the-art models.
Classifying Long Clinical Documents with Pre-trained Transformers
Authors: Xin Su, Timothy Miller, Xiyu Ding, Majid Afshar, Dmitriy Dligach
Date: 2021-05-14 00:00:00
Tasks: Sentence

Automatic phenotyping is a task of identifying cohorts of patients that match a predefined set of criteria. Phenotyping typically involves classifying long clinical documents that contain thousands of tokens. At the same time, recent state-of-art transformer-based pre-trained language models limit the input to a few hundred tokens (e.g. 512 tokens for BERT). We evaluate several strategies for incorporating pre-trained sentence encoders into document-level representations of clinical text, and find that hierarchical transformers without pre-training are competitive with task pre-trained models.
DaLAJ - a dataset for linguistic acceptability judgments for Swedish: Format, baseline, sharing
Authors: Elena Volodina, Yousuf Ali Mohammed, Julia Klezl
Date: 2021-05-14 00:00:00
Tasks: Binary Classification, Linguistic Acceptability, Sentence, TAG

We present DaLAJ 1.0, a Dataset for Linguistic Acceptability Judgments for Swedish, comprising 9 596 sentences in its first version; and the initial experiment using it for the binary classification task. DaLAJ is based on the SweLL second language learner data, consisting of essays at different levels of proficiency. To make sure the dataset can be freely available despite the GDPR regulations, we have sentence-scrambled learner essays and removed part of the metadata about learners, keeping for each sentence only information about the mother tongue and the level of the course where the essay has been written. We use the normalized version of learner language as the basis for the DaLAJ sentences, and keep only one error per sentence. We repeat the same sentence for each individual correction tag used in the sentence. For DaLAJ 1.0 we have used four error categories (out of 35 available in SweLL), all connected to lexical or word-building choices. Our baseline results for the binary classification show an accuracy of 58% for DaLAJ 1.0 using BERT embeddings. The dataset is included in the SwedishGlue (Swe. SuperLim) benchmark. Below, we describe the format of the dataset, first experiments, our insights and the motivation for the chosen approach to data sharing.
Dynamic Multi-Branch Layers for On-Device Neural Machine Translation
Authors: Zhixing Tan, Zeyuan Yang, Meng Zhang, Qun Liu, Maosong Sun, Yang Liu
Date: 2021-05-14 00:00:00
Tasks: Machine Translation, NMT, Translation

With the rapid development of artificial intelligence (AI), there is a trend in moving AI applications, such as neural machine translation (NMT), from cloud to mobile devices. Constrained by limited hardware resources and battery, the performance of on-device NMT systems is far from satisfactory. Inspired by conditional computation, we propose to improve the performance of on-device NMT systems with dynamic multi-branch layers. Specifically, we design a layer-wise dynamic multi-branch network with only one branch activated during training and inference. As not all branches are activated during training, we propose shared-private reparameterization to ensure sufficient training for each branch. At almost the same computational cost, our method achieves improvements of up to 1.7 BLEU points on the WMT14 English-German translation task and 1.8 BLEU points on the WMT20 Chinese-English translation task over the Transformer model, respectively. Compared with a strong baseline that also uses multiple branches, the proposed method is up to 1.5 times faster with the same number of parameters.
Adversarial Learning for Zero-Shot Stance Detection on Social Media
Authors: Emily Allaway, Malavika Srikanth, Kathleen McKeown
Date: 2021-05-14 00:00:00
Proceeding: NAACL 2021 4
Tasks: Stance Detection, Zero-Shot Stance Detection

Stance detection on social media can help to identify and understand slanted news or commentary in everyday life. In this work, we propose a new model for zero-shot stance detection on Twitter that uses adversarial learning to generalize across topics. Our model achieves state-of-the-art performance on a number of unseen test topics with minimal computational costs. In addition, we extend zero-shot stance detection to new topics, highlighting future directions for zero-shot transfer.
Shades of confusion: Lexical uncertainty modulates ad hoc coordination in an interactive communication task
Authors: Sonia K. Murthy, Thomas L. Griffiths, Robert D. Hawkins
Date: 2021-05-13 00:00:00

There is substantial variability in the expectations that communication partners bring into interactions, creating the potential for misunderstandings. To directly probe these gaps and our ability to overcome them, we propose a communication task based on color-concept associations. In Experiment 1, we establish several key properties of the mental representations of these expectations, or lexical priors, based on recent probabilistic theories. Associations are more variable for abstract concepts, variability is represented as uncertainty within each individual, and uncertainty enables accurate predictions about whether others are likely to share the same association. In Experiment 2, we then examine the downstream consequences of these representations for communication. Accuracy is initially low when communicating about concepts with more variable associations, but rapidly increases as participants form ad hoc conventions. Together, our findings suggest that people cope with variability by maintaining well-calibrated uncertainty about their partner and appropriately adaptable representations of their own.
Efficient Spiking Neural Networks with Radix Encoding
Authors: Zhehui Wang, Xiaozhe Gu, Rick Goh, Joey Tianyi Zhou, Tao Luo
Date: 2021-05-14 00:00:00

Spiking neural networks (SNNs) have advantages in latency and energy efficiency over traditional artificial neural networks (ANNs) due to its event-driven computation mechanism and replacement of energy-consuming weight multiplications with additions. However, in order to reach accuracy of its ANN counterpart, it usually requires long spike trains to ensure the accuracy. Traditionally, a spike train needs around one thousand time steps to approach similar accuracy as its ANN counterpart. This offsets the computation efficiency brought by SNNs because longer spike trains mean a larger number of operations and longer latency. In this paper, we propose a radix encoded SNN with ultra-short spike trains. In the new model, the spike train takes less than ten time steps. Experiments show that our method demonstrates 25X speedup and 1.1% increment on accuracy, compared with the state-of-the-art work on VGG-16 network architecture and CIFAR-10 dataset.
Multi-Objective Optimisation of Cortical Spiking Neural Networks With Genetic Algorithms
Authors: James Fitzgerald, KongFatt Wong-Lin
Date: 2021-05-14 00:00:00

Spiking neural networks (SNNs) communicate through the all-or-none spiking activity of neurons. However, fitting the large number of SNN model parameters to observed neural activity patterns, for example, in biological experiments, remains a challenge. Previous work using genetic algorithm (GA) optimisation on a specific efficient SNN model, using the Izhikevich neuronal model, was limited to a single parameter and objective. This work applied a version of GA, called non-dominated sorting GA (NSGA-III), to demonstrate the feasibility of performing multi-objective optimisation on the same SNN, focusing on searching network connectivity parameters to achieve target firing rates of excitatory and inhibitory neuronal types, including across different network connectivity sparsity. We showed that NSGA-III could readily optimise for various firing rates. Notably, when the excitatory neural firing rates were higher than or equal to that of inhibitory neurons, the errors were small. Moreover, when connectivity sparsity was considered as a parameter to be optimised, the optimal solutions required sparse network connectivity. We also found that for excitatory neural firing rates lower than that of inhibitory neurons, the errors were generally larger. Overall, we have successfully demonstrated the feasibility of implementing multi-objective GA optimisation on network parameters of recurrent and sparse SNN.
Quantifying the Impact of Boundary Constraint Handling Methods on Differential Evolution
Authors: Rick Boks, Anna V. Kononova, Hao Wang
Date: 2021-05-14 00:00:00
Tasks: Benchmarking

Constraint handling is one of the most influential aspects of applying metaheuristics to real-world applications, which can hamper the search progress if treated improperly. In this work, we focus on a particular case - the box constraints, for which many boundary constraint handling methods (BCHMs) have been proposed. We call for the necessity of studying the impact of BCHMs on metaheuristics' performance and behavior, which receives seemingly little attention in the field. We target quantifying such impacts through systematic benchmarking by investigating 28 major variants of Differential Evolution (DE) taken from the modular DE framework (by combining different mutation and crossover operators) and $13$ commonly applied BCHMs, resulting in $28 \times 13 = 364$ algorithm instances after pairing DE variants with BCHMs. After executing the algorithm instances on the well-known BBOB/COCO problem set, we analyze the best-reached objective function value (performance-wise) and the percentage of repaired solutions (behavioral) using statistical ranking methods for each combination of mutation, crossover, and BBOB function group. Our results clearly show that the choice of BCHMs substantially affects the empirical performance as well as the number of generated infeasible solutions, which allows us to provide general guidelines for selecting an appropriate BCHM for a given scenario.
Learning Gaussian Graphical Models with Latent Confounders
Authors: Ke Wang, Alexander Franks, Sang-Yun Oh
Date: 2021-05-14 00:00:00

Gaussian Graphical models (GGM) are widely used to estimate the network structures in many applications ranging from biology to finance. In practice, data is often corrupted by latent confounders which biases inference of the underlying true graphical structure. In this paper, we compare and contrast two strategies for inference in graphical models with latent confounders: Gaussian graphical models with latent variables (LVGGM) and PCA-based removal of confounding (PCA+GGM). While these two approaches have similar goals, they are motivated by different assumptions about confounding. In this paper, we explore the connection between these two approaches and propose a new method, which combines the strengths of these two approaches. We prove the consistency and convergence rate for the PCA-based method and use these results to provide guidance about when to use each method. We demonstrate the effectiveness of our methodology using both simulations and in two real-world applications.