pwc-archive/papers-with-abstracts huggingface dataset:
COSEA: Convolutional Code Search with Layer-wise Attention
Authors: Hao Wang, Jia Zhang, Yingce Xia, Jiang Bian, Chao Zhang, Tie-Yan Liu
Date: 2020-10-19 00:00:00
Tasks: Code Search

Semantic code search, which aims to retrieve code snippets relevant to a given natural language query, has attracted many research efforts with the purpose of accelerating software development. The huge amount of online publicly available code repositories has prompted the employment of deep learning techniques to build state-of-the-art code search models. Particularly, they leverage deep neural networks to embed codes and queries into a unified semantic vector space and then use the similarity between code's and query's vectors to approximate the semantic correlation between code and the query. However, most existing studies overlook the code's intrinsic structural logic, which indeed contains a wealth of semantic information, and fails to capture intrinsic features of codes. In this paper, we propose a new deep learning architecture, COSEA, which leverages convolutional neural networks with layer-wise attention to capture the valuable code's intrinsic structural logic. To further increase the learning efficiency of COSEA, we propose a variant of contrastive loss for training the code search model, where the ground-truth code should be distinguished from the most similar negative sample. We have implemented a prototype of COSEA. Extensive experiments over existing public datasets of Python and SQL have demonstrated that COSEA can achieve significant improvements over state-of-the-art methods on code search tasks.
Heads-up! Unsupervised Constituency Parsing via Self-Attention Heads
Authors: Bowen Li, Taeuk Kim, Reinald Kim Amplayo, Frank Keller
Date: 2020-10-19 00:00:00
Proceeding: Asian Chapter of the Association for Computational Linguistics 2020
Tasks: Constituency Parsing

Transformer-based pre-trained language models (PLMs) have dramatically improved the state of the art in NLP across many tasks. This has led to substantial interest in analyzing the syntactic knowledge PLMs learn. Previous approaches to this question have been limited, mostly using test suites or probes. Here, we propose a novel fully unsupervised parsing approach that extracts constituency trees from PLM attention heads. We rank transformer attention heads based on their inherent properties, and create an ensemble of high-ranking heads to produce the final tree. Our method is adaptable to low-resource languages, as it does not rely on development sets, which can be expensive to annotate. Our experiments show that the proposed method often outperform existing approaches if there is no development set present. Our unsupervised parser can also be used as a tool to analyze the grammars PLMs learn implicitly. For this, we use the parse trees induced by our method to train a neural PCFG and compare it to a grammar derived from a human-annotated treebank.
Improving Transformation Invariance in Contrastive Representation Learning
Authors: Adam Foster, Rattana Pukdee, Tom Rainforth
Date: 2020-10-19 00:00:00
Proceeding: ICLR 2021 1
Tasks: Contrastive Learning, Representation Learning

We propose methods to strengthen the invariance properties of representations obtained by contrastive learning. While existing approaches implicitly induce a degree of invariance as representations are learned, we look to more directly enforce invariance in the encoding process. To this end, we first introduce a training objective for contrastive learning that uses a novel regularizer to control how the representation changes under transformation. We show that representations trained with this objective perform better on downstream tasks and are more robust to the introduction of nuisance transformations at test time. Second, we propose a change to how test time representations are generated by introducing a feature averaging approach that combines encodings from multiple transformations of the original input, finding that this leads to across the board performance gains. Finally, we introduce the novel Spirograph dataset to explore our ideas in the context of a differentiable generative process with multiple downstream tasks, showing that our techniques for learning invariance are highly beneficial.
No-regret learning and mixed Nash equilibria: They do not mix
Authors: Lampros Flokas, Emmanouil-Vasileios Vlatakis-Gkaragkounis, Thanasis Lianeas, Panayotis Mertikopoulos, Georgios Piliouras
Date: 2020-10-19 00:00:00
Proceeding: NeurIPS 2020 12

Understanding the behavior of no-regret dynamics in general $N$-player games is a fundamental question in online learning and game theory. A folk result in the field states that, in finite games, the empirical frequency of play under no-regret learning converges to the game's set of coarse correlated equilibria. By contrast, our understanding of how the day-to-day behavior of the dynamics correlates to the game's Nash equilibria is much more limited, and only partial results are known for certain classes of games (such as zero-sum or congestion games). In this paper, we study the dynamics of "follow-the-regularized-leader" (FTRL), arguably the most well-studied class of no-regret dynamics, and we establish a sweeping negative result showing that the notion of mixed Nash equilibrium is antithetical to no-regret learning. Specifically, we show that any Nash equilibrium which is not strict (in that every player has a unique best response) cannot be stable and attracting under the dynamics of FTRL. This result has significant implications for predicting the outcome of a learning process as it shows unequivocally that only strict (and hence, pure) Nash equilibria can emerge as stable limit points thereof.
Softer Pruning, Incremental Regularization
Authors: Linhang Cai, Zhulin An, Chuanguang Yang, Yongjun Xu
Date: 2020-10-19 00:00:00
Tasks: Network Pruning

Network pruning is widely used to compress Deep Neural Networks (DNNs). The Soft Filter Pruning (SFP) method zeroizes the pruned filters during training while updating them in the next training epoch. Thus the trained information of the pruned filters is completely dropped. To utilize the trained pruned filters, we proposed a SofteR Filter Pruning (SRFP) method and its variant, Asymptotic SofteR Filter Pruning (ASRFP), simply decaying the pruned weights with a monotonic decreasing parameter. Our methods perform well across various networks, datasets and pruning rates, also transferable to weight pruning. On ILSVRC-2012, ASRFP prunes 40% of the parameters on ResNet-34 with 1.63% top-1 and 0.68% top-5 accuracy improvement. In theory, SRFP and ASRFP are an incremental regularization of the pruned filters. Besides, We note that SRFP and ASRFP pursue better results while slowing down the speed of convergence.
Stationary Activations for Uncertainty Calibration in Deep Learning
Authors: Lassi Meronen, Christabella Irwanto, Arno Solin
Date: 2020-10-19 00:00:00
Proceeding: NeurIPS 2020 12
Tasks: Deep Learning, General Classification

We introduce a new family of non-linear neural network activation functions that mimic the properties induced by the widely-used Mat\'ern family of kernels in Gaussian process (GP) models. This class spans a range of locally stationary models of various degrees of mean-square differentiability. We show an explicit link to the corresponding GP models in the case that the network consists of one infinitely wide hidden layer. In the limit of infinite smoothness the Mat\'ern family results in the RBF kernel, and in this case we recover RBF activations. Mat\'ern activation functions result in similar appealing properties to their counterparts in GP models, and we demonstrate that the local stationarity property together with limited mean-square differentiability shows both good performance and uncertainty calibration in Bayesian deep learning tasks. In particular, local stationarity helps calibrate out-of-distribution (OOD) uncertainty. We demonstrate these properties on classification and regression benchmarks and a radar emitter classification task.
Diving Deep into Context-Aware Neural Machine Translation
Authors: Jingjing Huo, Christian Herold, Yingbo Gao, Leonard Dahlmann, Shahram Khadivi, Hermann Ney
Date: 2020-10-19 00:00:00
Proceeding: WMT (EMNLP) 2020 11
Tasks: Machine Translation, NMT, Translation

Context-aware neural machine translation (NMT) is a promising direction to improve the translation quality by making use of the additional context, e.g., document-level translation, or having meta-information. Although there exist various architectures and analyses, the effectiveness of different context-aware NMT models is not well explored yet. This paper analyzes the performance of document-level NMT models on four diverse domains with a varied amount of parallel document-level bilingual data. We conduct a comprehensive set of experiments to investigate the impact of document-level NMT. We find that there is no single best approach to document-level NMT, but rather that different architectures come out on top on different tasks. Looking at task-specific problems, such as pronoun resolution or headline translation, we find improvements in the context-aware systems, even in cases where the corpus-level metrics like BLEU show no significant improvement. We also show that document-level back-translation significantly helps to compensate for the lack of document-level bi-texts.
SPECT Imaging Reconstruction Method Based on Deep Convolutional Neural Network
Authors: Charalambos Chrysostomou, Loizos Koutsantonis, Christos Lemesios, Costas N. Papanicolas
Date: 2020-10-19 00:00:00
Tasks: Image Reconstruction

In this paper, we explore a novel method for tomographic image reconstruction in the field of SPECT imaging. Deep Learning methodologies and more specifically deep convolutional neural networks (CNN) are employed in the new reconstruction method, which is referred to as "CNN Reconstruction - CNNR". For training of the CNNR Projection data from software phantoms were used. For evaluation of the efficacy of the CNNR method, both software and hardware phantoms were used. The resulting tomographic images are compared to those produced by filtered back projection (FBP) [1], the "Maximum Likelihood Expectation Maximization" (MLEM) [1] and ordered subset expectation maximization (OSEM) [2].
Dos and Don'ts of Machine Learning in Computer Security
Authors: Daniel Arp, Erwin Quiring, Feargus Pendlebury, Alexander Warnecke, Fabio Pierazzi, Christian Wressnegger, Lorenzo Cavallaro, Konrad Rieck
Date: 2020-10-19 00:00:00
Tasks: BIG-bench Machine Learning, Computer Security, Malware Detection

With the growing processing power of computing systems and the increasing availability of massive datasets, machine learning algorithms have led to major breakthroughs in many different areas. This development has influenced computer security, spawning a series of work on learning-based security systems, such as for malware detection, vulnerability discovery, and binary code analysis. Despite great potential, machine learning in security is prone to subtle pitfalls that undermine its performance and render learning-based systems potentially unsuitable for security tasks and practical deployment. In this paper, we look at this problem with critical eyes. First, we identify common pitfalls in the design, implementation, and evaluation of learning-based security systems. We conduct a study of 30 papers from top-tier security conferences within the past 10 years, confirming that these pitfalls are widespread in the current security literature. In an empirical analysis, we further demonstrate how individual pitfalls can lead to unrealistic performance and interpretations, obstructing the understanding of the security problem at hand. As a remedy, we propose actionable recommendations to support researchers in avoiding or mitigating the pitfalls where possible. Furthermore, we identify open problems when applying machine learning in security and provide directions for further research.
Chance-Constrained Control with Lexicographic Deep Reinforcement Learning
Authors: Alessandro Giuseppi, Antonio Pietrabissa
Date: 2020-10-19 00:00:00
Tasks: Deep Reinforcement Learning, reinforcement-learning, Reinforcement Learning, Reinforcement Learning (RL)

This paper proposes a lexicographic Deep Reinforcement Learning (DeepRL)-based approach to chance-constrained Markov Decision Processes, in which the controller seeks to ensure that the probability of satisfying the constraint is above a given threshold. Standard DeepRL approaches require i) the constraints to be included as additional weighted terms in the cost function, in a multi-objective fashion, and ii) the tuning of the introduced weights during the training phase of the Deep Neural Network (DNN) according to the probability thresholds. The proposed approach, instead, requires to separately train one constraint-free DNN and one DNN associated to each constraint and then, at each time-step, to select which DNN to use depending on the system observed state. The presented solution does not require any hyper-parameter tuning besides the standard DNN ones, even if the probability thresholds changes. A lexicographic version of the well-known DeepRL algorithm DQN is also proposed and validated via simulations.
ARENA: A Data-driven Radio Access Networks Analysis of Football Events
Authors: Lanfranco Zanzi, Vincenzo Sciancalepore, Andres Garcia-Saavedra, Xavier Costa-Perez, Georgios Agapiou, Hans D. Schotten
Date: 2020-10-19 00:00:00

Mass events represent one of the most challenging scenarios for mobile networks because, although their date and time are usually known in advance, the actual demand for resources is difficult to predict due to its dependency on many different factors. Based on data provided by a major European carrier during mass events in a football stadium comprising up to 30.000 people, 16 base station sectors and $1$Km$^2$ area, we performed a data-driven analysis of the radio access network infrastructure dynamics during such events. Given the insights obtained from the analysis, we developed ARENA, a model-free deep learning Radio Access Network (RAN) capacity forecasting solution that, taking as input past network monitoring data and events context information, provides guidance to mobile operators on the expected RAN capacity needed during a future event. Our results, validated against real events contained in the dataset, illustrate the effectiveness of our proposed solution.
Unsupervised Expressive Rules Provide Explainability and Assist Human Experts Grasping New Domains
Authors: Eyal Shnarch, Leshem Choshen, Guy Moshkowich, Noam Slonim, Ranit Aharonov
Date: 2020-10-19 00:00:00
Proceeding: Findings of the Association for Computational Linguistics 2020
Tasks: Domain Adaptation

Approaching new data can be quite deterrent; you do not know how your categories of interest are realized in it, commonly, there is no labeled data at hand, and the performance of domain adaptation methods is unsatisfactory. Aiming to assist domain experts in their first steps into a new task over a new corpus, we present an unsupervised approach to reveal complex rules which cluster the unexplored corpus by its prominent categories (or facets). These rules are human-readable, thus providing an important ingredient which has become in short supply lately - explainability. Each rule provides an explanation for the commonality of all the texts it clusters together. We present an extensive evaluation of the usefulness of these rules in identifying target categories, as well as a user study which assesses their interpretability.
Fast accuracy estimation of deep learning based multi-class musical source separation
Authors: Alexandru Mocanu, Benjamin Ricaud, Milos Cernak
Date: 2020-10-19 00:00:00
Tasks: Audio Source Separation, Music Source Separation

Music source separation represents the task of extracting all the instruments from a given song. Recent breakthroughs on this challenge have gravitated around a single dataset, MUSDB, only limited to four instrument classes. Larger datasets and more instruments are costly and time-consuming in collecting data and training deep neural networks (DNNs). In this work, we propose a fast method to evaluate the separability of instruments in any dataset without training and tuning a DNN. This separability measure helps to select appropriate samples for the efficient training of neural networks. Based on the oracle principle with an ideal ratio mask, our approach is an excellent proxy to estimate the separation performances of state-of-the-art deep learning approaches such as TasNet or Open-Unmix. Our results contribute to revealing two essential points for audio source separation: 1) the ideal ratio mask, although light and straightforward, provides an accurate measure of the audio separability performance of recent neural nets, and 2) new end-to-end learning methods such as Tasnet, that operate directly on waveforms, are, in fact, internally building a Time-Frequency (TF) representation, so that they encounter the same limitations as the TF based-methods when separating audio pattern overlapping in the TF plane.
ERIC: Extracting Relations Inferred from Convolutions
Authors: Joe Townsend, Theodoros Kasioumis, Hiroya Inakoshi
Date: 2020-10-19 00:00:00

Our main contribution is to show that the behaviour of kernels across multiple layers of a convolutional neural network can be approximated using a logic program. The extracted logic programs yield accuracies that correlate with those of the original model, though with some information loss in particular as approximations of multiple layers are chained together or as lower layers are quantised. We also show that an extracted program can be used as a framework for further understanding the behaviour of CNNs. Specifically, it can be used to identify key kernels worthy of deeper inspection and also identify relationships with other kernels in the form of the logical rules. Finally, we make a preliminary, qualitative assessment of rules we extract from the last convolutional layer and show that kernels identified are symbolic in that they react strongly to sets of similar images that effectively divide output classes into sub-classes with distinct characteristics.
Efficient Estimation and Evaluation of Prediction Rules in Semi-Supervised Settings under Stratified Sampling
Authors: Jessica Gronsbell, Molei Liu, Lu Tian, Tianxi Cai
Date: 2020-10-19 00:00:00
Tasks: Imputation, Missing Labels, Prediction, regression

In many contemporary applications, large amounts of unlabeled data are readily available while labeled examples are limited. There has been substantial interest in semi-supervised learning (SSL) which aims to leverage unlabeled data to improve estimation or prediction. However, current SSL literature focuses primarily on settings where labeled data is selected randomly from the population of interest. Non-random sampling, while posing additional analytical challenges, is highly applicable to many real world problems. Moreover, no SSL methods currently exist for estimating the prediction performance of a fitted model under non-random sampling. In this paper, we propose a two-step SSL procedure for evaluating a prediction rule derived from a working binary regression model based on the Brier score and overall misclassification rate under stratified sampling. In step I, we impute the missing labels via weighted regression with nonlinear basis functions to account for nonrandom sampling and to improve efficiency. In step II, we augment the initial imputations to ensure the consistency of the resulting estimators regardless of the specification of the prediction model or the imputation model. The final estimator is then obtained with the augmented imputations. We provide asymptotic theory and numerical studies illustrating that our proposals outperform their supervised counterparts in terms of efficiency gain. Our methods are motivated by electronic health records (EHR) research and validated with a real data analysis of an EHR-based study of diabetic neuropathy.
Fractal Autoencoders for Feature Selection
Authors: Xinxing Wu, Qiang Cheng
Date: 2020-10-19 00:00:00
Tasks: Diversity, feature selection

Feature selection reduces the dimensionality of data by identifying a subset of the most informative features. In this paper, we propose an innovative framework for unsupervised feature selection, called fractal autoencoders (FAE). It trains a neural network to pinpoint informative features for global exploring of representability and for local excavating of diversity. Architecturally, FAE extends autoencoders by adding a one-to-one scoring layer and a small sub-neural network for feature selection in an unsupervised fashion. With such a concise architecture, FAE achieves state-of-the-art performances; extensive experimental results on fourteen datasets, including very high-dimensional data, have demonstrated the superiority of FAE over existing contemporary methods for unsupervised feature selection. In particular, FAE exhibits substantial advantages on gene expression data exploration, reducing measurement cost by about $15$\% over the widely used L1000 landmark genes. Further, we show that the FAE framework is easily extensible with an application.
Neural Additive Vector Autoregression Models for Causal Discovery in Time Series
Authors: Bart Bussmann, Jannes Nys, Steven Latré
Date: 2020-10-19 00:00:00
Tasks: Causal Discovery, Time Series, Time Series Analysis

Causal structure discovery in complex dynamical systems is an important challenge for many scientific domains. Although data from (interventional) experiments is usually limited, large amounts of observational time series data sets are usually available. Current methods that learn causal structure from time series often assume linear relationships. Hence, they may fail in realistic settings that contain nonlinear relations between the variables. We propose Neural Additive Vector Autoregression (NAVAR) models, a neural approach to causal structure learning that can discover nonlinear relationships. We train deep neural networks that extract the (additive) Granger causal influences from the time evolution in multi-variate time series. The method achieves state-of-the-art results on various benchmark data sets for causal discovery, while providing clear interpretations of the mapped causal relations.
LANNS: A Web-Scale Approximate Nearest Neighbor Lookup System
Authors: Ishita Doshi, Dhritiman Das, Ashish Bhutani, Rajeev Kumar, Rushi Bhatt, Niranjan Balasubramanian
Date: 2020-10-19 00:00:00
Tasks: Information Retrieval, Playing the Game of 2048, Retrieval

Nearest neighbor search (NNS) has a wide range of applications in information retrieval, computer vision, machine learning, databases, and other areas. Existing state-of-the-art algorithm for nearest neighbor search, Hierarchical Navigable Small World Networks(HNSW), is unable to scale to large datasets of 100M records in high dimensions. In this paper, we propose LANNS, an end-to-end platform for Approximate Nearest Neighbor Search, which scales for web-scale datasets. Library for Large Scale Approximate Nearest Neighbor Search (LANNS) is deployed in multiple production systems for identifying topK ($100 \leq topK \leq 200$) approximate nearest neighbors with a latency of a few milliseconds per query, high throughput of 2.5k Queries Per Second (QPS) on a single node, on large ($\sim$180M data points) high dimensional (50-2048 dimensional) datasets.