pwc-archive/papers-with-abstracts huggingface dataset:
Exploring Neural Entity Representations for Semantic Information
Authors: Andrew Runge, Eduard Hovy
Date: 2020-11-17 00:00:00
Proceeding: EMNLP (BlackboxNLP) 2020 11
Tasks: Entity Linking

Neural methods for embedding entities are typically extrinsically evaluated on downstream tasks and, more recently, intrinsically using probing tasks. Downstream task-based comparisons are often difficult to interpret due to differences in task structure, while probing task evaluations often look at only a few attributes and models. We address both of these issues by evaluating a diverse set of eight neural entity embedding methods on a set of simple probing tasks, demonstrating which methods are able to remember words used to describe entities, learn type, relationship and factual information, and identify how frequently an entity is mentioned. We also compare these methods in a unified framework on two entity linking tasks and discuss how they generalize to different model architectures and datasets.
SHIELD: Defending Textual Neural Networks against Multiple Black-Box Adversarial Attacks with Stochastic Multi-Expert Patcher
Authors: Thai Le, Noseong Park, Dongwon Lee
Date: 2020-11-17 00:00:00
Proceeding: ACL 2022 5
Tasks: Adversarial Robustness

Even though several methods have proposed to defend textual neural network (NN) models against black-box adversarial attacks, they often defend against a specific text perturbation strategy and/or require re-training the models from scratch. This leads to a lack of generalization in practice and redundant computation. In particular, the state-of-the-art transformer models (e.g., BERT, RoBERTa) require great time and computation resources. By borrowing an idea from software engineering, in order to address these limitations, we propose a novel algorithm, SHIELD, which modifies and re-trains only the last layer of a textual NN, and thus it "patches" and "transforms" the NN into a stochastic weighted ensemble of multi-expert prediction heads. Considering that most of current black-box attacks rely on iterative search mechanisms to optimize their adversarial perturbations, SHIELD confuses the attackers by automatically utilizing different weighted ensembles of predictors depending on the input. In other words, SHIELD breaks a fundamental assumption of the attack, which is a victim NN model remains constant during an attack. By conducting comprehensive experiments, we demonstrate that all of CNN, RNN, BERT, and RoBERTa-based textual NNs, once patched by SHIELD, exhibit a relative enhancement of 15%--70% in accuracy on average against 14 different black-box attacks, outperforming 6 defensive baselines across 3 public datasets. All codes are to be released.
ZORB: A Derivative-Free Backpropagation Algorithm for Neural Networks
Authors: Varun Ranganathan, Alex Lewandowski
Date: 2020-11-17 00:00:00

Gradient descent and backpropagation have enabled neural networks to achieve remarkable results in many real-world applications. Despite ongoing success, training a neural network with gradient descent can be a slow and strenuous affair. We present a simple yet faster training algorithm called Zeroth-Order Relaxed Backpropagation (ZORB). Instead of calculating gradients, ZORB uses the pseudoinverse of targets to backpropagate information. ZORB is designed to reduce the time required to train deep neural networks without penalizing performance. To illustrate the speed up, we trained a feed-forward neural network with 11 layers on MNIST and observed that ZORB converged 300 times faster than Adam while achieving a comparable error rate, without any hyperparameter tuning. We also broaden the scope of ZORB to convolutional neural networks, and apply it to subsamples of the CIFAR-10 dataset. Experiments on standard classification and regression benchmarks demonstrate ZORB's advantage over traditional backpropagation with Gradient Descent.
MGIC: Multigrid-in-Channels Neural Network Architectures
Authors: Moshe Eliasof, Jonathan Ephrath, Lars Ruthotto, Eran Treister
Date: 2020-11-17 00:00:00
Proceeding: NeurIPS Workshop DLDE 2021 12
Tasks: image-classification, Image Classification, Point Cloud Classification

We present a multigrid-in-channels (MGIC) approach that tackles the quadratic growth of the number of parameters with respect to the number of channels in standard convolutional neural networks (CNNs). Thereby our approach addresses the redundancy in CNNs that is also exposed by the recent success of lightweight CNNs. Lightweight CNNs can achieve comparable accuracy to standard CNNs with fewer parameters; however, the number of weights still scales quadratically with the CNN's width. Our MGIC architectures replace each CNN block with an MGIC counterpart that utilizes a hierarchy of nested grouped convolutions of small group size to address this. Hence, our proposed architectures scale linearly with respect to the network's width while retaining full coupling of the channels as in standard CNNs. Our extensive experiments on image classification, segmentation, and point cloud classification show that applying this strategy to different architectures like ResNet and MobileNetV3 reduces the number of parameters while obtaining similar or better accuracy.
TreeGen -- a Monte Carlo generator for data frames
Authors: Agnieszka Niemczynowicz, Gabriela Białoskórska, Joanna Nieżurawska-Zając, Radosław A. Kycia
Date: 2020-11-17 00:00:00

The typical problem in Data Science is creating a structure that encodes the occurrence frequency of unique elements in rows and relations between different rows of a data frame. We present the probability tree abstract data structure, an extension of the decision tree, that facilitates more than two choices with assigned probabilities. Such a tree represents statistical relations between different rows of the data frame. The Probability Tree algorithmic structure is supplied with the Generator module that is a Monte Carlo generator that traverses through the tree. These two components are implemented in TreeGen Python package. The package can be used in increasing data multiplicity, compressing data preserving its statistical information, constructing hierarchical models, exploring data, and in feature extraction.
Distributed Online Learning with Multiple Kernels
Authors: Jeongmin Chae, SongNam Hong
Date: 2020-11-17 00:00:00
Tasks: Privacy Preserving, Time Series, Time Series Analysis, Time Series Prediction

In the Internet-of-Things (IoT) systems, there are plenty of informative data provided by a massive number of IoT devices (e.g., sensors). Learning a function from such data is of great interest in machine learning tasks for IoT systems. Focusing on streaming (or sequential) data, we present a privacy-preserving distributed online learning framework with multiplekernels (named DOMKL). The proposed DOMKL is devised by leveraging the principles of an online alternating direction of multipliers (OADMM) and a distributed Hedge algorithm. We theoretically prove that DOMKL over T time slots can achieve an optimal sublinear regret, implying that every learned function achieves the performance of the best function in hindsight as in the state-of-the-art centralized online learning method. Moreover, it is ensured that the learned functions of any two neighboring learners have a negligible difference as T grows, i.e., the so-called consensus constraints hold. Via experimental tests with various real datasets, we verify the effectiveness of the proposed DOMKL on regression and time-series prediction tasks.
Multi-agent Reinforcement Learning Accelerated MCMC on Multiscale Inversion Problem
Authors: Eric Chung, Yalchin Efendiev, Wing Tat Leung, Sai-Mang Pun, Zecheng Zhang
Date: 2020-11-17 00:00:00
Tasks: Multi-agent Reinforcement Learning, reinforcement-learning, Reinforcement Learning, Reinforcement Learning (RL)

In this work, we propose a multi-agent actor-critic reinforcement learning (RL) algorithm to accelerate the multi-level Monte Carlo Markov Chain (MCMC) sampling algorithms. The policies (actors) of the agents are used to generate the proposal in the MCMC steps; and the critic, which is centralized, is in charge of estimating the long term reward. We verify our proposed algorithm by solving an inverse problem with multiple scales. There are several difficulties in the implementation of this problem by using traditional MCMC sampling. Firstly, the computation of the posterior distribution involves evaluating the forward solver, which is very time consuming for a problem with heterogeneous. We hence propose to use the multi-level algorithm. More precisely, we use the generalized multiscale finite element method (GMsFEM) as the forward solver in evaluating a posterior distribution in the multi-level rejection procedure. Secondly, it is hard to find a function which can generate samplings which are meaningful. To solve this issue, we learn an RL policy as the proposal generator. Our experiments show that the proposed method significantly improves the sampling process
Contrastive Weight Regularization for Large Minibatch SGD
Authors: Qiwei Yuan, Weizhe Hua, Yi Zhou, Cunxi Yu
Date: 2020-11-17 00:00:00

The minibatch stochastic gradient descent method (SGD) is widely applied in deep learning due to its efficiency and scalability that enable training deep networks with a large volume of data. Particularly in the distributed setting, SGD is usually applied with large batch size. However, as opposed to small-batch SGD, neural network models trained with large-batch SGD can hardly generalize well, i.e., the validation accuracy is low. In this work, we introduce a novel regularization technique, namely distinctive regularization (DReg), which replicates a certain layer of the deep network and encourages the parameters of both layers to be diverse. The DReg technique introduces very little computation overhead. Moreover, we empirically show that optimizing the neural network with DReg using large-batch SGD achieves a significant boost in the convergence and improved generalization performance. We also demonstrate that DReg can boost the convergence of large-batch SGD with momentum. We believe that DReg can be used as a simple regularization trick to accelerate large-batch training in deep learning.
Deep Networks for Direction-of-Arrival Estimation in Low SNR
Authors: Georgios K. Papageorgiou, Mathini Sellathurai, Yonina C. Eldar
Date: 2020-11-17 00:00:00
Tasks: Direction of Arrival Estimation, Multi-Label Classification, MUlTI-LABEL-ClASSIFICATION

In this work, we consider direction-of-arrival (DoA) estimation in the presence of extreme noise using Deep Learning (DL). In particular, we introduce a Convolutional Neural Network (CNN) that is trained from mutli-channel data of the true array manifold matrix and is able to predict angular directions using the sample covariance estimate. We model the problem as a multi-label classification task and train a CNN in the low-SNR regime to predict DoAs across all SNRs. The proposed architecture demonstrates enhanced robustness in the presence of noise, and resilience to a small number of snapshots. Moreover, it is able to resolve angles within the grid resolution. Experimental results demonstrate significant performance gains in the low-SNR regime compared to state-of-the-art methods and without the requirement of any parameter tuning. We relax the assumption that the number of sources is known a priori and present a training method, where the CNN learns to infer the number of sources jointly with the DoAs. Simulation results demonstrate that the proposed CNN can accurately estimate off-grid angles in low SNR, while at the same time the number of sources is successfully inferred for a sufficient number of snapshots. Our robust solution can be applied in several fields, ranging from wireless array sensors to acoustic microphones or sonars.
Use HiResCAM instead of Grad-CAM for faithful explanations of convolutional neural networks
Authors: Rachel Lea Draelos, Lawrence Carin
Date: 2020-11-17 00:00:00
Tasks: General Classification, Image Classification

Explanation methods facilitate the development of models that learn meaningful concepts and avoid exploiting spurious correlations. We illustrate a previously unrecognized limitation of the popular neural network explanation method Grad-CAM: as a side effect of the gradient averaging step, Grad-CAM sometimes highlights locations the model did not actually use. To solve this problem, we propose HiResCAM, a novel class-specific explanation method that is guaranteed to highlight only the locations the model used to make each prediction. We prove that HiResCAM is a generalization of CAM and explore the relationships between HiResCAM and other gradient-based explanation methods. Experiments on PASCAL VOC 2012, including crowd-sourced evaluations, illustrate that while HiResCAM's explanations faithfully reflect the model, Grad-CAM often expands the attention to create bigger and smoother visualizations. Overall, this work advances convolutional neural network explanation approaches and may aid in the development of trustworthy models for sensitive applications.
Multimodal Prototypical Networks for Few-shot Learning
Authors: Frederik Pahde, Mihai Puscas, Tassilo Klein, Moin Nabi
Date: 2020-11-17 00:00:00
Tasks: Classification, Few-Shot Learning, General Classification

Although providing exceptional results for many computer vision tasks, state-of-the-art deep learning algorithms catastrophically struggle in low data scenarios. However, if data in additional modalities exist (e.g. text) this can compensate for the lack of data and improve the classification results. To overcome this data scarcity, we design a cross-modal feature generation framework capable of enriching the low populated embedding space in few-shot scenarios, leveraging data from the auxiliary modality. Specifically, we train a generative model that maps text data into the visual feature space to obtain more reliable prototypes. This allows to exploit data from additional modalities (e.g. text) during training while the ultimate task at test time remains classification with exclusively visual data. We show that in such cases nearest neighbor classification is a viable approach and outperform state-of-the-art single-modal and multimodal few-shot learning methods on the CUB-200 and Oxford-102 datasets.
Analyzing and Mitigating JPEG Compression Defects in Deep Learning
Authors: Max Ehrlich, Larry Davis, Ser-Nam Lim, Abhinav Shrivastava
Date: 2020-11-17 00:00:00
Tasks: Deep Learning

With the proliferation of deep learning methods, many computer vision problems which were considered academic are now viable in the consumer setting. One drawback of consumer applications is lossy compression, which is necessary from an engineering standpoint to efficiently and cheaply store and transmit user images. Despite this, there has been little study of the effect of compression on deep neural networks and benchmark datasets are often losslessly compressed or compressed at high quality. Here we present a unified study of the effects of JPEG compression on a range of common tasks and datasets. We show that there is a significant penalty on common performance metrics for high compression. We test several methods for mitigating this penalty, including a novel method based on artifact correction which requires no labels to train.
Dual-stream Multiple Instance Learning Network for Whole Slide Image Classification with Self-supervised Contrastive Learning
Authors: Bin Li, Yin Li, Kevin W. Eliceiri
Date: 2020-11-17 00:00:00
Proceeding: CVPR 2021 1
Tasks: Classification, Contrastive Learning, General Classification, image-classification, Image Classification, Multiple Instance Learning, whole slide images

We address the challenging problem of whole slide image (WSI) classification. WSIs have very high resolutions and usually lack localized annotations. WSI classification can be cast as a multiple instance learning (MIL) problem when only slide-level labels are available. We propose a MIL-based method for WSI classification and tumor detection that does not require localized annotations. Our method has three major components. First, we introduce a novel MIL aggregator that models the relations of the instances in a dual-stream architecture with trainable distance measurement. Second, since WSIs can produce large or unbalanced bags that hinder the training of MIL models, we propose to use self-supervised contrastive learning to extract good representations for MIL and alleviate the issue of prohibitive memory cost for large bags. Third, we adopt a pyramidal fusion mechanism for multiscale WSI features, and further improve the accuracy of classification and localization. Our model is evaluated on two representative WSI datasets. The classification accuracy of our model compares favorably to fully-supervised methods, with less than 2% accuracy gap across datasets. Our results also outperform all previous MIL-based methods. Additional benchmark results on standard MIL datasets further demonstrate the superior performance of our MIL aggregator on general MIL problems. GitHub repository: https://github.com/binli123/dsmil-wsi
A Study on MIMO Channel Estimation by 2D and 3D Convolutional Neural Networks
Authors: Ben Marinberg, Ariel Cohen, Eilam Ben-Dror, Haim Permuter
Date: 2020-11-12 00:00:00

In this paper, we study the usage of Convolutional Neural Network (CNN) estimators for the task of Multiple-Input-Multiple-Output Orthogonal Frequency Division Multiplexing (MIMO-OFDM) Channel Estimation (CE). Specifically, the CNN estimators interpolate the channel values of reference signals for estimating the channel of the full OFDM resource element (RE) matrix. We have designed a 2D CNN architecture based on U-net, and a 3D CNN architecture for handling spatial correlation. We investigate the performance of various CNN architectures fora diverse data set generated according to the 5G NR standard and in particular, we investigate the influence of spatial correlation, Doppler, and reference signal resource allocation. The CE CNN estimators are then integrated with MIMO detection algorithms for testing their influence on the system level Bit Error Rate(BER) performance.
Environmental Pollution Prediction of NOx by Process Analysis and Predictive Modelling in Natural Gas Turbine Power Plants
Authors: Alan Rezazadeh
Date: 2020-11-05 00:00:00
Tasks: BIG-bench Machine Learning

The main objective of this paper is to propose K-Nearest-Neighbor (KNN) algorithm for predicting NOx emissions from natural gas electrical generation turbines. The process of producing electricity is dynamic and rapidly changing due to many factors such as weather and electrical grid requirements. Gas turbine equipment are also a dynamic part of the electricity generation since the equipment characteristics and thermodynamics behavior change as the turbines age. Regular maintenance of turbines are also another dynamic part of the electrical generation process, affecting the performance of equipment. This analysis discovered using KNN, trained on relatively small dataset produces the most accurate prediction rates. This statement can be logically explained as KNN finds the K nearest neighbor to the current input parameters and estimates a rated average of historically similar observations as prediction. This paper incorporates ambient weather conditions, electrical output as well as turbine performance factors to build a machine learning model to predict NOx emissions. The model can be used to optimize the operational processes for reduction in harmful emissions and increasing overall operational efficiency. Latent algorithms such as Principle Component Algorithms (PCA) have been used for monitoring the equipment performance behavior change which deeply influences process paraments and consequently determines NOx emissions. Typical statistical methods of machine learning performance evaluations such as multivariate analysis, clustering and residual analysis have been used throughout the paper.
RAMP-CNN: A Novel Neural Network for Enhanced Automotive Radar Object Recognition
Authors: Xiangyu Gao, Guanbin Xing, Sumit Roy, Hui Liu
Date: 2020-11-13 00:00:00
Tasks: object-detection, Object Detection, Object Recognition

Millimeter-wave radars are being increasingly integrated into commercial vehicles to support new advanced driver-assistance systems by enabling robust and high-performance object detection, localization, as well as recognition - a key component of new environmental perception. In this paper, we propose a novel radar multiple-perspectives convolutional neural network (RAMP-CNN) that extracts the location and class of objects based on further processing of the range-velocity-angle (RVA) heatmap sequences. To bypass the complexity of 4D convolutional neural networks (NN), we propose to combine several lower-dimension NN models within our RAMP-CNN model that nonetheless approaches the performance upper-bound with lower complexity. The extensive experiments show that the proposed RAMP-CNN model achieves better average recall and average precision than prior works in all testing scenarios. Besides, the RAMP-CNN model is validated to work robustly under nighttime, which enables low-cost radars as a potential substitute for pure optical sensing under severe conditions.
Patient-Specific Seizure Prediction Using Single Seizure Electroencephalography Recording
Authors: Zaid Bin Tariq, Arun Iyengar, Lara Marcuse, Hui Su, Bülent Yener
Date: 2020-11-14 00:00:00
Tasks: EEG, Electroencephalogram (EEG), Prediction, Seizure prediction, Specificity

Electroencephalogram (EEG) is a prominent way to measure the brain activity for studying epilepsy, thereby helping in predicting seizures. Seizure prediction is an active research area with many deep learning based approaches dominating the recent literature for solving this problem. But these models require a considerable number of patient-specific seizures to be recorded for extracting the preictal and interictal EEG data for training a classifier. The increase in sensitivity and specificity for seizure prediction using the machine learning models is noteworthy. However, the need for a significant number of patient-specific seizures and periodic retraining of the model because of non-stationary EEG creates difficulties for designing practical device for a patient. To mitigate this process, we propose a Siamese neural network based seizure prediction method that takes a wavelet transformed EEG tensor as an input with convolutional neural network (CNN) as the base network for detecting change-points in EEG. Compared to the solutions in the literature, which utilize days of EEG recordings, our method only needs one seizure for training which translates to less than ten minutes of preictal and interictal data while still getting comparable results to models which utilize multiple seizures for seizure prediction.