pwc-archive/papers-with-abstracts huggingface dataset:
Stabilized Medical Attacks
Authors: Gege Qi, Lijun Gong, Yibing Song, Kai Ma, Yefeng Zheng
Date: 2021-01-01 00:00:00
Proceeding: ICLR 2021 1
Tasks: Adversarial Attack, Medical Diagnosis, Medical Image Analysis

Convolutional Neural Networks (CNNs) have advanced existing medical systems for automatic disease diagnosis. However, a threat to these systems arises that adversarial attacks make CNNs vulnerable. Inaccurate diagnosis results make a negative influence on human healthcare. There is a need to investigate potential adversarial attacks to robustify deep medical diagnosis systems. On the other side, there are several modalities of medical images (e.g., CT, fundus, and endoscopic image) of which each type is significantly different from others. It is more challenging to generate adversarial perturbations for different types of medical images. In this paper, we propose a universal medical adversarial attack method to consistently produce adversarial perturbations on medical images. The objective function of our method consists of a loss deviation term and a loss stabilization term. The loss deviation term increases the divergence between the CNN prediction of an adversarial example and its ground truth label. Meanwhile, the loss stabilization term ensures similar CNN predictions of this example and its smoothed input. From the perspective of the whole iterations for perturbation generation, the proposed loss stabilization term exhaustively searches the perturbation space to smooth the single spot for local optimal escape. We further analyze the KL-divergence of the proposed loss function and find that the loss stabilization term makes the perturbations updated towards a fixed objective spot while deviating from the ground truth. This stabilization ensures the proposed medical attack effective for different types of medical images while producing perturbations in small variance. Experiments on several medical image analysis benchmarks including the recent COVID-19 dataset show the stability of the proposed method.
Deep Manifold Computing and Visualization Using Elastic Locally Isometric Smoothness
Authors: Stan Z. Li, Zelin Zang, Lirong Wu
Date: 2021-01-01 00:00:00
Tasks: Dimensionality Reduction

The ability to preserve local geometry of highly nonlinear manifolds in high dimensional spaces and properly unfold them into lower dimensional hyperplanes is the key to the success of manifold computing, nonlinear dimensionality reduction (NLDR) and visualization. This paper proposes a novel method, called  elastic locally isometric smoothness (ELIS), to empower deep neural networks with such an ability. ELIS requires that a desired metric between points should be preserved across layers in order to preserve local geometry; such a smoothness constraint effectively regularizes vector-based transformations to become well-behaved local metric-preserving homeomorphisms. Moreover, ELIS requires that the smoothness should be imposed in a way to render sufficient flexibility for tackling complicated nonlinearity and non-Euclideanity; this is achieved layer-wisely via nonlinearity in both the similarity and activation functions. The ELIS method incorporates a class of suitable nonlinear similarity functions into a two-way divergence loss and uses hyperparameter continuation in finding optimal solutions. Extensive experiments, comparisons, and ablation study demonstrate that ELIS can deliver results not only superior to UMAP and t-SNE for  and visualization but also better than other leading counterparts of manifold and autoencoder learning for NLDR and manifold data generation.
THE EFFICACY OF L1 REGULARIZATION IN NEURAL NETWORKS
Authors: Gen Li, Yuantao Gu, Jie Ding
Date: 2021-01-01 00:00:00

A crucial problem in neural networks is to select the most appropriate number of hidden neurons and obtain tight statistical risk bounds. In this work, we present a new perspective towards the bias-variance tradeoff in neural networks. As an alternative to selecting the number of neurons, we theoretically show that $L_1$ regularization can control the generalization error and sparsify the input dimension. In particular, with an appropriate $L_1$ regularization on the output layer, the network can produce a statistical risk that is near minimax optimal. Moreover, an appropriate $L_1$ regularization on the input layer leads to a risk bound that does not involve the input data dimension. Our analysis is based on a new amalgamation of dimension-based and norm-based complexity analysis to bound the generalization error. A consequent observation from our results is that an excessively large number of neurons do not necessarily inflate generalization errors under a suitable regularization.
Transforming Recurrent Neural Networks with Attention and Fixed-point Equations
Authors: Zhaobin Xu, Baotian Hu, Buzhou Tang
Date: 2021-01-01 00:00:00
Tasks: Position

Transformer has achieved state of the art performance in multiple Natural Language Processing tasks recently. Yet the Feed Forward Network(FFN) in a Transformer block is computationally expensive. In this paper, we present a framework to transform Recurrent Neural Networks(RNNs) and their variants into self-attention-style models, with an approximation of Banach Fixed-point Theorem. Within this framework, we propose a new model, StarSaber, by solving a set of equations obtained from RNN with Fixed-point Theorem and further approximate it with a Multi-layer Perceptron. It provides a view of stacking layers. StarSaber achieves better performance than both the vanilla Transformer and an improved version called ReZero on three datasets and is more computationally efficient, due to the reduction of Transformer's FFN layer. It has two major parts. One is a way to encode position information with two different matrices. For every position in a sequence, we have a matrix operating on positions before it and another matrix operating on positions after it. The other is the introduction of direct paths from the input layer to the rest of layers. Ablation studies show the effectiveness of these two parts. We additionally show that other RNN variants such as RNNs with gates can also be transformed in the same way, outperforming the two kinds of Transformers as well.
Exploring Target Driven Image Classification
Authors: Aditya Singh, Alessandro Bay, Andrea Mirabile
Date: 2021-01-01 00:00:00
Tasks: Classification, Fine-Grained Image Classification, General Classification, image-classification, Image Classification

For a given image, traditional supervised image classification using deep neural networks is akin to answering the question 'what object category does this image belong to?'. The model takes in an image as input and produces the most likely label for it. However, there is an alternate approach to arrive at the final answer which we investigate in this paper. We argue that, for any arbitrary category $\mathit{\tilde{y}}$, the composed question 'Is this image of an object category $\mathit{\tilde{y}}$' serves as a viable approach for image classification via. deep neural networks. The difference lies in the supplied additional information in form of the target along with the image. Motivated by the curiosity to unravel the advantages and limitations of the addressed approach, we propose Indicator Neural Networks(INN). It utilizes a pair of image and label as input and produces a boolean response. INN consists of $2$ encoding components namely: label encoder and image encoder which learns latent representations for labels and images respectively. Predictor, the third component, combines the learnt individual label and image representations to make the final yes/no prediction. The network is trained end-to-end. We perform evaluations on image classification and fine-grained image classification datasets against strong baselines. We also investigate various components of INNs to understand their contribution in the final prediction of the model. Our probing of the modules reveals that, as opposed to traditionally trained deep counterpart, INN tends to much larger regions of the input image for generating the image features. The generated image feature is further refined by the generated label encoding prior to the final prediction.
Generating Furry Cars: Disentangling Object Shape and Appearance across Multiple Domains
Authors: Utkarsh Ojha, Krishna Kumar Singh, Yong Jae Lee
Date: 2021-01-01 00:00:00
Proceeding: ICLR 2021 1
Tasks: Disentanglement, Object

We consider the novel task of learning disentangled representations of object shape and appearance across multiple domains (e.g., dogs and cars).  The goal is to learn a generative model that learns an intermediate distribution, which borrows a subset of properties from each domain, enabling the generation of images that did not exist in any domain exclusively.  This challenging problem requires an accurate disentanglement of object shape, appearance, and background from each domain, so that the appearance and shape factors from the two domains can be interchanged. We augment an existing approach that can disentangle factors within a single domain but struggles to do so across domains.  Our key technical contribution is to represent object appearance with a differentiable histogram of visual features, and to optimize the generator so that two images with the same latent appearance factor but different latent shape factors produce similar histograms. On multiple multi-domain datasets, we demonstrate our method leads to accurate and consistent appearance and shape transfer across domains.
String Theory: Parsed Categoric Encodings with Automunge
Authors: Nicholas Teague
Date: 2021-01-01 00:00:00
Tasks: Feature Engineering

The Automunge open source python library platform for tabular data pre-processing automates feature engineering data transformations of numerical encoding and missing data infill to received tidy data on bases fit to properties of columns in a designated train set for consistent and efficient application to subsequent data pipelines such as for inference, where transformations may be applied to distinct columns in “family tree” sets with generations and branches of derivations. Included in the library of transformations are methods to extract structure from bounded categorical string sets by way of automated string parsing, in which comparisons between entries in the set of unique values are parsed to identify character subset overlaps which may be encoded by appended columns of boolean overlap detection activations or by replacing string entries with identified overlap partitions. Further string parsing options, which may also be applied to unbounded categoric sets, include extraction of numeric substring partitions from entries or search functions to identify presence of specified substring partitions. The aggregation of these methods into “family tree” sets of transformations are demonstrated for use to automatically extract structure from categoric string compositions in relation to the set of entries in a column, such as may be applied to prepare categoric string set encodings for machine learning without human intervention.
Probabilistic Path Hamiltonian Monte Carlo
Authors: Vu Dinh, Arman Bilge, Cheng Zhang, Frederick A. Matsen IV
Date: 2017-02-25 00:00:00
Proceeding: ICML 2017 8

Hamiltonian Monte Carlo (HMC) is an efficient and effective means of sampling posterior distributions on Euclidean space, which has been extended to manifolds with boundary. However, some applications require an extension to more general spaces. For example, phylogenetic (evolutionary) trees are defined in terms of both a discrete graph and associated continuous parameters; although one can represent these aspects using a single connected space, this rather complex space is not suitable for existing HMC algorithms. In this paper, we develop Probabilistic Path HMC (PPHMC) as a first step to sampling distributions on spaces with intricate combinatorial structure. We define PPHMC on orthant complexes, show that the resulting Markov chain is ergodic, and provide a promising implementation for the case of phylogenetic trees in open-source software. We also show that a surrogate function to ease the transition across a boundary on which the log-posterior has discontinuous derivatives can greatly improve efficiency.
Hyperspectral Super-Resolution: A Coupled Tensor Factorization Approach
Authors: Charilaos I. Kanatsoulis, Xiao Fu, Nicholas D. Sidiropoulos, Wing-Kin Ma
Date: 2018-04-15 00:00:00
Tasks: Super-Resolution

Hyperspectral super-resolution refers to the problem of fusing a hyperspectral image (HSI) and a multispectral image (MSI) to produce a super-resolution image (SRI) that has fine spatial and spectral resolution. State-of-the-art methods approach the problem via low-rank matrix approximations to the matricized HSI and MSI. These methods are effective to some extent, but a number of challenges remain. First, HSIs and MSIs are naturally third-order tensors (data "cubes") and thus matricization is prone to loss of structural information--which could degrade performance. Second, it is unclear whether or not these low-rank matrix-based fusion strategies can guarantee identifiability or exact recovery of the SRI. However, identifiability plays a pivotal role in estimation problems and usually has a significant impact on performance in practice. Third, the majority of the existing methods assume that there are known (or easily estimated) degradation operators applied to the SRI to form the corresponding HSI and MSI--which is hardly the case in practice. In this work, we propose to tackle the super-resolution problem from a tensor perspective. Specifically, we utilize the multidimensional structure of the HSI and MSI to propose a coupled tensor factorization framework that can effectively overcome the aforementioned issues. The proposed approach guarantees the identifiability of the SRI under mild and realistic conditions. Furthermore, it works with little knowledge of the degradation operators, which is clearly an advantage over the existing methods. Semi-real numerical experiments are included to show the effectiveness of the proposed approach.
Wikipedia graph mining: dynamic structure of collective memory
Authors: Volodymyr Miz, Kirell Benzi, Benjamin Ricaud, Pierre Vandergheynst
Date: 2017-10-01 00:00:00
Tasks: Articles, Event Extraction, Graph Mining, Learning Theory, Time Series, Time Series Analysis

Wikipedia is the biggest encyclopedia ever created and the fifth most visited website in the world. Tens of millions of people surf it every day, seeking answers to various questions. Collective user activity on its pages leaves publicly available footprints of human behavior, making Wikipedia an excellent source for analysis of collective behavior. In this work, we propose a distributed graph-based event extraction model, inspired by the Hebbian learning theory. The model exploits collective effect of the dynamics to discover events. We focus on data-streams with underlying graph structure and perform several large-scale experiments on the Wikipedia visitor activity data. We show that the presented model is scalable regarding time-series length and graph density, providing a distributed implementation of the proposed algorithm. We extract dynamical patterns of collective activity and demonstrate that they correspond to meaningful clusters of associated events, reflected in the Wikipedia articles. We also illustrate evolutionary dynamics of the graphs over time to highlight changing nature of visitors' interests. Finally, we discuss clusters of events that model collective recall process and represent collective memories - common memories shared by a group of people.
Low-dimensional functionality of complex network dynamics: Neuro-sensory integration in the Caenorhabditis elegans connectome
Authors: James Kunert, Eli Shlizerman, J. Nathan Kutz
Date: 2013-10-24 00:00:00

We develop a biophysical model of neuro-sensory integration in the model organism Caenorhabditis elegans. Building on recent experimental findings of the neuron conductances and their resolved connectome, we posit the first full dynamic model of the neural voltage excitations that allows for a characterization of input stimuli to behavioral responses. Thus a clear connection between receptory cell inputs to downstream motor-responses is illustrated, showing that robust, low-dimensional bifurcation structures dominate neural pathways of activity. The underlying bifurcation structures discovered, i.e. an induced Hopf bifurcation, are critical in explaining behavioral responses such as swimming and crawling. More broadly, we demonstrate that complex dynamical networks can produce robust functionality from underlying low-dimensional bifurcations.
Named Entity Recognition and Resolution in Legal Text
Authors: Christopher Dozier, Ravikumar Kondadadi, Marc Light, Arun Vachher, Sriharsha Veeramachaneni, Ramdev Wudali
Date: 2014-06-05 00:00:00
Tasks: Blocking, named-entity-recognition, Named Entity Recognition, Named Entity Recognition (NER)

Named entities in the text are persons, places, companies, etc. that are explicitly mentioned in the text using proper nouns. The process of finding named entities in a text and classifying them to a semantic type is called named entity recognition. The resolution of named entities is the process of linking a mention of a name in text to a pre-existing database entry. This grounds the mention in something analogous to a real-world entity. For example, a mention of a judge named Mary Smith might be
resolved to a database entry for a specific judge of a specific district of a specific state. This recognition and resolution of named entities can be leveraged in a number of ways including providing hypertext links to the information stored about a particular judge: their education, who appointed them, their other case opinions, etc. 
This paper discusses named entity recognition and resolution in legal documents such as US case law, depositions, and pleadings, and other trial documents. The types of entities include judges, attorneys, companies, jurisdictions, and courts. 
We outline three methods for named entity recognition, lookup, context rules, and statistical models. We then describe an actual system for finding named entities in legal text and evaluate its accuracy. Similarly, for the resolution, we discuss our blocking techniques, our resolution features, and the supervised and semi-supervised machine learning techniques we employ for the final matching.
Performing Image Classification for 10 Different Monkey Species using CNN
Authors: Emmanuel Maduwuba, Dharanikota Rajendra Kamal and Kamaljeet Singh Mann. Western University of Ontario
Date: 2020-06-28 00:00:00
Tasks: BIG-bench Machine Learning, Fine-Grained Image Classification, General Classification, image-classification, Image Classification

the main aim of this project is to achieve fine
grain image classification by applying a suitable machine
learning architecture to the set of images present in the
dataset. The chosen dataset is taken as a part of the Kaggle
competition and is selected from Wikipedia's monkey
cladogram and this dataset contains 10 different species of
monkeys which are to be classified with the help of a
machine learning architecture augmented by Image
processing. After having brief exposure and using several
architectures to classify this dataset, the Convolutional
Neural network was found to be the best fit.
Using Deep Learning to Identify Utility Poles with Crossarms and Estimate Their Locations from Google Street View Images
Authors: Weixing Zhang, Chandi Witharana, Weidong Li, Chuanrong Zhang, Xiaojiang Li, and Jason Parent
Date: 2018-08-01 00:00:00
Tasks: object-detection, Object Detection

Traditional methods of detecting and mapping utility poles are inefficient and costly because of the demand for visual interpretation with quality data sources or intense field inspection. The advent of deep learning for object detection provides an opportunity for detecting utility poles from side-view optical images. In this study, we proposed using a deep learning-based method for automatically mapping roadside utility poles with crossarms (UPCs) from Google Street View (GSV) images. The method combines the state-of-the-art DL object detection algorithm (i.e., the RetinaNet object detection algorithm) and a modified brute-force-based line-of-bearing (LOB, a LOB stands for the ray towards the location of the target [UPC at here] from the original location of the sensor [GSV mobile platform]) measurement method to estimate the locations of detected roadside UPCs from GSV. Experimental results indicate that: (1) both the average precision (AP) and the overall accuracy (OA) are around 0.78 when the intersection-over-union (IoU) threshold is greater than 0.3, based on the testing of 500 GSV images with a total number of 937 objects; and (2) around 2.6%, 47%, and 79% of estimated locations of utility poles are within 1 m, 5 m, and 10 m buffer zones, respectively, around the referenced locations of utility poles. In general, this study indicates that even in a complex background, most utility poles can be detected with the use of DL, and the LOB measurement method can estimate the locations of most UPCs.
Self-training Improves Pre-training for Natural Language Understanding
Authors: Jingfei Du, Edouard Grave, Beliz Gunel, Vishrav Chaudhary, Onur Celebi, Michael Auli, Ves Stoyanov, Alexis Conneau
Date: 2020-10-05 00:00:00
Proceeding: NAACL 2021 4
Tasks: Data Augmentation, Few-Shot Learning, Knowledge Distillation, Natural Language Understanding, text-classification, Text Classification, Unsupervised Pre-training

Unsupervised pre-training has led to much recent progress in natural language understanding. In this paper, we study self-training as another way to leverage unlabeled data through semi-supervised learning. To obtain additional data for a specific task, we introduce SentAugment, a data augmentation method which computes task-specific query embeddings from labeled data to retrieve sentences from a bank of billions of unlabeled sentences crawled from the web. Unlike previous semi-supervised methods, our approach does not require in-domain unlabeled data and is therefore more generally applicable. Experiments show that self-training is complementary to strong RoBERTa baselines on a variety of tasks. Our augmentation approach leads to scalable and effective self-training with improvements of up to 2.6% on standard text classification benchmarks. Finally, we also show strong gains on knowledge-distillation and few-shot learning.
Mastering Atari with Discrete World Models
Authors: Danijar Hafner, Timothy Lillicrap, Mohammad Norouzi, Jimmy Ba
Date: 2020-10-05 00:00:00
Proceeding: ICLR 2021 1
Tasks: Atari Games, GPU

Intelligent agents need to generalize from past experience to achieve goals in complex environments. World models facilitate such generalization and allow learning behaviors from imagined outcomes to increase sample-efficiency. While learning world models from image inputs has recently become feasible for some tasks, modeling Atari games accurately enough to derive successful behaviors has remained an open challenge for many years. We introduce DreamerV2, a reinforcement learning agent that learns behaviors purely from predictions in the compact latent space of a powerful world model. The world model uses discrete representations and is trained separately from the policy. DreamerV2 constitutes the first agent that achieves human-level performance on the Atari benchmark of 55 tasks by learning behaviors inside a separately trained world model. With the same computational budget and wall-clock time, Dreamer V2 reaches 200M frames and surpasses the final performance of the top single-GPU agents IQN and Rainbow. DreamerV2 is also applicable to tasks with continuous actions, where it learns an accurate world model of a complex humanoid robot and solves stand-up and walking from only pixel inputs.