pwc-archive/papers-with-abstracts huggingface dataset:
ICLR Reproducibility Challenge Report (Padam : Closing The Generalization Gap Of Adaptive Gradient Methods in Training Deep Neural Networks)
Authors: Harshal Mittal, Kartikey Pandey, Yash Kant
Date: 2019-01-28 00:00:00

This work is a part of ICLR Reproducibility Challenge 2019, we try to
reproduce the results in the conference submission PADAM: Closing The
Generalization Gap of Adaptive Gradient Methods In Training Deep Neural
Networks. Adaptive gradient methods proposed in past demonstrate a degraded
generalization performance than the stochastic gradient descent (SGD) with
momentum. The authors try to address this problem by designing a new
optimization algorithm that bridges the gap between the space of Adaptive
Gradient algorithms and SGD with momentum. With this method a new tunable
hyperparameter called partially adaptive parameter p is introduced that varies
between [0, 0.5]. We build the proposed optimizer and use it to mirror the
experiments performed by the authors. We review and comment on the empirical
analysis performed by the authors. Finally, we also propose a future direction
for further study of Padam. Our code is available at:
https://github.com/yashkant/Padam-Tensorflow
Black Box Submodular Maximization: Discrete and Continuous Settings
Authors: Lin Chen, Mingrui Zhang, Hamed Hassani, Amin Karbasi
Date: 2019-01-28 00:00:00

In this paper, we consider the problem of black box continuous submodular maximization where we only have access to the function values and no information about the derivatives is provided. For a monotone and continuous DR-submodular function, and subject to a bounded convex body constraint, we propose Black-box Continuous Greedy, a derivative-free algorithm that provably achieves the tight $[(1-1/e)OPT-\epsilon]$ approximation guarantee with $O(d/\epsilon^3)$ function evaluations. We then extend our result to the stochastic setting where function values are subject to stochastic zero-mean noise. It is through this stochastic generalization that we revisit the discrete submodular maximization problem and use the multi-linear extension as a bridge between discrete and continuous settings. Finally, we extensively evaluate the performance of our algorithm on continuous and discrete submodular objective functions using both synthetic and real data.
Improving Neural Network Quantization without Retraining using Outlier Channel Splitting
Authors: Ritchie Zhao, Yuwei Hu, Jordan Dotzel, Christopher De Sa, Zhiru Zhang
Date: 2019-01-28 00:00:00
Tasks: Language Modeling, Language Modelling, Neural Network Compression, Quantization

Quantization can improve the execution latency and energy efficiency of neural networks on both commodity GPUs and specialized accelerators. The majority of existing literature focuses on training quantized DNNs, while this work examines the less-studied topic of quantizing a floating-point model without (re)training. DNN weights and activations follow a bell-shaped distribution post-training, while practical hardware uses a linear quantization grid. This leads to challenges in dealing with outliers in the distribution. Prior work has addressed this by clipping the outliers or using specialized hardware. In this work, we propose outlier channel splitting (OCS), which duplicates channels containing outliers, then halves the channel values. The network remains functionally identical, but affected outliers are moved toward the center of the distribution. OCS requires no additional training and works on commodity hardware. Experimental evaluation on ImageNet classification and language modeling shows that OCS can outperform state-of-the-art clipping techniques with only minor overhead.
Principled analytic classifier for positive-unlabeled learning via weighted integral probability metric
Authors: Yongchan Kwon, Wonyoung Kim, Masashi Sugiyama, Myunghee Cho Paik
Date: 2019-01-28 00:00:00
Tasks: Hyperparameter Optimization

We consider the problem of learning a binary classifier from only positive and unlabeled observations (called PU learning). Recent studies in PU learning have shown superior performance theoretically and empirically. However, most existing algorithms may not be suitable for large-scale datasets because they face repeated computations of a large Gram matrix or require massive hyperparameter optimization. In this paper, we propose a computationally efficient and theoretically grounded PU learning algorithm. The proposed PU learning algorithm produces a closed-form classifier when the hypothesis space is a closed ball in reproducing kernel Hilbert space. In addition, we establish upper bounds of the estimation error and the excess risk. The obtained estimation error bound is sharper than existing results and the derived excess risk bound has an explicit form, which vanishes as sample sizes increase. Finally, we conduct extensive numerical experiments using both synthetic and real datasets, demonstrating improved accuracy, scalability, and robustness of the proposed algorithm.
Data-to-Text Generation with Style Imitation
Authors: Shuai Lin, Wentao Wang, Zichao Yang, Xiaodan Liang, Frank F. Xu, Eric Xing, Zhiting Hu
Date: 2019-01-28 00:00:00
Proceeding: Findings of the Association for Computational Linguistics 2020
Tasks: Data-to-Text Generation, Sentence, Style Transfer, Text Generation

Recent neural approaches to data-to-text generation have mostly focused on improving content fidelity while lacking explicit control over writing styles (e.g., word choices, sentence structures). More traditional systems use templates to determine the realization of text. Yet manual or automatic construction of high-quality templates is difficult, and a template acting as hard constraints could harm content fidelity when it does not match the record perfectly. We study a new way of stylistic control by using existing sentences as soft templates. That is, the model learns to imitate the writing style of any given exemplar sentence, with automatic adaptions to faithfully describe the content record. The problem is challenging due to the lack of parallel data. We develop a neural approach that includes a hybrid attention-copy mechanism, learns with weak supervisions, and is enhanced with a new content coverage constraint. We conduct experiments in restaurants and sports domains. Results show our approach achieves stronger performance than a range of comparison methods. Our approach balances well between content fidelity and style control given exemplars that match the records to varying degrees.
Characterizing the Shape of Activation Space in Deep Neural Networks
Authors: Thomas Gebhart, Paul Schrater, Alan Hylton
Date: 2019-01-28 00:00:00

The representations learned by deep neural networks are difficult to interpret in part due to their large parameter space and the complexities introduced by their multi-layer structure. We introduce a method for computing persistent homology over the graphical activation structure of neural networks, which provides access to the task-relevant substructures activated throughout the network for a given input. This topological perspective provides unique insights into the distributed representations encoded by neural networks in terms of the shape of their activation structures. We demonstrate the value of this approach by showing an alternative explanation for the existence of adversarial examples. By studying the topology of network activations across multiple architectures and datasets, we find that adversarial perturbations do not add activations that target the semantic structure of the adversarial class as previously hypothesized. Rather, adversarial examples are explainable as alterations to the dominant activation structures induced by the original image, suggesting the class representations learned by deep networks are problematically sparse on the input space.
Strong Black-box Adversarial Attacks on Unsupervised Machine Learning Models
Authors: Anshuman Chhabra, Abhishek Roy, Prasant Mohapatra
Date: 2019-01-28 00:00:00
Tasks: Adversarial Attack, BIG-bench Machine Learning, Clustering, Language Modelling

Machine Learning (ML) and Deep Learning (DL) models have achieved state-of-the-art performance on multiple learning tasks, from vision to natural language modelling. With the growing adoption of ML and DL to many areas of computer science, recent research has also started focusing on the security properties of these models. There has been a lot of work undertaken to understand if (deep) neural network architectures are resilient to black-box adversarial attacks which craft perturbed input samples that fool the classifier without knowing the architecture used. Recent work has also focused on the transferability of adversarial attacks and found that adversarial attacks are generally easily transferable between models, datasets, and techniques. However, such attacks and their analysis have not been covered from the perspective of unsupervised machine learning algorithms. In this paper, we seek to bridge this gap through multiple contributions. We first provide a strong (iterative) black-box adversarial attack that can craft adversarial samples which will be incorrectly clustered irrespective of the choice of clustering algorithm. We choose 4 prominent clustering algorithms, and a real-world dataset to show the working of the proposed adversarial algorithm. Using these clustering algorithms we also carry out a simple study of cross-technique adversarial attack transferability.
Neural Related Work Summarization with a Joint Context-driven Attention Mechanism
Authors: Yongzhen Wang, Xiaozhong Liu, Zheng Gao
Date: 2019-01-28 00:00:00
Proceeding: EMNLP 2018 10

Conventional solutions to automatic related work summarization rely heavily
on human-engineered features. In this paper, we develop a neural data-driven
summarizer by leveraging the seq2seq paradigm, in which a joint context-driven
attention mechanism is proposed to measure the contextual relevance within full
texts and a heterogeneous bibliography graph simultaneously. Our motivation is
to maintain the topic coherency between a related work section and its target
document, where both the textual and graphic contexts play a big role in
characterizing the relationship among scientific publications accurately.
Experimental results on a large dataset show that our approach achieves a
considerable improvement over a typical seq2seq summarizer and five classical
summarization baselines.
Stiffness: A New Perspective on Generalization in Neural Networks
Authors: Stanislav Fort, Paweł Krzysztof Nowak, Stanislaw Jastrzebski, Srini Narayanan
Date: 2019-01-28 00:00:00

In this paper we develop a new perspective on generalization of neural networks by proposing and investigating the concept of a neural network stiffness. We measure how stiff a network is by looking at how a small gradient step in the network's parameters on one example affects the loss on another example. Higher stiffness suggests that a network is learning features that generalize. In particular, we study how stiffness depends on 1) class membership, 2) distance between data points in the input space, 3) training iteration, and 4) learning rate. We present experiments on MNIST, FASHION MNIST, and CIFAR-10/100 using fully-connected and convolutional neural networks, as well as on a transformer-based NLP model. We demonstrate the connection between stiffness and generalization, and observe its dependence on learning rate. When training on CIFAR-100, the stiffness matrix exhibits a coarse-grained behavior indicative of the model's awareness of super-class membership. In addition, we measure how stiffness between two data points depends on their mutual input-space distance, and establish the concept of a dynamical critical length -- a distance below which a parameter update based on a data point influences its neighbors.
Stochastic Linear Bandits with Hidden Low Rank Structure
Authors: Sahin Lale, Kamyar Azizzadenesheli, Anima Anandkumar, Babak Hassibi
Date: 2019-01-28 00:00:00
Tasks: Decision Making, Dimensionality Reduction, image-classification, Image Classification, valid

High-dimensional representations often have a lower dimensional underlying
structure. This is particularly the case in many decision making settings. For
example, when the representation of actions is generated from a deep neural
network, it is reasonable to expect a low-rank structure whereas conventional
structures like sparsity are not valid anymore. Subspace recovery methods, such
as Principle Component Analysis (PCA) can find the underlying low-rank
structures in the feature space and reduce the complexity of the learning
tasks. In this work, we propose Projected Stochastic Linear Bandit (PSLB), an
algorithm for high dimensional stochastic linear bandits (SLB) when the
representation of actions has an underlying low-dimensional subspace structure.
PSLB deploys PCA based projection to iteratively find the low rank structure in
SLBs. We show that deploying projection methods assures dimensionality
reduction and results in a tighter regret upper bound that is in terms of the
dimensionality of the subspace and its properties, rather than the
dimensionality of the ambient space. We modify the image classification task
into the SLB setting and empirically show that, when a pre-trained DNN provides
the high dimensional feature representations, deploying PSLB results in
significant reduction of regret and faster convergence to an accurate model
compared to state-of-art algorithm.
End-to-End Discriminative Deep Network for Liver Lesion Classification
Authors: Francisco Perdigon Romero, Andre Diler, Gabriel Bisson-Gregoire, Simon Turcotte, Real Lapointe, Franck Vandenbroucke-Menu, An Tang, Samuel Kadoury
Date: 2019-01-28 00:00:00
Tasks: Classification, General Classification, Lesion Classification

Colorectal liver metastasis is one of most aggressive liver malignancies.
While the definition of lesion type based on CT images determines the diagnosis
and therapeutic strategy, the discrimination between cancerous and
non-cancerous lesions are critical and requires highly skilled expertise,
experience and time. In the present work we introduce an end-to-end deep
learning approach to assist in the discrimination between liver metastases from
colorectal cancer and benign cysts in abdominal CT images of the liver. Our
approach incorporates the efficient feature extraction of InceptionV3 combined
with residual connections and pre-trained weights from ImageNet. The
architecture also includes fully connected classification layers to generate a
probabilistic output of lesion type. We use an in-house clinical biobank with
230 liver lesions originating from 63 patients. With an accuracy of 0.96 and a
F1-score of 0.92, the results obtained with the proposed approach surpasses
state of the art methods. Our work provides the basis for incorporating machine
learning tools in specialized radiology software to assist physicians in the
early detection and treatment of liver lesions.
Bridging the Gap Between Computational Photography and Visual Recognition
Authors: Rosaura G. VidalMata, Sreya Banerjee, Brandon RichardWebster, Michael Albright, Pedro Davalos, Scott McCloskey, Ben Miller, Asong Tambo, Sushobhan Ghosh, Sudarshan Nagesh, Ye Yuan, Yueyu Hu, Junru Wu, Wenhan Yang, Xiaoshuai Zhang, Jiaying Liu, Zhangyang Wang, Hwann-Tzong Chen, Tzu-Wei Huang, Wen-Chi Chin, Yi-Chun Li, Mahmoud Lababidi, Charles Otto, Walter J. Scheirer
Date: 2019-01-28 00:00:00
Tasks: Image Restoration, Object Recognition

What is the current state-of-the-art for image restoration and enhancement applied to degraded images acquired under less than ideal circumstances? Can the application of such algorithms as a pre-processing step to improve image interpretability for manual analysis or automatic visual recognition to classify scene content? While there have been important advances in the area of computational photography to restore or enhance the visual quality of an image, the capabilities of such techniques have not always translated in a useful way to visual recognition tasks. Consequently, there is a pressing need for the development of algorithms that are designed for the joint problem of improving visual appearance and recognition, which will be an enabling factor for the deployment of visual recognition tools in many real-world scenarios. To address this, we introduce the UG^2 dataset as a large-scale benchmark composed of video imagery captured under challenging conditions, and two enhancement tasks designed to test algorithmic impact on visual quality and automatic object recognition. Furthermore, we propose a set of metrics to evaluate the joint improvement of such tasks as well as individual algorithmic advances, including a novel psychophysics-based evaluation regime for human assessment and a realistic set of quantitative measures for object recognition performance. We introduce six new algorithms for image restoration or enhancement, which were created as part of the IARPA sponsored UG^2 Challenge workshop held at CVPR 2018. Under the proposed evaluation regime, we present an in-depth analysis of these algorithms and a host of deep learning-based and classic baseline approaches. From the observed results, it is evident that we are in the early days of building a bridge between computational photography and visual recognition, leaving many opportunities for innovation in this area.
Causal Discovery with a Mixture of DAGs
Authors: Eric V. Strobl
Date: 2019-01-28 00:00:00
Tasks: Causal Discovery, Causal Inference

Causal processes in biomedicine may contain cycles, evolve over time or differ between populations. However, many graphical models cannot accommodate these conditions. We propose to model causation using a mixture of directed cyclic graphs (DAGs), where the joint distribution in a population follows a DAG at any single point in time but potentially different DAGs across time. We also introduce an algorithm called Causal Inference over Mixtures that uses longitudinal data to infer a graph summarizing the causal relations generated from a mixture of DAGs. Experiments demonstrate improved performance compared to prior approaches.
An End-to-End Solution for Effectively Demoting Watermarked Images in Image Search
Authors: Ning Ma, Xin Zhao, Mark Bolin
Date: 2019-01-28 00:00:00
Tasks: Image Retrieval

We propose an end-to-end solution, from watermark feature generation to metric design, for effectively demoting watermarked images surfed by a real world image search engine. We use a few fundamental techniques to obtain effective watermark features of images in the image search index, and utilize the signals in a commercial search engine to improve the image search quality. We collect a diverse and large set (about 1M) of images with human labels indicating whether the image contains visible watermark. We train a few deep convolutional neural networks to extract watermark information from the raw images. The deep CNN classifiers we trained can achieve high accuracy on the watermark test data set. We also analyze the images based on their domains to get watermark information from a domain-based watermark classifier. We design a new novel hybrid metric which includes the relevance, image attractiveness and watermark information all together. We demonstrate that using these watermark signals together with the new metric in image search ranker can significantly demote the watermarked images during the online image ranking.
Deconstructing Generative Adversarial Networks
Authors: Banghua Zhu, Jiantao Jiao, David Tse
Date: 2019-01-27 00:00:00

We deconstruct the performance of GANs into three components: 1. Formulation: we propose a perturbation view of the population target of GANs. Building on this interpretation, we show that GANs can be viewed as a generalization of the robust statistics framework, and propose a novel GAN architecture, termed as Cascade GANs, to provably recover meaningful low-dimensional generator approximations when the real distribution is high-dimensional and corrupted by outliers. 2. Generalization: given a population target of GANs, we design a systematic principle, projection under admissible distance, to design GANs to meet the population requirement using finite samples. We implement our principle in three cases to achieve polynomial and sometimes near-optimal sample complexities: (1) learning an arbitrary generator under an arbitrary pseudonorm; (2) learning a Gaussian location family under TV distance, where we utilize our principle provide a new proof for the optimality of Tukey median viewed as GANs; (3) learning a low-dimensional Gaussian approximation of a high-dimensional arbitrary distribution under Wasserstein distance. We demonstrate a fundamental trade-off in the approximation error and statistical error in GANs, and show how to apply our principle with empirical samples to predict how many samples are sufficient for GANs in order not to suffer from the discriminator winning problem. 3. Optimization: we demonstrate alternating gradient descent is provably not locally asymptotically stable in optimizing the GAN formulation of PCA. We diagnose the problem as the minimax duality gap being non-zero, and propose a new GAN architecture whose duality gap is zero, where the value of the game is equal to the previous minimax value (not the maximin value). We prove the new GAN architecture is globally asymptotically stable in optimization under alternating gradient descent.
A deep learning-based method for prostate segmentation in T2-weighted magnetic resonance imaging
Authors: Davood Karimi, Golnoosh Samei, Yanan Shao, Septimiu Salcudean
Date: 2019-01-27 00:00:00
Tasks: Data Augmentation, Segmentation

We propose a novel automatic method for accurate segmentation of the prostate in T2-weighted magnetic resonance imaging (MRI). Our method is based on convolutional neural networks (CNNs). Because of the large variability in the shape, size, and appearance of the prostate and the scarcity of annotated training data, we suggest training two separate CNNs. A global CNN will determine a prostate bounding box, which is then resampled and sent to a local CNN for accurate delineation of the prostate boundary. This way, the local CNN can effectively learn to segment the fine details that distinguish the prostate from the surrounding tissue using the small amount of available training data. To fully exploit the training data, we synthesize additional data by deforming the training images and segmentations using a learned shape model. We apply the proposed method on the PROMISE12 challenge dataset and achieve state of the art results. Our proposed method generates accurate, smooth, and artifact-free segmentations. On the test images, we achieve an average Dice score of 90.6 with a small standard deviation of 2.2, which is superior to all previous methods. Our two-step segmentation approach and data augmentation strategy may be highly effective in segmentation of other organs from small amounts of annotated medical images.