pwc-archive/papers-with-abstracts huggingface dataset:
A Multi-Modal and Multitask Benchmark in the Clinical Domain
Authors: Yong Huang, Edgar Mariano Marroquin, Volodymyr Kuleshov
Date: 2021-01-01 00:00:00
Tasks: BIG-bench Machine Learning, Decompensation, Deep Learning, Time Series, Time Series Analysis

Healthcare represents one of the most promising application areas for machine learning algorithms,  including modern methods based on deep learning.   Modern deep learning algorithms perform best on large datasets and on unstructured modalities such as text or image data; advances in deep learning have often been driven by the availability of such large datasets. Here, we introduce Multi-Modal Multitask MIMIC-III (M3) — a dataset and benchmark for evaluating machine learning algorithms in the healthcare domain.  This dataset contains multi-modal patient data collected from intensive care units — including physiological time series, clinical notes, ECG waveforms, and tabular inputs — and defines six clinical tasks — including predicting mortality, decompensation, readmission, and other outcomes — which serve as benchmarks for comparing algorithms. We introduce new multi-modal and multitask models for this dataset, and show that they outperform previous state-of-the-art results that only rely on a subset of all tasks and modalities. This highlights the potential of multitask and multi-modal learning to improve the performance of algorithms in the healthcare domain. More generally, we envision M3 as a general resource that will help accelerate research in applying machine learning to healthcare.
What are effective labels for augmented data? Improving robustness with AutoLabel
Authors: Yao Qin, Xuezhi Wang, Balaji Lakshminarayanan, Ed Chi, Alex Beutel
Date: 2021-01-01 00:00:00
Tasks: Adversarial Robustness, Data Augmentation

A wide breadth of research has devised data augmentation approaches that can improve both accuracy and generalization performance for neural networks. However, augmented data can end up being far from the clean data and what is the appropriate label is less clear. Despite this, most existing work simply reuses the original label from the clean data, and the choice of label accompanying the augmented data is relatively less explored. In this paper, we propose AutoLabel to automatically learn the labels for augmented data, based on the distance between the clean distribution and augmented distribution. AutoLabel is built on label smoothing and is guided by the calibration-performance over a hold-out validation set. We show that AutoLabel is a generic framework that can be easily applied to existing data augmentation methods, including AugMix, mixup, and adversarial training. Experiments on CIFAR-10, CIFAR-100 and ImageNet show that AutoLabel can improve models' accuracy and calibration performance, especially under distributional shift. Additionally, we demonstrate that AutoLabel can help adversarial training by bridging the gap between clean accuracy and adversarial robustness.
Individuality in the hive - Learning to embed lifetime social behaviour of honey bees
Authors: Benjamin Wild, David Dormagen, Michael L Smith, Tim Landgraf
Date: 2021-01-01 00:00:00

Honey bees are a popular model for complex social systems, in which global behavior emerges from the actions and interactions of thousands of individuals. While the average life of a bee is organized as a sequence of tasks roughly determined by age, there is substantial variation at the individual level. For example, young bees can become foragers early in life, depending on the colony’s needs. Using a unique dataset containing lifetime trajectories of all individuals over multiple generations in two honey bee colonies, we propose a new temporal matrix factorization model that jointly learns the average developmental path and structured variations of individuals in the social network over their entire lives. Our method yields inherently interpretable embeddings that are biologically plausible and consistent over time, which allow one to compare individuals regardless of when, or in which colony, they lived. Our method provides a quantitative framework for understanding behavioral heterogeneity in complex social systems applicable in fields such as behavioral biology, social sciences, neuroscience, and information science.
On Relating "Why?" and "Why Not?" Explanations
Authors: Alexey Ignatiev, Nina Narodytska, Nicholas Asher, Joao Marques-Silva
Date: 2021-01-01 00:00:00

Explanations of Machine Learning (ML) models often address a ‘Why?’ question. Such explanations can be related with selecting feature-value pairs which are sufficient for the prediction. Recent work has investigated explanations that address
a ‘Why Not?’ question, i.e. finding a change of feature values that guarantee a change of prediction. Given their goals, these two forms of explaining predictions of ML models appear to be mostly unrelated. However, this paper demonstrates otherwise, and establishes a rigorous formal relationship between ‘Why?’ and ‘Why Not?’ explanations. Concretely, the paper proves that, for any given instance, ‘Why?’ explanations are minimal hitting sets of ‘Why Not?’ explanations and vice-versa. Furthermore, the paper devises novel algorithms for extracting and enumerating both forms of explanations.
Learning Contextualized Knowledge Graph Structures for Commonsense Reasoning
Authors: Jun Yan, Mrigank Raman, Tianyu Zhang, Ryan Rossi, Handong Zhao, Sungchul Kim, Nedim Lipka, Xiang Ren
Date: 2021-01-01 00:00:00
Tasks: Knowledge Graphs, Natural Language Inference, Question Answering

Recently, neural-symbolic architectures have achieved success on commonsense reasoning through effectively encoding relational structures retrieved from external knowledge graphs (KGs) and obtained state-of-the-art results in tasks such as (commonsense) question answering and natural language inference. However, current neural-symbolic reasoning methods rely on quality and contextualized knowledge structures (i.e., fact triples) that can be retrieved at the pre-processing stage and overlook challenges such as dealing with incompleteness of a KG (low coverage), limited expressiveness of its relations, and irrelevant retrieved facts in the reasoning context. 
In this paper, we present a novel neural-symbolic approach, named Hybrid Graph Network (HGN), which jointly generates feature representations for new triples (as complement to the existing edges in the KG), determines relevance of the triples to the reasoning context, and learns graph model parameters for encoding the relational information. Our method learns a compact graph structure (comprising both retrieved and generated edges) through filtering edges that are unhelpful to the reasoning process. We show marked improvements on three commonsense reasoning benchmarks and demonstrate the superiority of the learned graph structures with user studies.
Robust Ensembles of Neural Networks using Itô Processes
Authors: Sumit Kumar Jha, Susmit Jha, Rickard Ewetz, Alvaro Velasquez
Date: 2021-01-01 00:00:00

Residual neural networks (ResNets) can be modeled as dynamical systems where the evolution of dynamical systems represents the inference in ResNets. We exploit this connection and the theory of stochastic dynamical systems to construct a novel ensemble of Itô processes as a new deep learning representation that is more robust than classical residual networks. An Itô process obtained by solving a suitably-formulated stochastic differential equation derived from a residual network has a probability density function that is not readily perturbed by small changes in the neural network’s inputs. Our robust stochastic Itô ensemble of neural networks achieve an accuracy of 73.91% on the CIFAR-10 dataset against the PGD attack with ε = 2.0 under the L2 norm, while the accuracy of Madry’s robustness toolbox on the same attack is 18.59%. Similarly, our stochastic Itô ensemble of neural networks achieves an accuracy of 79.66% on PGD attack with ε = 16/255 under the L∞ norm, while the accuracy of Madry’s robustness toolbox on the same attack is 18.13%. The Itô ensemble trained on ImageNet achieves an accuracy of 28.53% against PGD attacks under the L∞ norm with ε = 16/255 and accuracy of 65.74% under the L2 norm with ε = 3.0, respectively. This significantly improves state-of-the-art accuracy of 5% and 35.16% for Madry’s robustness tool against the same PGD attacks under the L∞ and L2 norms, respectively. Further, our approach achieves these high robustness values without any explicit adversarial training or a significant loss of accuracy on benign inputs.
Beyond COVID-19 Diagnosis: Prognosis with Hierarchical Graph Representation Learning
Authors: Chen Liu, Jinze Cui, Dailin Gan, Guosheng Yin
Date: 2021-01-01 00:00:00
Tasks: Computed Tomography (CT), COVID-19 Diagnosis, Decision Making, Graph Representation Learning, Prognosis, Representation Learning

Coronavirus disease 2019 (COVID-19), the pandemic that is spreading fast globally, has caused over 34 million confirmed cases. Apart from the reverse transcription polymerase chain reaction (RT-PCR), the chest computed tomography (CT) is viewed as a standard and effective tool for disease diagnosis and progression monitoring. We propose a diagnosis and prognosis model based on graph convolutional networks (GCNs). The chest CT scan of a patient, typically involving hundreds of sectional images in sequential order, is formulated as a densely connected weighted graph. A novel distance aware pooling is proposed to abstract the node information hierarchically, which is robust and efficient for such densely connected graphs. Our method, combining GCNs and distance aware pooling, can integrate the information from all slices in the chest CT scans for optimal decision making, which leads to the state-of-the-art accuracy in the COVID-19 diagnosis and prognosis. With less than 1% number of total parameters in the baseline 3D ResNet model, our method achieves 94.7% accuracy for diagnosis. It has a 2.3% improvement compared with the baseline model on the same dataset. In addition, we can localize the most informative slices with disease lesions for COVID-19 within a large sequence of chest CT images. The proposed model can produce visual explanations for the diagnosis and prognosis, making the decision more transparent and explainable, while RT-PCR only leads to the test result with no prognosis information. The prognosis analysis can help hospitals or clinical centers designate medical resources more efficiently and better support clinicians to determine the proper clinical treatment.
Unsupervised Simultaneous Depth-from-defocus and Depth-from-focus
Authors: Yawen Lu, Guoyu Lu
Date: 2021-01-01 00:00:00
Tasks: Depth Estimation

If the accuracy of depth estimation from a single RGB image could be improved it would be possible to eliminate the need for expensive and bulky depth sensing hardware. The majority of efforts toward this end have been focused on utilizing geometric constraints, image sequences, or stereo image pairs with the help of a deep neural network. In this work, we propose a framework for simultaneous depth estimation from a single image and image focal stacks using depth-from-defocus and depth-from-focus algorithms. The proposed network is able to learn optimal depth mapping from the information contained in the blurring of a single image, generate a simulated image focal stack and all-in-focus image, and train a depth estimator from an image focal stack. As there is no large dataset specifically designed for our problem, we first learned on a synthetic indoor dataset: NYUv2. Then we compare the performance by comparing with other existing methods on DSLR dataset. Finally, we collected our own dataset using a DSLR and further verify on it. Experiments demonstrate that our system is able to provide comparable results compared with other state-of-the-art methods.
H-divergence: A Decision-Theoretic Discrepancy Measure for Two Sample Tests
Authors: Shengjia Zhao, Abhishek Sinha, Yutong He, Aidan Perreault, Jiaming Song, Stefano Ermon
Date: 2021-01-01 00:00:00
Tasks: Vocal Bursts Valence Prediction

Measuring the discrepancy between two probability distributions is a fundamental problem in machine learning and statistics. Based on ideas from decision theory, we investigate a new class of discrepancies that are based on the optimal decision loss. Two probability distributions are different if the optimal decision loss is higher on the mixture distribution than on each individual distribution. We show that this generalizes popular notions of discrepancy measurements such as the Jensen Shannon divegence and the maximum mean discrepancy. We apply our approach to two-sample tests, which evaluates whether two sets of samples come from the same distribution. On various benchmark and real datasets, we demonstrate that tests based on our generalized notion of discrepancy is able to achieve superior test power. We also apply our approach to sample quality evaluation as an alternative to the FID score, and our discrepancy ranks sample quality consistently with human intuition.
Single Image Depth Estimation Based on Spectral Consistency and Predicted View
Authors: Anonymous
Date: 2021-01-01 00:00:00
Tasks: Depth Estimation

Single image depth estimation is a critical issue for robot vision, augmented reality, and many other applications when an image sequence is not available. Self-supervised single image depth estimation models target at predicting accurate disparity map just from one single image without ground truth supervision or stereo image pair during real applications. Compared with direct single image depth estimation, single image stereo algorithm can generate the depth from different camera perspectives. In this paper, we propose a novel architecture to infer accurate disparity by leveraging both spectral-consistency based learning model and view-prediction based stereo reconstruction algorithm. Direct spectral-consistency based method can avoid false positive matching in smooth regions. Single image stereo can preserve more distinct boundaries from another camera perspective. By learning confidence map and designing a fusion strategy, the two disparities from two approaches are able to be effectively fused to produce the refined disparity. Extensive experiments indicate that our method exploits both advantages of spectral consistency and view prediction, especially in constraining boundaries and correcting wrong predicting regions.
AttackDist: Characterizing Zero-day Adversarial Samples by Counter Attack
Authors: Simin Chen, Zihe Song, Lei Ma, Cong Liu, Wei Yang
Date: 2021-01-01 00:00:00

Deep Neural Networks (DNNs) have been shown vulnerable to adversarial attacks, which could produce adversarial samples that easily fool the state-of-the-art DNNs. The harmfulness of adversarial attacks calls for the defense mechanisms under fire. However, the relationship between adversarial attacks and defenses is like spear and shield. Whenever a defense method is proposed, a new attack would be followed to bypass the defense immediately. Devising a definitive defense against new attacks~(zero-day attacks) is proven to be challenging. We tackle this challenge by characterizing the intrinsic properties of adversarial samples, via measuring the norm of the perturbation after a counterattack. Our method is based on the idea that, from an optimization perspective, adversarial samples would be closer to the decision boundary; thus the perturbation to counterattack adversarial samples would be significantly smaller than normal cases. Motivated by this, we propose AttackDist, an attack-agnostic property to characterize adversarial samples. We first theoretically clarify under which condition AttackDist can provide a certified detecting performance, then show that a potential application of AttackDist is distinguishing zero-day adversarial examples without knowing the mechanisms of new attacks. As a proof-of-concept, we evaluate AttackDist on two widely used benchmarks. The evaluation results show that AttackDist can outperform the state-of-the-art detection measures by large margins in detecting zero-day adversarial attacks.
Unsupervised inference for optimizing deep feedforward neural network architecture
Authors: Anonymous
Date: 2021-01-01 00:00:00

We propose an unsupervised inference algorithm to find optimal deep feedforward neural network architecture by modeling hierarchical representations of given data set. Our algorithm learns the optimal neural network architecture that represents a system that generates observed data, in a forward manner without backpropagation. We hypothesize that a neural network architecture, which models the hierarchical representations of given data, provides the optimal feedforward neural network architecture with competitive predictive performance in a supervised manner when compared with supervised-based neural network architecture optimization models. To prove the hypothesis, we evaluated the predictive performance with the feedforward neural network architectures with intensive experiments using various well-known benchmark data sets, such as SPECT, FMNIST, CIFAR-10, heart disease, German Credit, and Statlog heart data. Our algorithm requires much smaller search space for feasible computation than conventional supervised-based neural network architecture optimization methods. The proposed unsupervised inference approach can fully enjoy optimizing neural network architecture from small size of labeled data, when a huge unlabeled data is available.
Learning Better Structured Representations Using Low-rank Adaptive Label Smoothing
Authors: Asish Ghoshal, Xilun Chen, Sonal Gupta, Luke Zettlemoyer, Yashar Mehdad
Date: 2021-01-01 00:00:00
Proceeding: ICLR 2021 1
Tasks: Generalization Bounds, Machine Translation, Semantic Parsing, Structured Prediction, text-classification, Text Classification

Training with soft targets instead of hard targets has been shown to improve performance and calibration of deep neural networks. Label smoothing is popular way of computing soft targets, where one-hot encoding of a class is smoothed with a uniform distribution. Owing to its simplicity, it has found wide-spread use for training deep neural networks on a wide variety of tasks, ranging from image and text classification to machine translation and semantic parsing. Complementing recent empirical justification for label smoothing, we obtain PAC-Bayesian generalization bounds for label smoothing and show that the generalization error depends on choice of the noise (smoothing) distribution. Then we propose low-rank adaptive label smoothing (LORAS): a simple yet novel method for training with learned soft targets that generalizes label smoothing and adapts to the latent structure of the label space in structured prediction tasks. Specifically, we evaluate our method on task-oriented semantic parsing tasks and show that just by training with appropriately smoothed soft targets, one can improve the accuracy of models by as much as 2% and reduce calibration error by 55% as compared to vanilla label smoothing. Used in conjunction with pre-trained sequence-to-sequence models, our method achieves state of the art performance on three semantic parsing data sets. LORAS can be used with any model, improves performance and implicit model calibration without increasing the number of model parameters, and can be scaled to problems with large label spaces containing tens of thousands of labels.
Adaptive Gradient Methods Can Be Provably Faster than SGD with Random Shuffling
Authors: Xunpeng Huang, Vicky Jiaqi Zhang, Hao Zhou, Lei LI
Date: 2021-01-01 00:00:00

Adaptive gradient methods have been shown to outperform SGD in many tasks of training neural networks. However, the acceleration effect is yet to be explained in the non-convex setting, since the best convergence rate of adaptive gradient methods is worse than that of SGD in literature. In this paper, we prove that adaptive gradient methods exhibit an $\small\tilde{O}(T^{-1/2})$-convergence rate for finding first-order stationary points under some mild assumptions, which improves previous best convergence results of adaptive gradient methods and SGD by factors of $\small O(T^{-1/4})$ and $\small O(T^{-1/6})$, respectively. In particular, we study two variants of AdaGrad with random shuffling and identify a novel consistency condition from general experiments result. Our analysis suggests that the combination of random shuffling and adaptive learning rates gives rise to better convergence.
Unsupervised Word Alignment via Cross-Lingual Contrastive Learning
Authors: Di wu, Liang Ding, Shuo Yang, DaCheng Tao
Date: 2021-01-01 00:00:00
Tasks: Contrastive Learning, Translation, Word Alignment

Word alignment is essential for the down-streaming cross-lingual language understanding and generation tasks. Recently, the performance of the neural word alignment models has exceeded that of statistical models. However, they heavily rely on sophisticated translation models. In this study, we propose a super lightweight unsupervised word alignment model, dubbed MirrorAlign, in which bidirectional symmetric attention trained with a contrastive learning objective is introduced, and an agreement loss is employed to bind the attention maps, such that the alignments follow mirror-like symmetry hypothesis. Experimental results on several public benchmarks demonstrate that our model achieves competitive, if not better, performance compared to the state of the art in word alignment while significantly reducing the training and decoding time on average. Further ablation analysis and case studies show the superiority of our proposed MirrorAlign. Notably, we recognize our model as a pioneer attempt to unify bilingual word embedding and word alignments. Encouragingly, our approach achieves 16.4X speedup against GIZA++, and 50X parameter compression compared with the Transformer-based alignment methods. We released our code to facilitate the community: https://github.com/ICLR20anonymous/mirroralign
A Unified Bayesian Framework for Discriminative and Generative Continual Learning
Authors: Abhishek Kumar, Sunabha Chatterjee, Piyush Rai
Date: 2021-01-01 00:00:00
Tasks: Continual Learning

Continual Learning is a learning paradigm where learning systems are trained on a sequence of tasks. The goal here is to perform well on the current task without suffering from a performance drop on the previous tasks. Two notable directions among the recent advances in continual learning with neural networks are (1) variational Bayes based regularization by learning priors from previous tasks, and, (2) learning the structure of deep networks to adapt to new tasks. So far, these two approaches have been orthogonal. We present a novel Bayesian framework for continual learning based on learning the structure of deep neural networks, addressing the shortcomings of both these approaches. The proposed framework learns the deep structure for each task by learning which weights to be used, and supports inter-task transfer through the overlapping of different sparse subsets of weights learned by different tasks. An appealing aspect of our proposed continual learning framework is that it is applicable to both discriminative (supervised) and generative (unsupervised) settings. Experimental results on supervised and unsupervised benchmarks shows that our model performs comparably or better than recent advances in continual learning.