pwc-archive/papers-with-abstracts huggingface dataset:
End-to-End Multi-Task Denoising for joint SDR and PESQ Optimization
Authors: Jaeyoung Kim, Mostafa El-Khamy, Jungwon Lee
Date: 2019-01-26 00:00:00
Tasks: Denoising, Speech Enhancement

Supervised learning based on a deep neural network recently has achieved substantial improvement on speech enhancement. Denoising networks learn mapping from noisy speech to clean one directly, or to a spectrum mask which is the ratio between clean and noisy spectra. In either case, the network is optimized by minimizing mean square error (MSE) between ground-truth labels and time-domain or spectrum output. However, existing schemes have either of two critical issues: spectrum and metric mismatches. The spectrum mismatch is a well known issue that any spectrum modification after short-time Fourier transform (STFT), in general, cannot be fully recovered after inverse short-time Fourier transform (ISTFT). The metric mismatch is that a conventional MSE metric is sub-optimal to maximize our target metrics, signal-to-distortion ratio (SDR) and perceptual evaluation of speech quality (PESQ). This paper presents a new end-to-end denoising framework with the goal of joint SDR and PESQ optimization. First, the network optimization is performed on the time-domain signals after ISTFT to avoid spectrum mismatch. Second, two loss functions which have improved correlations with SDR and PESQ metrics are proposed to minimize metric mismatch. The experimental result showed that the proposed denoising scheme significantly improved both SDR and PESQ performance over the existing methods.
Graphical-model based estimation and inference for differential privacy
Authors: Ryan McKenna, Daniel Sheldon, Gerome Miklau
Date: 2019-01-26 00:00:00

Many privacy mechanisms reveal high-level information about a data
distribution through noisy measurements. It is common to use this information
to estimate the answers to new queries. In this work, we provide an approach to
solve this estimation problem efficiently using graphical models, which is
particularly effective when the distribution is high-dimensional but the
measurements are over low-dimensional marginals. We show that our approach is
far more efficient than existing estimation techniques from the privacy
literature and that it can improve the accuracy and scalability of many
state-of-the-art mechanisms.
Progressive Label Distillation: Learning Input-Efficient Deep Neural Networks
Authors: Zhong Qiu Lin, Alexander Wong
Date: 2019-01-26 00:00:00
Tasks: Knowledge Distillation, speech-recognition, Speech Recognition

Much of the focus in the area of knowledge distillation has been on
distilling knowledge from a larger teacher network to a smaller student
network. However, there has been little research on how the concept of
distillation can be leveraged to distill the knowledge encapsulated in the
training data itself into a reduced form. In this study, we explore the concept
of progressive label distillation, where we leverage a series of
teacher-student network pairs to progressively generate distilled training data
for learning deep neural networks with greatly reduced input dimensions. To
investigate the efficacy of the proposed progressive label distillation
approach, we experimented with learning a deep limited vocabulary speech
recognition network based on generated 500ms input utterances distilled
progressively from 1000ms source training data, and demonstrated a significant
increase in test accuracy of almost 78% compared to direct learning.
Stacking and stability
Authors: Nino Arsov, Martin Pavlovski, Ljupco Kocarev
Date: 2019-01-26 00:00:00
Tasks: Meta-Learning

Stacking is a general approach for combining multiple models toward greater
predictive accuracy. It has found various application across different domains,
ensuing from its meta-learning nature. Our understanding, nevertheless, on how
and why stacking works remains intuitive and lacking in theoretical insight. In
this paper, we use the stability of learning algorithms as an elemental
analysis framework suitable for addressing the issue. To this end, we analyze
the hypothesis stability of stacking, bag-stacking, and dag-stacking and
establish a connection between bag-stacking and weighted bagging. We show that
the hypothesis stability of stacking is a product of the hypothesis stability
of each of the base models and the combiner. Moreover, in bag-stacking and
dag-stacking, the hypothesis stability depends on the sampling strategy used to
generate the training set replicates. Our findings suggest that 1) subsampling
and bootstrap sampling improve the stability of stacking, and 2) stacking
improves the stability of both subbagging and bagging.
Language Model Pre-training for Hierarchical Document Representations
Authors: Ming-Wei Chang, Kristina Toutanova, Kenton Lee, Jacob Devlin
Date: 2019-01-26 00:00:00
Proceeding: ICLR 2019 5
Tasks: Document Summarization, Extractive Document Summarization, Extractive Text Summarization, Language Modeling, Language Modelling, Question Answering, Segmentation, Sentence, Sentiment Analysis

Hierarchical neural architectures are often used to capture long-distance
dependencies and have been applied to many document-level tasks such as
summarization, document segmentation, and sentiment analysis. However,
effective usage of such a large context can be difficult to learn, especially
in the case where there is limited labeled data available. Building on the
recent success of language model pretraining methods for learning flat
representations of text, we propose algorithms for pre-training hierarchical
document representations from unlabeled data. Unlike prior work, which has
focused on pre-training contextual token representations or context-independent
{sentence/paragraph} representations, our hierarchical document representations
include fixed-length sentence/paragraph representations which integrate
contextual information from the entire documents. Experiments on document
segmentation, document-level question answering, and extractive document
summarization demonstrate the effectiveness of the proposed pre-training
algorithms.
Strong Equivalence and Program Structure in Arguing Essential Equivalence between Logic Programs
Authors: Yuliya Lierler
Date: 2019-01-26 00:00:00

Answer set programming is a prominent declarative programming paradigm used in formulating combinatorial search problems and implementing different knowledge representation formalisms. Frequently, several related and yet substantially different answer set programs exist for a given problem. Sometimes these encodings may display significantly different performance. Uncovering precise formal links between these programs is often important and yet far from trivial. This paper presents formal results carefully relating a number of interesting program rewritings. It also provides the proof of correctness of system Projector concerned with automatic program rewritings for the sake of efficiency. Under consideration in Theory and Practice of Logic Programming (TPLP).
The Use of Unlabeled Data versus Labeled Data for Stopping Active Learning for Text Classification
Authors: Garrett Beatty, Ethan Kochis, Michael Bloodgood
Date: 2019-01-26 00:00:00
Tasks: Active Learning, General Classification, text-classification, Text Classification

Annotation of training data is the major bottleneck in the creation of text
classification systems. Active learning is a commonly used technique to reduce
the amount of training data one needs to label. A crucial aspect of active
learning is determining when to stop labeling data. Three potential sources for
informing when to stop active learning are an additional labeled set of data,
an unlabeled set of data, and the training data that is labeled during the
process of active learning. To date, no one has compared and contrasted the
advantages and disadvantages of stopping methods based on these three
information sources. We find that stopping methods that use unlabeled data are
more effective than methods that use labeled data.
The informal semantics of Answer Set Programming: A Tarskian perspective
Authors: Marc Denecker, Yuliya Lierler, Miroslaw Truszczynski, Joost Vennekens
Date: 2019-01-26 00:00:00

In Knowledge Representation, it is crucial that knowledge engineers have a
good understanding of the formal expressions that they write. What formal
expressions state intuitively about the domain of discourse is studied in the
theory of the informal semantics of a logic. In this paper we study the
informal semantics of Answer Set Programming. The roots of answer set
programming lie in the language of Extended Logic Programming, which was
introduced initially as an epistemic logic for default and autoepistemic
reasoning. In 1999, the seminal papers on answer set programming proposed to
use this logic for a different purpose, namely, to model and solve search
problems. Currently, the language is used primarily in this new role. However,
the original epistemic intuitions lose their explanatory relevance in this new
context. How answer set programs are connected to the specifications of
problems they model is more easily explained in a classical Tarskian semantics,
in which models correspond to possible worlds, rather than to belief states of
an epistemic agent. In this paper, we develop a new theory of the informal
semantics of answer set programming, which is formulated in the Tarskian
setting and based on Frege's compositionality principle. It differs
substantially from the earlier epistemic theory of informal semantics,
providing a different view on the meaning of the connectives in answer set
programming and on its relation to other logics, in particular classical logic.
DeepSZ: A Novel Framework to Compress Deep Neural Networks by Using Error-Bounded Lossy Compression
Authors: Sian Jin, Sheng Di, Xin Liang, Jiannan Tian, Dingwen Tao, Franck Cappello
Date: 2019-01-26 00:00:00
Tasks: Network Pruning, Neural Network Compression

DNNs have been quickly and broadly exploited to improve the data analysis
quality in many complex science and engineering applications. Today's DNNs are
becoming deeper and wider because of increasing demand on the analysis quality
and more and more complex applications to resolve. The wide and deep DNNs,
however, require large amounts of resources, significantly restricting their
utilization on resource-constrained systems. Although some network
simplification methods have been proposed to address this issue, they suffer
from either low compression ratios or high compression errors, which may
introduce a costly retraining process for the target accuracy. In this paper,
we propose DeepSZ: an accuracy-loss bounded neural network compression
framework, which involves four key steps: network pruning, error bound
assessment, optimization for error bound configuration, and compressed model
generation, featuring a high compression ratio and low encoding time. The
contribution is three-fold. (1) We develop an adaptive approach to select the
feasible error bounds for each layer. (2) We build a model to estimate the
overall loss of accuracy based on the accuracy degradation caused by individual
decompressed layers. (3) We develop an efficient optimization algorithm to
determine the best-fit configuration of error bounds in order to maximize the
compression ratio under the user-set accuracy constraint. Experiments show that
DeepSZ can compress AlexNet and VGG-16 on the ImageNet by a compression ratio
of 46X and 116X, respectively, and compress LeNet-300-100 and LeNet-5 on the
MNIST by a compression ratio of 57X and 56X, respectively, with only up to 0.3%
loss of accuracy. Compared with other state-of-the-art methods, DeepSZ can
improve the compression ratio by up to 1.43X, the DNN encoding performance by
up to 4.0X (with four Nvidia Tesla V100 GPUs), and the decoding performance by
up to 6.2X.
Stopping Active Learning based on Predicted Change of F Measure for Text Classification
Authors: Michael Altschuler, Michael Bloodgood
Date: 2019-01-26 00:00:00
Tasks: Active Learning, General Classification, text-classification, Text Classification

During active learning, an effective stopping method allows users to limit
the number of annotations, which is cost effective. In this paper, a new
stopping method called Predicted Change of F Measure will be introduced that
attempts to provide the users an estimate of how much performance of the model
is changing at each iteration. This stopping method can be applied with any
base learner. This method is useful for reducing the data annotation bottleneck
encountered when building text classification systems.
Context in Neural Machine Translation: A Review of Models and Evaluations
Authors: Andrei Popescu-Belis
Date: 2019-01-25 00:00:00
Tasks: Machine Translation, NMT, Translation

This review paper discusses how context has been used in neural machine
translation (NMT) in the past two years (2017-2018). Starting with a brief
retrospect on the rapid evolution of NMT models, the paper then reviews studies
that evaluate NMT output from various perspectives, with emphasis on those
analyzing limitations of the translation of contextual phenomena. In a
subsequent version, the paper will then present the main methods that were
proposed to leverage context for improving translation quality, and
distinguishes methods that aim to improve the translation of specific phenomena
from those that consider a wider unstructured context.
Generative Adversarial Networks for Black-Box API Attacks with Limited Training Data
Authors: Yi Shi, Yalin E. Sagduyu, Kemal Davaslioglu, Jason H. Li
Date: 2019-01-25 00:00:00
Tasks: BIG-bench Machine Learning, Generative Adversarial Network, Inference Attack

As online systems based on machine learning are offered to public or paid
subscribers via application programming interfaces (APIs), they become
vulnerable to frequent exploits and attacks. This paper studies adversarial
machine learning in the practical case when there are rate limitations on API
calls. The adversary launches an exploratory (inference) attack by querying the
API of an online machine learning system (in particular, a classifier) with
input data samples, collecting returned labels to build up the training data,
and training an adversarial classifier that is functionally equivalent and
statistically close to the target classifier. The exploratory attack with
limited training data is shown to fail to reliably infer the target classifier
of a real text classifier API that is available online to the public. In
return, a generative adversarial network (GAN) based on deep learning is built
to generate synthetic training data from a limited number of real training data
samples, thereby extending the training data and improving the performance of
the inferred classifier. The exploratory attack provides the basis to launch
the causative attack (that aims to poison the training process) and evasion
attack (that aims to fool the classifier into making wrong decisions) by
selecting training and test data samples, respectively, based on the confidence
scores obtained from the inferred classifier. These stealth attacks with small
footprint (using a small number of API calls) make adversarial machine learning
practical under the realistic case with limited training data available to the
adversary.
DADAM: A Consensus-based Distributed Adaptive Gradient Method for Online Optimization
Authors: Parvin Nazari, Davoud Ataee Tarzanagh, George Michailidis
Date: 2019-01-25 00:00:00
Proceeding: ICLR 2019 5
Tasks: Stochastic Optimization

Adaptive gradient-based optimization methods such as \textsc{Adagrad}, \textsc{Rmsprop}, and \textsc{Adam} are widely used in solving large-scale machine learning problems including deep learning. A number of schemes have been proposed in the literature aiming at parallelizing them, based on communications of peripheral nodes with a central node, but incur high communications cost. To address this issue, we develop a novel consensus-based distributed adaptive moment estimation method (\textsc{Dadam}) for online optimization over a decentralized network that enables data parallelization, as well as decentralized computation. The method is particularly useful, since it can accommodate settings where access to local data is allowed. Further, as established theoretically in this work, it can outperform centralized adaptive algorithms, for certain classes of loss functions used in applications. We analyze the convergence properties of the proposed algorithm and provide a dynamic regret bound on the convergence rate of adaptive moment estimation methods in both stochastic and deterministic settings. Empirical results demonstrate that \textsc{Dadam} works also well in practice and compares favorably to competing online optimization methods.
Subspace Clustering of Very Sparse High-Dimensional Data
Authors: Hankui Peng, Nicos Pavlidis, Idris Eckley, Ioannis Tsalamanis
Date: 2019-01-25 00:00:00
Tasks: Clustering, Fraud Detection, Sentiment Analysis, Vocal Bursts Intensity Prediction

In this paper we consider the problem of clustering collections of very short
texts using subspace clustering. This problem arises in many applications such
as product categorisation, fraud detection, and sentiment analysis. The main
challenge lies in the fact that the vectorial representation of short texts is
both high-dimensional, due to the large number of unique terms in the corpus,
and extremely sparse, as each text contains a very small number of words with
no repetition. We propose a new, simple subspace clustering algorithm that
relies on linear algebra to cluster such datasets. Experimental results on
identifying product categories from product names obtained from the US Amazon
website indicate that the algorithm can be competitive against state-of-the-art
clustering algorithms.
Audio-Visual Scene-Aware Dialog
Authors: Huda Alamri, Vincent Cartillier, Abhishek Das, Jue Wang, Anoop Cherian, Irfan Essa, Dhruv Batra, Tim K. Marks, Chiori Hori, Peter Anderson, Stefan Lee, Devi Parikh
Date: 2019-01-25 00:00:00
Tasks: Scene-Aware Dialogue

We introduce the task of scene-aware dialog. Our goal is to generate a complete and natural response to a question about a scene, given video and audio of the scene and the history of previous turns in the dialog. To answer successfully, agents must ground concepts from the question in the video while leveraging contextual cues from the dialog history. To benchmark this task, we introduce the Audio Visual Scene-Aware Dialog (AVSD) Dataset. For each of more than 11,000 videos of human actions from the Charades dataset, our dataset contains a dialog about the video, plus a final summary of the video by one of the dialog participants. We train several baseline systems for this task and evaluate the performance of the trained models using both qualitative and quantitative metrics. Our results indicate that models must utilize all the available inputs (video, audio, question, and dialog history) to perform best on this dataset.
On Learning Meaningful Code Changes via Neural Machine Translation
Authors: Michele Tufano, Jevgenija Pantiuchina, Cody Watson, Gabriele Bavota, Denys Poshyvanyk
Date: 2019-01-25 00:00:00
Tasks: Bug fixing, Machine Translation, NMT, Translation

Recent years have seen the rise of Deep Learning (DL) techniques applied to
source code. Researchers have exploited DL to automate several development and
maintenance tasks, such as writing commit messages, generating comments and
detecting vulnerabilities among others. One of the long lasting dreams of
applying DL to source code is the possibility to automate non-trivial coding
activities. While some steps in this direction have been taken (e.g., learning
how to fix bugs), there is still a glaring lack of empirical evidence on the
types of code changes that can be learned and automatically applied by DL. Our
goal is to make this first important step by quantitatively and qualitatively
investigating the ability of a Neural Machine Translation (NMT) model to learn
how to automatically apply code changes implemented by developers during pull
requests. We train and experiment with the NMT model on a set of 236k pairs of
code components before and after the implementation of the changes provided in
the pull requests. We show that, when applied in a narrow enough context (i.e.,
small/medium-sized pairs of methods before/after the pull request changes), NMT
can automatically replicate the changes implemented by developers during pull
requests in up to 36% of the cases. Moreover, our qualitative analysis shows
that the model is capable of learning and replicating a wide variety of
meaningful code changes, especially refactorings and bug-fixing activities. Our
results pave the way for novel research in the area of DL on code, such as the
automatic learning and applications of refactoring.
Communication Complexity of Estimating Correlations
Authors: Uri Hadar, Jingbo Liu, Yury Polyanskiy, Ofer Shayevitz
Date: 2019-01-25 00:00:00

We characterize the communication complexity of the following distributed
estimation problem. Alice and Bob observe infinitely many iid copies of
$\rho$-correlated unit-variance (Gaussian or $\pm1$ binary) random variables,
with unknown $\rho\in[-1,1]$. By interactively exchanging $k$ bits, Bob wants
to produce an estimate $\hat\rho$ of $\rho$. We show that the best possible
performance (optimized over interaction protocol $\Pi$ and estimator $\hat
\rho$) satisfies $\inf_{\Pi \hat\rho}\sup_\rho \mathbb{E} [|\rho-\hat\rho|^2] =
\tfrac{1}{k} (\frac{1}{2 \ln 2} + o(1))$. Curiously, the number of samples in
our achievability scheme is exponential in $k$; by contrast, a naive scheme
exchanging $k$ samples achieves the same $\Omega(1/k)$ rate but with a
suboptimal prefactor. Our protocol achieving optimal performance is one-way
(non-interactive). We also prove the $\Omega(1/k)$ bound even when $\rho$ is
restricted to any small open sub-interval of $[-1,1]$ (i.e. a local minimax
lower bound). Our proof techniques rely on symmetric strong data-processing
inequalities and various tensorization techniques from information-theoretic
interactive common-randomness extraction. Our results also imply an $\Omega(n)$
lower bound on the information complexity of the Gap-Hamming problem, for which
we show a direct information-theoretic proof.
Automated Segmentation of CT Scans for Normal Pressure Hydrocephalus
Authors: Angela Zhang, Po-Yu Kao, Ronald Sahyouni, Ashutosh Shelat, Jefferson Chen, B. S. Manjunath
Date: 2019-01-25 00:00:00
Tasks: Computed Tomography (CT), General Classification

Normal Pressure Hydrocephalus (NPH) is one of the few reversible forms of dementia, Due to their low cost and versatility, Computed Tomography (CT) scans have long been used as an aid to help diagnose intracerebral anomalies such as NPH. However, no well-defined and effective protocol currently exists for the analysis of CT scan-based ventricular, cerebral mass and subarachnoid space volumes in the setting of NPH. The Evan's ratio, an approximation of the ratio of ventricle to brain volume using only one 2D slice of the scan, has been proposed but is not robust. Instead of manually measuring a 2-dimensional proxy for the ratio of ventricle volume to brain volume, this study proposes an automated method of calculating the brain volumes for better recognition of NPH from a radiological standpoint. The method first aligns the subject CT volume to a common space through an affine transformation, then uses a random forest classifier to mask relevant tissue types. A 3D morphological segmentation method is used to partition the brain volume, which in turn is used to train machine learning methods to classify the subjects into non-NPH vs. NPH based on volumetric information. The proposed algorithm has increased sensitivity compared to the Evan's ratio thresholding method.