pwc-archive/papers-with-abstracts huggingface dataset:
Constrained Output Embeddings for End-to-End Code-Switching Speech Recognition with Only Monolingual Data
Authors: Yerbolat Khassanov, Hai-Hua Xu, Van Tung Pham, Zhiping Zeng, Eng Siong Chng, Chongjia Ni, Bin Ma
Date: 2019-04-08 00:00:00
Tasks: Automatic Speech Recognition, Automatic Speech Recognition (ASR), speech-recognition, Speech Recognition

The lack of code-switch training data is one of the major concerns in the development of end-to-end code-switching automatic speech recognition (ASR) models. In this work, we propose a method to train an improved end-to-end code-switching ASR using only monolingual data. Our method encourages the distributions of output token embeddings of monolingual languages to be similar, and hence, promotes the ASR model to easily code-switch between languages. Specifically, we propose to use Jensen-Shannon divergence and cosine distance based constraints. The former will enforce output embeddings of monolingual languages to possess similar distributions, while the later simply brings the centroids of two distributions to be close to each other. Experimental results demonstrate high effectiveness of the proposed method, yielding up to 4.5% absolute mixed error rate improvement on Mandarin-English code-switching ASR task.
Enriching Rare Word Representations in Neural Language Models by Embedding Matrix Augmentation
Authors: Yerbolat Khassanov, Zhiping Zeng, Van Tung Pham, Hai-Hua Xu, Eng Siong Chng
Date: 2019-04-08 00:00:00
Tasks: speech-recognition, Speech Recognition

The neural language models (NLM) achieve strong generalization capability by learning the dense representation of words and using them to estimate probability distribution function. However, learning the representation of rare words is a challenging problem causing the NLM to produce unreliable probability estimates. To address this problem, we propose a method to enrich representations of rare words in pre-trained NLM and consequently improve its probability estimation performance. The proposed method augments the word embedding matrices of pre-trained NLM while keeping other parameters unchanged. Specifically, our method updates the embedding vectors of rare words using embedding vectors of other semantically and syntactically similar words. To evaluate the proposed method, we enrich the rare street names in the pre-trained NLM and use it to rescore 100-best hypotheses output from the Singapore English speech recognition system. The enriched NLM reduces the word error rate by 6% relative and improves the recognition accuracy of the rare words by 16% absolute as compared to the baseline NLM.
FoveaBox: Beyond Anchor-based Object Detector
Authors: Tao Kong, Fuchun Sun, Huaping Liu, Yuning Jiang, Lei LI, Jianbo Shi
Date: 2019-04-08 00:00:00
Tasks: Object, object-detection, Object Detection

We present FoveaBox, an accurate, flexible, and completely anchor-free framework for object detection. While almost all state-of-the-art object detectors utilize predefined anchors to enumerate possible locations, scales and aspect ratios for the search of the objects, their performance and generalization ability are also limited to the design of anchors. Instead, FoveaBox directly learns the object existing possibility and the bounding box coordinates without anchor reference. This is achieved by: (a) predicting category-sensitive semantic maps for the object existing possibility, and (b) producing category-agnostic bounding box for each position that potentially contains an object. The scales of target boxes are naturally associated with feature pyramid representations. In FoveaBox, an instance is assigned to adjacent feature levels to make the model more accurate.We demonstrate its effectiveness on standard benchmarks and report extensive experimental analysis. Without bells and whistles, FoveaBox achieves state-of-the-art single model performance on the standard COCO and Pascal VOC object detection benchmark. More importantly, FoveaBox avoids all computation and hyper-parameters related to anchor boxes, which are often sensitive to the final detection performance. We believe the simple and effective approach will serve as a solid baseline and help ease future research for object detection. The code has been made publicly available at https://github.com/taokong/FoveaBox .
Minimum Enclosing Ball Revisited: Stability and Sub-linear Time Algorithms
Authors: Hu Ding
Date: 2019-04-08 00:00:00

In this paper, we revisit the Minimum Enclosing Ball (MEB) problem and its robust version, MEB with outliers, in Euclidean space $\mathbb{R}^d$. Though the problem has been extensively studied before, most of the existing algorithms need at least linear time (in the number of input points $n$ and the dimensionality $d$) to achieve a $(1+\epsilon)$-approximation. Motivated by some recent developments on beyond worst-case analysis, we introduce the notion of stability for MEB (with outliers), which is natural and easy to understand. Roughly speaking, an instance of MEB is stable, if the radius of the resulting ball cannot be significantly reduced by removing a small fraction of the input points. Under the stability assumption, we present two sampling algorithms for computing approximate MEB with sample complexities independent of the number of input points $n$. In particular, the second algorithm has the sample complexity even independent of the dimensionality $d$. Further, we extend the idea to achieve a sub-linear time approximation algorithm for the MEB with outliers problem. Note that most existing sub-linear time algorithms for the problems of MEB and MEB with outliers usually result in bi-criteria approximations, where the "bi-criteria" means that the solution has to allow the approximations on the radius and the number of covered points. Differently, all the algorithms proposed in this paper yield single-criterion approximations (with respect to radius). We expect that our proposed notion of stability and techniques will be applicable to design sub-linear time algorithms for other optimization problems.
Bayesian Non-Parametric Multi-Source Modelling Based Determined Blind Source Separation
Authors: Chaitanya Narisetty, Tatsuya Komatsu, Reishi Kondo
Date: 2019-04-08 00:00:00
Tasks: blind source separation

This paper proposes a determined blind source separation method using
Bayesian non-parametric modelling of sources. Conventionally source signals are
separated from a given set of mixture signals by modelling them using
non-negative matrix factorization (NMF). However in NMF, a latent variable
signifying model complexity must be appropriately specified to avoid
over-fitting or under-fitting. As real-world sources can be of varying and
unknown complexities, we propose a Bayesian non-parametric framework which is
invariant to such latent variables. We show that our proposed method adapts to
different source complexities, while conventional methods require parameter
tuning for optimal separation.
Resource Constrained Neural Network Architecture Search: Will a Submodularity Assumption Help?
Authors: Yunyang Xiong, Ronak Mehta, Vikas Singh
Date: 2019-04-08 00:00:00
Proceeding: ICCV 2019 10
Tasks: Neural Architecture Search, Reinforcement Learning

The design of neural network architectures is frequently either based on human expertise using trial/error and empirical feedback or tackled via large scale reinforcement learning strategies performed over distinct discrete architecture choices. In the latter case, the optimization is often non-differentiable and also not very amenable to derivative-free optimization methods. Most methods in use today require sizable computational resources. And if we want networks that additionally satisfy resource constraints, the above challenges are exacerbated because the search must now balance accuracy with certain budget constraints on resources. We formulate this problem as the optimization of a set function -- we find that the empirical behavior of this set function often (but not always) satisfies marginal gain and monotonicity principles -- properties central to the idea of submodularity. Based on this observation, we adapt algorithms within discrete optimization to obtain heuristic schemes for neural network architecture search, where we have resource constraints on the architecture. This simple scheme when applied on CIFAR-100 and ImageNet, identifies resource-constrained architectures with quantifiably better performance than current state-of-the-art models designed for mobile devices. Specifically, we find high-performing architectures with fewer parameters and computations by a search method that is much faster.
Cluster Developing 1-Bit Matrix Completion
Authors: Chengkun Zhang. Junbin Gao, Stephen Lu
Date: 2019-04-07 00:00:00
Tasks: Clustering, Matrix Completion, Recommendation Systems

Matrix completion has a long-time history of usage as the core technique of
recommender systems. In particular, 1-bit matrix completion, which considers
the prediction as a ``Recommended'' or ``Not Recommended'' question, has proved
its significance and validity in the field. However, while customers and
products aggregate into interacted clusters, state-of-the-art model-based 1-bit
recommender systems do not take the consideration of grouping bias. To tackle
the gap, this paper introduced Group-Specific 1-bit Matrix Completion (GS1MC)
by first-time consolidating group-specific effects into 1-bit recommender
systems under the low-rank latent variable framework. Additionally, to empower
GS1MC even when grouping information is unobtainable, Cluster Developing Matrix
Completion (CDMC) was proposed by integrating the sparse subspace clustering
technique into GS1MC. Namely, CDMC allows clustering users/items and to
leverage their group effects into matrix completion at the same time.
Experiments on synthetic and real-world data show that GS1MC outperforms the
current 1-bit matrix completion methods. Meanwhile, it is compelling that CDMC
can successfully capture items' genre features only based on sparse binary
user-item interactive data. Notably, GS1MC provides a new insight to
incorporate and evaluate the efficacy of clustering methods while CDMC can be
served as a new tool to explore unrevealed social behavior or market
phenomenon.
ANTNets: Mobile Convolutional Neural Networks for Resource Efficient Image Classification
Authors: Yunyang Xiong, Hyunwoo J. Kim, Varsha Hedau
Date: 2019-04-07 00:00:00
Tasks: Classification, General Classification, image-classification, Image Classification

Deep convolutional neural networks have achieved remarkable success in computer vision. However, deep neural networks require large computing resources to achieve high performance. Although depthwise separable convolution can be an efficient module to approximate a standard convolution, it often leads to reduced representational power of networks. In this paper, under budget constraints such as computational cost (MAdds) and the parameter count, we propose a novel basic architectural block, ANTBlock. It boosts the representational power by modeling, in a high dimensional space, interdependency of channels between a depthwise convolution layer and a projection layer in the ANTBlocks. Our experiments show that ANTNet built by a sequence of ANTBlocks, consistently outperforms state-of-the-art low-cost mobile convolutional neural networks across multiple datasets. On CIFAR100, our model achieves 75.7% top-1 accuracy, which is 1.5% higher than MobileNetV2 with 8.3% fewer parameters and 19.6% less computational cost. On ImageNet, our model achieves 72.8% top-1 accuracy, which is 0.8% improvement, with 157.7ms (20% faster) on iPhone 5s over MobileNetV2.
Meta-Learning with Differentiable Convex Optimization
Authors: Kwonjoon Lee, Subhransu Maji, Avinash Ravichandran, Stefano Soatto
Date: 2019-04-07 00:00:00
Proceeding: CVPR 2019 6
Tasks: Few-Shot Image Classification, Few-Shot Learning, Meta-Learning

Many meta-learning approaches for few-shot learning rely on simple base
learners such as nearest-neighbor classifiers. However, even in the few-shot
regime, discriminatively trained linear predictors can offer better
generalization. We propose to use these predictors as base learners to learn
representations for few-shot learning and show they offer better tradeoffs
between feature size and performance across a range of few-shot recognition
benchmarks. Our objective is to learn feature embeddings that generalize well
under a linear classification rule for novel categories. To efficiently solve
the objective, we exploit two properties of linear classifiers: implicit
differentiation of the optimality conditions of the convex problem and the dual
formulation of the optimization problem. This allows us to use high-dimensional
embeddings with improved generalization at a modest increase in computational
overhead. Our approach, named MetaOptNet, achieves state-of-the-art performance
on miniImageNet, tieredImageNet, CIFAR-FS, and FC100 few-shot learning
benchmarks. Our code is available at https://github.com/kjunelee/MetaOptNet.
DeepGCNs: Can GCNs Go as Deep as CNNs?
Authors: Guohao Li, Matthias Müller, Ali Thabet, Bernard Ghanem
Date: 2019-04-07 00:00:00
Proceeding: ICCV 2019 10
Tasks: 3D Semantic Segmentation, Graph Classification, Node Classification, Semantic Segmentation

Convolutional Neural Networks (CNNs) achieve impressive performance in a wide variety of fields. Their success benefited from a massive boost when very deep CNN models were able to be reliably trained. Despite their merits, CNNs fail to properly address problems with non-Euclidean data. To overcome this challenge, Graph Convolutional Networks (GCNs) build graphs to represent non-Euclidean data, borrow concepts from CNNs, and apply them in training. GCNs show promising results, but they are usually limited to very shallow models due to the vanishing gradient problem. As a result, most state-of-the-art GCN models are no deeper than 3 or 4 layers. In this work, we present new ways to successfully train very deep GCNs. We do this by borrowing concepts from CNNs, specifically residual/dense connections and dilated convolutions, and adapting them to GCN architectures. Extensive experiments show the positive effect of these deep GCN frameworks. Finally, we use these new concepts to build a very deep 56-layer GCN, and show how it significantly boosts performance (+3.7% mIoU over state-of-the-art) in the task of point cloud semantic segmentation. We believe that the community can greatly benefit from this work, as it opens up many opportunities for advancing GCN-based research.
JumpReLU: A Retrofit Defense Strategy for Adversarial Attacks
Authors: N. Benjamin Erichson, Zhewei Yao, Michael W. Mahoney
Date: 2019-04-07 00:00:00

It has been demonstrated that very simple attacks can fool
highly-sophisticated neural network architectures. In particular, so-called
adversarial examples, constructed from perturbations of input data that are
small or imperceptible to humans but lead to different predictions, may lead to
an enormous risk in certain critical applications. In light of this, there has
been a great deal of work on developing adversarial training strategies to
improve model robustness. These training strategies are very expensive, in both
human and computational time. To complement these approaches, we propose a very
simple and inexpensive strategy which can be used to ``retrofit'' a
previously-trained network to improve its resilience to adversarial attacks.
More concretely, we propose a new activation function---the JumpReLU---which,
when used in place of a ReLU in an already-trained model, leads to a trade-off
between predictive accuracy and robustness. This trade-off is controlled by the
jump size, a hyper-parameter which can be tuned during the validation stage.
Our empirical results demonstrate that this increases model robustness,
protecting against adversarial attacks with substantially increased levels of
perturbations. This is accomplished simply by retrofitting existing networks
with our JumpReLU activation function, without the need for retraining the
model. Additionally, we demonstrate that adversarially trained (robust) models
can greatly benefit from retrofitting.
Unsupervised Recurrent Neural Network Grammars
Authors: Yoon Kim, Alexander M. Rush, Lei Yu, Adhiguna Kuncoro, Chris Dyer, Gábor Melis
Date: 2019-04-07 00:00:00
Proceeding: NAACL 2019 6
Tasks: Constituency Grammar Induction, Language Modeling, Language Modelling, Sentence, Variational Inference

Recurrent neural network grammars (RNNG) are generative models of language which jointly model syntax and surface structure by incrementally generating a syntax tree and sentence in a top-down, left-to-right order. Supervised RNNGs achieve strong language modeling and parsing performance, but require an annotated corpus of parse trees. In this work, we experiment with unsupervised learning of RNNGs. Since directly marginalizing over the space of latent trees is intractable, we instead apply amortized variational inference. To maximize the evidence lower bound, we develop an inference network parameterized as a neural CRF constituency parser. On language modeling, unsupervised RNNGs perform as well their supervised counterparts on benchmarks in English and Chinese. On constituency grammar induction, they are competitive with recent neural language models that induce tree structures from words through attention mechanisms.
Information Bottleneck and its Applications in Deep Learning
Authors: Hassan Hafez-Kolahi, Shohreh Kasaei
Date: 2019-04-07 00:00:00
Tasks: Deep Learning

Information Theory (IT) has been used in Machine Learning (ML) from early
days of this field. In the last decade, advances in Deep Neural Networks (DNNs)
have led to surprising improvements in many applications of ML. The result has
been a paradigm shift in the community toward revisiting previous ideas and
applications in this new framework. Ideas from IT are no exception. One of the
ideas which is being revisited by many researchers in this new era, is
Information Bottleneck (IB); a formulation of information extraction based on
IT. The IB is promising in both analyzing and improving DNNs. The goal of this
survey is to review the IB concept and demonstrate its applications in deep
learning. The information theoretic nature of IB, makes it also a good
candidate in showing the more general concept of how IT can be used in ML. Two
important concepts are highlighted in this narrative on the subject, i) the
concise and universal view that IT provides on seemingly unrelated methods of
ML, demonstrated by explaining how IB relates to minimal sufficient statistics,
stochastic gradient descent, and variational auto-encoders, and ii) the common
technical mistakes and problems caused by applying ideas from IT, which is
discussed by a careful study of some recent methods suffering from them.
A unifying approach for doubly-robust $\ell_1$ regularized estimation of causal contrasts
Authors: Ezequiel Smucler, Andrea Rotnitzky, James M. Robins
Date: 2019-04-07 00:00:00
Tasks: Causal Inference, Econometrics, regression

We consider inference about a scalar parameter under a non-parametric model based on a one-step estimator computed as a plug in estimator plus the empirical mean of an estimator of the parameter's influence function. We focus on a class of parameters that have influence function which depends on two infinite dimensional nuisance functions and such that the bias of the one-step estimator of the parameter of interest is the expectation of the product of the estimation errors of the two nuisance functions. Our class includes many important treatment effect contrasts of interest in causal inference and econometrics, such as ATE, ATT, an integrated causal contrast with a continuous treatment, and the mean of an outcome missing not at random. We propose estimators of the target parameter that entertain approximately sparse regression models for the nuisance functions allowing for the number of potential confounders to be even larger than the sample size. By employing sample splitting, cross-fitting and $\ell_1$-regularized regression estimators of the nuisance functions based on objective functions whose directional derivatives agree with those of the parameter's influence function, we obtain estimators of the target parameter with two desirable robustness properties: (1) they are rate doubly-robust in that they are root-n consistent and asymptotically normal when both nuisance functions follow approximately sparse models, even if one function has a very non-sparse regression coefficient, so long as the other has a sufficiently sparse regression coefficient, and (2) they are model doubly-robust in that they are root-n consistent and asymptotically normal even if one of the nuisance functions does not follow an approximately sparse model so long as the other nuisance function follows an approximately sparse model with a sufficiently sparse regression coefficient.
Unsupervised Dialog Structure Learning
Authors: Weiyan Shi, Tiancheng Zhao, Zhou Yu
Date: 2019-04-07 00:00:00
Proceeding: NAACL 2019 6
Tasks: Reinforcement Learning

Learning a shared dialog structure from a set of task-oriented dialogs is an important challenge in computational linguistics. The learned dialog structure can shed light on how to analyze human dialogs, and more importantly contribute to the design and evaluation of dialog systems. We propose to extract dialog structures using a modified VRNN model with discrete latent vectors. Different from existing HMM-based models, our model is based on variational-autoencoder (VAE). Such model is able to capture more dynamics in dialogs beyond the surface forms of the language. We find that qualitatively, our method extracts meaningful dialog structure, and quantitatively, outperforms previous models on the ability to predict unseen data. We further evaluate the model's effectiveness in a downstream task, the dialog system building task. Experiments show that, by integrating the learned dialog structure into the reward function design, the model converges faster and to a better outcome in a reinforcement learning setting.
Measuring Human Perception to Improve Handwritten Document Transcription
Authors: Samuel Grieggs, Bingyu Shen, Greta Rauch, Pei Li, Jiaqi Ma, David Chiang, Brian Price, Walter J. Scheirer
Date: 2019-04-07 00:00:00

The subtleties of human perception, as measured by vision scientists through the use of psychophysics, are important clues to the internal workings of visual recognition. For instance, measured reaction time can indicate whether a visual stimulus is easy for a subject to recognize, or whether it is hard. In this paper, we consider how to incorporate psychophysical measurements of visual perception into the loss function of a deep neural network being trained for a recognition task, under the assumption that such information can enforce consistency with human behavior. As a case study to assess the viability of this approach, we look at the problem of handwritten document transcription. While good progress has been made towards automatically transcribing modern handwriting, significant challenges remain in transcribing historical documents. Here we describe a general enhancement strategy, underpinned by the new loss formulation, which can be applied to the training regime of any deep learning-based document transcription system. Through experimentation, reliable performance improvement is demonstrated for the standard IAM and RIMES datasets for three different network architectures. Further, we go on to show feasibility for our approach on a new dataset of digitized Latin manuscripts, originally produced by scribes in the Cloister of St. Gall in the the 9th century.