pwc-archive/papers-with-abstracts huggingface dataset:
Learning Sparse Low-Precision Neural Networks With Learnable Regularization
Authors: Yoojin Choi, Mostafa El-Khamy, Jungwon Lee
Date: 2018-09-01 00:00:00
Tasks: Image Super-Resolution, L2 Regularization, Quantization, Super-Resolution

We consider learning deep neural networks (DNNs) that consist of low-precision weights and activations for efficient inference of fixed-point operations. In training low-precision networks, gradient descent in the backward pass is performed with high-precision weights while quantized low-precision weights and activations are used in the forward pass to calculate the loss function for training. Thus, the gradient descent becomes suboptimal, and accuracy loss follows. In order to reduce the mismatch in the forward and backward passes, we utilize mean squared quantization error (MSQE) regularization. In particular, we propose using a learnable regularization coefficient with the MSQE regularizer to reinforce the convergence of high-precision weights to their quantized values. We also investigate how partial L2 regularization can be employed for weight pruning in a similar manner. Finally, combining weight pruning, quantization, and entropy coding, we establish a low-precision DNN compression pipeline. In our experiments, the proposed method yields low-precision MobileNet and ShuffleNet models on ImageNet classification with the state-of-the-art compression ratios of 7.13 and 6.79, respectively. Moreover, we examine our method for image super resolution networks to produce 8-bit low-precision models at negligible performance loss.
Extractive Adversarial Networks: High-Recall Explanations for Identifying Personal Attacks in Social Media Posts
Authors: Samuel Carton, Qiaozhu Mei, Paul Resnick
Date: 2018-09-01 00:00:00
Proceeding: EMNLP 2018 10
Tasks: Hard Attention

We introduce an adversarial method for producing high-recall explanations of
neural text classifier decisions. Building on an existing architecture for
extractive explanations via hard attention, we add an adversarial layer which
scans the residual of the attention for remaining predictive signal. Motivated
by the important domain of detecting personal attacks in social media comments,
we additionally demonstrate the importance of manually setting a semantically
appropriate `default' behavior for the model by explicitly manipulating its
bias term. We develop a validation set of human-annotated personal attacks to
evaluate the impact of these changes.
Hierarchical CVAE for Fine-Grained Hate Speech Classification
Authors: Jing Qian, Mai ElSherief, Elizabeth Belding, William Yang Wang
Date: 2018-08-31 00:00:00
Proceeding: EMNLP 2018 10
Tasks: Binary Classification, Classification, General Classification, Hate Speech Detection

Existing work on automated hate speech detection typically focuses on binary
classification or on differentiating among a small set of categories. In this
paper, we propose a novel method on a fine-grained hate speech classification
task, which focuses on differentiating among 40 hate groups of 13 different
hate group categories. We first explore the Conditional Variational Autoencoder
(CVAE) as a discriminative model and then extend it to a hierarchical
architecture to utilize the additional hate category information for more
accurate prediction. Experimentally, we show that incorporating the hate
category information for training can significantly improve the classification
performance and our proposed model outperforms commonly-used discriminative
models.
A Simplified Approach to Deep Learning for Image Segmentation
Authors: Ishtar Nyawira, Kristi Bushman
Date: 2018-08-31 00:00:00
Tasks: Deep Learning, General Classification, Image Segmentation, Management, Semantic Segmentation

Leaping into the rapidly developing world of deep learning is an exciting and
sometimes confusing adventure. All of the advice and tutorials available can be
hard to organize and work through, especially when training specific models on
specific datasets, different from those originally used to train the network.
In this short guide, we aim to walk the reader through the techniques that we
have used to successfully train two deep neural networks for pixel-wise
classification, including some data management and augmentation approaches for
working with image data that may be insufficiently annotated or relatively
homogenous.
Understanding Neural Pathways in Zebrafish through Deep Learning and High Resolution Electron Microscope Data
Authors: Ishtar Nyawira, Kristi Bushman, Iris Qian, Annie Zhang
Date: 2018-08-31 00:00:00
Tasks: Deep Learning, Image Segmentation, Segmentation, Semantic Segmentation

The tracing of neural pathways through large volumes of image data is an
incredibly tedious and time-consuming process that significantly encumbers
progress in neuroscience. We are exploring deep learning's potential to
automate segmentation of high-resolution scanning electron microscope (SEM)
image data to remove that barrier. We have started with neural pathway tracing
through 5.1GB of whole-brain serial-section slices from larval zebrafish
collected by the Center for Brain Science at Harvard University. This kind of
manual image segmentation requires years of careful work to properly trace the
neural pathways in an organism as small as a zebrafish larva (approximately 5mm
in total body length). In automating this process, we would vastly improve
productivity, leading to faster data analysis and breakthroughs in
understanding the complexity of the brain. We will build upon prior attempts to
employ deep learning for automatic image segmentation extending methods for
unconventional deep learning data.
Predicting protein inter-residue contacts using composite likelihood maximization and deep learning
Authors: Haicang Zhang, Qi Zhang, Fusong Ju, Jianwei Zhu, Yujuan Gao, Ziwei Xie, Minghua Deng, Shiwei Sun, Wei-Mou Zheng, Dongbo Bu
Date: 2018-08-31 00:00:00
Tasks: Prediction

Accurate prediction of inter-residue contacts of a protein is important to
calcu- lating its tertiary structure. Analysis of co-evolutionary events among
residues has been proved effective to inferring inter-residue contacts. The
Markov ran- dom field (MRF) technique, although being widely used for contact
prediction, suffers from the following dilemma: the actual likelihood function
of MRF is accurate but time-consuming to calculate, in contrast, approximations
to the actual likelihood, say pseudo-likelihood, are efficient to calculate but
inaccu- rate. Thus, how to achieve both accuracy and efficiency simultaneously
remains a challenge. In this study, we present such an approach (called clmDCA)
for contact prediction. Unlike plmDCA using pseudo-likelihood, i.e., the
product of conditional probability of individual residues, our approach uses
composite- likelihood, i.e., the product of conditional probability of all
residue pairs. Com- posite likelihood has been theoretically proved as a better
approximation to the actual likelihood function than pseudo-likelihood.
Meanwhile, composite likelihood is still efficient to maximize, thus ensuring
the efficiency of clmDCA. We present comprehensive experiments on popular
benchmark datasets, includ- ing PSICOV dataset and CASP-11 dataset, to show
that: i) clmDCA alone outperforms the existing MRF-based approaches in
prediction accuracy. ii) When equipped with deep learning technique for
refinement, the prediction ac- curacy of clmDCA was further significantly
improved, suggesting the suitability of clmDCA for subsequent refinement
procedure. We further present successful application of the predicted contacts
to accurately build tertiary structures for proteins in the PSICOV dataset.
  Accessibility: The software clmDCA and a server are publicly accessible
through http://protein.ict.ac.cn/clmDCA/.
NEU: A Meta-Algorithm for Universal UAP-Invariant Feature Representation
Authors: Anastasis Kratsios, Cody Hyndman
Date: 2018-08-31 00:00:00
Tasks: Dimensionality Reduction, Meta-Learning

Effective feature representation is key to the predictive performance of any algorithm. This paper introduces a meta-procedure, called Non-Euclidean Upgrading (NEU), which learns feature maps that are expressive enough to embed the universal approximation property (UAP) into most model classes while only outputting feature maps that preserve any model class's UAP. We show that NEU can learn any feature map with these two properties if that feature map is asymptotically deformable into the identity. We also find that the feature-representations learned by NEU are always submanifolds of the feature space. NEU's properties are derived from a new deep neural model that is universal amongst all orientation-preserving homeomorphisms on the input space. We derive qualitative and quantitative approximation guarantees for this architecture. We quantify the number of parameters required for this new architecture to memorize any set of input-output pairs while simultaneously fixing every point of the input space lying outside some compact set, and we quantify the size of this set as a function of our model's depth. Moreover, we show that no deep feed-forward network with commonly used activation function has all these properties. NEU's performance is evaluated against competing machine learning methods on various regression and dimension reduction tasks both with financial and simulated data.
3D Segmentation with Exponential Logarithmic Loss for Highly Unbalanced Object Sizes
Authors: Ken C. L. Wong, Mehdi Moradi, Hui Tang, Tanveer Syeda-Mahmood
Date: 2018-08-31 00:00:00
Tasks: Brain Segmentation, Image Segmentation, Medical Image Segmentation, Segmentation, Semantic Segmentation

With the introduction of fully convolutional neural networks, deep learning
has raised the benchmark for medical image segmentation on both speed and
accuracy, and different networks have been proposed for 2D and 3D segmentation
with promising results. Nevertheless, most networks only handle relatively
small numbers of labels (<10), and there are very limited works on handling
highly unbalanced object sizes especially in 3D segmentation. In this paper, we
propose a network architecture and the corresponding loss function which
improve segmentation of very small structures. By combining skip connections
and deep supervision with respect to the computational feasibility of 3D
segmentation, we propose a fast converging and computationally efficient
network architecture for accurate segmentation. Furthermore, inspired by the
concept of focal loss, we propose an exponential logarithmic loss which
balances the labels not only by their relative sizes but also by their
segmentation difficulties. We achieve an average Dice coefficient of 82% on
brain segmentation with 20 labels, with the ratio of the smallest to largest
object sizes as 0.14%. Less than 100 epochs are required to reach such
accuracy, and segmenting a 128x128x128 volume only takes around 0.4 s.
RxNN: A Framework for Evaluating Deep Neural Networks on Resistive Crossbars
Authors: Shubham Jain, Abhronil Sengupta, Kaushik Roy, Anand Raghunathan
Date: 2018-08-31 00:00:00

Resistive crossbars designed with non-volatile memory devices have emerged as promising building blocks for Deep Neural Network (DNN) hardware, due to their ability to compactly and efficiently realize vector-matrix multiplication (VMM), the dominant computational kernel in DNNs. However, a key challenge with resistive crossbars is that they suffer from a range of device and circuit level non-idealities such as interconnect parasitics, peripheral circuits, sneak paths, and process variations. These non-idealities can lead to errors in VMMs, eventually degrading the DNN's accuracy. It is therefore critical to study the impact of crossbar non-idealities on the accuracy of large-scale DNNs. However, this is challenging because existing device and circuit models are too slow to use in application-level evaluations. We present RxNN, a fast and accurate simulation framework to evaluate large-scale DNNs on resistive crossbar systems. RxNN splits and maps the computations involved in each DNN layer into crossbar operations, and evaluates them using a Fast Crossbar Model (FCM) that accurately captures the errors arising due to crossbar non-idealities while being four-to-five orders of magnitude faster than circuit simulation. FCM models a crossbar-based VMM operation using three stages - non-linear models for the input and output peripheral circuits (DACs and ADCs), and an equivalent non-ideal conductance matrix for the core crossbar array. We implement RxNN by extending the Caffe machine learning framework and use it to evaluate a suite of six large-scale DNNs developed for the ImageNet Challenge. Our experiments reveal that resistive crossbar non-idealities can lead to significant accuracy degradations (9.6%-32%) for these large-scale DNNs. To the best of our knowledge, this work is the first quantitative evaluation of the accuracy of large-scale DNNs on resistive crossbar based hardware.
Nightmare at test time: How punctuation prevents parsers from generalizing
Authors: Anders Søgaard, Miryam de Lhoneux, Isabelle Augenstein
Date: 2018-08-31 00:00:00
Proceeding: WS 2018 11

Punctuation is a strong indicator of syntactic structure, and parsers trained
on text with punctuation often rely heavily on this signal. Punctuation is a
diversion, however, since human language processing does not rely on
punctuation to the same extent, and in informal texts, we therefore often leave
out punctuation. We also use punctuation ungrammatically for emphatic or
creative purposes, or simply by mistake. We show that (a) dependency parsers
are sensitive to both absence of punctuation and to alternative uses; (b)
neural parsers tend to be more sensitive than vintage parsers; (c) training
neural parsers without punctuation outperforms all out-of-the-box parsers
across all scenarios where punctuation departs from standard punctuation. Our
main experiments are on synthetically corrupted data to study the effect of
punctuation in isolation and avoid potential confounds, but we also show
effects on out-of-domain data.
When to Finish? Optimal Beam Search for Neural Text Generation (modulo beam size)
Authors: Liang Huang, Kai Zhao, Mingbo Ma
Date: 2018-08-31 00:00:00
Proceeding: EMNLP 2017 9
Tasks: Image Captioning, Machine Translation, Text Generation, Translation

In neural text generation such as neural machine translation, summarization,
and image captioning, beam search is widely used to improve the output text
quality. However, in the neural generation setting, hypotheses can finish in
different steps, which makes it difficult to decide when to end beam search to
ensure optimality. We propose a provably optimal beam search algorithm that
will always return the optimal-score complete hypothesis (modulo beam size),
and finish as soon as the optimality is established (finishing no later than
the baseline). To counter neural generation's tendency for shorter hypotheses,
we also introduce a bounded length reward mechanism which allows a modified
version of our beam search algorithm to remain optimal. Experiments on neural
machine translation demonstrate that our principled beam search algorithm leads
to improvement in BLEU score over previously proposed alternatives.
Denoising Neural Machine Translation Training with Trusted Data and Online Data Selection
Authors: Wei Wang, Taro Watanabe, Macduff Hughes, Tetsuji Nakagawa, Ciprian Chelba
Date: 2018-08-31 00:00:00
Proceeding: WS 2018 10
Tasks: Denoising, Machine Translation, NMT, Translation

Measuring domain relevance of data and identifying or selecting well-fit
domain data for machine translation (MT) is a well-studied topic, but denoising
is not yet. Denoising is concerned with a different type of data quality and
tries to reduce the negative impact of data noise on MT training, in
particular, neural MT (NMT) training. This paper generalizes methods for
measuring and selecting data for domain MT and applies them to denoising NMT
training. The proposed approach uses trusted data and a denoising curriculum
realized by online data selection. Intrinsic and extrinsic evaluations of the
approach show its significant effectiveness for NMT to train on data with
severe noise.
Indicatements that character language models learn English morpho-syntactic units and regularities
Authors: Yova Kementchedjhieva, Adam Lopez
Date: 2018-08-31 00:00:00
Tasks: Language Modeling, Language Modelling

Character language models have access to surface morphological patterns, but
it is not clear whether or how they learn abstract morphological regularities.
We instrument a character language model with several probes, finding that it
can develop a specific unit to identify word boundaries and, by extension,
morpheme boundaries, which allows it to capture linguistic properties and
regularities of these units. Our language model proves surprisingly good at
identifying the selectional restrictions of English derivational morphemes, a
task that requires both morphological and syntactic awareness. Thus we conclude
that, when morphemes overlap extensively with the words of a language, a
character language model can perform morphological abstraction.
MULDEF: Multi-model-based Defense Against Adversarial Examples for Neural Networks
Authors: Siwakorn Srisakaokul, Yuhao Zhang, Zexuan Zhong, Wei Yang, Tao Xie, Bo Li
Date: 2018-08-31 00:00:00
Tasks: Diversity

Despite being popularly used in many applications, neural network models have been found to be vulnerable to adversarial examples, i.e., carefully crafted examples aiming to mislead machine learning models. Adversarial examples can pose potential risks on safety and security critical applications. However, existing defense approaches are still vulnerable to attacks, especially in a white-box attack scenario. To address this issue, we propose a new defense approach, named MulDef, based on robustness diversity. Our approach consists of (1) a general defense framework based on multiple models and (2) a technique for generating these multiple models to achieve high defense capability. In particular, given a target model, our framework includes multiple models (constructed from the target model) to form a model family. The model family is designed to achieve robustness diversity (i.e., an adversarial example successfully attacking one model cannot succeed in attacking other models in the family). At runtime, a model is randomly selected from the family to be applied on each input example. Our general framework can inspire rich future research to construct a desirable model family achieving higher robustness diversity. Our evaluation results show that MulDef (with only up to 5 models in the family) can substantially improve the target model's accuracy on adversarial examples by 22-74% in a white-box attack scenario, while maintaining similar accuracy on legitimate examples.
Generalizing Procrustes Analysis for Better Bilingual Dictionary Induction
Authors: Yova Kementchedjhieva, Sebastian Ruder, Ryan Cotterell, Anders Søgaard
Date: 2018-08-31 00:00:00
Proceeding: CONLL 2018 10

Most recent approaches to bilingual dictionary induction find a linear alignment between the word vector spaces of two languages. We show that projecting the two languages onto a third, latent space, rather than directly onto each other, while equivalent in terms of expressivity, makes it easier to learn approximate alignments. Our modified approach also allows for supporting languages to be included in the alignment process, to obtain an even better performance in low resource settings.
Aesthetic Features for Personalized Photo Recommendation
Authors: Yu Qing Zhou, Ga Wu, Scott Sanner, Putra Manggala
Date: 2018-08-31 00:00:00
Tasks: Collaborative Filtering, Image Retrieval, Style Transfer

Many photography websites such as Flickr, 500px, Unsplash, and Adobe Behance
are used by amateur and professional photography enthusiasts. Unlike
content-based image search, such users of photography websites are not just
looking for photos with certain content, but more generally for photos with a
certain photographic "aesthetic". In this context, we explore personalized
photo recommendation and propose two aesthetic feature extraction methods based
on (i) color space and (ii) deep style transfer embeddings. Using a dataset
from 500px, we evaluate how these features can be best leveraged by
collaborative filtering methods and show that (ii) provides a significant boost
in photo recommendation performance.
Your Actions or Your Associates? Predicting Certification and Dropout in MOOCs with Behavioral and Social Features
Authors: Niki Gitinabard, Farzaneh Khoshnevisan, Collin F. Lynch, Elle Yuan Wang
Date: 2018-08-31 00:00:00
Tasks: feature selection, Survival Analysis

The high level of attrition and low rate of certification in Massive Open
Online Courses (MOOCs) has prompted a great deal of research. Prior researchers
have focused on predicting dropout based upon behavioral features such as
student confusion, click-stream patterns, and social interactions. However, few
studies have focused on combining student logs with forum data.
  In this work, we use data from two different offerings of the same MOOC. We
conduct a survival analysis to identify likely dropouts. We then examine two
classes of features, social and behavioral, and apply a combination of modeling
and feature-selection methods to identify the most relevant features to predict
both dropout and certification. We examine the utility of three different model
types and we consider the impact of different definitions of dropout on the
predictors. Finally, we assess the reliability of the models over time by
evaluating whether or not models from week 1 can predict dropout in week 2, and
so on. The outcomes of this study will help instructors identify students
likely to fail or dropout as soon as the first two weeks and provide them with
more support.
What do RNN Language Models Learn about Filler-Gap Dependencies?
Authors: Ethan Wilcox, Roger Levy, Takashi Morita, Richard Futrell
Date: 2018-08-31 00:00:00

RNN language models have achieved state-of-the-art perplexity results and
have proven useful in a suite of NLP tasks, but it is as yet unclear what
syntactic generalizations they learn. Here we investigate whether
state-of-the-art RNN language models represent long-distance filler-gap
dependencies and constraints on them. Examining RNN behavior on experimentally
controlled sentences designed to expose filler-gap dependencies, we show that
RNNs can represent the relationship in multiple syntactic positions and over
large spans of text. Furthermore, we show that RNNs learn a subset of the known
restrictions on filler-gap dependencies, known as island constraints: RNNs show
evidence for wh-islands, adjunct islands, and complex NP islands. These studies
demonstrates that state-of-the-art RNN models are able to learn and generalize
about empty syntactic positions.
A Supervised Learning Approach For Heading Detection
Authors: Sahib Singh Budhiraja, Vijay Mago
Date: 2018-08-31 00:00:00
Tasks: Sensitivity, Specificity

As the Portable Document Format (PDF) file format increases in popularity,
research in analysing its structure for text extraction and analysis is
necessary. Detecting headings can be a crucial component of classifying and
extracting meaningful data. This research involves training a supervised
learning model to detect headings with features carefully selected through
recursive feature elimination. The best performing classifier had an accuracy
of 96.95%, sensitivity of 0.986 and a specificity of 0.953. This research into
heading detection contributes to the field of PDF based text extraction and can
be applied to the automation of large scale PDF text analysis in a variety of
professional and policy based contexts.