pwc-archive/papers-with-abstracts huggingface dataset:
Making Sense of Hidden Layer Information in Deep Networks by Learning Hierarchical Targets
Authors: Abhinav Tushar
Date: 2015-05-03 00:00:00
Tasks: General Classification, text-classification, Text Classification

This paper proposes an architecture for deep neural networks with hidden
layer branches that learn targets of lower hierarchy than final layer targets.
The branches provide a channel for enforcing useful information in hidden layer
which helps in attaining better accuracy, both for the final layer and hidden
layers. The shared layers modify their weights using the gradients of all cost
functions higher than the branching layer. This model provides a flexible
inference system with many levels of targets which is modular and can be used
efficiently in situations requiring different levels of results according to
complexity. This paper applies the idea to a text classification task on 20
Newsgroups data set with two level of hierarchical targets and a comparison is
made with training without the use of hidden layer branches.
Aspect Level Sentiment Classification with Deep Memory Network
Authors: Duyu Tang, Bing Qin, Ting Liu
Date: 2016-05-28 00:00:00
Proceeding: EMNLP 2016 11
Tasks: Aspect-Based Sentiment Analysis (ABSA), CPU, General Classification, Sentiment Classification

We introduce a deep memory network for aspect level sentiment classification.
Unlike feature-based SVM and sequential neural models such as LSTM, this
approach explicitly captures the importance of each context word when inferring
the sentiment polarity of an aspect. Such importance degree and text
representation are calculated with multiple computational layers, each of which
is a neural attention model over an external memory. Experiments on laptop and
restaurant datasets demonstrate that our approach performs comparable to
state-of-art feature based SVM system, and substantially better than LSTM and
attention-based LSTM architectures. On both datasets we show that multiple
computational layers could improve the performance. Moreover, our approach is
also fast. The deep memory network with 9 layers is 15 times faster than LSTM
with a CPU implementation.
A Character-level Convolutional Neural Network for Distinguishing Similar Languages and Dialects
Authors: Yonatan Belinkov, James Glass
Date: 2016-09-24 00:00:00
Proceeding: WS 2016 12
Tasks: Dialect Identification

Discriminating between closely-related language varieties is considered a
challenging and important task. This paper describes our submission to the DSL
2016 shared-task, which included two sub-tasks: one on discriminating similar
languages and one on identifying Arabic dialects. We developed a
character-level neural network for this task. Given a sequence of characters,
our model embeds each character in vector space, runs the sequence through
multiple convolutions with different filter widths, and pools the convolutional
representations to obtain a hidden vector representation of the text that is
used for predicting the language or dialect. We primarily focused on the Arabic
dialect identification task and obtained an F1 score of 0.4834, ranking 6th out
of 18 participants. We also analyze errors made by our system on the Arabic
data in some detail, and point to challenges such an approach is faced with.
Cultural Shift or Linguistic Drift? Comparing Two Computational Measures of Semantic Change
Authors: William L. Hamilton, Jure Leskovec, Dan Jurafsky
Date: 2016-06-09 00:00:00
Proceeding: EMNLP 2016 11
Tasks: Cultural Vocal Bursts Intensity Prediction

Words shift in meaning for many reasons, including cultural factors like new
technologies and regular linguistic processes like subjectification.
Understanding the evolution of language and culture requires disentangling
these underlying causes. Here we show how two different distributional measures
can be used to detect two different types of semantic change. The first
measure, which has been used in many previous works, analyzes global shifts in
a word's distributional semantics, it is sensitive to changes due to regular
processes of linguistic drift, such as the semantic generalization of promise
("I promise." -> "It promised to be exciting."). The second measure, which we
develop here, focuses on local changes to a word's nearest semantic neighbors;
it is more sensitive to cultural shifts, such as the change in the meaning of
cell ("prison cell" -> "cell phone"). Comparing measurements made by these two
methods allows researchers to determine whether changes are more cultural or
linguistic in nature, a distinction that is essential for work in the digital
humanities and historical linguistics.
Word Ordering Without Syntax
Authors: Allen Schmaltz, Alexander M. Rush, Stuart M. Shieber
Date: 2016-04-28 00:00:00
Proceeding: EMNLP 2016 11
Tasks: Language Modeling, Language Modelling, Sentence

Recent work on word ordering has argued that syntactic structure is
important, or even required, for effectively recovering the order of a
sentence. We find that, in fact, an n-gram language model with a simple
heuristic gives strong results on this task. Furthermore, we show that a long
short-term memory (LSTM) language model is even more effective at recovering
order, with our basic model outperforming a state-of-the-art syntactic model by
11.5 BLEU points. Additional data and larger beams yield further gains, at the
expense of training and search time.
Inducing Domain-Specific Sentiment Lexicons from Unlabeled Corpora
Authors: William L. Hamilton, Kevin Clark, Jure Leskovec, Dan Jurafsky
Date: 2016-06-09 00:00:00
Proceeding: EMNLP 2016 11
Tasks: Word Embeddings

A word's sentiment depends on the domain in which it is used. Computational
social science research thus requires sentiment lexicons that are specific to
the domains being studied. We combine domain-specific word embeddings with a
label propagation framework to induce accurate domain-specific sentiment
lexicons using small sets of seed words, achieving state-of-the-art performance
competitive with approaches that rely on hand-curated resources. Using our
framework we perform two large-scale empirical studies to quantify the extent
to which sentiment varies across time and between communities. We induce and
release historical sentiment lexicons for 150 years of English and
community-specific sentiment lexicons for 250 online communities from the
social media forum Reddit. The historical lexicons show that more than 5% of
sentiment-bearing (non-neutral) English words completely switched polarity
during the last 150 years, and the community-specific lexicons highlight how
sentiment varies drastically between different communities.
Distilling an Ensemble of Greedy Dependency Parsers into One MST Parser
Authors: Adhiguna Kuncoro, Miguel Ballesteros, Lingpeng Kong, Chris Dyer, Noah A. Smith
Date: 2016-09-24 00:00:00
Proceeding: EMNLP 2016 11
Tasks: Dependency Parsing

We introduce two first-order graph-based dependency parsers achieving a new
state of the art. The first is a consensus parser built from an ensemble of
independently trained greedy LSTM transition-based parsers with different
random initializations. We cast this approach as minimum Bayes risk decoding
(under the Hamming cost) and argue that weaker consensus within the ensemble is
a useful signal of difficulty or ambiguity. The second parser is a
"distillation" of the ensemble into a single model. We train the distillation
parser using a structured hinge loss objective with a novel cost that
incorporates ensemble uncertainty estimates for each possible attachment,
thereby avoiding the intractable cross-entropy computations required by
applying standard distillation objectives to problems with structured outputs.
The first-order distillation parser matches or surpasses the state of the art
on English, Chinese, and German.
Informative Planning and Online Learning with Sparse Gaussian Processes
Authors: Kai-Chieh Ma, Lantao Liu, Gaurav S. Sukhatme
Date: 2016-09-24 00:00:00
Tasks: Gaussian Processes

A big challenge in environmental monitoring is the spatiotemporal variation
of the phenomena to be observed. To enable persistent sensing and estimation in
such a setting, it is beneficial to have a time-varying underlying
environmental model. Here we present a planning and learning method that
enables an autonomous marine vehicle to perform persistent ocean monitoring
tasks by learning and refining an environmental model. To alleviate the
computational bottleneck caused by large-scale data accumulated, we propose a
framework that iterates between a planning component aimed at collecting the
most information-rich data, and a sparse Gaussian Process learning component
where the environmental model and hyperparameters are learned online by taking
advantage of only a subset of data that provides the greatest contribution. Our
simulations with ground-truth ocean data shows that the proposed method is both
accurate and efficient.
Multimodal Compact Bilinear Pooling for Visual Question Answering and Visual Grounding
Authors: Akira Fukui, Dong Huk Park, Daylen Yang, Anna Rohrbach, Trevor Darrell, Marcus Rohrbach
Date: 2016-06-06 00:00:00
Proceeding: EMNLP 2016 11
Tasks: Phrase Grounding, Visual Grounding, Visual Question Answering, Visual Question Answering (VQA)

Modeling textual or visual information with vector representations trained
from large language or visual datasets has been successfully explored in recent
years. However, tasks such as visual question answering require combining these
vector representations with each other. Approaches to multimodal pooling
include element-wise product or sum, as well as concatenation of the visual and
textual representations. We hypothesize that these methods are not as
expressive as an outer product of the visual and textual vectors. As the outer
product is typically infeasible due to its high dimensionality, we instead
propose utilizing Multimodal Compact Bilinear pooling (MCB) to efficiently and
expressively combine multimodal features. We extensively evaluate MCB on the
visual question answering and grounding tasks. We consistently show the benefit
of MCB over ablations without MCB. For visual question answering, we present an
architecture which uses MCB twice, once for predicting attention over spatial
features and again to combine the attended representation with the question
representation. This model outperforms the state-of-the-art on the Visual7W
dataset and the VQA challenge.
Global Neural CCG Parsing with Optimality Guarantees
Authors: Kenton Lee, Mike Lewis, Luke Zettlemoyer
Date: 2016-07-05 00:00:00
Proceeding: EMNLP 2016 11
Tasks: Sentence

We introduce the first global recursive neural parsing model with optimality
guarantees during decoding. To support global features, we give up dynamic
programs and instead search directly in the space of all possible subtrees.
Although this space is exponentially large in the sentence length, we show it
is possible to learn an efficient A* parser. We augment existing parsing
models, which have informative bounds on the outside score, with a global model
that has loose bounds but only needs to model non-local phenomena. The global
model is trained with a new objective that encourages the parser to explore a
tiny fraction of the search space. The approach is applied to CCG parsing,
improving state-of-the-art accuracy by 0.4 F1. The parser finds the optimal
parse for 99.9% of held-out sentences, exploring on average only 190 subtrees.
Morphological Priors for Probabilistic Neural Word Embeddings
Authors: Parminder Bhatia, Robert Guthrie, Jacob Eisenstein
Date: 2016-08-03 00:00:00
Proceeding: EMNLP 2016 11
Tasks: Part-Of-Speech Tagging, Word Embeddings, Word Similarity

Word embeddings allow natural language processing systems to share
statistical information across related words. These embeddings are typically
based on distributional statistics, making it difficult for them to generalize
to rare or unseen words. We propose to improve word embeddings by incorporating
morphological information, capturing shared sub-word features. Unlike previous
work that constructs word embeddings directly from morphemes, we combine
morphological and distributional information in a unified probabilistic
framework, in which the word embedding is a latent variable. The morphological
information provides a prior distribution on the latent word embeddings, which
in turn condition a likelihood function over an observed corpus. This approach
yields improvements on intrinsic word similarity evaluations, and also in the
downstream task of part-of-speech tagging.
Training Deep Networks for Facial Expression Recognition with Crowd-Sourced Label Distribution
Authors: Emad Barsoum, Cha Zhang, Cristian Canton Ferrer, Zhengyou Zhang
Date: 2016-08-03 00:00:00
Tasks: Facial Expression Recognition, Facial Expression Recognition (FER), Multi-Label Learning

Crowd sourcing has become a widely adopted scheme to collect ground truth
labels. However, it is a well-known problem that these labels can be very
noisy. In this paper, we demonstrate how to learn a deep convolutional neural
network (DCNN) from noisy labels, using facial expression recognition as an
example. More specifically, we have 10 taggers to label each input image, and
compare four different approaches to utilizing the multiple labels: majority
voting, multi-label learning, probabilistic label drawing, and cross-entropy
loss. We show that the traditional majority voting scheme does not perform as
well as the last two approaches that fully leverage the label distribution. An
enhanced FER+ data set with multiple labels for each face image will also be
shared with the research community.
Distributed Cooperative Decision-Making in Multiarmed Bandits: Frequentist and Bayesian Algorithms
Authors: Peter Landgren, Vaibhav Srivastava, Naomi Ehrich Leonard
Date: 2016-06-02 00:00:00
Tasks: Decision Making

We study distributed cooperative decision-making under the explore-exploit tradeoff in the multiarmed bandit (MAB) problem. We extend the state-of-the-art frequentist and Bayesian algorithms for single-agent MAB problems to cooperative distributed algorithms for multi-agent MAB problems in which agents communicate according to a fixed network graph. We rely on a running consensus algorithm for each agent's estimation of mean rewards from its own rewards and the estimated rewards of its neighbors. We prove the performance of these algorithms and show that they asymptotically recover the performance of a centralized agent. Further, we rigorously characterize the influence of the communication graph structure on the decision-making performance of the group.
Semantic Parsing to Probabilistic Programs for Situated Question Answering
Authors: Jayant Krishnamurthy, Oyvind Tafjord, Aniruddha Kembhavi
Date: 2016-06-22 00:00:00
Proceeding: EMNLP 2016 11
Tasks: Question Answering, Semantic Parsing

Situated question answering is the problem of answering questions about an
environment such as an image or diagram. This problem requires jointly
interpreting a question and an environment using background knowledge to select
the correct answer. We present Parsing to Probabilistic Programs (P3), a novel
situated question answering model that can use background knowledge and global
features of the question/environment interpretation while retaining efficient
approximate inference. Our key insight is to treat semantic parses as
probabilistic programs that execute nondeterministically and whose possible
executions represent environmental uncertainty. We evaluate our approach on a
new, publicly-released data set of 5000 science diagram questions,
outperforming several competitive classical and neural baselines.
Derivative Delay Embedding: Online Modeling of Streaming Time Series
Authors: Zhifei Zhang, Yang song, Wei Wang, Hairong Qi
Date: 2016-09-24 00:00:00
Tasks: General Classification, Time Series, Time Series Analysis

The staggering amount of streaming time series coming from the real world
calls for more efficient and effective online modeling solution. For time
series modeling, most existing works make some unrealistic assumptions such as
the input data is of fixed length or well aligned, which requires extra effort
on segmentation or normalization of the raw streaming data. Although some
literature claim their approaches to be invariant to data length and
misalignment, they are too time-consuming to model a streaming time series in
an online manner. We propose a novel and more practical online modeling and
classification scheme, DDE-MGM, which does not make any assumptions on the time
series while maintaining high efficiency and state-of-the-art performance. The
derivative delay embedding (DDE) is developed to incrementally transform time
series to the embedding space, where the intrinsic characteristics of data is
preserved as recursive patterns regardless of the stream length and
misalignment. Then, a non-parametric Markov geographic model (MGM) is proposed
to both model and classify the pattern in an online manner. Experimental
results demonstrate the effectiveness and superior classification accuracy of
the proposed DDE-MGM in an online setting as compared to the state-of-the-art.
A Tutorial on Distributed (Non-Bayesian) Learning: Problem, Algorithms and Results
Authors: Angelia Nedić, Alex Olshevsky, César A. Uribe
Date: 2016-09-23 00:00:00

We overview some results on distributed learning with focus on a family of
recently proposed algorithms known as non-Bayesian social learning. We consider
different approaches to the distributed learning problem and its algorithmic
solutions for the case of finitely many hypotheses. The original centralized
problem is discussed at first, and then followed by a generalization to the
distributed setting. The results on convergence and convergence rate are
presented for both asymptotic and finite time regimes. Various extensions are
discussed such as those dealing with directed time-varying networks, Nesterov's
acceleration technique and a continuum sets of hypothesis.
Fast Learning of Clusters and Topics via Sparse Posteriors
Authors: Michael C. Hughes, Erik B. Sudderth
Date: 2016-09-23 00:00:00
Tasks: Articles, Topic Models

Mixture models and topic models generate each observation from a single
cluster, but standard variational posteriors for each observation assign
positive probability to all possible clusters. This requires dense storage and
runtime costs that scale with the total number of clusters, even though
typically only a few clusters have significant posterior mass for any data
point. We propose a constrained family of sparse variational distributions that
allow at most $L$ non-zero entries, where the tunable threshold $L$ trades off
speed for accuracy. Previous sparse approximations have used hard assignments
($L=1$), but we find that moderate values of $L>1$ provide superior
performance. Our approach easily integrates with stochastic or incremental
optimization algorithms to scale to millions of examples. Experiments training
mixture models of image patches and topic models for news articles show that
our approach produces better-quality models in far less time than baseline
methods.
Combinatorial Energy Learning for Image Segmentation
Authors: Jeremy Maitin-Shepard, Viren Jain, Michal Januszewski, Peter Li, Pieter Abbeel
Date: 2015-06-13 00:00:00
Proceeding: NeurIPS 2016 12
Tasks: Image Segmentation, Segmentation, Semantic Segmentation

We introduce a new machine learning approach for image segmentation that uses
a neural network to model the conditional energy of a segmentation given an
image. Our approach, combinatorial energy learning for image segmentation
(CELIS) places a particular emphasis on modeling the inherent combinatorial
nature of dense image segmentation problems. We propose efficient algorithms
for learning deep neural networks to model the energy function, and for local
optimization of this energy in the space of supervoxel agglomerations. We
extensively evaluate our method on a publicly available 3-D microscopy dataset
with 25 billion voxels of ground truth data. On an 11 billion voxel test set,
we find that our method improves volumetric reconstruction accuracy by more
than 20% as compared to two state-of-the-art baseline methods: graph-based
segmentation of the output of a 3-D convolutional neural network trained to
predict boundaries, as well as a random forest classifier trained to
agglomerate supervoxels that were generated by a 3-D convolutional neural
network.
Lifted Rule Injection for Relation Embeddings
Authors: Thomas Demeester, Tim Rocktäschel, Sebastian Riedel
Date: 2016-06-27 00:00:00
Proceeding: EMNLP 2016 11
Tasks: Knowledge Base Construction, Relation, Representation Learning

Methods based on representation learning currently hold the state-of-the-art
in many natural language processing and knowledge base inference tasks. Yet, a
major challenge is how to efficiently incorporate commonsense knowledge into
such models. A recent approach regularizes relation and entity representations
by propositionalization of first-order logic rules. However,
propositionalization does not scale beyond domains with only few entities and
rules. In this paper we present a highly efficient method for incorporating
implication rules into distributed representations for automated knowledge base
construction. We map entity-tuple embeddings into an approximately Boolean
space and encourage a partial ordering over relation embeddings based on
implication rules mined from WordNet. Surprisingly, we find that the strong
restriction of the entity-tuple embedding space does not hurt the
expressiveness of the model and even acts as a regularizer that improves
generalization. By incorporating few commonsense rules, we achieve an increase
of 2 percentage points mean average precision over a matrix factorization
baseline, while observing a negligible increase in runtime.
Relations such as Hypernymy: Identifying and Exploiting Hearst Patterns in Distributional Vectors for Lexical Entailment
Authors: Stephen Roller, Katrin Erk
Date: 2016-05-18 00:00:00
Proceeding: EMNLP 2016 11
Tasks: Lexical Entailment

We consider the task of predicting lexical entailment using distributional
vectors. We perform a novel qualitative analysis of one existing model which
was previously shown to only measure the prototypicality of word pairs. We find
that the model strongly learns to identify hypernyms using Hearst patterns,
which are well known to be predictive of lexical relations. We present a novel
model which exploits this behavior as a method of feature extraction in an
iterative procedure similar to Principal Component Analysis. Our model combines
the extracted features with the strengths of other proposed models in the
literature, and matches or outperforms prior work on multiple data sets.
Speaker Recognition for Children's Speech
Authors: Saeid Safavi, Maryam Najafian, Abualsoud Hanani, Martin J Russell, Peter Jancovic, Michael J. Carey
Date: 2016-09-23 00:00:00
Tasks: Speaker Recognition

This paper presents results on Speaker Recognition (SR) for children's
speech, using the OGI Kids corpus and GMM-UBM and GMM-SVM SR systems. Regions
of the spectrum containing important speaker information for children are
identified by conducting SR experiments over 21 frequency bands. As for adults,
the spectrum can be split into four regions, with the first (containing primary
vocal tract resonance information) and third (corresponding to high frequency
speech sounds) being most useful for SR. However, the frequencies at which
these regions occur are from 11% to 38% higher for children. It is also noted
that subband SR rates are lower for younger children. Finally results are
presented of SR experiments to identify a child in a class (30 children,
similar age) and school (288 children, varying ages). Class performance depends
on age, with accuracy varying from 90% for young children to 99% for older
children. The identification rate achieved for a child in a school is 81%.