pwc-archive/papers-with-abstracts huggingface dataset:
Neuramanteau: A Neural Network Ensemble Model for Lexical Blends
Authors: Kollol Das, Shaona Ghosh
Date: 2017-11-01 00:00:00
Proceeding: IJCNLP 2017 11

The problem of blend formation in generative linguistics is interesting in the context of neologism, their quick adoption in modern life and the creative generative process guiding their formation. Blend quality depends on multitude of factors with high degrees of uncertainty. In this work, we investigate if the modern neural network models can sufficiently capture and recognize the creative blend composition process. We propose recurrent neural network sequence-to-sequence models, that are evaluated on multiple blend datasets available in the literature. We propose an ensemble neural and hybrid model that outperforms most of the baselines and heuristic models upon evaluation on test data.
Multi-Agent Dual Learning
Authors: Yiren Wang, Yingce Xia, Tianyu He, Fei Tian, Tao Qin, ChengXiang Zhai, Tie-Yan Liu
Date: 2019-05-01 00:00:00
Proceeding: ICLR 2019 5
Tasks: Machine Translation, Translation

Dual learning has attracted much attention in machine learning, computer vision and natural language processing communities. The core idea of dual learning is to leverage the duality between the primal task (mapping from domain X to domain Y) and dual task (mapping from domain Y to X) to boost the performances of both tasks. Existing dual learning framework forms a system with two agents (one primal model and one dual model) to utilize such duality. In this paper, we extend this framework by introducing multiple primal and dual models, and propose the multi-agent dual learning framework. Experiments on neural machine translation and image translation tasks demonstrate the effectiveness of the new framework. 
In particular, we set a new record on IWSLT 2014 German-to-English translation with a 35.44 BLEU score, achieve a 31.03 BLEU score on WMT 2014 English-to-German translation with over 2.6 BLEU improvement over the strong Transformer baseline, and set a new record of 49.61 BLEU score on the recent WMT 2018 English-to-German translation.
RNNs implicitly implement tensor-product representations
Authors: R. Thomas McCoy, Tal Linzen, Ewan Dunbar, Paul Smolensky
Date: 2019-05-01 00:00:00
Proceeding: ICLR 2019 5
Tasks: Representation Learning, Sentence

Recurrent neural networks (RNNs) can learn continuous vector representations of symbolic structures such as sequences and sentences; these representations often exhibit linear regularities (analogies).  Such regularities motivate our hypothesis that RNNs that show such regularities implicitly compile symbolic structures into tensor product representations (TPRs; Smolensky, 1990), which additively combine tensor products of vectors representing roles (e.g.,  sequence positions) and vectors representing fillers (e.g., particular words). To test this hypothesis, we introduce Tensor Product Decomposition Networks (TPDNs), which use TPRs to approximate existing vector representations. We demonstrate using synthetic data that TPDNs can successfully approximate linear and tree-based RNN autoencoder representations, suggesting that these representations exhibit interpretable compositional structure; we explore the settings that lead RNNs to induce such structure-sensitive representations.  By contrast, further TPDN experiments show that the representations of four models trained to encode naturally-occurring sentences can be largely approximated with a bag of words, with only marginal improvements from more sophisticated structures. We conclude that TPDNs provide a powerful method for interpreting vector representations, and that standard RNNs can induce compositional sequence representations that are remarkably well approximated byTPRs; at the same time, existing training tasks for sentence representation learning may not be sufficient for inducing robust structural representations
An Empirical study of Binary Neural Networks' Optimisation
Authors: Milad Alizadeh, Javier Fernández-Marqués, Nicholas D. Lane, Yarin Gal
Date: 2019-05-01 00:00:00
Proceeding: ICLR 2019 5

Binary neural networks using the Straight-Through-Estimator (STE) have been shown to achieve state-of-the-art results, but their training process is not well-founded. This is due to the discrepancy between the evaluated function in the forward path, and the weight updates in the back-propagation, updates which do not correspond to gradients of the forward path. Efficient convergence and accuracy of binary models often rely on careful fine-tuning and various ad-hoc techniques. In this work, we empirically identify and study the effectiveness of the various ad-hoc techniques commonly used in the literature, providing best-practices for efficient training of binary models. We show that adapting learning rates using second moment methods is crucial for the successful use of the STE, and that other optimisers can easily get stuck in local minima. We also find that many of the commonly employed tricks are only effective towards the end of the training, with these methods making early stages of the training considerably slower. Our analysis disambiguates necessary from unnecessary ad-hoc techniques for training of binary neural networks, paving the way for future development of solid theoretical foundations for these. Our newly-found insights further lead to new procedures which make training of existing binary neural networks notably faster.
Improving deep forest by confidence screening
Authors: Ming Pang, Kai-Ming Ting, Peng Zhao, Zhi-Hua Zhou
Date: 2019-11-17 00:00:00
Proceeding: the 18th IEEE International Conference on Data Mining 2019 11
Tasks: Representation Learning

Most studies about deep learning are based on neural network models, where many layers of parameterized nonlinear differentiable modules are trained by back propagation. Recently, it has been shown that deep learning can also be realized by non-differentiable modules without back propagation training called deep forest. The developed representation learning process is based on a cascade of cascades of decision tree forests, where the high memory requirement and the high time cost inhibit the training of large models. In this paper, we propose a simple yet effective approach to improve the efficiency of deep forest. The key idea is to pass the instances with high confidence directly to the final stage rather than passing through all the levels. We also provide a theoretical analysis suggesting a means to vary the model complexity from low to high as the level increases in the cascade, which further reduces the memory requirement and time cost. Our experiments show that the proposed approach achieves highly competitive predictive performance with significantly reduced time cost and memory requirement by up to one order of magnitude.
Evaluating Preprocessing Strategies for Time Series Prediction Using Deep Learning Architectures
Authors: Sajitha Naduvil-Vadukootu, Rafal A. Angryk, Pete Riley
Date: 2017-01-01 00:00:00
Proceeding: Proceedings of the Thirtieth International Florida Artificial Intelligence Research Society Conference 520 2017 1
Tasks: Prediction, Time Series, Time Series Analysis, Time Series Prediction

We propose a novel approach to combine state-of-the-art time
series data processing methods, such as symbolic aggregate
approximation (SAX), with very recently developed deep
neural network architectures, such as deep recurrent neural
networks (DRNN), for time series data modeling and prediction. Time series data appear extensively in various scientific
domains and industrial applications, yet the challenges in accurate modeling and prediction from such data remain open.
Deep recurrent neural networks (DRNN) have been proposed
as promising approaches to sequence prediction. We extend
this research to the new challenge of the time series prediction
space, building a system that effectively combines recurrent
neural networks (RNN) with time series specific preprocessing techniques. Our experiments show comparisons of model
performance with various data preprocessing techniques. We
demonstrate that preprocessed inputs can steer us towards
simpler (and therefore more computationally efficient) architectures of neural networks (when compared to original inputs).
Feed-forward Propagation in Probabilistic Neural Networks with Categorical and Max Layers
Authors: Alexander Shekhovtsov, Boris Flach
Date: 2019-05-01 00:00:00
Proceeding: ICLR 2019 5

Probabilistic Neural Networks deal with various sources of stochasticity: input noise, dropout, stochastic neurons, parameter uncertainties modeled as random variables, etc.
In this paper we revisit a feed-forward propagation approach that allows one to estimate for each neuron its mean and variance w.r.t. all mentioned sources of stochasticity. In contrast, standard NNs propagate only point estimates, discarding the uncertainty.
Methods propagating also the variance have been proposed by several authors in different context. The view presented here attempts to clarify the assumptions and derivation behind such methods, relate them to classical NNs and broaden their scope of applicability.
The main technical contributions are new approximations for the distributions of argmax and max-related transforms, which allow for fully analytic uncertainty propagation in networks with softmax and max-pooling layers as well as leaky ReLU activations.
We evaluate the accuracy of the approximation and suggest a simple calibration. Applying the method to networks with dropout allows for faster training and gives improved test likelihoods without the need of sampling.
Human-level Protein Localization with Convolutional Neural Networks
Authors: Elisabeth Rumetshofer, Markus Hofmarcher, Clemens Röhrl, Sepp Hochreiter, Günter Klambauer
Date: 2019-05-01 00:00:00
Proceeding: ICLR 2019 5

Localizing a specific protein in a human cell is essential for understanding cellular functions and biological processes of underlying diseases. A promising, low-cost,and time-efficient biotechnology for localizing proteins is high-throughput fluorescence microscopy imaging (HTI). This imaging technique stains the protein of interest in a cell with fluorescent antibodies and subsequently takes a microscopic image.  Together with images of other stained proteins or cell organelles and the annotation by the Human Protein Atlas project, these images provide a rich source of information on the protein location which can be utilized by computational methods.  It is yet unclear how precise such methods are and whether they can compete with human experts.   We here focus on deep learning image analysis methods and, in particular, on Convolutional Neural Networks (CNNs)since they showed overwhelming success across different imaging tasks. We pro-pose a novel CNN architecture “GapNet-PL” that has been designed to tackle the characteristics of HTI data and uses global averages of filters at different abstraction levels.   We present the largest comparison of CNN architectures including GapNet-PL for protein localization in HTI images of human cells.  GapNet-PL outperforms all other competing methods and reaches close to perfect localization in all 13 tasks with an average AUC of 98% and F1 score of 78%.  On a separate test set the performance of GapNet-PL was compared with three human experts and 25 scholars. GapNet-PL achieved an accuracy of 91%, significantly (p-value 1.1e−6) outperforming the best human expert with an accuracy of 72%.
Explicit Document Modeling through Weighted Multiple-Instance Learning
Authors: Nikolaos Pappas, Andrei Popescu-Belis
Date: 2017-02-17 00:00:00
Proceeding: Journal of Artificial Intelligence Research (JAIR) 2017 2
Tasks: Multiple Instance Learning, Sentence, Sentiment Analysis, Weakly-supervised Learning

Representing documents is a crucial component in many NLP tasks, for instance predicting aspect ratings in reviews.  Previous methods for this task treat documents globally and do not acknowledge that target categories are often assigned by their authors with generally no indication of the specific sentences that motivate them.  To address this issue, we adopt a weakly supervised learning model, which jointly learns to focus on relevant parts of a document according to the context along with a classifier for the target categories. Originated from the weighted multiple-instance regression (MIR)  framework, the modellearns decomposable document vectors for each individual category and thus overcomesthe representational bottleneck in previous methods due to a fixed-length document vector.  During prediction, the estimated relevance or saliency weights explicitly capture the contribution of each sentence to the predicted rating, thus offering an explanation of the rating.  Our model achieves state-of-the-art performance on multi-aspect sentiment analysis, improving over several baselines.  Moreover, the predicted saliency weights are close to human estimates obtained by crowdsourcing and increase the performance of lexical and topical features for review segmentation and summarization.
Improved Neural Machine Translation using Side Information
Authors: Cong Duy Vu Hoang, Gholamreza Haffari, Trevor Cohn
Date: 2018-12-01 00:00:00
Proceeding: ALTA 2018 12
Tasks: Machine Translation, NMT, Translation

In this work, we investigate whether side information is helpful in neural machine translation (NMT). We study various kinds of side information, including topical information, personal trait, then propose different ways of incorporating them into the existing NMT models. Our experimental results show the benefits of side information in improving the NMT models.
Text-dependent Forensic Voice Comparison: Likelihood Ratio Estimation with the Hidden Markov Model (HMM) and Gaussian Mixture Model
Authors: Satoru Tsuge, Shunichi Ishihara
Date: 2018-12-01 00:00:00
Proceeding: ALTA 2018 12
Tasks: model

Among the more typical forensic voice comparison (FVC) approaches, the acoustic-phonetic statistical approach is suitable for text-dependent FVC, but it does not fully exploit available time-varying information of speech in its modelling. The automatic approach, on the other hand, essentially deals with text-independent cases, which means temporal information is not explicitly incorporated in the modelling. Text-dependent likelihood ratio (LR)-based FVC studies, in particular those that adopt the automatic approach, are few. This preliminary LR-based FVC study compares two statistical models, the Hidden Markov Model (HMM) and the Gaussian Mixture Model (GMM), for the calculation of forensic LRs using the same speech data. FVC experiments were carried out using different lengths of Japanese short words under a forensically realistic, but challenging condition: only two speech tokens for model training and LR estimation. Log-likelihood-ratio cost (Cllr) was used as the assessment metric. The study demonstrates that the HMM system constantly outperforms the GMM system in terms of average Cllr values. However, words longer than three mora are needed if the advantage of the HMM is to become evident. With a seven-mora word, for example, the HMM outperformed the GMM by a Cllr value of 0.073.
Development of Natural Language Processing Tools for Cook Islands M\=aori
Authors: Rol Solano, o Coto, Sally Akevai Nicholas, Samantha Wray
Date: 2018-12-01 00:00:00
Proceeding: ALTA 2018 12
Tasks: Machine Translation, Part-Of-Speech Tagging, POS, POS Tagging, Speech Recognition, Speech-to-Text

This paper presents three ongoing projects for NLP in Cook Islands Maori: Untrained Forced Alignment (approx. 9{\%} error when detecting the center of words), speech-to-text (37{\%} WER in the best trained models) and POS tagging (92{\%} accuracy for the best performing model). Included as part of these projects are new resources filling in a gap in Australasian languages, including gold standard POS-tagged written corpora, transcribed speech corpora, time-aligned corpora down to the level of phonemes. These are part of efforts to accelerate the documentation of Cook Islands Maori and to increase its vitality amongst its users.
Unsupervised Mining of Analogical Frames by Constraint Satisfaction
Authors: Lance De Vine, Shlomo Geva, Peter Bruza
Date: 2018-12-01 00:00:00
Proceeding: ALTA 2018 12
Tasks: Language Acquisition, Machine Translation, Word Embeddings

It has been demonstrated that vector-based representations of words trained on large text corpora encode linguistic regularities that may be exploited via the use of vector space arithmetic. This capability has been extensively explored and is generally measured via tasks which involve the automated completion of linguistic proportional analogies. The question remains, however, as to what extent it is possible to induce relations from word embeddings in a principled and systematic way, without the provision of exemplars or seed terms. In this paper we propose an extensible and efficient framework for inducing relations via the use of constraint satisfaction. The method is efficient, unsupervised and can be customized in various ways. We provide both quantitative and qualitative analysis of the results.
Specifying Conceptual Models Using Restricted Natural Language
Authors: Bayzid Ashik Hossain, Rolf Schwitter
Date: 2018-12-01 00:00:00
Proceeding: ALTA 2018 12
Tasks: General Knowledge

The key activity to design an information system is conceptual modelling which brings out and describes the general knowledge that is required to build a system. In this paper we propose a novel approach to conceptual modelling where the domain experts will be able to specify and construct a model using a restricted form of natural language. A restricted natural language is a subset of a natural language that has well-defined computational properties and therefore can be translated unambiguously into a formal notation. We will argue that a restricted natural language is suitable for writing precise and consistent specifications that lead to executable conceptual models. Using a restricted natural language will allow the domain experts to describe a scenario in the terminology of the application domain without the need to formally encode this scenario. The resulting textual specification can then be automatically translated into the language of the desired conceptual modelling framework.
Extracting structured data from invoices
Authors: Xavier Holt, Andrew Chisholm
Date: 2018-12-01 00:00:00
Proceeding: ALTA 2018 12
Tasks: BIG-bench Machine Learning, Optical Character Recognition (OCR)

Business documents encode a wealth of information in a format tailored to human consumption {--} i.e. aesthetically disbursed natural language text, graphics and tables. We address the task of extracting key fields (e.g. the amount due on an invoice) from a wide-variety of potentially unseen document formats. In contrast to traditional template driven extraction systems, we introduce a content-driven machine-learning approach which is both robust to noise and generalises to unseen document formats. In a comparison of our approach with alternative invoice extraction systems, we observe an absolute accuracy gain of 20{\textbackslash}{\%} across compared fields, and a 25{\textbackslash}{\%}{--}94{\textbackslash}{\%} reduction in extraction latency.
Cluster Labeling by Word Embeddings and WordNet's Hypernymy
Authors: Hanieh Poostchi, Massimo Piccardi
Date: 2018-12-01 00:00:00
Proceeding: ALTA 2018 12
Tasks: Clustering, Descriptive, Document Classification, Learning Word Embeddings, Word Embeddings

Cluster labeling is the assignment of representative labels to clusters obtained from the organization of a document collection. Once assigned, the labels can play an important role in applications such as navigation, search and document classification. However, finding appropriately descriptive labels is still a challenging task. In this paper, we propose various approaches for assigning labels to word clusters by leveraging word embeddings and the synonymity and hypernymy relations in the WordNet lexical ontology. Experiments carried out using the WebAP document dataset have shown that one of the approaches stand out in the comparison and is capable of selecting labels that are reasonably aligned with those chosen by a pool of four human annotators.
A Comparative Study of Embedding Models in Predicting the Compositionality of Multiword Expressions
Authors: N, Navnita akumar, Bahar Salehi, Timothy Baldwin
Date: 2018-12-01 00:00:00
Proceeding: ALTA 2018 12
Tasks: Information Retrieval, Representation Learning, Word Embeddings

In this paper, we perform a comparative evaluation of off-the-shelf embedding models over the task of compositionality prediction of multiword expressions(``MWEs''). Our experimental results suggest that character- and document-level models capture knowledge of MWE compositionality and are effective in modelling varying levels of compositionality, with the advantage over word-level models that they do not require token-level identification of MWEs in the training corpus.
Towards Efficient Machine Translation Evaluation by Modelling Annotators
Authors: Nitika Mathur, Timothy Baldwin, Trevor Cohn
Date: 2018-12-01 00:00:00
Proceeding: ALTA 2018 12
Tasks: Machine Translation, Translation

Accurate evaluation of translation has long been a difficult, yet important problem. Current evaluations use direct assessment (DA), based on crowd sourcing judgements from a large pool of workers, along with quality control checks, and a robust method for combining redundant judgements. In this paper we show that the quality control mechanism is overly conservative, which increases the time and expense of the evaluation. We propose a model that does not rely on a pre-processing step to filter workers and takes into account varying annotator reliabilities. Our model effectively weights each worker's scores based on the inferred precision of the worker, and is much more reliable than the mean of either the raw scores or the standardised scores. We also show that DA does not deliver on the promise of longitudinal evaluation, and propose redesigning the structure of the annotation tasks that can solve this problem.
Overview of the 2018 ALTA Shared Task: Classifying Patent Applications
Authors: Diego Moll{\'a}, Dilesha Seneviratne
Date: 2018-12-01 00:00:00
Proceeding: ALTA 2018 12
Tasks: General Classification

We present an overview of the 2018 ALTA shared task. This is the 9th of the series of shared tasks organised by ALTA since 2010. The task was to classify Australian patent classifications following the sections defined by the International Patient Classification (IPC), using data made available by IP Australia. We introduce the task, describe the data and present the results of the participating teams. Some of the participating teams outperformed state of the art.
Universal Language Model Fine-tuning for Patent Classification
Authors: Jason Hepburn
Date: 2018-12-01 00:00:00
Proceeding: ALTA 2018 12
Tasks: Classification, General Classification, Language Modeling, Language Modelling, model, Patent classification, Text Classification

This paper describes the methods used for the 2018 ALTA Shared Task. The task this year was to automatically classify Australian patents into their main International Patent Classification section. Our final submission used a Support Vector Machine (SVM) and Universal Language Model with Fine-tuning (ULMFiT). Our system achieved the best results in the student category.