pwc-archive/papers-with-abstracts huggingface dataset:
A Random Matrix Theory Approach to Damping in Deep Learning
Authors: Diego Granziol, Nicholas Baskerville
Date: 2020-11-15 00:00:00
Tasks: Deep Learning

We conjecture that the inherent difference in generalisation between adaptive and non-adaptive gradient methods in deep learning stems from the increased estimation noise in the flattest directions of the true loss surface. We demonstrate that typical schedules used for adaptive methods (with low numerical stability or damping constants) serve to bias relative movement towards flat directions relative to sharp directions, effectively amplifying the noise-to-signal ratio and harming generalisation. We further demonstrate that the numerical damping constant used in these methods can be decomposed into a learning rate reduction and linear shrinkage of the estimated curvature matrix. We then demonstrate significant generalisation improvements by increasing the shrinkage coefficient, closing the generalisation gap entirely in both logistic regression and several deep neural network experiments. Extending this line further, we develop a novel random matrix theory based damping learner for second order optimiser inspired by linear shrinkage estimation. We experimentally demonstrate our learner to be very insensitive to the initialised value and to allow for extremely fast convergence in conjunction with continued stable training and competitive generalisation.
Automatic selection of clustering algorithms using supervised graph embedding
Authors: Noy Cohen-Shapira, Lior Rokach
Date: 2020-11-16 00:00:00
Tasks: AutoML, Clustering, Graph Embedding, Meta-Learning

The widespread adoption of machine learning (ML) techniques and the extensive expertise required to apply them have led to increased interest in automated ML solutions that reduce the need for human intervention. One of the main challenges in applying ML to previously unseen problems is algorithm selection - the identification of high-performing algorithm(s) for a given dataset, task, and evaluation measure. This study addresses the algorithm selection challenge for data clustering, a fundamental task in data mining that is aimed at grouping similar objects. We present MARCO-GE, a novel meta-learning approach for the automated recommendation of clustering algorithms. MARCO-GE first transforms datasets into graphs and then utilizes a graph convolutional neural network technique to extract their latent representation. Using the embedding representations obtained, MARCO-GE trains a ranking meta-model capable of accurately recommending top-performing algorithms for a new dataset and clustering evaluation measure. Extensive evaluation on 210 datasets, 13 clustering algorithms, and 10 clustering measures demonstrates the effectiveness of our approach and its superiority in terms of predictive and generalization performance over state-of-the-art clustering meta-learning approaches.
Foundations of Bayesian Learning from Synthetic Data
Authors: Harrison Wilde, Jack Jewson, Sebastian Vollmer, Chris Holmes
Date: 2020-11-16 00:00:00
Tasks: Synthetic Data Generation

There is significant growth and interest in the use of synthetic data as an enabler for machine learning in environments where the release of real data is restricted due to privacy or availability constraints. Despite a large number of methods for synthetic data generation, there are comparatively few results on the statistical properties of models learnt on synthetic data, and fewer still for situations where a researcher wishes to augment real data with another party's synthesised data. We use a Bayesian paradigm to characterise the updating of model parameters when learning in these settings, demonstrating that caution should be taken when applying conventional learning algorithms without appropriate consideration of the synthetic data generating process and learning task. Recent results from general Bayesian updating support a novel and robust approach to Bayesian synthetic-learning founded on decision theory that outperforms standard approaches across repeated experiments on supervised learning and inference problems.
Recursive Importance Sketching for Rank Constrained Least Squares: Algorithms and High-order Convergence
Authors: Yuetian Luo, Wen Huang, Xudong Li, Anru R. Zhang
Date: 2020-11-17 00:00:00
Tasks: Retrieval

In this paper, we propose {\it \underline{R}ecursive} {\it \underline{I}mportance} {\it \underline{S}ketching} algorithm for {\it \underline{R}ank} constrained least squares {\it \underline{O}ptimization} (RISRO). The key step of RISRO is recursive importance sketching, a new sketching framework based on deterministically designed recursive projections, which significantly differs from the randomized sketching in the literature \citep{mahoney2011randomized,woodruff2014sketching}. Several existing algorithms in the literature can be reinterpreted under this new sketching framework and RISRO offers clear advantages over them. RISRO is easy to implement and computationally efficient, where the core procedure in each iteration is to solve a dimension-reduced least squares problem. We establish the local quadratic-linear and quadratic rate of convergence for RISRO under some mild conditions. We also discover a deep connection of RISRO to the Riemannian Gauss-Newton algorithm on fixed rank matrices. The effectiveness of RISRO is demonstrated in two applications in machine learning and statistics: low-rank matrix trace regression and phase retrieval. Simulation studies demonstrate the superior numerical performance of RISRO.
Association Rules Enhanced Knowledge Graph Attention Network
Authors: Zhenghao Zhang, Jianbin Huang, Qinglin Tan
Date: 2020-11-14 00:00:00
Tasks: Graph Attention, Graph Embedding, Knowledge Base Completion, Knowledge Graph Completion, Knowledge Graph Embedding, Knowledge Graphs, Link Prediction, Triplet

Most existing knowledge graphs suffer from incompleteness. Embedding knowledge graphs into continuous vector spaces has recently attracted increasing interest in knowledge base completion. However, in most existing embedding methods, only fact triplets are utilized, and logical rules have not been thoroughly studied for the knowledge base completion task. To overcome the problem, we propose an association rules enhanced knowledge graph attention network (AR-KGAT). The AR-KGAT captures both entity and relation features for high-order neighborhoods of any given entity in an end-to-end manner under the graph attention network framework. The major component of AR-KGAT is an encoder of an effective neighborhood aggregator, which addresses the problems by aggregating neighbors with both association-rules-based and graph-based attention weights. Additionally, the proposed model also encapsulates the representations from multi-hop neighbors of nodes to refine their embeddings. The decoder enables AR-KGAT to be translational between entities and relations while keeping the superior link prediction performance. A logic-like inference pattern is utilized as constraints for knowledge graph embedding. Then, the global loss is minimized over both atomic and complex formulas to achieve the embedding task. In this manner, we learn embeddings compatible with triplets and rules, which are certainly more predictive for knowledge acquisition and inference. We conduct extensive experiments on two benchmark datasets: WN18RR and FB15k-237, for two knowledge graph completion tasks: the link prediction and triplet classification to evaluate the proposed AR-KGAT model. The results show that the proposed AR-KGAT model achieves significant and consistent improvements over state-of-the-art methods.
End-to-end spoken language understanding using transformer networks and self-supervised pre-trained features
Authors: Edmilson Morais, Hong-Kwang J. Kuo, Samuel Thomas, Zoltan Tuske, Brian Kingsbury
Date: 2020-11-16 00:00:00
Tasks: Spoken Language Understanding

Transformer networks and self-supervised pre-training have consistently delivered state-of-art results in the field of natural language processing (NLP); however, their merits in the field of spoken language understanding (SLU) still need further investigation. In this paper we introduce a modular End-to-End (E2E) SLU transformer network based architecture which allows the use of self-supervised pre-trained acoustic features, pre-trained model initialization and multi-task training. Several SLU experiments for predicting intent and entity labels/values using the ATIS dataset are performed. These experiments investigate the interaction of pre-trained model initialization and multi-task training with either traditional filterbank or self-supervised pre-trained acoustic features. Results show not only that self-supervised pre-trained acoustic features outperform filterbank features in almost all the experiments, but also that when these features are used in combination with multi-task training, they almost eliminate the necessity of pre-trained model initialization.
A Probabilistic Approach in Historical Linguistics Word Order Change in Infinitival Clauses: from Latin to Old French
Authors: Olga Scrivner
Date: 2020-11-16 00:00:00

This research offers a new interdisciplinary approach to the field of Linguistics by using Computational Linguistics, NLP, Bayesian Statistics and Sociolinguistics methods. This thesis investigates word order change in infinitival clauses from Object-Verb (OV) to Verb-Object (VO) in the history of Latin and Old French. By applying a variationist approach, I examine a synchronic word order variation in each stage of language change, from which I infer the character, periodization and constraints of diachronic variation. I also show that in discourse-configurational languages, such as Latin and Early Old French, it is possible to identify pragmatically neutral contexts by using information structure annotation. I further argue that by mapping pragmatic categories into a syntactic structure, we can detect how word order change unfolds. For this investigation, the data are extracted from annotated corpora spanning several centuries of Latin and Old French and from additional resources created by using computational linguistic methods. The data are then further codified for various pragmatic, semantic, syntactic and sociolinguistic factors. This study also evaluates previous factors proposed to account for word order alternation and change. I show how information structure and syntactic constraints change over time and propose a method that allows researchers to differentiate a stable word order alternation from alternation indicating a change. Finally, I present a three-stage probabilistic model of word order change, which also conforms to traditional language change patterns.
A Two-Phase Approach for Abstractive Podcast Summarization
Authors: Chujie Zheng, Kunpeng Zhang, Harry Jiannan Wang, Ling Fan
Date: 2020-11-16 00:00:00
Tasks: Decoder, Sentence, Sentence Similarity, Vocal Bursts Valence Prediction

Podcast summarization is different from summarization of other data formats, such as news, patents, and scientific papers in that podcasts are often longer, conversational, colloquial, and full of sponsorship and advertising information, which imposes great challenges for existing models. In this paper, we focus on abstractive podcast summarization and propose a two-phase approach: sentence selection and seq2seq learning. Specifically, we first select important sentences from the noisy long podcast transcripts. The selection is based on sentence similarity to the reference to reduce the redundancy and the associated latent topics to preserve semantics. Then the selected sentences are fed into a pre-trained encoder-decoder framework for the summary generation. Our approach achieves promising results regarding both ROUGE-based measures and human evaluations.
Facebook AI's WMT20 News Translation Task Submission
Authors: Peng-Jen Chen, Ann Lee, Changhan Wang, Naman Goyal, Angela Fan, Mary Williamson, Jiatao Gu
Date: 2020-11-16 00:00:00
Tasks: Data Augmentation, Reranking, Translation

This paper describes Facebook AI's submission to WMT20 shared news translation task. We focus on the low resource setting and participate in two language pairs, Tamil <-> English and Inuktitut <-> English, where there are limited out-of-domain bitext and monolingual data. We approach the low resource problem using two main strategies, leveraging all available data and adapting the system to the target news domain. We explore techniques that leverage bitext and monolingual data from all languages, such as self-supervised model pretraining, multilingual models, data augmentation, and reranking. To better adapt the translation system to the test domain, we explore dataset tagging and fine-tuning on in-domain data. We observe that different techniques provide varied improvements based on the available data of the language pair. Based on the finding, we integrate these techniques into one training pipeline. For En->Ta, we explore an unconstrained setup with additional Tamil bitext and monolingual data and show that further improvement can be obtained. On the test set, our best submitted systems achieve 21.5 and 13.7 BLEU for Ta->En and En->Ta respectively, and 27.9 and 13.0 for Iu->En and En->Iu respectively.
Don't Patronize Me! An Annotated Dataset with Patronizing and Condescending Language towards Vulnerable Communities
Authors: Carla Pérez-Almendros, Luis Espinosa-Anke, Steven Schockaert
Date: 2020-11-16 00:00:00
Proceeding: COLING 2020 8

In this paper, we introduce a new annotated dataset which is aimed at supporting the development of NLP models to identify and categorize language that is patronizing or condescending towards vulnerable communities (e.g. refugees, homeless people, poor families). While the prevalence of such language in the general media has long been shown to have harmful effects, it differs from other types of harmful language, in that it is generally used unconsciously and with good intentions. We furthermore believe that the often subtle nature of patronizing and condescending language (PCL) presents an interesting technical challenge for the NLP community. Our analysis of the proposed dataset shows that identifying PCL is hard for standard NLP models, with language models such as BERT achieving the best results.
Refining Automatic Speech Recognition System for older adults
Authors: Liu Chen, Meysam Asgari
Date: 2020-11-17 00:00:00
Tasks: Automatic Speech Recognition, Automatic Speech Recognition (ASR), speech-recognition, Speech Recognition, Transfer Learning

Building a high quality automatic speech recognition (ASR) system with limited training data has been a challenging task particularly for a narrow target population. Open-sourced ASR systems, trained on sufficient data from adults, are susceptible on seniors' speech due to acoustic mismatch between adults and seniors. With 12 hours of training data, we attempt to develop an ASR system for socially isolated seniors (80+ years old) with possible cognitive impairments. We experimentally identify that ASR for the adult population performs poorly on our target population and transfer learning (TL) can boost the system's performance. Standing on the fundamental idea of TL, tuning model parameters, we further improve the system by leveraging an attention mechanism to utilize the model's intermediate information. Our approach achieves 1.58% absolute improvements over the TL model.
Personalized Cardiovascular Disease Risk Mitigation via Longitudinal Inverse Classification
Authors: Michael T. Lash, W. Nick Street
Date: 2020-11-16 00:00:00
Tasks: Classification, Decision Making, General Classification

Cardiovascular disease (CVD) is a serious illness affecting millions world-wide and is the leading cause of death in the US. Recent years, however, have seen tremendous growth in the area of personalized medicine, a field of medicine that places the patient at the center of the medical decision-making and treatment process. Many CVD-focused personalized medicine innovations focus on genetic biomarkers, which provide person-specific CVD insights at the genetic level, but do not focus on the practical steps a patient could take to mitigate their risk of CVD development. In this work we propose longitudinal inverse classification, a recommendation framework that provides personalized lifestyle recommendations that minimize the predicted probability of CVD risk. Our framework takes into account historical CVD risk, as well as other patient characteristics, to provide recommendations. Our experiments show that earlier adoption of the recommendations elicited from our framework produce significant CVD risk reduction.
Machine Learning and Soil Humidity Sensing: Signal Strength Approach
Authors: Lea Dujić Rodić, Tomislav Županović, Toni Perković, Petar Šolić, Joel J. P. C. Rodrigues
Date: 2020-11-16 00:00:00
Tasks: BIG-bench Machine Learning

The IoT vision of ubiquitous and pervasive computing gives rise to future smart irrigation systems comprising physical and digital world. Smart irrigation ecosystem combined with Machine Learning can provide solutions that successfully solve the soil humidity sensing task in order to ensure optimal water usage. Existing solutions are based on data received from the power hungry/expensive sensors that are transmitting the sensed data over the wireless channel. Over time, the systems become difficult to maintain, especially in remote areas due to the battery replacement issues with large number of devices. Therefore, a novel solution must provide an alternative, cost and energy effective device that has unique advantage over the existing solutions. This work explores a concept of a novel, low-power, LoRa-based, cost-effective system which achieves humidity sensing using Deep learning techniques that can be employed to sense soil humidity with the high accuracy simply by measuring signal strength of the given underground beacon device.
Avoiding Communication in Logistic Regression
Authors: Aditya Devarakonda, James Demmel
Date: 2020-11-16 00:00:00
Tasks: regression

Stochastic gradient descent (SGD) is one of the most widely used optimization methods for solving various machine learning problems. SGD solves an optimization problem by iteratively sampling a few data points from the input data, computing gradients for the selected data points, and updating the solution. However, in a parallel setting, SGD requires interprocess communication at every iteration. We introduce a new communication-avoiding technique for solving the logistic regression problem using SGD. This technique re-organizes the SGD computations into a form that communicates every $s$ iterations instead of every iteration, where $s$ is a tuning parameter. We prove theoretical flops, bandwidth, and latency upper bounds for SGD and its new communication-avoiding variant. Furthermore, we show experimental results that illustrate that the new Communication-Avoiding SGD (CA-SGD) method can achieve speedups of up to $4.97\times$ on a high-performance Infiniband cluster without altering the convergence behavior or accuracy.
Distilling a Hierarchical Policy for Planning and Control via Representation and Reinforcement Learning
Authors: Jung-Su Ha, Young-Jin Park, Hyeok-Joo Chae, Soon-Seo Park, Han-Lim Choi
Date: 2020-11-16 00:00:00
Tasks: reinforcement-learning, Reinforcement Learning (RL)

We present a hierarchical planning and control framework that enables an agent to perform various tasks and adapt to a new task flexibly. Rather than learning an individual policy for each particular task, the proposed framework, DISH, distills a hierarchical policy from a set of tasks by representation and reinforcement learning. The framework is based on the idea of latent variable models that represent high-dimensional observations using low-dimensional latent variables. The resulting policy consists of two levels of hierarchy: (i) a planning module that reasons a sequence of latent intentions that would lead to an optimistic future and (ii) a feedback control policy, shared across the tasks, that executes the inferred intention. Because the planning is performed in low-dimensional latent space, the learned policy can immediately be used to solve or adapt to new tasks without additional training. We demonstrate the proposed framework can learn compact representations (3- and 1-dimensional latent states and commands for a humanoid with 197- and 36-dimensional state features and actions) while solving a small number of imitation tasks, and the resulting policy is directly applicable to other types of tasks, i.e., navigation in cluttered environments. Video: https://youtu.be/HQsQysUWOhg
Redesigning the classification layer by randomizing the class representation vectors
Authors: Gabi Shalev, Gal-Lev Shalev, Joseph Keshet
Date: 2020-11-16 00:00:00
Tasks: Classification, General Classification, image-classification, Image Classification

Neural image classification models typically consist of two components. The first is an image encoder, which is responsible for encoding a given raw image into a representative vector. The second is the classification component, which is often implemented by projecting the representative vector onto target class vectors. The target class vectors, along with the rest of the model parameters, are estimated so as to minimize the loss function. In this paper, we analyze how simple design choices for the classification layer affect the learning dynamics. We show that the standard cross-entropy training implicitly captures visual similarities between different classes, which might deteriorate accuracy or even prevents some models from converging. We propose to draw the class vectors randomly and set them as fixed during training, thus invalidating the visual similarities encoded in these vectors. We analyze the effects of keeping the class vectors fixed and show that it can increase the inter-class separability, intra-class compactness, and the overall model accuracy, while maintaining the robustness to image corruptions and the generalization of the learned concepts.
Robust Analysis of Stock Price Time Series Using CNN and LSTM-Based Deep Learning Models
Authors: Sidra Mehtab, Jaydip Sen, Subhasis Dasgupta
Date: 2020-11-07 00:00:00
Tasks: Stock Price Prediction, Time Series, Time Series Analysis

Prediction of stock price and stock price movement patterns has always been a critical area of research. While the well-known efficient market hypothesis rules out any possibility of accurate prediction of stock prices, there are formal propositions in the literature demonstrating accurate modeling of the predictive systems that can enable us to predict stock prices with a very high level of accuracy. In this paper, we present a suite of deep learning-based regression models that yields a very high level of accuracy in stock price prediction. To build our predictive models, we use the historical stock price data of a well-known company listed in the National Stock Exchange (NSE) of India during the period December 31, 2012 to January 9, 2015. The stock prices are recorded at five minutes intervals of time during each working day in a week. Using these extremely granular stock price data, we build four convolutional neural network (CNN) and five long- and short-term memory (LSTM)-based deep learning models for accurate forecasting of the future stock prices. We provide detailed results on the forecasting accuracies of all our proposed models based on their execution time and their root mean square error (RMSE) values.