pwc-archive/papers-with-abstracts huggingface dataset:
TextNAS: A Neural Architecture Search Space tailored for Text Representation
Authors: Yujing Wang, Yaming Yang, Yiren Chen, Jing Bai, Ce Zhang, Guinan Su, Xiaoyu Kou, Yunhai Tong, Mao Yang, Lidong Zhou
Date: 2019-12-23 00:00:00
Tasks: General Classification, Natural Language Inference, Neural Architecture Search, text-classification, Text Classification

Learning text representation is crucial for text classification and other language related tasks. There are a diverse set of text representation networks in the literature, and how to find the optimal one is a non-trivial problem. Recently, the emerging Neural Architecture Search (NAS) techniques have demonstrated good potential to solve the problem. Nevertheless, most of the existing works of NAS focus on the search algorithms and pay little attention to the search space. In this paper, we argue that the search space is also an important human prior to the success of NAS in different applications. Thus, we propose a novel search space tailored for text representation. Through automatic search, the discovered network architecture outperforms state-of-the-art models on various public datasets on text classification and natural language inference tasks. Furthermore, some of the design principles found in the automatic network agree well with human intuition.
An optical diffractive deep neural network with multiple frequency-channels
Authors: Yingshi Chen, Jinfeng Zhu
Date: 2019-12-23 00:00:00

Diffractive deep neural network (DNNet) is a novel machine learning framework on the modulation of optical transmission. Diffractive network would get predictions at the speed of light. It's pure passive architecture, no additional power consumption. We improved the accuracy of diffractive network with optical waves at different frequency. Each layers have multiple frequency-channels (optical distributions at different frequency). These channels are merged at the output plane to get final output. The experiment in the fasion-MNIST and EMNIST datasets showed multiple frequency-channels would increase the accuracy a lot. We also give detailed analysis to show the difference between DNNet and MLP. The modulation process in DNNet is actually optical activation function. We develop an open source package ONNet. The source codes are available at https://github.com/closest-git/ONNet.
AutoML: Exploration v.s. Exploitation
Authors: Hassan Eldeeb, Abdelrhman Eldallal
Date: 2019-12-23 00:00:00
Tasks: AutoML, Meta-Learning

Building a machine learning (ML) pipeline in an automated way is a crucial and complex task as it is constrained with the available time budget and resources. This encouraged the research community to introduce several solutions to utilize the available time and resources. A lot of work is done to suggest the most promising classifiers for a given dataset using sundry of techniques including meta-learning based techniques. This gives the autoML framework the chance to spend more time exploiting those classifiers and tuning their hyper-parameters. In this paper, we empirically study the hypothesis of improving the pipeline performance by exploiting the most promising classifiers within the limited time budget. We also study the effect of increasing the time budget over the pipeline performance. The empirical results across autoSKLearn, TPOT and ATM, show that exploiting the most promising classifiers does not achieve a statistically better performance than exploring the entire search space. The same conclusion is also applied for long time budgets.
Learn-able parameter guided Activation Functions
Authors: S. Balaji, T. Kavya, Natasha Sebastian
Date: 2019-12-23 00:00:00

In this paper, we explore the concept of adding learn-able slope and mean shift parameters to an activation function to improve the total response region. The characteristics of an activation function depend highly on the value of parameters. Making the parameters learn-able, makes the activation function more dynamic and capable to adapt as per the requirements of its neighboring layers. The introduced slope parameter is independent of other parameters in the activation function. The concept was applied to ReLU to develop Dual Line and DualParametric ReLU activation function. Evaluation on MNIST and CIFAR10 show that the proposed activation function Dual Line achieves top-5 position for mean accuracy among 43 activation functions tested with LENET4, LENET5, and WideResNet architectures. This is the first time more than 40 activation functions were analyzed on MNIST andCIFAR10 dataset at the same time. The study on the distribution of positive slope parameter beta indicates that the activation function adapts as per the requirements of the neighboring layers. The study shows that model performance increases with the proposed activation functions
Layerwise Noise Maximisation to Train Low-Energy Deep Neural Networks
Authors: Sébastien Henwood, François Leduc-Primeau, Yvon Savaria
Date: 2019-12-23 00:00:00

Deep neural networks (DNNs) depend on the storage of a large number of parameters, which consumes an important portion of the energy used during inference. This paper considers the case where the energy usage of memory elements can be reduced at the cost of reduced reliability. A training algorithm is proposed to optimize the reliability of the storage separately for each layer of the network, while incurring a negligible complexity overhead compared to a conventional stochastic gradient descent training. For an exponential energy-reliability model, the proposed training approach can decrease the memory energy consumption of a DNN with binary parameters by 3.3$\times$ at isoaccuracy, compared to a reliable implementation.
A Survey of Deep Learning Applications to Autonomous Vehicle Control
Authors: Sampo Kuutti, Richard Bowden, Yaochu Jin, Phil Barber, Saber Fallah
Date: 2019-12-23 00:00:00
Tasks: Autonomous Vehicles, Deep Learning, object-detection, Object Detection, Semantic Segmentation, Survey

Designing a controller for autonomous vehicles capable of providing adequate performance in all driving scenarios is challenging due to the highly complex environment and inability to test the system in the wide variety of scenarios which it may encounter after deployment. However, deep learning methods have shown great promise in not only providing excellent performance for complex and non-linear control problems, but also in generalising previously learned rules to new scenarios. For these reasons, the use of deep learning for vehicle control is becoming increasingly popular. Although important advancements have been achieved in this field, these works have not been fully summarised. This paper surveys a wide range of research works reported in the literature which aim to control a vehicle through deep learning methods. Although there exists overlap between control and perception, the focus of this paper is on vehicle control, rather than the wider perception problem which includes tasks such as semantic segmentation and object detection. The paper identifies the strengths and limitations of available deep learning methods through comparative analysis and discusses the research challenges in terms of computation, architecture selection, goal specification, generalisation, verification and validation, as well as safety. Overall, this survey brings timely and topical information to a rapidly evolving field relevant to intelligent transportation systems.
Deep learning predictions of sand dune migration
Authors: Kelly Kochanski, Divya Mohan, Jenna Horrall, Barry Rountree, Ghaleb Abdulla
Date: 2019-12-13 00:00:00
Tasks: Deep Learning, Sand

A dry decade in the Navajo Nation has killed vegetation, dessicated soils, and released once-stable sand into the wind. This sand now covers one-third of the Nation's land, threatening roads, gardens and hundreds of homes. Many arid regions have similar problems: global warming has increased dune movement across farmland in Namibia and Angola, and the southwestern US. Current dune models, unfortunately, do not scale well enough to provide useful forecasts for the $\sim$5\% of land surfaces covered by mobile sand. We test the ability of two deep learning algorithms, a GAN and a CNN, to model the motion of sand dunes. The models are trained on simulated data from community-standard cellular automaton model of sand dunes. Preliminary results show the GAN producing reasonable forward predictions of dune migration at ten million times the speed of the existing model.
Deteção de estruturas permanentes a partir de dados de séries temporais Sentinel 1 e 2
Authors: André Neves, Carlos Damásio, João Pires, Fernando Birra
Date: 2019-12-11 00:00:00
Tasks: Management, Time Series, Time Series Analysis

Mapping structures such as settlements, roads, individual houses and any other types of artificial structures is of great importance for the analysis of urban growth, masking, image alignment and, especially in the studied use case, the definition of Fuel Management Networks (FGC), which protect buildings from forest fires. Current cartography has a low generation frequency and their resolution may not be suitable for extracting small structures such as small settlements or roads, which may lack forest fire protection. In this paper, we use time series data, extracted from Sentinel-1 and 2 constellations, over Santar\'em, Ma\c{c}\~ao, to explore the detection of permanent structures at a resolution of 10 by 10 meters. For this purpose, a XGBoost classification model is trained with 133 attributes extracted from the time series from all the bands, including normalized radiometric indices. The results show that the use of time series data increases the accuracy of the extraction of permanent structures when compared using only static data, using multitemporal data also increases the number of detected roads. In general, the final result has a permanent structure mapping with a higher resolution than state of the art settlement maps, small structures and roads are also more accurately represented. Regarding the use case, by using our final map for the creation of FGC it is possible to simplify and accelerate the process of delimitation of the official FGC.
Majorization Minimization Technique for Optimally Solving Deep Dictionary Learning
Authors: Vanika Singhal, Angshul Majumdar
Date: 2019-12-11 00:00:00
Tasks: Denoising, Dictionary Learning

The concept of deep dictionary learning has been recently proposed. Unlike shallow dictionary learning which learns single level of dictionary to represent the data, it uses multiple layers of dictionaries. So far, the problem could only be solved in a greedy fashion; this was achieved by learning a single layer of dictionary in each stage where the coefficients from the previous layer acted as inputs to the subsequent layer (only the first layer used the training samples as inputs). This was not optimal; there was feedback from shallower to deeper layers but not the other way. This work proposes an optimal solution to deep dictionary learning whereby all the layers of dictionaries are solved simultaneously. We employ the Majorization Minimization approach. Experiments have been carried out on benchmark datasets; it shows that optimal learning indeed improves over greedy piecemeal learning. Comparison with other unsupervised deep learning tools (stacked denoising autoencoder, deep belief network, contractive autoencoder and K-sparse autoencoder) show that our method supersedes their performance both in accuracy and speed.
Predicting the Outcome of Judicial Decisions made by the European Court of Human Rights
Authors: Conor O'Sullivan, Joeran Beel
Date: 2019-12-16 00:00:00
Tasks: Articles, BIG-bench Machine Learning, Binary Classification, Overall - Test, Word Embeddings

In this study, machine learning models were constructed to predict whether judgments made by the European Court of Human Rights (ECHR) would lead to a violation of an Article in the Convention on Human Rights. The problem is framed as a binary classification task where a judgment can lead to a "violation" or "non-violation" of a particular Article. Using auto-sklearn, an automated algorithm selection package, models were constructed for 12 Articles in the Convention. To train these models, textual features were obtained from the ECHR Judgment documents using N-grams, word embeddings and paragraph embeddings. Additional documents, from the ECHR, were incorporated into the models through the creation of a word embedding (echr2vec) and a doc2vec model. The features obtained using the echr2vec embedding provided the highest cross-validation accuracy for 5 of the Articles. The overall test accuracy, across the 12 Articles, was 68.83%. As far as we could tell, this is the first estimate of the accuracy of such machine learning models using a realistic test set. This provides an important benchmark for future work. As a baseline, a simple heuristic of always predicting the most common outcome in the past was used. The heuristic achieved an overall test accuracy of 86.68% which is 29.7% higher than the models. Again, this was seemingly the first study that included such a heuristic with which to compare model results. The higher accuracy achieved by the heuristic highlights the importance of including such a baseline.
Differentiable Reasoning on Large Knowledge Bases and Natural Language
Authors: Pasquale Minervini, Matko Bošnjak, Tim Rocktäschel, Sebastian Riedel, Edward Grefenstette
Date: 2019-12-17 00:00:00
Tasks: Link Prediction, Question Answering, Reading Comprehension

Reasoning with knowledge expressed in natural language and Knowledge Bases (KBs) is a major challenge for Artificial Intelligence, with applications in machine reading, dialogue, and question answering. General neural architectures that jointly learn representations and transformations of text are very data-inefficient, and it is hard to analyse their reasoning process. These issues are addressed by end-to-end differentiable reasoning systems such as Neural Theorem Provers (NTPs), although they can only be used with small-scale symbolic KBs. In this paper we first propose Greedy NTPs (GNTPs), an extension to NTPs addressing their complexity and scalability limitations, thus making them applicable to real-world datasets. This result is achieved by dynamically constructing the computation graph of NTPs and including only the most promising proof paths during inference, thus obtaining orders of magnitude more efficient models. Then, we propose a novel approach for jointly reasoning over KBs and textual mentions, by embedding logic facts and natural language sentences in a shared embedding space. We show that GNTPs perform on par with NTPs at a fraction of their cost while achieving competitive link prediction results on large datasets, providing explanations for predictions, and inducing interpretable models. Source code, datasets, and supplementary material are available online at https://github.com/uclnlp/gntp.
Optimize Cash Collection: Use Machine learning to Predicting Invoice Payment
Authors: Ana Paula Appel, Victor Oliveira, Bruno Lima, Gabriel Louzada Malfatti, Vagner Figueredo de Santana, Rogerio de Paula
Date: 2019-12-20 00:00:00
Tasks: BIG-bench Machine Learning, Decision Making

Predicting invoice payment is valuable in multiple industries and supports decision-making processes in most financial workflows. However, the challenge in this realm involves dealing with complex data and the lack of data related to decisions-making processes not registered in the accounts receivable system. This work presents a prototype developed as a solution devised during a partnership with a multinational bank to support collectors in predicting invoices payment. The proposed prototype reached up to 77\% of accuracy, which improved the prioritization of customers and supported the daily work of collectors. With the presented results, one expects to support researchers dealing with the problem of invoice payment prediction to get insights and examples of how to tackle issues present in real data.
Variable-lag Granger Causality for Time Series Analysis
Authors: Chainarong Amornbunchornvej, Elena Zheleva, Tanya Y. Berger-Wolf
Date: 2019-12-18 00:00:00
Tasks: Causal Inference, Leadership Inference, Time Series, Time Series Analysis

Granger causality is a fundamental technique for causal inference in time series data, commonly used in the social and biological sciences. Typical operationalizations of Granger causality make a strong assumption that every time point of the effect time series is influenced by a combination of other time series with a fixed time delay. However, the assumption of the fixed time delay does not hold in many applications, such as collective behavior, financial markets, and many natural phenomena. To address this issue, we develop variable-lag Granger causality, a generalization of Granger causality that relaxes the assumption of the fixed time delay and allows causes to influence effects with arbitrary time delays. In addition, we propose a method for inferring variable-lag Granger causality relations. We demonstrate our approach on an application for studying coordinated collective behavior and show that it performs better than several existing methods in both simulated and real-world datasets. Our approach can be applied in any domain of time series analysis.
An Attention-based Graph Neural Network for Heterogeneous Structural Learning
Authors: Huiting Hong, Hantao Guo, Yu-Cheng Lin, Xiaoqing Yang, Zang Li, Jieping Ye
Date: 2019-12-19 00:00:00
Tasks: Graph Embedding, Graph Neural Network, Graph Representation Learning, Heterogeneous Node Classification, Multi-Task Learning, Node Classification, Representation Learning

In this paper, we focus on graph representation learning of heterogeneous information network (HIN), in which various types of vertices are connected by various types of relations. Most of the existing methods conducted on HIN revise homogeneous graph embedding models via meta-paths to learn low-dimensional vector space of HIN. In this paper, we propose a novel Heterogeneous Graph Structural Attention Neural Network (HetSANN) to directly encode structural information of HIN without meta-path and achieve more informative representations. With this method, domain experts will not be needed to design meta-path schemes and the heterogeneous information can be processed automatically by our proposed model. Specifically, we implicitly represent heterogeneous information using the following two methods: 1) we model the transformation between heterogeneous vertices through a projection in low-dimensional entity spaces; 2) afterwards, we apply the graph neural network to aggregate multi-relational information of projected neighborhood by means of attention mechanism. We also present three extensions of HetSANN, i.e., voices-sharing product attention for the pairwise relationships in HIN, cycle-consistency loss to retain the transformation between heterogeneous entity spaces, and multi-task learning with full use of information. The experiments conducted on three public datasets demonstrate that our proposed models achieve significant and consistent improvements compared to state-of-the-art solutions.
Does Symbolic Knowledge Prevent Adversarial Fooling?
Authors: Stefano Teso
Date: 2019-12-19 00:00:00

Arguments in favor of injecting symbolic knowledge into neural architectures abound. When done right, constraining a sub-symbolic model can substantially improve its performance and sample complexity and prevent it from predicting invalid configurations. Focusing on deep probabilistic (logical) graphical models -- i.e., constrained joint distributions whose parameters are determined (in part) by neural nets based on low-level inputs -- we draw attention to an elementary but unintended consequence of symbolic knowledge: that the resulting constraints can propagate the negative effects of adversarial examples.
Learned SVD: solving inverse problems via hybrid autoencoding
Authors: Yoeri E. Boink, Christoph Brune
Date: 2019-12-20 00:00:00
Tasks: Dimensionality Reduction

Our world is full of physics-driven data where effective mappings between data manifolds are desired. There is an increasing demand for understanding combined model-based and data-driven methods. We propose a nonlinear, learned singular value decomposition (L-SVD), which combines autoencoders that simultaneously learn and connect latent codes for desired signals and given measurements. We provide a convergence analysis for a specifically structured L-SVD that acts as a regularisation method. In a more general setting, we investigate the topic of model reduction via data dimensionality reduction to obtain a regularised inversion. We present a promising direction for solving inverse problems in cases where the underlying physics are not fully understood or have very complex behaviour. We show that the building blocks of learned inversion maps can be obtained automatically, with improved performance upon classical methods and better interpretability than black-box methods.
The State of Knowledge Distillation for Classification
Authors: Fabian Ruffy, Karanbir Chahal
Date: 2019-12-20 00:00:00
Tasks: Classification, Data Augmentation, General Classification, Knowledge Distillation

We survey various knowledge distillation (KD) strategies for simple classification tasks and implement a set of techniques that claim state-of-the-art accuracy. Our experiments using standardized model architectures, fixed compute budgets, and consistent training schedules indicate that many of these distillation results are hard to reproduce. This is especially apparent with methods using some form of feature distillation. Further examination reveals a lack of generalizability where these techniques may only succeed for specific architectures and training settings. We observe that appropriately tuned classical distillation in combination with a data augmentation training scheme gives an orthogonal improvement over other techniques. We validate this approach and open-source our code.
ET-USB: Transformer-Based Sequential Behavior Modeling for Inbound Customer Service
Authors: Ta-Chun Su, Guan-Ying Chen
Date: 2019-12-20 00:00:00
Tasks: Deep Learning, Recommendation Systems

Deep learning models with attention mechanisms have achieved exceptional results for many tasks, including language tasks and recommendation systems. Whereas previous studies have emphasized allocation of phone agents, we focused on inbound call prediction for customer service. A common method of analyzing user history behaviors is to extract all types of aggregated feature over time, but that method may fail to detect users' behavioral sequences. Therefore, we created a new approach, ET-USB, that incorporates users' sequential and nonsequential features; we apply the powerful Transformer encoder, a self-attention network model, to capture the information underlying user behavior sequences. ET-USB is helpful in various business scenarios at Cathay Financial Holdings. We conducted experiments to test the proposed network structure's ability to process various dimensions of behavior data; the results suggest that ET-USB delivers results superior to those of delivered by other deep-learning models.
Soft Q Network
Authors: Jingbin Liu, Shuai Liu, Xinyang Gu
Date: 2019-12-20 00:00:00
Tasks: Q-Learning

Deep Q Network (DQN) is a very successful algorithm, yet the inherent problem of reinforcement learning, i.e. the exploit-explore balance, remains. In this work, we introduce entropy regularization into DQN and propose SQN. We find that the backup equation of soft Q learning can enjoy the corrective feedback if we view the soft backup as policy improvement in the form of Q, instead of policy evaluation. We show that Soft Q Learning with Corrective Feedback (SQL-CF) underlies the on-plicy nature of SQL and the equivalence of SQL and Soft Policy Gradient (SPG). With these insights, we propose an on-policy version of deep Q learning algorithm, i.e. Q On-Policy (QOP). We experiment with QOP on a self-play environment called Google Research Football (GRF). The QOP algorithm exhibits great stability and efficiency in training GRF agents.