pwc-archive/papers-with-abstracts huggingface dataset:
Knowledge Amalgamation from Heterogeneous Networks by Common Feature Learning
Authors: Sihui Luo, Xinchao Wang, Gongfan Fang, Yao Hu, Dapeng Tao, Mingli Song
Date: 2019-06-24 00:00:00

An increasing number of well-trained deep networks have been released online by researchers and developers, enabling the community to reuse them in a plug-and-play way without accessing the training annotations. However, due to the large number of network variants, such public-available trained models are often of different architectures, each of which being tailored for a specific task or dataset. In this paper, we study a deep-model reusing task, where we are given as input pre-trained networks of heterogeneous architectures specializing in distinct tasks, as teacher models. We aim to learn a multitalented and light-weight student model that is able to grasp the integrated knowledge from all such heterogeneous-structure teachers, again without accessing any human annotation. To this end, we propose a common feature learning scheme, in which the features of all teachers are transformed into a common space and the student is enforced to imitate them all so as to amalgamate the intact knowledge. We test the proposed approach on a list of benchmarks and demonstrate that the learned student is able to achieve very promising performance, superior to those of the teachers in their specialized tasks.
Gauge theory and twins paradox of disentangled representations
Authors: X. Dong, L. Zhou
Date: 2019-06-24 00:00:00
Tasks: BIG-bench Machine Learning

Achieving disentangled representations of information is one of the key goals of deep network based machine learning system. Recently there are more discussions on this issue. In this paper, by comparing the geometric structure of disentangled representation and the geometry of the evolution of mixed states in quantum mechanics, we give a fibre bundle based geometric picture of disentangled representation which can be regarded as a kind of gauge theory. From this perspective we can build a connection between the disentangled representations and the twins paradox in relativity. This can help to clarify some problems about disentangled representation.
Dynamic Network Embeddings for Network Evolution Analysis
Authors: Chuanchang Chen, Yubo Tao, Hai Lin
Date: 2019-06-24 00:00:00
Tasks: Link Prediction, Network Embedding

Network embeddings learn to represent nodes as low-dimensional vectors to preserve the proximity between nodes and communities of the network for network analysis. The temporal edges (e.g., relationships, contacts, and emails) in dynamic networks are important for network evolution analysis, but few existing methods in network embeddings can capture the dynamic information from temporal edges. In this paper, we propose a novel dynamic network embedding method to analyze evolution patterns of dynamic networks effectively. Our method uses random walk to keep the proximity between nodes and applies dynamic Bernoulli embeddings to train discrete-time network embeddings in the same vector space without alignments to preserve the temporal continuity of stable nodes. We compare our method with several state-of-the-art methods by link prediction and evolving node detection, and the experiments demonstrate that our method generally has better performance in these tasks. Our method is further verified by two real-world dynamic networks via detecting evolving nodes and visualizing their temporal trajectories in the embedded space.
Assessing the Applicability of Authorship Verification Methods
Authors: Oren Halvani, Christian Winter, Lukas Graner
Date: 2019-06-24 00:00:00
Tasks: Authorship Verification

Authorship verification (AV) is a research subject in the field of digital text forensics that concerns itself with the question, whether two documents have been written by the same person. During the past two decades, an increasing number of proposed AV approaches can be observed. However, a closer look at the respective studies reveals that the underlying characteristics of these methods are rarely addressed, which raises doubts regarding their applicability in real forensic settings. The objective of this paper is to fill this gap by proposing clear criteria and properties that aim to improve the characterization of existing and future AV approaches. Based on these properties, we conduct three experiments using 12 existing AV approaches, including the current state of the art. The examined methods were trained, optimized and evaluated on three self-compiled corpora, where each corpus focuses on a different aspect of applicability. Our results indicate that part of the methods are able to cope with very challenging verification cases such as 250 characters long informal chat conversations (72.7% accuracy) or cases in which two scientific documents were written at different times with an average difference of 15.6 years (> 75% accuracy). However, we also identified that all involved methods are prone to cross-topic verification cases.
Embedding Projection for Targeted Cross-Lingual Sentiment: Model Comparisons and a Real-World Study
Authors: Jeremy Barnes, Roman Klinger
Date: 2019-06-24 00:00:00
Tasks: Machine Translation, Sentence, Sentiment Analysis

Sentiment analysis benefits from large, hand-annotated resources in order to train and test machine learning models, which are often data hungry. While some languages, e.g., English, have a vast array of these resources, most under-resourced languages do not, especially for fine-grained sentiment tasks, such as aspect-level or targeted sentiment analysis. To improve this situation, we propose a cross-lingual approach to sentiment analysis that is applicable to under-resourced languages and takes into account target-level information. This model incorporates sentiment information into bilingual distributional representations, by jointly optimizing them for semantics and sentiment, showing state-of-the-art performance at sentence-level when combined with machine translation. The adaptation to targeted sentiment analysis on multiple domains shows that our model outperforms other projection-based bilingual embedding methods on binary targeted sentiment tasks. Our analysis on ten languages demonstrates that the amount of unlabeled monolingual data has surprisingly little effect on the sentiment results. As expected, the choice of annotated source language for projection to a target leads to better results for source-target language pairs which are similar. Therefore, our results suggest that more efforts should be spent on the creation of resources for less similar languages to those which are resource-rich already. Finally, a domain mismatch leads to a decreased performance. This suggests resources in any language should ideally cover varieties of domains.
The Value of Collaboration in Convex Machine Learning with Differential Privacy
Authors: Nan Wu, Farhad Farokhi, David Smith, Mohamed Ali Kaafar
Date: 2019-06-24 00:00:00
Tasks: BIG-bench Machine Learning

In this paper, we apply machine learning to distributed private data owned by multiple data owners, entities with access to non-overlapping training datasets. We use noisy, differentially-private gradients to minimize the fitness cost of the machine learning model using stochastic gradient descent. We quantify the quality of the trained model, using the fitness cost, as a function of privacy budget and size of the distributed datasets to capture the trade-off between privacy and utility in machine learning. This way, we can predict the outcome of collaboration among privacy-aware data owners prior to executing potentially computationally-expensive machine learning algorithms. Particularly, we show that the difference between the fitness of the trained machine learning model using differentially-private gradient queries and the fitness of the trained machine model in the absence of any privacy concerns is inversely proportional to the size of the training datasets squared and the privacy budget squared. We successfully validate the performance prediction with the actual performance of the proposed privacy-aware learning algorithms, applied to: financial datasets for determining interest rates of loans using regression; and detecting credit card frauds using support vector machines.
An AGI with Time-Inconsistent Preferences
Authors: James D. Miller, Roman Yampolskiy
Date: 2019-06-23 00:00:00
Tasks: Decision Making

This paper reveals a trap for artificial general intelligence (AGI) theorists who use economists' standard method of discounting. This trap is implicitly and falsely assuming that a rational AGI would have time-consistent preferences. An agent with time-inconsistent preferences knows that its future self will disagree with its current self concerning intertemporal decision making. Such an agent cannot automatically trust its future self to carry out plans that its current self considers optimal.
Metaheuristics optimized feedforward neural networks for efficient stock price prediction
Authors: Bradley J. Pillay, Absalom E. Ezugwu
Date: 2019-06-23 00:00:00
Tasks: Decision Making, Stock Price Prediction, Time Series, Time Series Analysis

The prediction of stock prices is an important task in economics, investment and making financial decisions. This has, for decades, spurred the interest of many researchers to make focused contributions to the design of accurate stock price predictive models; of which some have been utilized to predict the next day opening and closing prices of the stock indices. This paper proposes the design and implementation of a hybrid symbiotic organisms search trained feedforward neural network model for effective and accurate stock price prediction. The symbiotic organisms search algorithm is used as an efficient optimization technique to train the feedforward neural networks, while the resulting training process is used to build a better stock price prediction model. Furthermore, the study also presents a comparative performance evaluation of three different stock price forecasting models; namely, the particle swarm optimization trained feedforward neural network model, the genetic algorithm trained feedforward neural network model and the well-known ARIMA model. The system developed in support of this study utilizes sixteen stock indices as time series datasets for training and testing purpose. Three statistical evaluation measures are used to compare the results of the implemented models, namely the root mean squared error, the mean absolute percentage error and the mean absolution deviation. The computational results obtained reveal that the symbiotic organisms search trained feedforward neural network model exhibits outstanding predictive performance compared to the other models. However, the performance study shows that the three metaheuristics trained feedforward neural network models have promising predictive competence for solving problems of high dimensional nonlinear time series data, which are difficult to capture by traditional models.
A Review on Deep Learning in Medical Image Reconstruction
Authors: Haimiao Zhang, Bin Dong
Date: 2019-06-23 00:00:00
Tasks: Deep Learning, Image Reconstruction, Image Restoration, Rolling Shutter Correction

Medical imaging is crucial in modern clinics to guide the diagnosis and treatment of diseases. Medical image reconstruction is one of the most fundamental and important components of medical imaging, whose major objective is to acquire high-quality medical images for clinical usage at minimal cost and risk to the patients. Mathematical models in medical image reconstruction or, more generally, image restoration in computer vision, have been playing a prominent role. Earlier mathematical models are mostly designed by human knowledge or hypothesis on the image to be reconstructed, and we shall call these models handcrafted models. Later, handcrafted plus data-driven modeling started to emerge which still mostly relies on human designs, while part of the model is learned from the observed data. More recently, as more data and computation resources are made available, deep learning based models (or deep models) pushed data-driven modeling to the extreme where the models are mostly based on learning with minimal human designs. Both handcrafted and data-driven modeling have their own advantages and disadvantages. One of the major research trends in medical imaging is to combine handcrafted modeling with deep modeling so that we can enjoy benefits from both approaches. The major part of this article is to provide a conceptual review of some recent works on deep modeling from the unrolling dynamics viewpoint. This viewpoint stimulates new designs of neural network architectures with inspiration from optimization algorithms and numerical differential equations. Given the popularity of deep modeling, there are still vast remaining challenges in the field, as well as opportunities which we shall discuss at the end of this article.
Assembly line balancing with task division
Authors: Carlos Alexandre X. Silva, Les Foulds, Humberto J. Longo
Date: 2019-06-22 00:00:00

In a commonly-used version of the Simple Assembly Line Balancing Problem (SALBP-1) tasks are assigned to stations along an assembly line with a fixed cycle time in order to minimize the required number of stations. It has traditionally been assumed that the total work needed for each product unit has been partitioned into economically indivisible tasks. However, in practice, it is sometimes possible to divide particular tasks in limited ways at additional time penalty cost. Despite the penalties, task division where possible, now and then leads to a reduction in the minimum number of stations. Deciding which allowable tasks to divide creates a new assembly line balancing problem, TDALBP (Task Division Assembly Line Balancing Problem). We propose a mathematical model of the TDALBP, an exact solution procedure for it and present promising computational results for the adaptation of some classical SALBP instances from the research literature. The results demonstrate that the TDALBP sometimes has the potential to significantly improve assembly line performance.
Beneficial perturbation network for continual learning
Authors: Shixian Wen, Laurent Itti
Date: 2019-06-22 00:00:00
Tasks: Continual Learning

Sequential learning of multiple tasks in artificial neural networks using gradient descent leads to catastrophic forgetting, whereby previously learned knowledge is erased during learning of new, disjoint knowledge. Here, we propose a fundamentally new type of method - Beneficial Perturbation Network (BPN). We add task-dependent memory (biasing) units to allow the network to operate in different regimes for different tasks. We compute the most beneficial directions to train these units, in a manner inspired by recent work on adversarial examples. At test time, beneficial perturbations for a given task bias the network toward that task to overcome catastrophic forgetting. BPN is not only more parameter-efficient than network expansion methods, but also does not need to store any data from previous tasks, in contrast with episodic memory methods. Experiments on variants of the MNIST, CIFAR-10, CIFAR-100 datasets demonstrate strong performance of BPN when compared to the state-of-the-art.
The Finite-Horizon Two-Armed Bandit Problem with Binary Responses: A Multidisciplinary Survey of the History, State of the Art, and Myths
Authors: Peter Jacko
Date: 2019-06-20 00:00:00
Tasks: Reinforcement Learning

In this paper we consider the two-armed bandit problem, which often naturally appears per se or as a subproblem in some multi-armed generalizations, and serves as a starting point for introducing additional problem features. The consideration of binary responses is motivated by its widespread applicability and by being one of the most studied settings. We focus on the undiscounted finite-horizon objective, which is the most relevant in many applications. We make an attempt to unify the terminology as this is different across disciplines that have considered this problem, and present a unified model cast in the Markov decision process framework, with subject responses modelled using the Bernoulli distribution, and the corresponding Beta distribution for Bayesian updating. We give an extensive account of the history and state of the art of approaches from several disciplines, including design of experiments, Bayesian decision theory, naive designs, reinforcement learning, biostatistics, and combination designs. We evaluate these designs, together with a few newly proposed, accurately computationally (using a newly written package in Julia programming language by the author) in order to compare their performance. We show that conclusions are different for moderate horizons (typical in practice) than for small horizons (typical in academic literature reporting computational results). We further list and clarify a number of myths about this problem, e.g., we show that, computationally, much larger problems can be designed to Bayes-optimality than what is commonly believed.
Benchmarking Neural Machine Translation for Southern African Languages
Authors: Laura Martinus, Jade Z. Abbott
Date: 2019-06-17 00:00:00
Proceeding: WS 2019 8
Tasks: Benchmarking, Machine Translation, Translation

Unlike major Western languages, most African languages are very low-resourced. Furthermore, the resources that do exist are often scattered and difficult to obtain and discover. As a result, the data and code for existing research has rarely been shared. This has lead a struggle to reproduce reported results, and few publicly available benchmarks for African machine translation models exist. To start to address these problems, we trained neural machine translation models for 5 Southern African languages on publicly-available datasets. Code is provided for training the models and evaluate the models on a newly released evaluation set, with the aim of spur future research in the field for Southern African languages.
Refuting Strong AI: Why Consciousness Cannot Be Algorithmic
Authors: Andrew Knight
Date: 2019-06-11 00:00:00

While physicalism requires only that a conscious state depends entirely on an underlying physical state, it is often assumed that consciousness is algorithmic and that conscious states can be copied, such as by copying or digitizing the human brain. In an effort to further elucidate the physical nature of consciousness, I challenge these assumptions and attempt to prove the Single Stream of Consciousness Theorem (SSCT): that a conscious entity cannot experience more than one stream of consciousness from a given conscious state. Assuming only that consciousness is a purely physical phenomenon, it is shown that both Special Relativity and Multiverse theory independently imply SSCT and that the Many Worlds Interpretation of quantum mechanics is inadequate to counter it. Then, SSCT is shown to be incompatible with Strong Artificial Intelligence, implying that consciousness cannot be created or simulated by a computer. Finally, SSCT is shown to imply that a conscious state cannot be physically reset to an earlier conscious state nor can it be duplicated by any physical means. The profound but counterintuitive implications of these conclusions are briefly discussed.
Zero-Shot Image Classification Using Coupled Dictionary Embedding
Authors: Mohammad Rostami, Soheil Kolouri, Zak Murez, Yuri Owekcho, Eric Eaton, Kuyngnam Kim
Date: 2019-06-10 00:00:00
Tasks: Attribute, Classification, Dictionary Learning, General Classification, image-classification, Image Classification, Zero-Shot Image Classification, Zero-Shot Learning

Zero-shot learning (ZSL) is a framework to classify images belonging to unseen classes based on solely semantic information about these unseen classes. In this paper, we propose a new ZSL algorithm using coupled dictionary learning. The core idea is that the visual features and the semantic attributes of an image can share the same sparse representation in an intermediate space. We use images from seen classes and semantic attributes from seen and unseen classes to learn two dictionaries that can represent sparsely the visual and semantic feature vectors of an image. In the ZSL testing stage and in the absence of labeled data, images from unseen classes can be mapped into the attribute space by finding the joint sparse representation using solely the visual data. The image is then classified in the attribute space given semantic descriptions of unseen classes. We also provide an attribute-aware formulation to tackle domain shift and hubness problems in ZSL. Extensive experiments are provided to demonstrate the superior performance of our approach against the state of the art ZSL algorithms on benchmark ZSL datasets.
Invariance-inducing regularization using worst-case transformations suffices to boost accuracy and spatial robustness
Authors: Fanny Yang, Zuowen Wang, Christina Heinze-Deml
Date: 2019-06-26 00:00:00
Proceeding: NeurIPS 2019 12

This work provides theoretical and empirical evidence that invariance-inducing regularizers can increase predictive accuracy for worst-case spatial transformations (spatial robustness). Evaluated on these adversarially transformed examples, we demonstrate that adding regularization on top of standard or adversarial training reduces the relative error by 20% for CIFAR10 without increasing the computational cost. This outperforms handcrafted networks that were explicitly designed to be spatial-equivariant. Furthermore, we observe for SVHN, known to have inherent variance in orientation, that robust training also improves standard accuracy on the test set. We prove that this no-trade-off phenomenon holds for adversarial examples from transformation groups in the infinite data limit.
Compositional Transfer in Hierarchical Reinforcement Learning
Authors: Markus Wulfmeier, Abbas Abdolmaleki, Roland Hafner, Jost Tobias Springenberg, Michael Neunert, Tim Hertweck, Thomas Lampe, Noah Siegel, Nicolas Heess, Martin Riedmiller
Date: 2019-06-26 00:00:00
Tasks: General Reinforcement Learning, Hierarchical Reinforcement Learning, reinforcement-learning, Reinforcement Learning, Reinforcement Learning (RL), Scheduling, Transfer Learning

The successful application of general reinforcement learning algorithms to real-world robotics applications is often limited by their high data requirements. We introduce Regularized Hierarchical Policy Optimization (RHPO) to improve data-efficiency for domains with multiple dominant tasks and ultimately reduce required platform time. To this end, we employ compositional inductive biases on multiple levels and corresponding mechanisms for sharing off-policy transition data across low-level controllers and tasks as well as scheduling of tasks. The presented algorithm enables stable and fast learning for complex, real-world domains in the parallel multitask and sequential transfer case. We show that the investigated types of hierarchy enable positive transfer while partially mitigating negative interference and evaluate the benefits of additional incentives for efficient, compositional task solutions in single task domains. Finally, we demonstrate substantial data-efficiency and final performance gains over competitive baselines in a week-long, physical robot stacking experiment.