pwc-archive/papers-with-abstracts huggingface dataset:
Equilibrium Propagation with Continual Weight Updates
Authors: Maxence Ernoult, Julie Grollier, Damien Querlioz, Yoshua Bengio, Benjamin Scellier
Date: 2020-04-29 00:00:00

Equilibrium Propagation (EP) is a learning algorithm that bridges Machine Learning and Neuroscience, by computing gradients closely matching those of Backpropagation Through Time (BPTT), but with a learning rule local in space. Given an input $x$ and associated target $y$, EP proceeds in two phases: in the first phase neurons evolve freely towards a first steady state; in the second phase output neurons are nudged towards $y$ until they reach a second steady state. However, in existing implementations of EP, the learning rule is not local in time: the weight update is performed after the dynamics of the second phase have converged and requires information of the first phase that is no longer available physically. In this work, we propose a version of EP named Continual Equilibrium Propagation (C-EP) where neuron and synapse dynamics occur simultaneously throughout the second phase, so that the weight update becomes local in time. Such a learning rule local both in space and time opens the possibility of an extremely energy efficient hardware implementation of EP. We prove theoretically that, provided the learning rates are sufficiently small, at each time step of the second phase the dynamics of neurons and synapses follow the gradients of the loss given by BPTT (Theorem 1). We demonstrate training with C-EP on MNIST and generalize C-EP to neural networks where neurons are connected by asymmetric connections. We show through experiments that the more the network updates follows the gradients of BPTT, the best it performs in terms of training. These results bring EP a step closer to biology by better complying with hardware constraints while maintaining its intimate link with backpropagation.
Continual Weight Updates and Convolutional Architectures for Equilibrium Propagation
Authors: Maxence Ernoult, Julie Grollier, Damien Querlioz, Yoshua Bengio, Benjamin Scellier
Date: 2020-04-29 00:00:00

Equilibrium Propagation (EP) is a biologically inspired alternative algorithm to backpropagation (BP) for training neural networks. It applies to RNNs fed by a static input x that settle to a steady state, such as Hopfield networks. EP is similar to BP in that in the second phase of training, an error signal propagates backwards in the layers of the network, but contrary to BP, the learning rule of EP is spatially local. Nonetheless, EP suffers from two major limitations. On the one hand, due to its formulation in terms of real-time dynamics, EP entails long simulation times, which limits its applicability to practical tasks. On the other hand, the biological plausibility of EP is limited by the fact that its learning rule is not local in time: the synapse update is performed after the dynamics of the second phase have converged and requires information of the first phase that is no longer available physically. Our work addresses these two issues and aims at widening the spectrum of EP from standard machine learning models to more bio-realistic neural networks. First, we propose a discrete-time formulation of EP which enables to simplify equations, speed up training and extend EP to CNNs. Our CNN model achieves the best performance ever reported on MNIST with EP. Using the same discrete-time formulation, we introduce Continual Equilibrium Propagation (C-EP): the weights of the network are adjusted continually in the second phase of training using local information in space and time. We show that in the limit of slow changes of synaptic strengths and small nudging, C-EP is equivalent to BPTT (Theorem 1). We numerically demonstrate Theorem 1 and C-EP training on MNIST and generalize it to the bio-realistic situation of a neural network with asymmetric connections between neurons.
A Neuromorphic Paradigm for Online Unsupervised Clustering
Authors: James E. Smith
Date: 2020-04-25 00:00:00
Tasks: Clustering

A computational paradigm based on neuroscientific concepts is proposed and shown to be capable of online unsupervised clustering. Because it is an online method, it is readily amenable to streaming realtime applications and is capable of dynamically adjusting to macro-level input changes. All operations, both training and inference, are localized and efficient. The paradigm is implemented as a cognitive column that incorporates five key elements: 1) temporal coding, 2) an excitatory neuron model for inference, 3) winner-take-all inhibition, 4) a column architecture that combines excitation and inhibition, 5) localized training via spike timing de-pendent plasticity (STDP). These elements are described and discussed, and a prototype column is given. The prototype column is simulated with a semi-synthetic benchmark and is shown to have performance characteristics on par with classic k-means. Simulations reveal the inner operation and capabilities of the column with emphasis on excitatory neuron response functions and STDP implementations.
Swarm Programming Using Moth-Flame Optimization and Whale Optimization Algorithms
Authors: Tapas Si
Date: 2020-04-25 00:00:00
Tasks: Symbolic Regression

Automatic programming (AP) is an important area of Machine Learning (ML) where computer programs are generated automatically. Swarm Programming (SP), a newly emerging research area in AP, automatically generates the computer programs using Swarm Intelligence (SI) algorithms. This paper presents two grammar-based SP methods named as Grammatical Moth-Flame Optimizer (GMFO) and Grammatical Whale Optimizer (GWO). The Moth-Flame Optimizer and Whale Optimization algorithm are used as search engines or learning algorithms in GMFO and GWO respectively. The proposed methods are tested on Santa Fe Ant Trail, quartic symbolic regression, and 3-input multiplexer problems. The results are compared with Grammatical Bee Colony (GBC) and Grammatical Fireworks algorithm (GFWA). The experimental results demonstrate that the proposed SP methods can be used in automatic computer program generation.
Hybrid 2-stage Imperialist Competitive Algorithm with Ant Colony Optimization for Solving Multi-Depot Vehicle Routing Problem
Authors: Ivars Dzalbs, Tatiana Kalganova
Date: 2020-04-07 00:00:00

The Multi-Depot Vehicle Routing Problem (MDVRP) is a real-world model of the simplistic Vehicle Routing Problem (VRP) that considers how to satisfy multiple customer demands from numerous depots. This paper introduces a hybrid 2-stage approach based on two population-based algorithms - Ant Colony Optimization (ACO) that mimics ant behaviour in nature and the Imperialist Competitive Algorithm (ICA) that is based on geopolitical relationships between countries. In the proposed hybrid algorithm, ICA is responsible for customer assignment to the depots while ACO is routing and sequencing the customers. The algorithm is compared to non-hybrid ACO and ICA as well as four other state-of-the-art methods across 23 common Cordreaus benchmark instances. Results show clear improvement over simple ACO and ICA and demonstrate very competitive results when compared to other rival algorithms.
Hyperparameter Optimization in Binary Communication Networks for Neuromorphic Deployment
Authors: Maryam Parsa, Catherine D. Schuman, Prasanna Date, Derek C. Rose, Bill Kay, J. Parker Mitchell, Steven R. Young, Ryan Dellana, William Severa, Thomas E. Potok, Kaushik Roy
Date: 2020-04-21 00:00:00
Tasks: Hyperparameter Optimization

Training neural networks for neuromorphic deployment is non-trivial. There have been a variety of approaches proposed to adapt back-propagation or back-propagation-like algorithms appropriate for training. Considering that these networks often have very different performance characteristics than traditional neural networks, it is often unclear how to set either the network topology or the hyperparameters to achieve optimal performance. In this work, we introduce a Bayesian approach for optimizing the hyperparameters of an algorithm for training binary communication networks that can be deployed to neuromorphic hardware. We show that by optimizing the hyperparameters on this algorithm for each dataset, we can achieve improvements in accuracy over the previous state-of-the-art for this algorithm on each dataset (by up to 15 percent). This jump in performance continues to emphasize the potential when converting traditional neural networks to binary communication applicable to neuromorphic hardware.
DeePore: a deep learning workflow for rapid and comprehensive characterization of porous materials
Authors: Arash Rabbani, Masoud Babaei, Reza Shams, Ying Da Wang, Traiwit Chung
Date: 2020-05-03 00:00:00
Tasks: Physical Simulations

DeePore is a deep learning workflow for rapid estimation of a wide range of porous material properties based on the binarized micro-tomography images. By combining naturally occurring porous textures we generated 17700 semi-real 3-D micro-structures of porous geo-materials with size of 256^3 voxels and 30 physical properties of each sample are calculated using physical simulations on the corresponding pore network models. Next, a designed feed-forward convolutional neural network (CNN) is trained based on the dataset to estimate several morphological, hydraulic, electrical, and mechanical characteristics of the porous material in a fraction of a second. In order to fine-tune the CNN design, we tested 9 different training scenarios and selected the one with the highest average coefficient of determination (R^2) equal to 0.885 for 1418 testing samples. Additionally, 3 independent synthetic images as well as 3 realistic tomography images have been tested using the proposed method and results are compared with pore network modelling and experimental data, respectively. Tested absolute permeabilities had around 13 % relative error compared to the experimental data which is noticeable considering the accuracy of the direct numerical simulation methods such as Lattice Boltzmann and Finite Volume. The workflow is compatible with any physical size of the images due to its dimensionless approach and can be used to characterize large-scale 3-D images by averaging the model outputs for a sliding window that scans the whole geometry.
Dual-stream shallow networks for facial micro-expression recognition
Authors: Huai-Qian Khor, John See, Sze-Teng Liong, Raphael C. -W. Phan, Weiyao Lin
Date: 2019-08-26 00:00:00
Proceeding: 2019 IEEE International Conference on Image Processing (ICIP) 2019 8
Tasks: Micro Expression Recognition, Micro-Expression Recognition

Micro-expressions are spontaneous, brief and subtle facial muscle movements that exposes underlying emotions. Motivated by recent exploits into deep learning for micro-expression analysis, we propose a lightweight dual-stream shallow network in the form of a pair of truncated CNNs with heterogeneous input features. The merging of the convolutional features allows for discriminative learning of micro-expression classes stemming from both streams. Using activation heatmaps, we further demonstrate that salient facial areas are well emphasized, and correspond closely to relevant action units belonging to emotion classes. We empirically validate the proposed network on three benchmark databases, obtaining state-of-the-art performance on the CASME II and SAMM while remaining competitive on the SMIC. Further observations point towards the sufficiency of utilizing shallower deep networks for micro-expression recognition.
Recurrent Interaction Network for Jointly Extracting Entities andClassifying Relations
Authors: Kai Sun, 2 Richong Zhang, 2∗Samuel Mensah, 1, 2 Yongyi Mao, 3 Xudong Liu
Date: 2020-05-01 00:00:00
Proceeding: arXiv 2020 5
Tasks: Multi-Task Learning, named-entity-recognition, Named Entity Recognition, Named Entity Recognition (NER), NER, Relation Extraction

Named entity recognition (NER) and Relation
extraction (RE) are two fundamental tasks in
natural language processing applications. In
practice, these two tasks are often to be solved
simultaneously. Traditional multi-task learning models implicitly capture the correlations
between NER and RE. However, there exist intrinsic connections between the output of NER
and RE. In this study, we argue that an explicit
interaction between the NER model and the
RE model will better guide the training of both
models. Based on the traditional multi-task
learning framework, we design an interactive
feature encoding method to capture the intrinsic connections between NER and RE tasks.
In addition, we propose a recurrent interaction
network to progressively capture the correlation between the two models. Empirical studies on two real-world datasets confirm the superiority of the proposed model
Real-time Single-channel Dereverberation and Separation with Time-domainAudio Separation Network
Authors: Yi Luo, Nima Mesgarani
Date: 2018-09-02 00:00:00
Proceeding: ISCA Interspeech 2018 9
Tasks: Denoising, Speech Dereverberation, Speech Separation

We investigate the recently proposed Time-domain Audio Sep-aration  Network  (TasNet)  in  the  task  of  real-time  single-channel speech dereverberation. Unlike systems that take time-frequency representation of the audio as input, TasNet learns anadaptive  front-end  in  replacement  of  the  time-frequency  rep-resentation  by  a  time-domain  convolutional  non-negative  au-toencoder.   We  show  that  by  formulating  the  dereverberationproblem as a denoising problem where the direct path is sepa-rated from the reverberations, a TasNet denoising autoencodercan outperform a deep LSTM baseline on log-power magnitudespectrogram input in both causal and non-causal settings.  Wefurther show that adjusting the stride size in the convolutionalautoencoder helps both the dereverberation and separation per-formance.
Alternative Objective Functions for Deep Clustering
Authors: Wang, Z.-Q.; Le Roux, J.; Hershey, J.R
Date: 2018-04-01 00:00:00
Proceeding: ICASSP 2018 4
Tasks: Clustering, Deep Clustering, Speech Separation

The recently proposed deep clustering framework represents a significant step towards solv-ing the cocktail party problem. This study proposes and compares a variety of alternativeobjective functions for training deep clustering networks. In addition, whereas the originaldeep clustering work relied on k-means clustering for test-time inference, here we investigateinference methods that are matched to the training objective. Furthermore, we explore theuse of an improved chimera network architecture for speech separation, which combines deepclustering with mask-inference networks in a multiobjective training scheme. The deep clus-tering loss acts as a regularizer while training the end-to-end mask inference network for bestseparation. With further iterative phase reconstruction, our best proposed method achievesa state-of-the-art 11.5 dB signal-to-distortion ratio (SDR) result on the publicly availablewsj0-2mix dataset, with a much simpler architecture than the previous best approach.
Delay-aware Resource Allocation in Fog-assisted IoT Networks Through Reinforcement Learning
Authors: Qiang Fan, Jianan Bai, Hongxia Zhang, Yang Yi, Lingjia Liu
Date: 2020-04-30 00:00:00
Tasks: reinforcement-learning, Reinforcement Learning (RL)

Fog nodes in the vicinity of IoT devices are promising to provision low latency services by offloading tasks from IoT devices to them. Mobile IoT is composed by mobile IoT devices such as vehicles, wearable devices and smartphones. Owing to the time-varying channel conditions, traffic loads and computing loads, it is challenging to improve the quality of service (QoS) of mobile IoT devices. As task delay consists of both the transmission delay and computing delay, we investigate the resource allocation (i.e., including both radio resource and computation resource) in both the wireless channel and fog node to minimize the delay of all tasks while their QoS constraints are satisfied. We formulate the resource allocation problem into an integer non-linear problem, where both the radio resource and computation resource are taken into account. As IoT tasks are dynamic, the resource allocation for different tasks are coupled with each other and the future information is impractical to be obtained. Therefore, we design an on-line reinforcement learning algorithm to make the sub-optimal decision in real time based on the system's experience replay data. The performance of the designed algorithm has been demonstrated by extensive simulation results.
Lightwave Power Transfer for Federated Learning-based Wireless Networks
Authors: Ha-Vu Tran, Georges Kaddoum, Hany Elgala, Chadi Abou-Rjeily, Hemani Kaushal
Date: 2020-04-11 00:00:00
Tasks: Federated Learning

Federated Learning (FL) has been recently presented as a new technique for training shared machine learning models in a distributed manner while respecting data privacy. However, implementing FL in wireless networks may significantly reduce the lifetime of energy-constrained mobile devices due to their involvement in the construction of the shared learning models. To handle this issue, we propose a novel approach at the physical layer based on the application of lightwave power transfer in the FL-based wireless network and a resource allocation scheme to manage the network's power efficiency. Hence, we formulate the corresponding optimization problem and then propose a method to obtain the optimal solution. Numerical results reveal that, the proposed scheme can provide sufficient energy to a mobile device for performing FL tasks without using any power from its own battery. Hence, the proposed approach can support the FL-based wireless network to overcome the issue of limited energy in mobile devices.
Forgetting Exceptions is Harmful in Language Learning
Authors: Walter Daelemans, Antal Van den Bosch, Jakub Zavrel
Date: 1998-12-22 00:00:00
Tasks: Chunking, Grapheme-to-Phoneme Conversion, Part-Of-Speech Tagging, Prepositional Phrase Attachment

We show that in language learning, contrary to received wisdom, keeping exceptional training instances in memory can be beneficial for generalization accuracy. We investigate this phenomenon empirically on a selection of benchmark natural language processing tasks: grapheme-to-phoneme conversion, part-of-speech tagging, prepositional-phrase attachment, and base noun phrase chunking. In a first series of experiments we combine memory-based learning with training set editing techniques, in which instances are edited based on their typicality and class prediction strength. Results show that editing exceptional instances (with low typicality or low class prediction strength) tends to harm generalization accuracy. In a second series of experiments we compare memory-based learning and decision-tree learning methods on the same selection of tasks, and find that decision-tree learning often performs worse than memory-based learning. Moreover, the decrease in performance can be linked to the degree of abstraction from exceptions (i.e., pruning or eagerness). We provide explanations for both results in terms of the properties of the natural language processing tasks and the learning algorithms.
SOLOIST: Building Task Bots at Scale with Transfer Learning and Machine Teaching
Authors: Baolin Peng, Chunyuan Li, Jinchao Li, Shahin Shayandeh, Lars Liden, Jianfeng Gao
Date: 2020-05-11 00:00:00
Tasks: End-To-End Dialogue Modelling, Few-Shot Learning, Language Modeling, Language Modelling, Response Generation, Transfer Learning, World Knowledge

We present a new method SOLOIST that uses transfer learning and machine teaching to build task bots at scale. We parameterize classical modular task-oriented dialog systems using a Transformer-based auto-regressive language model, which subsumes different dialog modules into a single neural model. We pre-train, on heterogeneous dialog corpora, a task-grounded response generation model, which can generate dialog responses grounded in user goals and real-world knowledge for task completion. The pre-trained model can be efficiently adapted to accomplish new tasks with a handful of task-specific dialogs via machine teaching, where training samples are generated by human teachers interacting with the system. Experiments show that (i) SOLOIST creates new state-of-the-art on well-studied task-oriented dialog benchmarks, including CamRest676 and MultiWOZ; (ii) in the few-shot fine-tuning settings, SOLOIST significantly outperforms existing methods, and (iii) the use of machine teaching substantially reduces the labeling cost of fine-tuning. The pre-trained models and codes are available at https://aka.ms/soloist.
Ring Reservoir Neural Networks for Graphs
Authors: Claudio Gallicchio, Alessio Micheli
Date: 2020-05-11 00:00:00
Tasks: Graph Classification

Machine Learning for graphs is nowadays a research topic of consolidated relevance. Common approaches in the field typically resort to complex deep neural network architectures and demanding training algorithms, highlighting the need for more efficient solutions. The class of Reservoir Computing (RC) models can play an important role in this context, enabling to develop fruitful graph embeddings through untrained recursive architectures. In this paper, we study progressive simplifications to the design strategy of RC neural networks for graphs. Our core proposal is based on shaping the organization of the hidden neurons to follow a ring topology. Experimental results on graph classification tasks indicate that ring-reservoirs architectures enable particularly effective network configurations, showing consistent advantages in terms of predictive performance.
The Safari of Update Structures: Visiting the Lens and Quantum Enclosures
Authors: Matthew Wilson, James Hefford, Guillaume Boisseau, Vincent Wang
Date: 2020-05-11 00:00:00

We build upon our recently introduced concept of an update structure to show that it is a generalisation of very-well-behaved lenses, that is, there is a bijection between a strict subset of update structures and vwb lenses in cartesian categories. We show that update structures are also sufficiently general to capture quantum observables, pinpointing the additional assumptions required to make the two coincide. In doing so, we shift the focus from special commutative dagger-Frobenius algebras to interacting (co)magma (co)module pairs, showing that the algebraic properties of the (co)multiplication arise from the module-comodule interaction, rather than direct assumptions about the magma-comagma pair. We then begin to investigate the zoo of possible update structures, introducing the notions of classical security-flagged databases, and databases of quantum systems. This work is of foundational interest as update structures place previously distinct areas of research in a general class of operationally motivated structures, we expect the taming of this class to illuminate novel relationships between separately studied topics in computer science, physics and mathematics.