pwc-archive/papers-with-abstracts huggingface dataset:
RL with KL penalties is better viewed as Bayesian inference
Authors: Tomasz Korbak, Ethan Perez, Christopher L Buckley
Date: 2022-05-23 00:00:00
Tasks: Bayesian Inference, Language Modelling, Reinforcement Learning (RL), Variational Inference

Reinforcement learning (RL) is frequently employed in fine-tuning large language models (LMs), such as GPT-3, to penalize them for undesirable features of generated sequences, such as offensiveness, social bias, harmfulness or falsehood. The RL formulation involves treating the LM as a policy and updating it to maximise the expected value of a reward function which captures human preferences, such as non-offensiveness. In this paper, we analyze challenges associated with treating a language model as an RL policy and show how avoiding those challenges requires moving beyond the RL paradigm. We start by observing that the standard RL approach is flawed as an objective for fine-tuning LMs because it leads to distribution collapse: turning the LM into a degenerate distribution. Then, we analyze KL-regularised RL, a widely used recipe for fine-tuning LMs, which additionally constrains the fine-tuned LM to stay close to its original distribution in terms of Kullback-Leibler (KL) divergence. We show that KL-regularised RL is equivalent to variational inference: approximating a Bayesian posterior which specifies how to update a prior LM to conform with evidence provided by the reward function. We argue that this Bayesian inference view of KL-regularised RL is more insightful than the typically employed RL perspective. The Bayesian inference view explains how KL-regularised RL avoids the distribution collapse problem and offers a first-principles derivation for its objective. While this objective happens to be equivalent to RL (with a particular choice of parametric reward), there exist other objectives for fine-tuning LMs which are no longer equivalent to RL. That observation leads to a more general point: RL is not an adequate formal framework for problems such as fine-tuning language models. These problems are best viewed as Bayesian inference: approximating a pre-defined target distribution.
Fed-DART and FACT: A solution for Federated Learning in a production environment
Authors: Nico Weber, Patrick Holzer, Tania Jacob, Enislay Ramentol
Date: 2022-05-23 00:00:00
Tasks: Federated Learning

Federated Learning as a decentralized artificial intelligence (AI) solution solves a variety of problems in industrial applications. It enables a continuously self-improving AI, which can be deployed everywhere at the edge. However, bringing AI to production for generating a real business impact is a challenging task. Especially in the case of Federated Learning, expertise and resources from multiple domains are required to realize its full potential. Having this in mind we have developed an innovative Federated Learning framework FACT based on Fed-DART, enabling an easy and scalable deployment, helping the user to fully leverage the potential of their private and decentralized data.
Logarithmic regret bounds for continuous-time average-reward Markov decision processes
Authors: Xuefeng Gao, Xun Yu Zhou
Date: 2022-05-23 00:00:00
Tasks: Point Processes, reinforcement-learning, Reinforcement Learning, Reinforcement Learning (RL)

We consider reinforcement learning for continuous-time Markov decision processes (MDPs) in the infinite-horizon, average-reward setting. In contrast to discrete-time MDPs, a continuous-time process moves to a state and stays there for a random holding time after an action is taken. With unknown transition probabilities and rates of exponential holding times, we derive instance-dependent regret lower bounds that are logarithmic in the time horizon. Moreover, we design a learning algorithm and establish a finite-time regret bound that achieves the logarithmic growth rate. Our analysis builds upon upper confidence reinforcement learning, a delicate estimation of the mean holding times, and stochastic comparison of point processes.
Time-series Transformer Generative Adversarial Networks
Authors: Padmanaba Srinivasan, William J. Knottenbelt
Date: 2022-05-23 00:00:00
Tasks: Question Answering, Time Series, Time Series Analysis, Time Series Generation

Many real-world tasks are plagued by limitations on data: in some instances very little data is available and in others, data is protected by privacy enforcing regulations (e.g. GDPR). We consider limitations posed specifically on time-series data and present a model that can generate synthetic time-series which can be used in place of real data. A model that generates synthetic time-series data has two objectives: 1) to capture the stepwise conditional distribution of real sequences, and 2) to faithfully model the joint distribution of entire real sequences. Autoregressive models trained via maximum likelihood estimation can be used in a system where previous predictions are fed back in and used to predict future ones; in such models, errors can accrue over time. Furthermore, a plausible initial value is required making MLE based models not really generative. Many downstream tasks learn to model conditional distributions of the time-series, hence, synthetic data drawn from a generative model must satisfy 1) in addition to performing 2). We present TsT-GAN, a framework that capitalises on the Transformer architecture to satisfy the desiderata and compare its performance against five state-of-the-art models on five datasets and show that TsT-GAN achieves higher predictive performance on all datasets.
Learning to Advise and Learning from Advice in Cooperative Multi-Agent Reinforcement Learning
Authors: Yue Jin, Shuangqing Wei, Jian Yuan, Xudong Zhang
Date: 2022-05-23 00:00:00
Tasks: Multi-agent Reinforcement Learning, reinforcement-learning, Reinforcement Learning (RL)

Learning to coordinate is a daunting problem in multi-agent reinforcement learning (MARL). Previous works have explored it from many facets, including cognition between agents, credit assignment, communication, expert demonstration, etc. However, less attention were paid to agents' decision structure and the hierarchy of coordination. In this paper, we explore the spatiotemporal structure of agents' decisions and consider the hierarchy of coordination from the perspective of multilevel emergence dynamics, based on which a novel approach, Learning to Advise and Learning from Advice (LALA), is proposed to improve MARL. Specifically, by distinguishing the hierarchy of coordination, we propose to enhance decision coordination at meso level with an advisor and leverage a policy discriminator to advise agents' learning at micro level. The advisor learns to aggregate decision information in both spatial and temporal domains and generates coordinated decisions by employing a spatiotemporal dual graph convolutional neural network with a task-oriented objective function. Each agent learns from the advice via a policy generative adversarial learning method where a discriminator distinguishes between the policies of the agent and the advisor and boosts both of them based on its judgement. Experimental results indicate the advantage of LALA over baseline approaches in terms of both learning efficiency and coordination capability. Coordination mechanism is investigated from the perspective of multilevel emergence dynamics and mutual information point of view, which provides a novel perspective and method to analyze and improve MARL algorithms.
Split personalities in Bayesian Neural Networks: the case for full marginalisation
Authors: David Yallup, Will Handley, Mike Hobson, Anthony Lasenby, Pablo Lemos
Date: 2022-05-23 00:00:00

The true posterior distribution of a Bayesian neural network is massively multimodal. Whilst most of these modes are functionally equivalent, we demonstrate that there remains a level of real multimodality that manifests in even the simplest neural network setups. It is only by fully marginalising over all posterior modes, using appropriate Bayesian sampling tools, that we can capture the split personalities of the network. The ability of a network trained in this manner to reason between multiple candidate solutions dramatically improves the generalisability of the model, a feature we contend is not consistently captured by alternative approaches to the training of Bayesian neural networks. We provide a concise minimal example of this, which can provide lessons and a future path forward for correctly utilising the explainability and interpretability of Bayesian neural networks.
Human-in-the-loop: Provably Efficient Preference-based Reinforcement Learning with General Function Approximation
Authors: Xiaoyu Chen, Han Zhong, Zhuoran Yang, Zhaoran Wang, LiWei Wang
Date: 2022-05-23 00:00:00
Tasks: Reinforcement Learning (RL)

We study human-in-the-loop reinforcement learning (RL) with trajectory preferences, where instead of receiving a numeric reward at each step, the agent only receives preferences over trajectory pairs from a human overseer. The goal of the agent is to learn the optimal policy which is most preferred by the human overseer. Despite the empirical successes, the theoretical understanding of preference-based RL (PbRL) is only limited to the tabular case. In this paper, we propose the first optimistic model-based algorithm for PbRL with general function approximation, which estimates the model using value-targeted regression and calculates the exploratory policies by solving an optimistic planning problem. Our algorithm achieves the regret of $\tilde{O} (\operatorname{poly}(d H) \sqrt{K} )$, where $d$ is the complexity measure of the transition and preference model depending on the Eluder dimension and log-covering numbers, $H$ is the planning horizon, $K$ is the number of episodes, and $\tilde O(\cdot)$ omits logarithmic terms. Our lower bound indicates that our algorithm is near-optimal when specialized to the linear setting. Furthermore, we extend the PbRL problem by formulating a novel problem called RL with $n$-wise comparisons, and provide the first sample-efficient algorithm for this new setting. To the best of our knowledge, this is the first theoretical result for PbRL with (general) function approximation.
GraphAD: A Graph Neural Network for Entity-Wise Multivariate Time-Series Anomaly Detection
Authors: Xu Chen, Qiu Qiu, Changshan Li, Kunqing Xie
Date: 2022-05-23 00:00:00
Tasks: Anomaly Detection, Graph Neural Network, Management, Time Series, Time Series Analysis, Time Series Anomaly Detection

In recent years, the emergence and development of third-party platforms have greatly facilitated the growth of the Online to Offline (O2O) business. However, the large amount of transaction data raises new challenges for retailers, especially anomaly detection in operating conditions. Thus, platforms begin to develop intelligent business assistants with embedded anomaly detection methods to reduce the management burden on retailers. Traditional time-series anomaly detection methods capture underlying patterns from the perspectives of time and attributes, ignoring the difference between retailers in this scenario. Besides, similar transaction patterns extracted by the platforms can also provide guidance to individual retailers and enrich their available information without privacy issues. In this paper, we pose an entity-wise multivariate time-series anomaly detection problem that considers the time-series of each unique entity. To address this challenge, we propose GraphAD, a novel multivariate time-series anomaly detection model based on the graph neural network. GraphAD decomposes the Key Performance Indicator (KPI) into stable and volatility components and extracts their patterns in terms of attributes, entities and temporal perspectives via graph neural networks. We also construct a real-world entity-wise multivariate time-series dataset from the business data of Ele.me. The experimental results on this dataset show that GraphAD significantly outperforms existing anomaly detection methods.
DistilCamemBERT: a distillation of the French model CamemBERT
Authors: Cyrile Delestre, Abibatou Amar
Date: 2022-05-23 00:00:00
Tasks: model

Modern Natural Language Processing (NLP) models based on Transformer structures represent the state of the art in terms of performance on very diverse tasks. However, these models are complex and represent several hundred million parameters for the smallest of them. This may hinder their adoption at the industrial level, making it difficult to scale up to a reasonable infrastructure and/or to comply with societal and environmental responsibilities. To this end, we present in this paper a model that drastically reduces the computational cost of a well-known French model (CamemBERT), while preserving good performance.
Learning to branch with Tree MDPs
Authors: Lara Scavuzzo, Feng Yang Chen, Didier Chételat, Maxime Gasse, Andrea Lodi, Neil Yorke-Smith, Karen Aardal
Date: 2022-05-23 00:00:00
Tasks: Reinforcement Learning (RL)

State-of-the-art Mixed Integer Linear Program (MILP) solvers combine systematic tree search with a plethora of hard-coded heuristics, such as the branching rule. The idea of learning branching rules from data has received increasing attention recently, and promising results have been obtained by learning fast approximations of the strong branching expert. In this work, we instead propose to learn branching rules from scratch via Reinforcement Learning (RL). We revisit the work of Etheve et al. (2020) and propose tree Markov Decision Processes, or tree MDPs, a generalization of temporal MDPs that provides a more suitable framework for learning to branch. We derive a tree policy gradient theorem, which exhibits a better credit assignment compared to its temporal counterpart. We demonstrate through computational experiments that tree MDPs improve the learning convergence, and offer a promising framework for tackling the learning-to-branch problem in MILPs.
An improved neural network model for treatment effect estimation
Authors: Niki Kiriakidou, Christos Diou
Date: 2022-05-23 00:00:00

Nowadays, in many scientific and industrial fields there is an increasing need for estimating treatment effects and answering causal questions. The key for addressing these problems is the wealth of observational data and the processes for leveraging this data. In this work, we propose a new model for predicting the potential outcomes and the propensity score, which is based on a neural network architecture. The proposed model exploits the covariates as well as the outcomes of neighboring instances in training data. Numerical experiments illustrate that the proposed model reports better treatment effect estimation performance compared to state-of-the-art models.
Generalization, Mayhems and Limits in Recurrent Proximal Policy Optimization
Authors: Marco Pleines, Matthias Pallasch, Frank Zimmer, Mike Preuss
Date: 2022-05-23 00:00:00
Tasks: Benchmarking, Deep Reinforcement Learning

At first sight it may seem straightforward to use recurrent layers in Deep Reinforcement Learning algorithms to enable agents to make use of memory in the setting of partially observable environments. Starting from widely used Proximal Policy Optimization (PPO), we highlight vital details that one must get right when adding recurrence to achieve a correct and efficient implementation, namely: properly shaping the neural net's forward pass, arranging the training data, correspondingly selecting hidden states for sequence beginnings and masking paddings for loss computation. We further explore the limitations of recurrent PPO by benchmarking the contributed novel environments Mortar Mayhem and Searing Spotlights that challenge the agent's memory beyond solely capacity and distraction tasks. Remarkably, we can demonstrate a transition to strong generalization in Mortar Mayhem when scaling the number of training seeds, while the agent does not succeed on Searing Spotlights, which seems to be a tough challenge for memory-based agents.
Bézier Flow: a Surface-wise Gradient Descent Method for Multi-objective Optimization
Authors: Akiyoshi Sannai, Yasunari Hikima, Ken Kobayashi, Akinori Tanaka, Naoki Hamada
Date: 2022-05-23 00:00:00
Tasks: PAC learning

In this paper, we propose a strategy to construct a multi-objective optimization algorithm from a single-objective optimization algorithm by using the B\'ezier simplex model. Also, we extend the stability of optimization algorithms in the sense of Probability Approximately Correct (PAC) learning and define the PAC stability. We prove that it leads to an upper bound on the generalization with high probability. Furthermore, we show that multi-objective optimization algorithms derived from a gradient descent-based single-objective optimization algorithm are PAC stable. We conducted numerical experiments and demonstrated that our method achieved lower generalization errors than the existing multi-objective optimization algorithm.
YouTube Ad View Sentiment Analysis using Deep Learning and Machine Learning
Authors: Tanvi Mehta, Ganesh Deshmukh
Date: 2022-05-23 00:00:00
Tasks: BIG-bench Machine Learning, Data Visualization, Sentiment Analysis

Sentiment Analysis is currently a vital area of research. With the advancement in the use of the internet, the creation of social media, websites, blogs, opinions, ratings, etc. has increased rapidly. People express their feedback and emotions on social media posts in the form of likes, dislikes, comments, etc. The rapid growth in the volume of viewer-generated or user-generated data or content on YouTube has led to an increase in YouTube sentiment analysis. Due to this, analyzing the public reactions has become an essential need for information extraction and data visualization in the technical domain. This research predicts YouTube Ad view sentiments using Deep Learning and Machine Learning algorithms like Linear Regression (LR), Support Vector Machine (SVM), Decision Tree (DT), Random Forest (RF), and Artificial Neural Network (ANN). Finally, a comparative analysis is done based on experimental results acquired from different models.
Beyond EM Algorithm on Over-specified Two-Component Location-Scale Gaussian Mixtures
Authors: Tongzheng Ren, Fuheng Cui, Sujay Sanghavi, Nhat Ho
Date: 2022-05-23 00:00:00
Tasks: Open-Ended Question Answering, parameter estimation

The Expectation-Maximization (EM) algorithm has been predominantly used to approximate the maximum likelihood estimation of the location-scale Gaussian mixtures. However, when the models are over-specified, namely, the chosen number of components to fit the data is larger than the unknown true number of components, EM needs a polynomial number of iterations in terms of the sample size to reach the final statistical radius; this is computationally expensive in practice. The slow convergence of EM is due to the missing of the locally strong convexity with respect to the location parameter on the negative population log-likelihood function, i.e., the limit of the negative sample log-likelihood function when the sample size goes to infinity. To efficiently explore the curvature of the negative log-likelihood functions, by specifically considering two-component location-scale Gaussian mixtures, we develop the Exponential Location Update (ELU) algorithm. The idea of the ELU algorithm is that we first obtain the exact optimal solution for the scale parameter and then perform an exponential step-size gradient descent for the location parameter. We demonstrate theoretically and empirically that the ELU iterates converge to the final statistical radius of the models after a logarithmic number of iterations. To the best of our knowledge, it resolves the long-standing open question in the literature about developing an optimization algorithm that has optimal statistical and computational complexities for solving parameter estimation even under some specific settings of the over-specified Gaussian mixture models.
WOGAN at the SBST 2022 CPS Tool Competition
Authors: Jarkko Peltomäki, Frankie Spencer, Ivan Porres
Date: 2022-05-23 00:00:00

WOGAN is an online test generation algorithm based on Wasserstein generative adversarial networks. In this note, we present how WOGAN works and summarize its performance in the SBST 2022 CPS tool competition concerning the AI of a self-driving car.
Wasserstein Generative Adversarial Networks for Online Test Generation for Cyber Physical Systems
Authors: Jarkko Peltomäki, Frankie Spencer, Ivan Porres
Date: 2022-05-23 00:00:00

We propose a novel online test generation algorithm WOGAN based on Wasserstein Generative Adversarial Networks. WOGAN is a general-purpose black-box test generator applicable to any system under test having a fitness function for determining failing tests. As a proof of concept, we evaluate WOGAN by generating roads such that a lane assistance system of a car fails to stay on the designated lane. We find that our algorithm has a competitive performance respect to previously published algorithms.
Falsification of Multiple Requirements for Cyber-Physical Systems Using Online Generative Adversarial Networks and Multi-Armed Bandits
Authors: Jarkko Peltomäki, Ivan Porres
Date: 2022-05-23 00:00:00
Tasks: Multi-Armed Bandits

We consider the problem of falsifying safety requirements of Cyber-Physical Systems expressed in signal temporal logic (STL). This problem can be turned into an optimization problem via STL robustness functions. In this paper, our focus is in falsifying systems with multiple requirements. We propose to solve such conjunctive requirements using online generative adversarial networks (GANs) as test generators. Our main contribution is an algorithm which falsifies a conjunctive requirement $\varphi_1 \land \cdots \land \varphi_n$ by using a GAN for each requirement $\varphi_i$ separately. Using ideas from multi-armed bandit algorithms, our algorithm only trains a single GAN at every step, which saves resources. Our experiments indicate that, in addition to saving resources, this multi-armed bandit algorithm can falsify requirements with fewer number of executions on the system under test when compared to (i) an algorithm training a single GAN for the complete conjunctive requirement and (ii) an algorithm always training $n$ GANs at each step.
TempLM: Distilling Language Models into Template-Based Generators
Authors: Tianyi Zhang, Mina Lee, Lisa Li, Ende Shen, Tatsunori B. Hashimoto
Date: 2022-05-23 00:00:00
Tasks: Text Generation

While pretrained language models (PLMs) have greatly improved text generation, they have also been known to produce unfaithful or inappropriate content. In contrast, classic template-based systems provide strong guarantees of faithfulness at the cost of fluency. We propose TempLM, which achieves the best of both worlds by distilling a PLM into a template-based generator. On the E2E and SynthBio data-to-text datasets, we show that TempLM is more faithful than the original PLM and is more fluent than prior template systems. Notably, on an out-of-domain evaluation, TempLM reduces a finetuned BART model's unfaithfulness rate from 83% to 0%. In a human study, we find that TempLM's templates substantially improve upon human-written ones in BERTScore.