The Concept of Criticality in Reinforcement Learning This paper introduces a novel idea in humanaided reinforcement learning  the concept of criticality. The criticality of a state indicates how much the choice of action in that particular state influences the expected return. In order to develop an intuition for the concept, we present examples of plausible criticality functions in multiple environments. Furthermore, we formulate a practical application of criticality in reinforcement learning: the criticalitybased varying stepnumber algorithm (CVS)  a flexible stepnumber algorithm that utilizes the criticality function, provided by a human, in order to avoid the problem of choosing an appropriate stepnumber in nstep algorithms such as nstep SARSA and nstep Tree Backup. We present experiments in the Atari Pong environment demonstrating that CVS is able to outperform popular learning algorithms such as Deep QLearning and Monte Carlo.

Entropic Risk Measure in Policy Search With the increasing pace of automation, modern robotic systems need to act in stochastic, nonstationary, partially observable environments. A range of algorithms for finding parameterized policies that optimize for longterm average performance have been proposed in the past. However, the majority of the proposed approaches does not explicitly take into account the variability of the performance metric, which may lead to finding policies that although performing well on average, can perform spectacularly bad in a particular run or over a period of time. To address this shortcoming, we study an approach to policy optimization that explicitly takes into account higher order statistics of the reward function. In this paper, we extend policy gradient methods to include the entropic risk measure in the objective function and evaluate their performance in simulation experiments and on a realrobot task of learning a hitting motion in robot badminton.