Motivating Physical Activity via Competitive Human-Robot Interaction
Abstract: This project aims to motivate research in competitive human-robot interaction by creating a robot competitor that can challenge human users in certainscenarios such as physical exercise and games.
With this goal in mind, we introduce the Fencing Game, a human-robot competition used to evaluate both thecapabilities of the robot competitor and user experience.
We develop the robotcompetitor through iterative multi-agent reinforcement learning, and show that itcan perform well against human competitors.
Our user study additionally foundthat our system was able to continuously create challenging and enjoyable interactions for humans and the majority of human subjects considered the system tobe entertaining and useful for improving the quality of their exercise.
Introduction
Competition is ubiquitous in the natural world [1, 2] and in human society [3, 4, 5].
Despite itsuniversality, competitive interaction has rarely been investigated in the ﬁeld of Human Robot Interaction, which has mainly focused on cooperative interactions such as collaborative manipulation,mobility assistance, feeding, and so on [6, 7, 8, 9, 10].
In some ways it is not surprising that competitive interaction has been overlooked: of course everyone wants a robot that can assist them; whowould want a robot that thwarts their intentions?
Yet, we also accept that human-human competitioncan be healthy and productive, for example in structured contexts such as sports.
In this paper weexplore the idea that human-robot competition can provide similar beneﬁts.
We believe that physical exercise settings such as athletic practice, ﬁtness training, and physicaltherapy are scenarios in which competitive HRI can beneﬁt users.
Physical exercise is essential toour physical and mental health.
In addition, the quality of some physical rehabilitation is closelyrelated to patients’ commitment to exercising regularly.
However, the major cause of poor adherenceto physical exercise is due to a lack of motivation [11, 12, 13].
It has been shown that enjoyment,competition, and challenges are important motivational factors for the practice of physical exercise[14, 12].
In this project, our goal is to create a robot that can provide these motivations to humanusers via adversarial behaviors.
The main contributions of this paper are listed below:
Motivating Competitive-HRI Research.
We discuss how competitive interaction can inﬂuencepeople in a positive manner and the technical difﬁculties competitive-HRI tasks represent.
Thisdiscussion motivates our proposed Fencing Game, a competitive physically interactive game, as anevaluation environment for competitive-HRI algorithms and user study.
System Design and Implementation.
A highly generalizable robotic system is created to supportour competitive-HRI research.
A multi-agent reinforcement learning(RL) method is used to train arobot to play competitive games with multiple gameplay styles.
Two User Studies.
Our ﬁrst user study found that more than 80% of subjects found competitiveinteraction with our robot to be entertaining and desirable.
Our system was able to provide chal- lenging gameplay experiences that signiﬁcantly increase human players’ heart rates.
Furthermore, we observed that human subjects who constantly explore different strategies tend to achieve higherrewards in the long run.
A second user study demonstrated that an RL-trained policy made it signiﬁcantly more challenging for subjects to make quantitative improvement in a long sequence of gamescompared to a carefully designed heuristic baseline policy.
The RL-trained agent also appeared tobe more intelligent to the human subjects.
Competitive Interaction
In this section, we discuss the signiﬁcance of competitive interaction and justify why we believe thatit deserves increased attention in the HRI community.
We will ﬁrst discuss how competition caninﬂuence people in a positive manner from a psychological perspective.
Afterward, we will discussthe technical challenges in competitive-HRI tasks and propose the use of the Fencing Game as themain task that this project will focus on.
Positive Inﬂuences of Competitive Interaction.
Competition between players provides motivationand can foster improvement in performance at a given task.
Plass et al.
[15] compared competitiveversus cooperative interactions in an educational mathematics video game.
The results revealed that,compared to working individually, the subjects performed signiﬁcantly better when working competitively than cooperatively.
In particular, competitive players demonstrated higher effectivenessin problem solving compared to non-competitive players.
This study also observed that subjectsexperienced a higher level of enjoyment via an increased tendency to engage in the game [16].
Furthermore, they also displayed higher situational interest, i.e. the subject paid more attention andhad more interaction during the game [17].
Viru et al.
[18] showed that competitive exercises canimprove athletic performance in a treadmill running test.
Subjects’ average running duration wasincreased by 4.2%.
During cycling and planking exercise, Feltz et al.
[4] were able to inspire higherperformance from subjects by placing them in competition with a manipulated virtual partner.
Inspired by these studies, we envision that a personal robot can become a competitive partner thatprovides enjoyment, increases motivation, and motivates improvement in activities such as physicalexercise.
For this reason, we initiate our competitive-HRI research by focusing on creating a physicalexercise companion.
Technical Challenges.
Creating an actual robot that can compete with a human physically is challenging.
The robot needs to constantly reason about the human’s intent via their actions, and strategically control its high degree-of-freedom body to counteract the opponent’s adversarial behavior andmaximize its own return.
Therefore, a big part of our competitive-HRI research focuses on solvingthese technical challenges to create a robotic system with real-time decision making capability andbody agility that is comparable to that of humans.
The Fencing Game.
Based on the expected technical challenges, we designed a two-player zerosum physically interactive game.
The Fencing Game is an attack and defense game where the humanplayer is the antagonist with the goal of maximizing their game score.
The robot is the protagonistwho aims to minimize the antagonist’s score.
Fig.
1 shows three images of human subjects playingthe game, and the algorithm in Appendix A.1 summarizes the scoring mechanism for this game.
The orange spherical area located between the two players denotes the target area of the game.
Theantagonist on the right earns 1 point for every 0.01 seconds that their bat is placed within the targetarea without contacting the opponent’s bat.
The antagonist will lose 10 points if the antagonist’sbat is placed within the target area and makes contact with the protagonist’s bat simultaneously.
Moreover, the antagonist will get 10 points of reward if the protagonist’s bat is placed within thetarget area, waiting for the antagonist to attack, for more than 2 seconds.
Each game will last for 20seconds.
The observation space for both agents includes the Cartesian pose and velocity of the twobats, as well as the game time in seconds.
For the sake of simplicity, both agents in this project arenon-mobile.
Yet, mobility can be easily integrated into future iterations of this game.
Related Work
Reinforcement Learning in Competitive Games.
Competitive games have been used as benchmarks to evaluate the ability of algorithms to train an agent to make rational and strategic decisions [19, 20, 21].
Multi-agent reinforcement learning methods allow agents to learn emergent andcomplex behavior by interacting with each other and co-evolving together [22, 23, 24, 25, 26, 27].
Many recent efforts have used multi-agent RL methods to learn continuous control polices that canachieve high complexity tasks.
Bansal et al.
[28] created control policies for simulated humanoidand quadrupedal robots to play competitive games such as soccer and wrestling.
Lowe et al.
[29]has extended DDPG [30] to multi-agent settings by employing a centralized action-value function.
Human-Robot Competition.
There are a few studies focusing on human-robot interaction in competitive games.
For instance, Kshirsagar et al.
[31] studied the affect of “co-worker” robots onhuman performance when they are performing in the same environment and competing for a monetary prize.
This study showed that humans were slightly discouraged when competing against ahigh performing robot.
On the other hand, humans exhibited a positive attitude towards the low performing robot.
Mutlu et al.
[32] showed that male subjects were more engaged in competitive videogames when they played with an ASIMO robot.
However, the majority of the subjects preferredcooperative games when they played with this robot.
Short et al.
[33] analysed the “rock-paperscissors” game and found that human subjects were more socially and mentally engaged when therobot cheated during the game.
Robots in Physical Training.
In the context of using robots to assist humans in physical exercises,a robot developed by Fasola and Mataric [34] was able to provide real-time coaching and encouragement for a seated arm exercise.
S ¨ ussenbach et al.
[35] developed a motivational robot for indoorcycling exercise that employed a set of communication techniques in response to the subject’s physical condition.
The results showed that the robot sufﬁciently increased the users’ workout efﬁciencyand intensity.
Sato et al.
[36] created a system capable of imitating the motion and strategy of topvolleyball blockers for assisting vollyball training.
These works are typically limited to a few competitive scenarios that require simple and repetitive motions.
In this work, we leverage reinforcement learning to create a robotic system that canpotentially play various physically competitive games against human players.
System Design and Implementation
Humans are highly efﬁcient at recognizing patterns and learning skills from just a small number ofexamples [37].
Existing research also shows that human subjects can adapt to robots and improvetheir performance in just a few trials in physical HRI tasks [38, 39].
However, games against a robotcompetitor are less challenging if a human player can easily predict its behavior and quickly ﬁnd anoptimal counter-strategy.
We hypothesized that human players can quickly learn to improve theirperformance against a given robot policy, but a change in the robot’s gameplay style could interruptthis learning effect and keep the games challenging.
A gameplay style is characterized by patternsin the agent’s end-effector motion trajectories.
For example, one antagonist may prefer using morestabbing movements, while another antagonist may prefer slashing movements.
Therefore, there aretwo primary objectives that govern the generation of robot control policies.
First, a policy shouldallow the robot to play the Fencing Game sufﬁciently well, such that the games are intense andengaging to the human users.
Second, we propose to obtain three policies with unique gameplaystyles for our user study.
A multi-agent reinforcement method created the robot control policiesthat are used in the user study.
We designed and implemented the physical system based on a PR2general-purpose robot.
The implementation of our physical system is detailed in Appendix A.2.
4.1 Learning to Compete
To generate robot control policies that comply with the two aforementioned requirements, we formulate the Fencing Game as a multi-agent Markov game problem [40].
Both the antagonist agent andthe protagonist agent are represented by a PR2 robot model in a Mujoco simulation environment.
Both agents are trained in a co-evolving manner by playing games against each other.
We breakdown the multi-agent proximal policy optimization method proposed by Bansal et al.
[28] into twoseparate training processes, which reduces the computational processing needed to obtain multiplepairs of agents with acceptable performance.
As a result, we were able to complete all samplingand training processes on a desktop computer (cpu: 1 × i 7 , gpu: 1 × GTX 970 ).
Our version ofmulti-agent PPO, the two-phase iterative training algorithm, is presented in Algo.
Learning to Move and Play.
The ﬁrst phase of the training can be seen as a pre-training process,which aims to allow both agents to quickly learn the motor skills required for joint control and therules of the game.
The agents are rewarded by both the weighted sum of a continuous reward andthe game score at each timestep.
The continuous reward encourages the agents’ exploration in thetask space.
The policy of the antagonist µ with parameters θ µi will ﬁrst be trained by collectingtrajectories that result from playing against the protagonist with its most recent policy.
This processcontinues until timeout or µ has converged.
The protagonist’s policy ν with parameter θ νi willthen be trained against the antagonist with its most recent policy.
We were able to acquire robotpolicies that play the Fencing Game sufﬁciently well by only running this training sequence twice( N iter = 2 ).
The pair of resulting policies from phase one will be called the warm-start policies.
Creating Characterized Policies.
It has been shown that by using different random seeds to guidepolicy optimization toward different local optima, an agent can learn different behaviors and approaches to complete the same task [41, 26].
The highly variable nature of multi-agent systemsenhances this random effect.
The agents in multi-agent systems are more likely to learn drastically different strategies and emergent behaviors when they continuously learn by competing with eachother [28, 42, 43, 44].
The second phase of training generates a policy with random characteristicsby exploiting this fact.
Both agents are initialized with their warm-start policies from phase oneand trained in the same iterative scheme shown in Algo.
Now the agents are solely rewardedby the game scores.
When training each of the agents, instead of having an agent face against theopponent’s latest policy, one of the previous versions of the opponent’s policy in the history will berandomly selected in phase two.
We created a policy library that contains six pairs of randomly characterized policies that resulted from six different rounds of phase two training.
Fig.
2 demonstratesthe change of game scores during the two-phase training process.
4.2 Selecting Agents With Distinct Gameplay Styles
Algorithm
Input: Environment E ; Stochastic policies µ and ν ; Instantaneous Reward Function r( · )
Initialize: Parameters θ µ 0 for µ and θ ν 0 for ν for i = 1 , 2 , ..
N iter do if phase one training then θ νi ← θ νi −else θ νi ← θ νrandom from history endfor j = 1 , 2 , ..
N µ do rollout ← roll ( E , µ θ µi , ν θ νi − 1 , r( · ) )
θ µi ← PPO Update(rollout) endif phase one training then θ µi ← θ µi else θ µi ← θ µrandom from history endfor j = 1 , 2 , ..
N ν do rollout ← roll ( E , µ θ µi , ν θ νi , r( · ) )
θ νi ← PPO Update(rollout) end end
In order to identify the most distinctive protagonist policies in the library described in the lastsubsection, each agent’s gameplay style needsto be quantiﬁed and compared.
We ﬁrst generate trajectories for all protagonist policies ina tournament [45], where each agent plays 100games with each of the six opponents.
Eightend effector trajectory features are selected toquantify agent’s gameplay styles: total displacement change on x , y , and z axis, averagevelocity, average acceleration, average jerk, total kinetic energy, and trajectory smoothness.
These features are calculated for each of thegames played by each of the protagonists.
Thequantiﬁed style of a protagonist agent is theaveraged features across 600 games.
The features of all protagonists are then compared viatheir three most signiﬁcant principal components (less than 2% of information loss)[46],and the three most separable policies are selected for the user study.
The protagonist’sgameplay style for the warm-start policy andthe three selected characterized policies are visualized in Fig.
4.3 Sim2Real
Due to the kinematic and dynamic mismatch between simulation and reality, policies that are trainedsolely on simulated data can perform poorly in reality.
We use a combination of a Jacobian Transpose end-effector controller and a system identiﬁcation (systemID) process to solve this problem.
Instead of specifying the torque values for each joint, the policy outputs an offset from the current end-effector pose.
The new desired pose is executed by the end-effector controller.
We usedthe CMA-ES algorithm to optimize the following objective over the parameter space of both thecontroller and the robot model in the simulation.
[Equation]
Where θ m represents the simulated robot model parameters: damping, armature, and friction
loss.θ c represents the proportional gains and derivative gains of the end-effector controller.
T r and T sare trajectories sampled from the real robotic system and simulation respectively that result fromthe same control sequence.
s tr ∈ T r and s ts ∈ T s are the robot’s end-effector pose in reality andsimulation at time t respectively.
As a result, the difference in end-effector dynamics between thesimulated robot and the real PR2 robot is reduced.
The maximum controller output is boundedconservatively to prevent possible human injury.
[Table]
Experiments and Analysis
We performed two in-lab user studies in this work. The ﬁrst user study performed a broad exploration on the idea of competitive-HRI under the fencing game setting. Sixteen human subjects wereasked to play ﬁve games with each of the four RL trained policies that resulted from Sec. 4. Subjects’game scores, heart rates, arm movements, and their responses to a modiﬁed technology acceptancemodel (TAM)[47] were used to evaluate our system from the following three perspectives: 1.
Is acompetitive robot accepted by human users under the scenarios of physical games and exercise?
Can our system effectively create challenging and intense gameplay experience?
Can the robotinterrupt the human learning effect by switching its gameplay style?
The second user study compared characterized policy 1 with a carefully designed heuristic-basedpolicy.
By placing its bat in between the target area and the point on the human’s bat that is closestto the target area, the robot exploits embedded knowledge of the game’s rules in order to execute astrong baseline heuristic policy.
Action noise was added to the heuristic policy to create randomnessin the robot’s behavior.
The design details of the heuristic-based policy are described in Appendix
A.3.
Ten human subjects were asked to play 10 games with each robot policy.
This experiment compared the two policies via game scores, subjects’ TAM responses, subjects’ perception of difﬁculty,enjoyment, and robot intelligence.
Details about the experiment procedures, subjective questiondesign, and the demographic information for both user studies are discussed in Appendix A.3.
5.1 User Study One Result: A Broad Exploration
Our ﬁrst experiment demonstrates that participants highly accept the use of a competitive robot asan exercise partner.
The majority of the subjects considered competitive games with our robot to beuseful, entertaining, desirable, and motivating.
Our system was able to provide a challenging andintensive interactive experience that signiﬁcantly increased subjects’ heart rates.
While competingagainst our RL trained robot, most subjects struggled to signiﬁcantly improve their performanceover time.
Yet, a subset of subjects who constantly explored different strategies achieved higherscores in the long run.
User Acceptance.
Table 1 summarizes the subjects’ responses to the technology acceptance model.
The majority of the subjects ( i.e. 68.75%) agreed that a competitive robot could improve the quality of their physical exercise.
Moreover, 87.5% of subjects agreed that competitive human-robotinteractive games are entertaining, and 81.25% of subjects agreed that a competitive robot exercise companion would be desirable in the future.
Interestingly, the intention to use (62.5% agree +strongly agree) and increased engagement (56.25% agree + strongly agree) metrics are not as highas the vast agreement on perceived enjoyment and desirability.
Some subjects who didn’t have astrong intention to use our system explained their reasons in the open-ended question.
They statedthat it is not immediately clear how competitive robots can play a role in their routine exercises, suchas jogging, and weight training.
On the other hand, most subjects ( i.e. , 71%) who exercise less thanthree hours per week agreed that a competitive robot partner will increase their engagement withphysical exercise.
Therefore, our competitive robot is more effective in motivating people who dorelatively little exercise.
Future research should explore how to effectively apply competitive-HRI
Increased Heart Rate.
Most of the gameplay with our competitive robot increased the humansubjects’ heart rates signiﬁcantly.
Subjects’ peak heart rates were higher than their resting heartrates in more than 99% of the games, and their peak heart rates in 92.6% of the games were higherthan their walking baseline heart rates.
Since the subjects were asked to keep their feet planted onthe ground during a game, their body maneuverability was very limited.
Despite this movementlimitation, subjects’ peak heart rates were signiﬁcantly higher ( i.e. , 20% to 58%) than their walkingbaseline in 29% of the games.
We found that, other than physical effort, subjects’ cognitive effortand reported emotions also corresponded to a rise in heart rate.
Figure 4.
b. shows that sections withhigher average heart rate corresponded to people feeling cognitive demand, motivated, frustrated,and intimidated by the robot.
In short, playing competitive games with our robot can be cognitivelydemanding and can trigger noticeable emotional reactions.
The Human Learning Effect.
As mentioned in Sec. 4, we wanted to test if changing the robot’sgameplay style interrupts the human learning process and keeps the interaction challenging throughout the whole experiment.
This hypothesis was based on our assumption that most human playerscan make signiﬁcant improvement within ﬁve games against a ﬁxed robot policy.
Surprisingly,this assumption was invalidated by our user study data.
Fig.
c. shows the average game scoresand standard deviation of all subjects in the ﬁve consecutive games against each of the four robotpolicies.
An analysis of variance (ANOVA) suggests that subjects have no signiﬁcant performanceincrease between the ﬁve games for a given policy (Warm-start Policy: p = 0 .
96 , Characterized
Policy 1: p = 0 .
74 , Characterized Policy 2: p = 0 .
85 , Characterized Policy 3: p = 0 . 43 ).
Thered horizontal dashed line in each subplot of Fig.
c. represents the best mean score achievedby a subject, which approximates the performance of an observed best human policy.
The averageperformance of all subjects were lower than ( i.e. , much lower in most cases) the performance of thebest human policies, and most subjects still have much room for performance improvement.
Ourassumption on human learning was based on evidence from noncompetitive HRI experiments.
Incontrast, our competitive setting creates a much more dynamical environment and resulted in a morechallenging learning environment for the human.
Since no signiﬁcant performance variance wasobserved across the four sections either (ANOVA F = 1 . 51 , p = 0 . 26 ), our system was able to remain challenging to users throughout the whole experiment.
However, more studies are needed toanalyze how changes in gameplay style interrupt human learning.
Human Performance.
Despite the fact that no signiﬁcant learning effect was found in the subjectpopulation, we observed that some subjects tried multiple strategies in the experiment, which couldbe interpreted as exploration within a learning framework.
This observation motivated us to analyzethe relationship between strategy exploration and performance, which we ﬁrst quantiﬁed as variancein game scores, and then as the featurized gameplay styles from Sec. 4.
As shown in Fig 4.
a., a largervariance in score corresponds to better performance in terms of maximum and mean scores.
We thencompared the variance of the gameplay style between the ﬁve subjects with the highest maximumscores and the ﬁve subjects with the lowest maximum scores.
For the sake of simplicity, we onlycompared the end-effector displacement change on x , y , and z axes and the averaged velocity.
Thevariance of each selected feature for subjects with high performance were at least 29% higher thansubjects with low performance.
This suggested that subjects who have high variance in score tendto take risks by constantly exploring different strategies, resulting in better rewards in the long run.
Future research could examine whether a robot can help human users achieve better performance byverbally or implicitly encouraging them to explore different strategies. 5.2 User Study Two Result: Baseline Comparison
This experiment compared an RL trained policy (characterized policy 1) to a strong heuristic baseline policy in a long gameplay sequence setting (10 games per policy).
Compared to the baseline,the RL trained policy has a slightly better game score performance.
In contrast to the ﬁrst experiment, the human learning effect was observed in both policies.
However, the RL trained policywas signiﬁcantly better at suppressing human subjects from learning to make progress even withoutswitching gameplay styles during the experiment.
While the responses to the TAM model were verysimilar for both policies, the subjects considered the RL policy to be more intelligent because of its“defensive” and “diverse” behavior. Game Scores: When playing against the baseline policy, the subject population’s average gamescores ( Baseline mean: 383.5, RL mean: 349.1), the maximum game score ( Baseline max: 929.0, RL max: 744.0), and the minimum game score ( Baseline min: -291.0, RL min: -320.0) are higherthan than those against the RL policy. Instead of switching policies every ﬁve games, we used alonger sequence of game-plays to evaluate each policy in this study.
We were able to ﬁnd a positivecorrelation between game scores and the amount of game-play experience against a speciﬁc policy.
A linear regression over the Baseline method data (slope=30.0, coefﬁcient=0.56, p value= 9 . 3 e- 10 )has a larger positive slope, stronger correlation coefﬁcient, and a smaller statistical signiﬁcant pvalue compared to that of the RL policy (slope=15.7, coefﬁcient=0.34, p value=0.0005).
Subjective Responses: Subjects’ responses to most of the modiﬁed TAM questions for the twopolicies are very similar as shown in Table A.5.
However, 70% of the population considered the
RL policy to be more intelligent.
In the responses for short questions 2, 3, and 4 in Table 2, the
Baseline policy was described as “fast” by 2 subjects, “follows my movement” by 4 subjects, andhas “repetitive/predictable” behavior by 4 subjects.
Meanwhile, the RL policy was considered to be“defensive” by 5 subjects, “strategic” by 2 subjects, and to have “diverse behavior” by 2 subjects.
This work motivated research in competitive-HRI by discussing how competition can be beneﬁcialto people and the technical difﬁculties competitive-HRI tasks represent.
The Fencing Game, a physically interactive zero-sum game is proposed to evaluate system capability in certain competitive-HRIscenarios.
We created a competitive robot using an iterative multi-agent RL algorithm, which is ableto challenge human users in various competitive scenarios, including the Fencing Game.
Our ﬁrstuser study found that human subjects are very accepting of a competitive robot exercise companion.
Our competitive robot provides entertaining, challenging, and intense gameplay experiencesthat signiﬁcantly increases the subject’s heart rate.
In our second user study, one of the policiesresulting from the proposed RL method was compared to a strong heuristic baseline policy. The RL-trained policy outperformed the baseline policy in terms of game scores and suppression of thehuman learning affect, and appeared to be more intelligent to 70% of the population. A Appendix
COVID-19 Effects.
We were fortunate to be able to perform an in-person experiment on a realworld robot and human subjects, yet the subject recruiting process was particularly challengingduring the current COVID pandemic.
With limited access to potential experiment participants, wewere only able to run a pilot test with two people prior to the actual experiment reported here.
Solelyfrom the pilot study data, we were not able to disqualify our eventual experimental assumption thatthere is a signiﬁcant learning effect among participants even within only ﬁve games.
Instead, asdescribed in Sec. 5.1, it was not until obtaining our full experimental results that we were able todiscard this assumption due to observing a large variance in the degree of human learning acrossparticipants.
Future research that aims to understand the human learning effect in competitive-HRItasks should collect more data for each individual and facilitate the participant’s ability to learn.
In particular, increased learning can be achieved by reducing environmental complexity, such asdecreasing the robot’s joint velocity, reducing the manipulator’s reachability in task space, using arobot with a lower degree of freedom (DoF) and so on.
A.1 The Fencing Game
Algo.
2 summarizes the scoring mechanism for the Fencing Game described in Sec. 2.
Algorithm
Initialize:
Game score s = 0; Timestep = 0.01 Sec; Game horizon =
20 Sec bat a → Antagonist’s bat bat p
→ Protagonist’s bat target → Target Area for every timestep in this game do if bat a in target then if bat a contacts bat p then s -=else s +=end if bat p in target for more than 200 consecutive timesteps then s += 10 end
A.2 System Implementation Details
The PR2 robot has two 7 degree of freedom (DoF) arms and its overall form-factor is similar to ahuman adult.
It is comparable to a human player in a competitive game in terms of body size andarm ﬂexibility.
The human player’s bat is attached to an infrared photo-diode array tracker, and itsposition and orientation is perceived by the robot via two tracking base stations.
An audible scoringfeedback system is created to report the scoring situation of each game in real-time.
The participantswill hear a higher frequency (440 Hz) signal sound when scoring, and a lower frequency (300 Hz)signal sound when receiving penalty from the robot.
The system implementation pipeline is shownin Fig.
Algorithm Details.
This paragraph provides extra technical details on the two-phase iterative training algorithm described in Sec. 4.
There are two major differences between phase one and twotraining in Algo.
(1) Phase one training uses a continuous reward to facilitate agents’ development of basic motor skills.
In phase two training, the agents are solely rewarded by the game scores.(2)
In phase one training, each agent is always learning against the latest version of the opponent.
But in phase two training, an agent would constantly and randomly load a previous version of opponent from the history, after a short period of training.
The use of continuous reward encouragesthe robot to quickly explore the task space, which makes the phase one training good for quicklyinitializing the policies for the antagonist and protagonist.
However, continued used of the iterationstrategy in phase one training would likely trap both agents in a low quality local equilibrium and/orchasing each other in circles in parameter space in a long sequence of training [48].
In contrast, thereward and iteration mechanisms of phase two training create a high variance learning environment which effectively mitigates the circling problem.
In addition, because of the high variance natureof the phase two training, agents are more likely to learn emergent behaviors and converge to moresophisticated policies.
A.
[Figure]
Demographics There were 16 human subjects (10 male, 6 female, of age M = 28 .
8 years, SD =5 . 56 ) recruited for the ﬁrst human-subject studies.
Nine out of 16 subjects reported doing more thanthree hours of physical exercise per week, and seven subjects reported less than 3 hours of weeklyexercise.
Jogging, walking, cardio, and weight training were the most common exercises chosenamong all subjects.
Ten subjects from this population participated the second user study.
Before the Experiment.
Before the experiment, each subject was asked to sit for 3 minutes andthen walk for 1 minute to record two average heart rate baseline values.
In the experiment, both therobot and a subject held a polystyrene bat to play the games.
A safety line was drawn on the ground,and every subject stood behind it to prevent a potential collision.
Since the robot was not mobile,each subject also kept his/her feet planted on the ground during a game.
The target area was notdirectly visible; an audible scoring feedback system notiﬁed a subject when his/her bat was placedwithin the target area.
A high-frequency signal sound indicated that the human player was scoring,and a low-frequency signal sound indicated that the human player was getting penalization.
Beforethe experiment, a subject had 5 minutes to explore this target area with their bat so that the targetarea’s location and the scoring mechanism were clear to the subject.
Afterward, the subject playedtwo warm-up games with the robot to get further familiarized with the system.
In these warm-upgames, the robot’s actions were slowed down and no data was collected.
Experiment Procedure for User Study One.
The experiment contains four sections.
In eachsection, a subject will play ﬁve consecutive games with the robot (with a ﬁxed robot policy) and restfor approximately 30 seconds between games.
Subjects’ heart rates will be recorded during each20 second game.
Due to the short duration of the games, our discussions in Sec.
5.1 use peak heartrate as a summarizing statistic of this recorded data.
In the ﬁrst section of the experiment, the robotwill use the warm-start policy resulting from the phase one training.
For the rest of the sections, therobot will load one of the three selected characterized policies in each section by following a randomorder.
This ensures that all subjects will ﬁrst learn to play with the robot at a regular speed, and thenplay with the robot with a new game style in each of the sections.
After each section, a subject willbe asked to describe the interaction in the last 5 games by selecting one or more of the followingadjectives: ‘Exciting’, ‘Joyful’, ‘Frustrating’, ‘Motivating’, ‘Amusing’, ‘Intimidating’, ‘Physically
Demanding’, ‘Cognitively Demanding’, ‘Boring’, ‘Others (please describe: )’.
Finally, when asubject ﬁnishes all four sections of games, they will complete the last part of the questionnaire thatassesses their acceptance of the competitive robot, and their subjective feelings towards the games.
We modiﬁed the technology acceptance model (TAM) [47] and created the questions in Table.
Weintroduced two extra questions ( i.e. , DE and IE) to understand the human subjects’ acceptance and desirability of a competitive robot companion in the future, and if a competitive robot can motivatethem to engage in physical exercise more frequently.
{+Other than the open-ended question, all TAMquestions were measured on a 5-point scale where 1 = “Strongly Disagree,” 
+}
= “Neutral,” 5 =“Strongly Agree”.
At the end of each gameplay section, a subject is also asked to consider andcompare both the enjoyability and the difﬁculty of the games between the ﬁnished sections.
Onesection can be equally, less or more enjoyable and difﬁcult than another section.
This will allowthe participant to have two rankings of the four sections based on their perceived enjoyment anddifﬁculty by the end of the experiment.
Experiment Procedure for User Study Two.
This experiment contains two sections.
In eachsection, a subject is asked to play 10 consecutive games with the same robot policy and rest for approximately 15 seconds between games.
The order in which each subject plays against the baselinepolicy and characterized RL policy is randomized.
After each section, a subject is asked to answerthe modiﬁed TAM questions.
After the ﬁnal section, a subject is also asked to answer short questions2, 3, and 4 in Table.
Baseline Heuristic Policy.
We aimed to design a strong baseline heuristic policy to create an intense human robot gameplay experience.
Given an observation of the world, the robot orients itsbat perpendicular to the human’s bat with random angular offsets drawn uniformly from -25 to 25degrees on the x, y, and z axes.
In order to ensure that the robot is always executing a competitivedefense, the policy commands the robot to position the center of its bat in between the target areaand the point on the human’s bat that is closest to the target area:
[Equation]
− hum l ) /length
s (cid:1)(cid:1)
Where bot p , target , hum u and hum l represent the position of the robot’s bat frame, the centerof the target area, the upper end of human’s bat, and the lower end of human’s bat respectively.hum close indicates the point on the human’s bat that is closest to the center of the target area, andlength s indicates the length of a bat.
The function uniform (0 . 5 , 1) randomly determines howfar apart the robot’s bat should be from the human’s bat.
In addition, there is a 50% chance forthe robot to execute the desired bat position calculated from the last time step instead of the latestdesired pose.
The added uncertainties introduce randomness to the robot’s behavior.
This heuristicallows the robot to dominate the fencing game when it can move faster or as fast as the antagonist.
However, human subjects are able to move slightly faster than our PR2 robot, which leaves room forhuman subjects to discover counter strategies.
A.4 Heart Rate
[Figure]
Fig. 4.
b. compares the subjective descriptions between four groups of gameplay sections withdifferent levels of average human heart rates.
For each section in the user study, we ﬁrst calculatethe average peak heart rate over the corresponding ﬁve games in the section.
A section’s heart ratelevel l is calculated by dividing the section’s average peak heart rate by the corresponding user’swalking baseline heart rate, which results in a percentage value describing how much more or lessthe average section heart rate is compared to the baseline.
The low, medium, high, and ultra-highheart rate groups contain the sections that l ≤ 100% , 100% < l ≤ 120% , 120% < l ≤ 140% , and140% < l respectively.
A.5 Experimental Results
Perceived Ease of Use.
Interestingly, although we did not observe signiﬁcant performance improvement within each section nor between sections, 62.5% of subjects perceived that it was easy for themto make progress when playing against the robot.
Furthermore, some participants expressed a desireto beat the best score of previous participants.
It is possible that some subjects focused only onattaining their perception of a “high” score for a small number of games rather than maintaininggood performance in all 20 games.
On the other hand, in a competitive game setting, we are stillnot sure exactly what it means for people to feel that getting a better score would be easy.
In ourexperiment, it probably suggested that most participants considered the robot opponent to be surmountable because only a very small number of participants described their gameplay experience
[Table]
Perceived Usefulness
Having a competitive robot companion would improve the quality (PU) of my physical exercise.
Perceived Ease of Use
Learning to earn higher score (make progress) in the games with (PEOU) a competitive robot would be easy for me.
Attitude
Using a competitive robot exercise partner to improve my exercise (ATT) quality is a good idea.
Intention to Use
Assuming I had access to a competitive robot for exercise, (ITU)
I would intend to use it.
Perceived Enjoyment
I would ﬁnd competitive human-robot gameplays are (PENJ) entertaining.
Desirability
Based on your experience today, future physical exercises and (DE) games with a competitive robot will be desirable.
Increased Engagement
Having a competitive robot companion would make me more likely (IE) to engage in physical exercise.
Short Question
Are there anything you would like to change to improve the inter- (used in study one) action experience?
Short Question
Which robot (part 1, part 2, or equally) do you think is more (used in study two) challenging/difﬁcult to play against? and why?
Short Question
Which robot (part 1, part 2, or equally) do you think is more (used in study two) enjoyable/fun to play against? and why?
Short Question
Which robot (part 1, part 2, or equally) do you think is more (used in study two) intelligent?
and why?
[Table]
IE
[Table]
Comparison Between Baseline Policy and RL Policy in Pair-T Tests as “Frustrating”, “Intimidating”, or “Boring”.
However, future work could study how the perceivedease of use affects the participants’ effort in competitive games.
Enjoyment and Difﬁculty.
We found that perceived enjoyment and difﬁculty do not signiﬁcantlyvary as the amount of player experience increases, but the amount of data used to train the corresponding policy does have an affect.
Fig.
6 shows the average ranking comparison for enjoyabilityand difﬁculty across both policies and experiment sections.
Among the utilized policies, no significant variance was observed across the three characterized polices for both enjoyability( p = 0 . 40 )and difﬁculty( p = 0 . 35 ).
Paired-T tests show that the warm-start policy is less enjoyable and difﬁcult than characterized policies ( p < 0 . 01 ), except for the enjoyability of the warm-start policyand policy 2 ( p = 0 . 13 ).
The result is similar in the time domain - no signiﬁcant variance wasfound in the last three sections, in which characterized policies were randomly ordered (enjoyability: p = 0 .
5 , difﬁculty: p = 0 . 42 ).
Section one, which only uses the warm-start policy, is lessenjoyable and difﬁcult than other sections( p < 0 . 05 ).
Across all sections, the perceived difﬁculty ispositively correlated to perceived enjoyment with a moderate coefﬁcient of 0 . 6 .