Original: C ROWD P LAY : C ROWDSOURCING HUMAN DEMONSTRATIONS FOR OFFLINE LEARNING
Revised: C ROWD P LAY : C ROWDSOURCING HUMAN DEMONSTRATIONS FOR OFFLINE LEARNING

Original: Anonymous authors
Revised: Anonymous authors

Original: Paper under double-blind review
Revised: Paper under double-blind review

Original: A BSTRACT
Revised: A BSTRACT

Original: Crowdsourcing has been instrumental for driving AI advances that rely on large scale data.
Revised: Crowdsourcing has been instrumental for driving AI advances that rely on large scale data.

Original: At the same time reinforcement learning has seen rapid progress through benchmark environments that strike a balance between tractability and real-world complexity, such as ALE and OpenAI Gym.
Revised: At the same time reinforcement learning has seen rapid progress through benchmark environments that strike a balance between tractability and real-world complexity, such as ALE and OpenAI Gym.

Original: In this paper we aim to ﬁll a gap at the intersection of these two:
Revised: In this paper we aim to ﬁll a gap at the intersection of these two:

Original: The use of crowdsourcing to gener
Revised: The use of crowdsourcing to gener

Original: - ate large
Revised: - ate large

Original: -scale human demonstration data in the support of advancing research into imitation learning and ofﬂine learning.
Revised: -scale human demonstration data in the support of advancing research into imitation learning and ofﬂine learning.

Original: To this end
Revised: To this end

Original: , we present
Revised: , we present

Original: CrowdPlay , a complete crowdsourcing pipeline for any standard
Revised: CrowdPlay , a complete crowdsourcing pipeline for any standard

Original: RL environment including
Revised: RL environment including

Original: OpenAI Gym (made available under an open-source license)
Revised: OpenAI Gym (made available under an open-source license)

Original: ; a large
Revised: ; a large

Original: -scale pub
Revised: -scale pub

Original: - licly available crowdsourced dataset of human gameplay demonstrations in
Revised: - licly available crowdsourced dataset of human gameplay demonstrations in

Original: Atari 2600 games
Revised: Atari 2600 games

Original: , including multimodal behavior and human human and human-
Revised: , including multimodal behavior and human human and human-

Original: AI multiagent data ofﬂine learning benchmarks with extensive human data evalua- tion and a detailed study of incentives
Revised: AI multiagent data ofﬂine learning benchmarks with extensive human data evalua- tion and a detailed study of incentives

Original: , including real-time feedback to drive high quality data.
Revised: , including real-time feedback to drive high quality data.

Original: We hope that this will drive the improvement in design of algorithms that account for the complexity of human behavioral data and thereby enable a step forward in direction of effective learning for real world settings.
Revised: We hope that this will drive the improvement in design of algorithms that account for the complexity of human behavioral data and thereby enable a step forward in direction of effective learning for real world settings.

Original:  I NTRODUCTION
Revised:  I NTRODUCTION

Original: Crowdsourcing has been instrumental in many AI advances, especially recent rapid progress in deep neural network models, which often rely on large training sets.
Revised: Crowdsourcing has been instrumental in many AI advances, especially recent rapid progress in deep neural network models, which often rely on large training sets.

Original: For instance ImageNet (Deng et al., 2009), a large database of annotated images, has enabled a number of breakthroughs in image classiﬁcation (Krizhevsky et al., 2012).
Revised: For instance ImageNet (Deng et al., 2009), a large database of annotated images, has enabled a number of breakthroughs in image classiﬁcation (Krizhevsky et al., 2012).

Original: At the same time, reinforcement learning (RL) has seen rapid progress in the last few years, fueled in part by the development of standard, easily accessible, benchmark environments like the Arcade Learning Environment (ALE)
Revised: At the same time, reinforcement learning (RL) has seen rapid progress in the last few years, fueled in part by the development of standard, easily accessible, benchmark environments like the Arcade Learning Environment (ALE)

Original: (Bellemare et al., 2013; Machado et al., 2018) and
Revised: (Bellemare et al., 2013; Machado et al., 2018) and

Original: OpenAI Gym (Brockman et al., 2016).
Revised: OpenAI Gym (Brockman et al., 2016).

Original: What has been underexplored is the intersection of the two: Using large-scale crowdsourced human data for ofﬂine learning, including imitation learning and ofﬂine RL.
Revised: What has been underexplored is the intersection of the two: Using large-scale crowdsourced human data for ofﬂine learning, including imitation learning and ofﬂine RL.

Original: We present CrowdPlay , a framework, methodology, and dataset that we hope will do for ofﬂine learning what ALE and Gym what ALE and AI Gym did for online learning.
Revised: We present CrowdPlay , a framework, methodology, and dataset that we hope will do for ofﬂine learning what ALE and OpenAI Gym did for online learning.

Original: CrowdPlay supports ﬂexible and scalable crowdsourcing that is geared towards multi-channel recruitment, and is able to interface with any OpenAI Gym or Gym-like Markov decision process (MDP) environment.
Revised: CrowdPlay supports ﬂexible and scalable crowdsourcing that is geared towards multi-channel recruitment, and is able to interface with any OpenAI Gym or Gym-like Markov decision process (MDP) environment.

Original: It supports real-time feedback to participants that can be used to boost data quality, as well as both purely human and mixed human-AI multiagent environments.
Revised: It supports realtime feedback to participants that can be used to boost data quality, as well as both purely human and mixed human-AI multiagent environments.

Original: CrowdPlay is also the ﬁrst dataset based on Atari 2600 games that features multimodal and multiagent behavior.
Revised: CrowdPlay is also the ﬁrst dataset based on Atari 2600 games that features multimodal and multiagent behavior.

Original: It includes both data from normal gameplay as well as explicitly multimodal behavioral data, where players are given instructions to follow a speciﬁc behavior.
Revised: It includes both data from normal gameplay as well as explicitly multimodal behavioral data, where players are given instructions to follow a speciﬁc behavior.

Original: In addition to single-agent data, the dataset includes data from two-player, human-AI and human-human games, and with both competitive and cooperative rewards.
Revised: In addition to single-agent data, the dataset includes data from two-player, human-AI and human-human games, and with both competitive and cooperative rewards.

Original: Participants were recruited through multiple channels (under IRB, details withheld for blind submission) including Amazon Mechanical Turk,
Revised: Participants were recruited through multiple channels (under IRB, details withheld for blind submission) including Amazon Mechanical Turk,

Original: Lab in the Wild, undergraduate students, and multiple social media channels.
Revised: Lab in the Wild, undergraduate students, and multiple social media channels.

Original: For some platforms we also include data with a range of different incentive structures for participants.
Revised: For some platforms we also include data with a range of different incentive structures for participants.

Original: The Atari games were run using ALE and a multiagent version of OpenAI Gym, guaranteeing that transitions are identical to what would be seen in standard Atari RL environments.
Revised: The Atari games were run using ALE and a multiagent version of OpenAI Gym, guaranteeing that transitions are identical to what would be seen in standard Atari RL environments.

Original: In this paper we focus on the use of CrowdPlay for Atari 2600 games, but a major advantage of the approach is that it works for any Gym-like MDP environment.
Revised: In this paper we focus on the use of CrowdPlay for Atari 2600 games, but a major advantage of the approach is that it works for any Gym-like MDP environment.

Original: We believe that Atari 2600 games are interesting for imitation learning (IL) and ofﬂine RL for the same reasons that they were seminal as a challenge problem in the development of RL: they offer a balance between achievable short-term research advances and sufﬁcient complexity.
Revised: We believe that Atari 2600 games are interesting for imitation learning (IL) and ofﬂine RL for the same reasons that they were seminal as a challenge problem in the development of RL: they offer a balance between achievable short-term research advances and sufﬁcient complexity.

Original: More recent work in psychology has also shown them to be of sufﬁcient richness to support the study of human learning behavior (Tsividis et al., 2017).
Revised: More recent work in psychology has also shown them to be of sufﬁcient richness to support the study of human learning behavior (Tsividis et al., 2017).

Original: Further, and despite the richness of the data, it is easy to collect at scale through the use of crowdsourcing and a web browser, and moreover, it can be used together with an established simulator for evaluation purposes.
Revised: Further, and despite the richness of the data, it is easy to collect at scale through the use of crowdsourcing and a web browser, and moreover, it can be used together with an established simulator for evaluation purposes.

Original: The CrowdPlay pipeline directly interfaces with standard RL environments.
Revised: The CrowdPlay pipeline directly interfaces with standard RL environments.

Original: In particular this means that trajectories and transitions are guaranteed to be the same for human data as they are for RL environments; that ofﬂine learning methods automatically have access to a simulator; and that crowdsourcing with human players need not develop environments and tasks from scratch, but can make use of AI agents that can interact with human agents in a benchmark environment.
Revised: In particular this means that trajectories and transitions are guaranteed to be the same for human data as they are for RL environments; that ofﬂine learning methods automatically have access to a simulator; and that crowdsourcing with human players need not develop environments and tasks from scratch, but can make use of AI agents that can interact with human agents in a benchmark environment.

Original: 1. R ELATED W ORK
Revised: 1. R ELATED W ORK

Original: Crowdsourcing Previous platforms for crowdsourcing such as TurkServer (Mao et al., 2012) have focused on supporting synchronous participation through Amazon Mechanical Turk participants and used to study economic behavior in simple experimental environments (and not for the generation of human behavioral data).
Revised: Crowdsourcing Previous platforms for crowdsourcing such as TurkServer (Mao et al., 2012) have focused on supporting synchronous participation through Amazon Mechanical Turk participants and used to study economic behavior in simple experimental environments (and not for the generation of human behavioral data).

Original: Tylkin et al.
Revised: Tylkin et al.

Original: (2021) use a combination of ALE and Javatari for crowdsourcing Atari data in the context of evaluating the performance of an AI agent in human-AI collaboration.
Revised: (2021) use a combination of ALE and Javatari for crowdsourcing Atari data in the context of evaluating the performance of an AI agent in human-AI collaboration.

Original: They propose modifying two-player Space Invaders to make it cooperative, and to train
Revised: They propose modifying two-player Space Invaders to make it cooperative, and to train

Original: AI agents using randomized starting position, both of which we adopt in (some of) our multiagent environments.
Revised: AI agents using randomized starting position, both of which we adopt in (some of) our multiagent environments.

Original: Their crowdsourcing approach is Atari-speciﬁc and not publicly available.
Revised: Their crowdsourcing approach is Atari-speciﬁc and not publicly available.

Original: Much work has been done on the study of incentives for participants in paid crowdsourcing studies, and this is also part of the focus of our work.
Revised: Much work has been done on the study of incentives for participants in paid crowdsourcing studies, and this is also part of the focus of our work.

Original: Prior work (Mao et al., 2013; Mason & Watts, 2009; Yin et al., 2013; Harris, 2011; Shaw et al., 2011) has largely found that quality-dependent payments may increase quantity of work more than quality of work, and has not looked at real-time feedback on work quality.
Revised: Prior work (Mao et al., 2013; Mason & Watts, 2009; Yin et al., 2013; Harris, 2011; Shaw et al., 2011) has largely found that quality-dependent payments may increase quantity of work more than quality of work, and has not looked at real-time feedback on work quality.

Original: Ofﬂine Learning.
Revised: Ofﬂine Learning.

Original: Much work has been done on ofﬂine learning (Rashidinejad et al., 2021), including Behavior Cloning (BC) (Pomerleau, 1989), Batch Constrained Q-Learning (BCQ) (Fujimoto et al., 2019), Conservative Q-Learning (CQL) (Kumar et al., 2020), Implicit Quantile Network (IQN) (Dabney et al., 2018), DQN (Mnih, 2015) and Off-policy version of Soft ActorCritic (Haarnoja et al., 2018)
Revised: Much work has been done on ofﬂine learning (Rashidinejad et al., 2021), including Behavior Cloning (BC) (Pomerleau, 1989), Batch Constrained Q-Learning (BCQ) (Fujimoto et al., 2019), Conservative Q-Learning (CQL) (Kumar et al., 2020), Implicit Quantile Network (IQN) (Dabney et al., 2018), DQN (Mnih, 2015) and Off-policy version of Soft ActorCritic (Haarnoja et al., 2018)

Original: Speciﬁc to Atari, Kanervisto et al.
Revised: Speciﬁc to Atari, Kanervisto et al.

Original: (2020) benchmark behavioral cloning algorithms on existing data from several video games, including Atari 2600 games.
Revised: (2020) benchmark behavioral cloning algorithms on existing data from several video games, including Atari 2600 games.

Original: They do not use memory in the policy nor framestacking in the observation, which we believe limits the potential performance of their approach on Atari games.
Revised: They do not use memory in the policy nor framestacking in the observation, which we believe limits the potential performance of their approach on Atari games.

Original: Laurens & Kazmi (2021) clone River Raid agents using existing datasets, and develop evaluation metrics based on action distribution and playstyle.
Revised: Laurens & Kazmi (2021) clone River Raid agents using existing datasets, and develop evaluation metrics based on action distribution and playstyle.

Original: They note that their approach struggles to clone a human agent.
Revised: They note that their approach struggles to clone a human agent.

Original: Atari 2600 datasets Atari Grand Challenge (AGC) (Kurin et al., 2017) is a dataset consisting of 45 hours of standard gameplay from ﬁve Atari 2600 games. 
Revised: Datasets Atari Grand Challenge (AGC) (Kurin et al., 2017) is a dataset consisting of 45 hours of standard gameplay from ﬁve Atari 2600 games. The authors also make their Atari-speciﬁc data collection software available.

Original: They use a browser app running the Atari emulator in the browser based on Javatari.
Revised: They use a browser app running the Atari emulator in the browser based on Javatari.

Original: It is unclear to us if this can be guaranteed to always have identical execution to the ALE emulator.
Revised: It is unclear to us if this can be guaranteed to always have identical execution to the ALE emulator.

Original: Atari-Head (Zhang et al., 2020) features 117 hours of gameplay data and includes eye-tracking data.
Revised: Atari-Head (Zhang et al., 2020) features 117 hours of gameplay data and includes eye-tracking data.

Original: This data was collected using ALE.
Revised: This data was collected using ALE.

Original: However, the emulator was run in a semi-frame-by-frame mode, advancing the emulator state only when a key was pressed, and at a maximum of 20 frames per second.
Revised: However, the emulator was run in a semi-frame-by-frame mode, advancing the emulator state only when a key was pressed, and at a maximum of 20 frames per second.

Original: The focus of the study was on attention tracking, and is not intended to be representative of natural human gameplay behavior.
Revised: The focus of the study was on attention tracking, and is not intended to be representative of natural human gameplay behavior.

Revised: D4RL (Fu et al., 2020) and RL Unplugged (Gulcehre et al., 2020) both also provide datasets for ofﬂine learning, but both focus on synthetic data.

Original: Database
Revised: Database

Original: Backend
Revised: Backend

Original: Load Balancer
Revised: Load Balancer

Original: Main [Figure]
Revised: Main Process

Original: Environment Process
Revised: Environment Process

Original: Gym Env
Revised: Gym Env

Revised: O

Original: [Figure]
Revised: [Figure]

Revised: ine Learning Pipeline

Revised: Gym Env

Original: Local Dataset Package
Revised: Local Dataset Package

Original: Query access to metadata, trajectory decompression and deserialization
Revised: Query access to metadata, trajectory decompression and deserialization

Original: Main MDP Loop: fetch user action query AIs call env.step() calculate real-time statistics observation, statistics to client enqueue trajectory async trajectory compression & DB insert
Revised: Main MDP Loop: fetch user action query AIs call env.step() calculate real-time statistics observation, statistics to client enqueue trajectory async trajectory compression & DB insert

Original: Trajectory Blobs Structured Metadata
Revised: Trajectory Blobs Structured Metadata

Original: Observations Statistics
Revised: Observations Statistics

Original: Keypresses
Revised: Keypresses

Original: AI 
Revised: AI •

Original: [Figure]
Revised: [Figure]

Original: managementand
Revised: management

Revised: [Figure]

Revised: Within-instance scaling • User management and API

Original: [Figure]
Revised: [Figure]

Original:  C ROWD P LAY : T HE P IPELINE 2. O VERVIEW
Revised:  C ROWD P LAY : T HE P IPELINE 2. O VERVIEW

Original: The heart of our pipeline are the CrowdPlay backend and frontend, which is a client-server architecture that streams OpenAI Gym environments and similar MDP environments to web browser clients.
Revised: The heart of our pipeline are the CrowdPlay backend and frontend, which is a client-server architecture that streams OpenAI Gym environments and similar MDP environments to web browser clients.

Original: It is highly extensible, scalable to hundreds or thousands of concurrent users, and allows the real-time capture of both trajectories as well as related statistics.
Revised: It is highly extensible, scalable to hundreds or thousands of concurrent users, and allows the real-time capture of both trajectories as well as related statistics.

Original: It is geared toward multi-channel recruitment of participants and strong incentives.
Revised: It is geared toward multi-channel recruitment of participants and strong incentives.

Original: As its most important feature, it interfaces directly with OpenAI Gym and similar environments, thus opening up the entire array of standard RL environments to rapid crowdsourcing of behavioral data into the support of research into IL and ofﬂine
Revised: As its most important feature, it interfaces directly with OpenAI Gym and similar environments, thus opening up the entire array of standard RL environments to rapid crowdsourcing of behavioral data into the support of research into IL and ofﬂine

Original: RL.
Revised: RL.

Original: It also supports multi-agent environments, including mixed human-AI environments.
Revised: It also supports multi-agent environments, including mixed human-AI environments.

Original: Complementing this is an engine to support the local download of the generated dataset, including storage of metadata in a relational database for fast access, and compressed trajectories for storage efﬁciency.
Revised: Complementing this is an engine to support the local download of the generated dataset, including storage of metadata in a relational database for fast access, and compressed trajectories for storage efﬁciency.

Original: The data download can be real-time and is incremental.
Revised: The data download can be real-time and is incremental.

Original: We give a short overview of the CrowdPlay architecture, and refer the reader to Appendix A.1 for more information.
Revised: We give a short overview of the CrowdPlay architecture, and refer the reader to Appendix A.1 for more information.

Original: 2. S OFTWARE A RCHITECTURE
Revised: 2. S OFTWARE A RCHITECTURE

Original: CrowdPlay provides a highly extensible, high performance client-server architecture for streaming
Revised: CrowdPlay provides a highly extensible, high performance client-server architecture for streaming

Original: MDP environments to remote web browser clients.
Revised: MDP environments to remote web browser clients.

Original: The backend interfaces with OpenAI Gym and similar MDP environments.
Revised: The backend interfaces with OpenAI Gym and similar MDP environments.

Original: Actions are collected as keypresses in the browser client, and sent to the backend where they are fed into the MDP’s “step” function.
Revised: Actions are collected as keypresses in the browser client, and sent to the backend where they are fed into the MDP’s “step” function.

Original: The returned observation is sent back to the browser client for display.
Revised: The returned observation is sent back to the browser client for display.

Original: This is repeated to generate an episode trajectory.
Revised: This is repeated to generate an episode trajectory.

Original: The remainder of the CrowdPlay software infrastructure is built to make this basic loop into a structure that is robust, performant, extensible, user-friendly and scalable.
Revised: The remainder of the CrowdPlay software infrastructure is built to make this basic loop into a structure that is robust, performant, extensible, user-friendly and scalable.

Original: Figure 1 shows the key parts of the CrowdPlay architecture.
Revised: Figure 1 shows the key parts of the CrowdPlay architecture.

Original: Communication between the browser client and backend is through high-performance socket connections.
Revised: Communication between the browser client and backend is through high-performance socket connections.

Original: The backend is built to be scalable both within-instance, using multiple processes, as well as across-instance using a load balancer and autoscaling instance groups.
Revised: The backend is built to be scalable both within-instance, using multiple processes, as well as across-instance using a load balancer and autoscaling instance groups.

Original: Trajectories are stored di-
Revised: Trajectories are stored di-

Original: rectly as compressed, serialized Python objects, allowing both very easy modiﬁcation of data capture as well as immediate decoding for use in existing Python-based learning pipelines.
Revised: rectly as compressed, serialized Python objects, allowing both very easy modiﬁcation of data capture as well as immediate decoding for use in existing Python-based learning pipelines.

Original: CrowdPlay also supports multi-agent environments.
Revised: CrowdPlay also supports multi-agent environments.

Original: It allows multiple human participants by routing multiple browser clients to a single MDP environment.
Revised: It allows multiple human participants by routing multiple browser clients to a single MDP environment.

Original: Mixed human-AI environments are supported through pre-trained neural network policies.
Revised: Mixed human-AI environments are supported through pre-trained neural network policies.

Original: For robustness, AI agents can also take over control of a human agent on the ﬂy, in the case that a human player disconnects from a multiagent environment, allowing uninterrupted gameplay for the remaining human players.
Revised: For robustness, AI agents can also take over control of a human agent on the ﬂy, in the case that a human player disconnects from a multiagent environment, allowing uninterrupted gameplay for the remaining human players.

Revised: A major focus in the design of CrowdPlay is providing the ease of access of the generated data in downstream ML pipelines.

Original: 2. I NCENTIVES AND R ECRUITMENT
Revised: We believe it is crucial for these pipelines to have access not only to the same simulator as the crowdsourcing pipeline, but also to the same observation pre-processing tools that are used in state-of-the-art RL methods on these simulators. Addressing these design goals, CrowdPlay includes a local metadata search engine and a custom, Deepmind-style (Mnih et al., 2015) observation processing function for ofﬂine data. We give more details in Appendix A.1.

Original: CrowdPlay provides an extensible and easy-to-use framework for collecting structured metadata and real-time statistics per user, session, episode, and individual steps.
Revised: CrowdPlay provides an extensible and easy-to-use framework for collecting structured metadata and real-time statistics per user, session, episode, and individual steps.

Original: This is used for capturing data quality information, as well as for driving real-time incentives for participants.
Revised: This is used for capturing data quality information, as well as for driving real-time incentives for participants, depending on recruitment platform.

Original: On platforms where users are paid or otherwise compensated (e.g. through entry into a rafﬂe), we use CrowdPlay’s real-time statistics engine to create strong incentives, including minimum effort requirements for compensation, and bonus payments that scale with performance.

Original: On platforms where users are not compensated, we use the same real-time statistics to reinforce intrinsic motivation through live “high score” pages and by reframing non-standard gameplay tasks as “challenges.”

Original: We discuss the platforms that we target in more detail in Appendix A.3, and the various incentive designs and their effect on data in Section 4.
Revised: We discuss the platforms that we target in more detail in Appendix A.3, and the various incentive designs and their effect on data in Section 4.

Original: 2. D ATA P OST - PROCESSING FOR ML

Original: A major focus in the design of CrowdPlay is providing the ease of access of the generated data in downstream ML pipelines.

Original: We believe it is crucial for these pipelines to have access not only to the same simulator as the crowdsourcing pipeline, but also to the same observation pre-processing tools that are used in state-of-the-art RL methods on these simulators.

Original: Addressing these design goals, CrowdPlay includes a custom, Deepmind-style observation processing function.

Original: This is implemented through a custom frame buffer that replaces the usual “Deepmind” wrappers, is veriﬁed to give the same output frame-by-frame and pixel-by-pixel as standard implementations of wrappers such as those in OpenAI Gym, and can be applied post-hoc to already recorded trajectories.

Original: This same observation processing is also used for the AI agents that we bring into mixed human-AI environments.

Original:  C ROWD P LAY A TARI : T HE D ATASET 3. S COPE AND M OTIVATION
Revised:  C ROWD P LAY A TARI : T HE D ATASET 3. S COPE AND M OTIVATION

Original: Our main focus in creating the ﬁrst CrowdPlay dataset has been on Atari 2600 games, as we believe that human data for these environments can enable advances in IL and ofﬂine RL (just as they have been in driving advances in online RL, and for the same reasons).
Revised: Our main focus in creating the ﬁrst CrowdPlay dataset has been on Atari 2600 games, as we believe that human data for these environments can enable advances in IL and ofﬂine RL (just as they have been in driving advances in online RL, and for the same reasons).

Original: In curating the dataset, we have been especially motivated by diversity—we have used multiple recruitment channels, each with different extrinsic and intrinsic user motivation and incentive models, and have reached overusers in 1300 sessions over a three week period in September 2021.
Revised: In curating the dataset, we have been especially motivated by diversity—we have used multiple recruitment channels, each with different extrinsic and intrinsic user motivation and incentive models, and have reached overusers in 1300 sessions over a three week period in September 2021.

Original: The CrowdPlay Atari dataset currently holds over 250 hours, or 54 million transitions, and was collected across six different games.
Revised: The CrowdPlay Atari dataset currently holds over 250 hours, or 54 million transitions, and was collected across six different games.

Original: We have targeted not only implicitly multimodal behavior through recruiting different users and through different channels, but also explicit multimodal behavior through explicit instructions that are reinforced through tight incentives and real-time feedback.
Revised: We have targeted not only implicitly multimodal behavior through recruiting different users and through different channels, but also explicit multimodal behavior through explicit instructions that are reinforced through tight incentives and real-time feedback.

Original: We include both single agent as well as multi-agent data, with the latter reported for both human-human and human-AI gameplay, and with both competitive as well as cooperative rewards.
Revised: We include both single agent as well as multi-agent data, with the latter reported for both human-human and human-AI gameplay, and with both competitive as well as cooperative rewards.

Original: We believe this is the ﬁrst multi-agent human behavioral dataset, at the very least for Atari 2600 games.
Revised: We believe this is the ﬁrst multi-agent human behavioral dataset, at the very least for Atari 2600 games.

Original: 3. T ASK D ESIGN
Revised: 3. T ASK D ESIGN

Original: We used the following Atari 2600 games for our dataset: Space Invaders, RiverRaid, Montezuma’s
Revised: We used the following Atari 2600 games for our dataset: Space Invaders, River Raid, Montezuma’s

Original: Revenge, Q*bert, Breakout and Beamrider.
Revised: Revenge, Q*bert, Breakout and Beamrider.

Original: Space Invaders makes up the largest part of our dataset.
Revised: Space Invaders makes up the largest part of our dataset.

Original: We chose this game as a focus for several reasons: it is well-known and popular both among Atari players as well as in the RL community, it was easy to come up with several speciﬁc and distinct behaviors that are still compatible with progressing in the game, and it has a native two-player mode.
Revised: We chose this game as a focus for several reasons: it is well-known and popular both among Atari players as well as in the RL community, it was easy to come up with several speciﬁc and distinct behaviors that are still compatible with progressing in the game, and it has a native two-player mode.

Original: RiverRaid provides the second largest amount of data and was chosen for similar reasons, in that it is accessible and well understood from a learning perspective, and has obvious opportunities to promote multimodal behavior.
Revised: River Raid provides the second largest amount of data and was chosen for similar reasons, in that it is accessible and well understood from a learning perspective, and has obvious opportunities to promote multimodal behavior.

Original: The remaining games were chosen for diversity of game types and their popularity in previous RL and IL work.
Revised: The remaining games were chosen for diversity of game types and their popularity in previous RL and IL work.

Revised: Table 1 shows a list of available data by game and recruitment channel.

Original: Multimodal Behavior For Space Invaders and RiverRaid, in addition to standard gameplay, we asked some participants to follow speciﬁc behavior in the game.
Revised: Multimodal Behavior For Space Invaders and River Raid, in addition to standard gameplay, we asked some participants to follow speciﬁc behavior in the game.

Original: We believe that this data will be useful for multiple research strands.
Revised: We believe that this data will be useful for multiple research strands.

Original: In imitation learning, this type of data can provide a testbed to understand whether algorithms are robust to divergent expert behavior.
Revised: In imitation learning, this type of data can provide a testbed to understand whether algorithms are robust to divergent expert behavior.

Original: In ofﬂine RL, this can allow for controlled experiments with different reward functions in an otherwise identical environment.
Revised: In ofﬂine RL, this can allow for controlled experiments with different reward functions in an otherwise identical environment.

Original: For most of this data, we recruited participants via Mechanical Turk, and gave incentives via both a minimum level of task adherence required to complete the HIT (unit of task), as well as an explicit reward function tied to a bonus payment, and with real-time feedback on the same.
Revised: For most of this data, we recruited participants via Mechanical Turk, and gave incentives via both a minimum level of task adherence required to complete the HIT (unit of task), as well as an explicit reward function tied to a bonus payment, and with real-time feedback on the same.

Original: The behavior types we instructed participants to follow were to either stay on either half of the game screen (in both Space Invaders and Riverraid, or to shoot aliens in a particular order (row by row from the bottom, column by column from the outside in, or column by column from the inside out; in Space
Revised: The behavior types we instructed participants to follow were to either stay on either half of the game screen (in both Space Invaders and River Raid, or to shoot aliens in a particular order (row by row from the bottom, column by column from the outside in, or column by column from the inside out; in Space

Original: Invaders only.)
Revised: Invaders only.)

Original: We discuss these tasks in more detail in Appendix A.2.
Revised: We discuss these tasks in more detail in Appendix A.2.

Original: Multiplayer Games CrowdPlay also contains a set of trajectories from multi-agent environments.
Revised: Multiplayer Games CrowdPlay also contains a set of trajectories from multi-agent environments.

Original: For these, we used the two-player mode of Space Invaders.
Revised: For these, we used the two-player mode of Space Invaders.

Original: We include data from two variants: (1) the standard two-player Space Invaders mode, and (2) a custom cooperative mode.
Revised: We include data from two variants: (1) the standard two-player Space Invaders mode, and (2) a custom cooperative mode.

Original: In the latter, we modify the original game in two ways.
Revised: In the latter, we modify the original game in two ways.

Original: First, in the standard two-player mode there is a score bonus if the other player loses a live.
Revised: First, in the standard two-player mode there is a score bonus if the other player loses a live.

Original: We removed this score bonus.
Revised: We removed this score bonus.

Original: Second, we gave both players the sum of their individual scores.
Revised: Second, we gave both players the sum of their individual scores.

Original: These modiﬁcations were done entirely in Python, in the case of the score bonus detecting these events from emulator state.
Revised: These modiﬁcations were done entirely in Python, in the case of the score bonus detecting these events from emulator state.

Original: Participants were instructed to ignore the score shown on the game screen, and were instead shown their “cooperative” score next to the game screen.
Revised: Participants were instructed to ignore the score shown on the game screen, and were instead shown their “cooperative” score next to the game screen.

Original: Most of our multiplayer data is from mTurk, and people were incentivized through bonus payments to maximize this score.
Revised: Most of our multiplayer data is from mTurk, and people were incentivized through bonus payments to maximize this score.

Original: Further, we include data from games with two human players, as well as games with one human and one AI player.
Revised: Further, we include data from games with two human players, as well as games with one human and one AI player.

Original: AI agents were trained in two-agent environments with two separate A2C policies.
Revised: We give details on this in Appendix A.2.

Original: We used RLLib’s implementation ofA2C (Mnih et al., 2016; Liang et al., 2018) for 40 million timesteps, using standard Deepmind-style observation processing and hyperparameters. We trained separate agents for the standard and cooperative variants.
Revised: A unique feature of the CrowdPlay Atari dataset is its size and diversity. In addition to explicit multimodal behavior, it also comprises data generated by participants recruited via multiple different channels, and different demographics.

Original: Following ﬁndings in (Tylkin et al., 2021) we trained AI agents using randomized starting positions each episode for better exploration of  the states likely encountered in human-AI environments.
Revised: Figure 3 (left) shows a t-SNE embedding of action distributions in episodes of standard Space Invaders gameplay. Episodes are colored according to recruitment channel (MTurk, Email or social media). We notice that there is a clear divide between action distributions of MTurk users and those of Email and social media users, which we take as evidence that different demographics already lead to interesting diversity in the data. This shows that different recruitment channels lead to diversity in the data, this already apparent in aggregate statistics such as action distributions.

Original: AI agents were also used in nominally human-human environments with MTurk participants, in two circumstances.
Revised: For multiagent data, Figure 3 shows a t-SNE embedding for cooperative and standard human-AI Space Invaders for MTurk users.

Original: One, if one human player disconnected, an AI agent took over control.
Revised: We note several things. First, there is a clear distinction between human and AI actions.

Original: This is to allow the remaining MTurk worker to complete their task. Similarly, if an MTurk worker was waiting for 15 minutes
Revised: Second, there are two clear clusters within the AI data corresponding to cooperative and competitive behavior. Third, there is also a clear relationship between cooperative

Original: [Table]
Revised: [Table]

Revised: 80 tsne

Revised: [Figure]

Revised: 2d

Original: Table 1 shows a list of available data by game and recruitment channel.
Revised: [Figure] t s ne - 2d - t w o [Figure] vs standard settings and action distribution for human data, and it is more complex than in the AI data

Original: 3. C HALLENGES AND L IMITATIONS
Revised: 3. C HALLENGES AND L IMITATIONS

Original: We discuss here some challenges that we experienced in collecting this data and the corresponding limitations in the dataset.
Revised: We discuss here some challenges that we experienced in collecting this data and the corresponding limitations in the dataset.

Original: One, our incentive design has continued to evolve, while a signiﬁcant amount of the data has been collected with suboptimal incentives.
Revised: One, our incentive design has continued to evolve, while a signiﬁcant amount of the data has been collected with suboptimal incentives.

Original: This is part of why we discuss incentive design in the next section in detail, to allow other researchers to build on what we have learned.
Revised: This is part of why we discuss incentive design in the next section in detail, to allow other researchers to build on what we have learned.

Original: Two, some of our dataset is unbalanced, especially for the different behavior types in the multimodal Space Invaders data, where we have 2-3 times as much data for some behaviors as for others.
Revised: Two, some of our dataset is unbalanced, especially for the different behavior types in the multimodal Space Invaders data, where we have 2-3 times as much data for some behaviors as for others.

Original: This is partly due to suboptimal incentives and some tasks being harder than we had intended [Table] and partly because we improved the way the pipeline assigned MTurk workers to tasks over time.
Revised: This is partly due to suboptimal incentives and some tasks being harder than we had intended, and partly because we improved the way the pipeline assigned MTurk workers to tasks over time.

Original: We have since then improved the automatic task assignment logic to more carefully take into account the amount of data already collected.
Revised: We have since then improved the automatic task assignment logic to more carefully take into account the amount of data already collected.

Original: Three, we have found that collecting humanhuman multiagent data is a difﬁcult challenge; e.g., we expected that the ability to play with friends would be a major draw in social media recruitment, but found that we had virtually zero uptake on this.
Revised: Three, we have found that collecting humanhuman multiagent data is a difﬁcult challenge; e.g., we expected that the ability to play with friends would be a major draw in social media recruitment, but found that we had virtually zero uptake on this.

Original: On the other hand, we have made signiﬁcant advances in making multiplayer data collection more feasible on MTurk.
Revised: On the other hand, we have made signiﬁcant advances in making multiplayer data collection more feasible on MTurk.

Original: In particular, the use of fallback AI agents has solved many practical problems with workers needing to depend on one another for successful task completion.
Revised: In particular, the use of fallback AI agents has solved many practical problems with workers needing to depend on one another for successful task completion.

Original: Still, we were able to collect much more single-player and human-AI data than human-human data.
Revised: Still, we were able to collect much more single-player and human-AI data than human-human data.

Original:  I NCENTIVE D ESIGN 4. I NCENTIVE D ESIGN AND R ECRUITMENT
Revised:  I NCENTIVE D ESIGN 4. I NCENTIVE D ESIGN AND R ECRUITMENT

Original: CrowdPlay has been designed with multichannel recruitment and strong participant incentives in mind.
Revised: CrowdPlay has been designed with multichannel recruitment and strong participant incentives in mind.

Original: It supports conﬁguring multiple environments and “tasks” for participants to complete, including different user instructions, incentives, and AI agent conﬁguration.
Revised: It supports conﬁguring multiple environments and “tasks” for participants to complete, including different user instructions, incentives, and AI agent conﬁguration.

Original: At the heart of its capabilities is its ability to capture an extensible list of metadata live during gameplay, including data such as playtime, score, and various in-game behavioral characteristics.
Revised: At the heart of its capabilities is its ability to capture an extensible list of metadata live during gameplay, including data such as playtime, score, and various in-game behavioral characteristics.

Original: In platforms where users are paid for their participation, we use this data to deﬁne both a minimum acceptable effort by users to get paid at all, as well as a dynamic bonus payment that can depend on a ﬁne-grained analysis human
Revised: On platforms where users are paid for their participation, we use this data to deﬁne both a minimum acceptable effort by users to get paid at all, as well as a dynamic bonus payment that can depend on a ﬁne-grained analysis of user effort.

Original: , a would start an of user effort.

Original: Both progress toward minimum required effort as well as bonus payments can be displayed to users during live gameplay.
Revised: Both progress toward minimum required effort as well as bonus payments can be displayed to users during live gameplay.

Revised: On platforms where users are not compensated, we use the same real-time statistics to reinforce intrinsic motivation through live “high score” pages and by reframing non-standard gameplay tasks as “challenges.”

Original: For instance, in one Space Invaders task we asked participants to shoot aliens in a speciﬁc order.
Revised: For instance, in one Space Invaders task we asked participants to shoot aliens in a speciﬁc order.

Original: Our architecture is able to determine adherence to these instructions by evaluating emulator state at every frame, detecting when an alien has been shot and which one, and keeping a running count of aliens shot in and out of the prescribed order.
Revised: Our architecture is able to determine adherence to these instructions by evaluating emulator state at every frame, detecting when an alien has been shot and which one, and keeping a running count of aliens shot in and out of the prescribed order.

Original: Users were rewarded based on both the total number of aliens they shot in the correct order, as well as the fraction of aliens that they shot correctly.
Revised: Users were rewarded based on both the total number of aliens they shot in the correct order, as well as the fraction of aliens that they shot correctly.

Original: Using this framework, we can mimic the reward structure of an MDP through monetary incentives and also make use of realtime feedback; and we can additionally shape incentives to achieve particular modalities.
Revised: Using this framework, we can mimic the reward structure of an MDP through monetary incentives and also make use of realtime feedback; and we can additionally shape incentives to achieve particular modalities.

Original: Figure 2 (right) shows an example of this realtime feedback.
Revised: Figure 2 (right) shows an example of this realtime feedback.

Original: 4. I NCENTIVE M ODELS
Revised: 4. I NCENTIVE M ODELS

Original: We provide various incentive models, as well as real-time feedback to participants.
Revised: We provide various incentive models, as well as real-time feedback to participants.

Original: For social media participants, this was feedback-only.
Revised: For social media participants, this was feedback-only.

Original: For students we made use of a rafﬂe, and required a minimum time and effort, e.g. 80% of aliens shot in the correct order.
Revised: For students we made use of a rafﬂe, and required a minimum time and effort, e.g. 80% of aliens shot in the correct order.

Original: For participants from Mechanical Turk, we experimented with multiple models of incentives.
Revised: For participants from Mechanical Turk, we experimented with multiple models of incentives.

Original: Experimental design Speciﬁcally, we report the results of an extensive experiment with the Space
Revised: Experimental design Speciﬁcally, we report the results of an extensive experiment with the Space

Original: Invaders “Inside-Out” task.
Revised: Invaders “Inside-Out” task.

Original: In this, we gave participants identical instructions for the behavior they were asked to follow, but adopted ﬁve different ways of remunerating them:
Revised: In this, we gave participants identical instructions for the behavior they were asked to follow, but adopted ﬁve different ways of remunerating them:

Original: No incentives: In this treatment, we paid participants a lump sum after 5 minutes of time of running the game, irrespective of their actions or performance in the game.
Revised: No incentives: In this treatment, we paid participants a lump sum after 5 minutes of time of running the game, irrespective of their actions or performance in the game.

Original: Active Time: As above, however we only count time if there was at least one user keypress in the last 30 seconds, and at least one non-zero game reward in the last 60 seconds.
Revised: Active Time: As above, however we only count time if there was at least one user keypress in the last 30 seconds, and at least one non-zero game reward in the last 60 seconds.

Original: Minimum Requirement:
Revised: Minimum Requirement:

Original: In addition to 5 minutes of play time, we require that participants follow instructions to at least some minimal degree.
Revised: In addition to 5 minutes of play time, we require that participants follow instructions to at least some minimal degree.

Original: We require they shoot a minimum of 50 aliens in the correct order, and that at least 80% of aliens shot were in the correct order (i.e. from the innermost columns that still had any aliens present).4.
Revised: We require they shoot a minimum of 50 aliens in the correct order, and that at least 80% of aliens shot were in the correct order (i e. from the innermost columns that still had any aliens present).4.

Original: Sliding Bonus: We pay participants a much smaller lump sum “base payment” afterminutes of active playtime and a bonus payment that reaches a maximum if participants shot at least 200 aliens in the correct order, and 100% of aliens short were in the correct order.
Revised: Sliding Bonus: We pay participants a much smaller lump sum “base payment” afterminutes of active playtime and a bonus payment that reaches a maximum if participants shot at least 200 aliens in the correct order, and 100% of aliens short were in the correct order.

Original: The base and full bonus payment were chosen to equal the lump sum payment of the previous incentive regimes.
Revised: The base and full bonus payment were chosen to equal the lump sum payment of the previous incentive regimes.

Original: The bonus payment p scaled linearly in both the number of  aliens shot in the correct order n correct , as well as the fraction of aliens shot in the correct order above one-half, 
Revised: The bonus payment p scaled linearly in both the number of mind low aliens shot in the correct order n correct , as well as the fraction of aliens shot in the correct order above one-half, i

Original: i e. n correct /n total −0 .
Revised: e. n correct /n total −

Original: [Equation]
Revised: [Equation]

Original: All Incentives: We pay a small lump sum subject to the conditions in the minimum requirements regime and a bonus as in the sliding bonus regime.
Revised: All Incentives: We pay a small lump sum subject to the conditions in the minimum requirements regime and a bonus as in the sliding bonus regime.

Original: For both task requirements and bonus payments, participants were given real-time feedback.
Revised: For both task requirements and bonus payments, participants were given real-time feedback.

Original: For minimum requirements, this took the form of an itemized list of requirements and their performance, e.g. “Correct aliens shot: 40 (required: 50)”.
Revised: For minimum requirements, this took the form of an itemized list of requirements and their performance, e.g. “Correct aliens shot: 40 (required: 50)”.

Original: For bonus payments, this took the form of a live estimate of the expected payment, but did not include details on how the payment was computed.
Revised: For bonus payments, this took the form of a live estimate of the expected payment, but did not include details on how the payment was computed.

Original: Task instructions stated that the bonus payment would depend on both the number of aliens shot as well as how many were shot in the correct order.
Revised: Task instructions stated that the bonus payment would depend on both the number of aliens shot as well as how many were shot in the correct order.

Original: that submitted a HIT, the feedback was overwhelmingly positive and often described the tasks as “fun” or similar.

Original: N o i n c en t i v e s

Original: A c t i v e T

Original: i m e

Original: Q u a li t y

Original: R equ i r e m en t

Original: S c a li ng

Original: B onu s

Original: A ll I n c en t i v e s

Original: S o c i a l M ed i a U s e r s

Original: E m a il R a ff l e

Original: Treatment / User Group

Original: T o t a l a m oun t

Original: o f good da t a pe r pa r t

Original: i c

Original: i pan t ( s )

Original: Measurement

Original: Total

Original: Average 0.

Original: A v e r age f r a c t i on o f good da t a pe r pa r t i c

Original: i pan t We also compare data from these treatments with data collected from social media users, as well as students reached on campus via email.
Revised: We also compare data from these treatments with data collected from social media users, as well as students reached on campus via email.

Original: For email participants, we enforced a similar minimum requirement as for the minimum requirement treatment on MTurk, with successful completion of the requirement earning an entry into a rafﬂe.
Revised: For email participants, we enforced a similar minimum requirement as for the minimum requirement treatment on MTurk, with successful completion of the requirement earning an entry into a rafﬂe.

Revised: For social media participants, we did not provide any monetary incentives, and instead phrased the task as a “challenge”.

Original: 4. R ESULTS
Revised: 4. R ESULTS

Original: In regard to the structure of incentives and effect on data, we report several main ﬁndings.
Revised: In regard to the structure of incentives and effect on data, we report several main ﬁndings.

Original: One, data quality  was signiﬁcantly higher when using any kind of quality-based incentives, be it a minimum requirement tied to a ﬁxed-amount payment, a sliding bonus payment, or a combination of both.
Revised: One, data quality (measured as the fraction of data per participant meeting a threshold of at least 80% of aliens shot in the speciﬁed order) was signiﬁcantly higher when using any kind of quality-based incentives, be it a minimum requirement tied to a ﬁxed-amount payment, a sliding bonus payment, or a combination of both.

Original: Non-incentivized data was low quality even when enforcing that participants actively play the game, thus ﬁltering out any users that put in no effort at all (e.g. by starting the game and then switching to a different window).
Revised: Non-incentivized data was low quality even when enforcing that participants actively play the game, thus ﬁltering out any users that put in no effort at all (e.g. by starting the game and then switching to a different window).

Original: The orange bars in Figure 3 show this, being higher for the middle three treatment groups than for the left two.
Revised: The orange bars in Figure 4 show this, being higher for the middle three treatment groups than for the left two.

Original: Compared to the previous literature on incentives on MTurk, our model differs in that bonus payments are on a sliding scale rather than an all-or-nothing ﬁxed payment, and that participants are given real-time feedback.
Revised: Compared to the previous literature on incentives on MTurk, our model differs in that bonus payments are on a sliding scale rather than an all-or-nothing ﬁxed payment, and that participants are given real-time feedback.

Original: This might explain why we see a larger effect from bonus payments than previous studies.
Revised: This might explain why we see a larger effect from bonus payments than previous studies.

Original: Two, the incentive treatments led to a larger amount of good data being collected per participant.
Revised: Two, the incentive treatments led to a larger amount of good data being collected per participant.

Original: Further, the analysis suggests that the combination of minimum requirement and sliding bonus payments can lead to a larger amount of good data collected per participant than either treatment by itself.
Revised: Further, the analysis suggests that the combination of minimum requirement and sliding bonus payments can lead to a larger amount of good data collected per participant than either treatment by itself.

Original: In other words, while the quality was already high and did not improve much further, the combination of both treatments leads people to play much longer.
Revised: In other words, while the quality was already high and did not improve much further, the combination of both treatments leads people to play much longer.

Original: We caution however that this effect is not shown with statistical conﬁdence, and that more investigation is needed.
Revised: We caution however that this effect is not shown with statistical conﬁdence, and that more investigation is needed.

Original: The blue bars in Figure 3 show this.
Revised: The blue bars in Figure 4 show this.

Original: Three, participants recruited via social media and email had the highest data quality, but we found these to also be the least scalable recruitment channels (speciﬁcs on mechanisms of social media recruitment omitted for blind review).
Revised: Three, participants recruited via social media and email had the highest data quality, but we found these to also be the least scalable recruitment channels (speciﬁcs on mechanisms of social media recruitment omitted for blind review).

Original: For social media, the amount of data collected per participant was the lowest of these channels, and the total number of users from social media and email was also lower than those from MTurk, and it would easily be possible to recruit more MTurk users.
Revised: For social media, the amount of data collected per participant was the lowest of these channels, and the total number of users from social media and email was also lower than those from MTurk, and it would easily be possible to recruit more MTurk users.

Original:  B ENCHMARK R ESULTS
Revised:  B ENCHMARK R ESULTS

Original: BC

Original: BCQ

Original: CQL

Original: IQN

Original: DQN

Original: SAC Algorithm

Original: N o r m a li z e d P e r f o r m a n c e

Original: Our framework provides a variety of data compositions that exhibit real-world intricacies and diversities.
Revised: Our framework provides a variety of data compositions that exhibit real-world intricacies and diversities.

Original: We envision that this would help to facilitate rapid progress for ofﬂine learning methods that rely on learning from diverse data compositions without access to online exploration (Rashidinejad et al., 2021).
Revised: We envision that this would help to facilitate rapid progress for ofﬂine learning methods that rely on learning from diverse data compositions without access to online exploration (Rashidinejad et al., 2021).

Original: We therefore evaluate our datasets on recently proposed ofﬂine learning algorithms and hope that it will inspire potential future directions in this area.
Revised: We therefore evaluate our datasets on recently proposed ofﬂine learning algorithms and hope that it will inspire potential future directions in this area.

Original: As the Atari is a discrete action domain, we focus on algorithms that can handle discrete actions effectively.
Revised: As the Atari is a discrete action domain, we focus on algorithms that can handle discrete actions effectively.

Original: Speciﬁcally, our baseline algorithms include Behavior Cloning (BC) (Pomerleau, 1989), Batch Constrained QLearning (BCQ) (Fujimoto et al., 2019), Conservative Q-Learning (CQL) (Kumar et al., 2020), Implicit Quantile Network (IQN) (Dabney et al., 2018), DQN (Mnih, 2015) and Offpolicy version of Soft Actor-Critic (Haarnoja et al., 2018).
Revised: Speciﬁcally, our baseline algorithms include Behavior Cloning (BC) (Pomerleau, 1989), Batch Constrained Q-Learning (BCQ) (Fujimoto et al., 2019), Conservative Q-Learning (CQL) (Kumar et al., 2020), Implicit Quantile Network (IQN) (Dabney et al., 2018) N o i n c en t i v e s A c t i v e T i m e

Revised: Q ua li t y R equ i r

Original: As evaluate the algorithms on different task, in Figure 4, we provide the normalized version of the performance across tasks (Agarwal et al., 2020).
Revised: e m en t S c a li ng B onu s A ll I n c en t i v e s S o c i a l M ed i a U s e r s

Revised: E m a il R a ff l e

Revised: [Figure]

Revised: / User

Revised: [Figure]

Revised: T o t a l a m oun t

Revised: o f good da t a pe r pa r t

Revised: i c

Revised: i pan t ( s )

Revised: [Figure]

Revised: [Figure]

Revised: Critic (Haarnoja et al., 2018).

Revised: As we evaluate the algorithms on different tasks, in Figure 5, we provide the normalized version of the performance across tasks (Agarwal et al., 2020).

Original: We provide individual normalized and unnormalized score for each task and experiment details in Appendix B.
Revised: We provide individual normalized and unnormalized scores for each task and experiment details in Appendix B.

Original: From the results, we observe that ofﬂine version of reinforcement learning algorithms outperforms behavioral regularized and constrained algorithms.
Revised: From the results we observe that ofﬂine versions of reinforcement learning algorithms outperform behavioral regularized and constrained algorithms.

Original: Further, while DQN does not perform consistently well across all tasks, it still fares better than both the distributed methods and comparable to the behavior regularized versions.
Revised: Further, while DQN does not perform consistently well across all tasks, it still fares better than both the distributed methods and comparable to the behavior regularized versions.

Original: Behavior cloning shows surprisingly well performance which may be due to artifact of those tasks where the collected demonstrations are of high quality for that task but does not cover the state space well, making it difﬁcult for the ofﬂine reinforcement learning algorithms.
Revised: Behavior cloning shows surprisingly good performance which may be due to artifact of those tasks where the collected demonstrations are of high quality for that task but does not cover the state space well, making it difﬁcult for the ofﬂine reinforcement learning algorithms.

Original: Overall, the results for all the algorithms demonstrate the open challenges on learning from human demonstrations where the data is both limited and noisy which raises the need for more robust and sample efﬁcient methods.
Revised: Overall, the results for all the algorithms demonstrate the open challenges on learning from human demonstrations where the data is both limited and noisy which raises the need for more robust and sample efﬁcient methods.

Revised: We contrast this to the benchmark results in the RL Unplugged paper (Gulcehre et al., 2020), where the same algorithms performed signiﬁcantly better both in general (Figure 6 therein) and on the speciﬁc games we benchmark here (Table 9 and Figure 7 therein).

Revised: This demonstrates that human data poses an interesting challenge to current state-of-the-art ofﬂine learning algorithms that synthetic data does not.

Original:  D ISCUSSION
Revised:  D ISCUSSION

Original: We have presented CrowdPlay, a pipeline for crowdsourcing human demonstration data in standard
Revised: We have presented CrowdPlay, a pipeline for crowdsourcing human demonstration data in standard

Original: RL environments, and a ﬁrst dataset along with benchmarks on Atari 2600 games.
Revised: RL environments, and a ﬁrst dataset along with benchmarks on Atari 2600 games.

Original: A key advantage of our approach is that it allows easily accessible crowdsourcing for any existing RL environment, and it can provide realtime feedback and incentives that mirror arbitrary RL reward functions or other kinds of desired, complex behavior types.
Revised: A key advantage of our approach is that it allows easily accessible crowdsourcing for any existing RL environment, and it can provide realtime feedback and incentives that mirror arbitrary RL reward functions or other kinds of desired, complex behavior types.

Original: Further, by using standard RL environments we can streamline the end-to-end crowdsourcing and learning pipeline: CrowdPlay can use the same environment simulator that is used by an existing RL pipeline, enabling for instance mixed human-AI environments that use the same algorithms already used to train AI agents.
Revised: Further, by using standard RL environments we can streamline the end-to-end crowdsourcing and learning pipeline: CrowdPlay can use the same environment simulator that is used by an existing RL pipeline, enabling for instance mixed human-AI environments that use the same algorithms already used to train AI agents.

Original: We hope that this new framework and dataset will enable rapid progress in ofﬂine learning on large-scale human demonstrations.
Revised: We hope that this new framework and dataset will enable rapid progress in ofﬂine learning on large-scale human demonstrations.

Original: Atari 2600 games were seminal in driving progress in online RL, and we believe this dataset and pipeline can advance IL and ofﬂine RL research for the same reasons that Atari 2600 is interesting for online RL.
Revised: Atari 2600 games were seminal in driving progress in online RL, and we believe this dataset and pipeline can advance IL and ofﬂine RL research for the same reasons that Atari 2600 is interesting for online RL.

Original: Moreover, this pipeline is not constrained to Atari games, and we hope that it will be utilized for other environments.
Revised: Moreover, this pipeline is not constrained to Atari games, and we hope that it will be utilized for other environments.

Original: Beyond supporting progress in IL and ofﬂine RL, we also believe that CrowdPlay can enable other novel applications, both in computer science (such as task classiﬁcation), but also in other ﬁelds such as psychology or behavioral economics.
Revised: Beyond supporting progress in IL and ofﬂine RL, we also believe that CrowdPlay can enable other novel applications, both in computer science (such as task classiﬁcation), but also in other ﬁelds such as psychology or behavioral economics.

Original:  R EPRODUCIBILITY S TATEMENT
Revised:  R EPRODUCIBILITY S TATEMENT

Original: Our entire framework and current version of the dataset will be available publicly under an opensource license and we have provided the links to the implementations used to produce the benchmark results in Appendix B.
Revised: Our entire framework and current version of the dataset will be available publicly under an opensource license and we have provided the links to the implementations used to produce the benchmark results in Appendix B.

Original: A A PPENDIX A. T ECHNICAL D ETAILS
Revised: A A PPENDIX A. T ECHNICAL D ETAILS

Original: Architecture Overview CrowdPlay uses a client-server architecture, where an MDP environment is run in a simulator and its output streamed to a web browser client.
Revised: Architecture Overview CrowdPlay uses a client-server architecture, where an MDP environment is run in a simulator and its output streamed to a web browser client.

Original: The browser client collects user keypresses which are streamed to the server, where they are decoded into actions which are fed into the MDP simulator.
Revised: The browser client collects user keypresses which are streamed to the server, where they are decoded into actions which are fed into the MDP simulator.

Original: The backend hosts the MDP simulator, records trajectories into a database, and provides additional functionality.
Revised: The backend hosts the MDP simulator, records trajectories into a database, and provides additional functionality.

Original: Communication uses a http API for management tasks such as establishing a connection and a websockets for streaming keypresses and observations for better performance.
Revised: Communication uses a http API for management tasks such as establishing a connection, and a websockets for streaming keypresses and observations for better performance.

Original: The backend can optionally interface with an external load balancer and MySQL instance for scalability.
Revised: The backend can optionally interface with an external load balancer and MySQL instance for scalability.

Original: Backend Architecture
Revised: Backend Architecture

Original: The backend is written in Python for best compatibility with existing
Revised: The backend is written in Python for best compatibility with existing

Original: Python-based RL simulators and easy data interchange with ofﬂine learning pipelines.
Revised: Python-based RL simulators and easy data interchange with ofﬂine learning pipelines.

Original: It is highly parallelized across multiple processes: A main process handles http requests to the API, routes websockets communications, and manages the multiple concurrent MDP environments.
Revised: It is highly parallelized across multiple processes: A main process handles http requests to the API, routes websockets communications, and manages the multiple concurrent MDP environments.

Original: Each environment simulator is run in its own process dedicated in essence to just the main “fetch actionenvironment step - get observation” MDP loop.
Revised: Each environment simulator is run in its own process dedicated in essence to just the main “fetch actionenvironment step - get observation” MDP loop.

Original: Once an episode is complete, the entire trajectory is asynchronously serialized using Pickle and compressed using Bzip2 in a separate process and inserted into a database.
Revised: Once an episode is complete, the entire trajectory is asynchronously serialized using Pickle and compressed using Bzip2 in a separate process and inserted into a database.

Original: MDP simulator interface CrowdPlay interfaces with standard MDP simulators such as OpenAI
Revised: MDP simulator interface CrowdPlay interfaces with standard MDP simulators such as OpenAI

Original: Gym environments, but also similar interfaces.
Revised: Gym environments, but also similar interfaces.

Original: For our current set of experiments, we use a Gymlike interface implemented using RLLib’s multiagent environment deﬁnition.
Revised: For our current set of experiments, we use a Gymlike interface implemented using RLLib’s multiagent environment deﬁnition, and built on a multiagent version of ALE Terry et al.

Original: We use this even in single-agent games, although this is for convenience and not due to an architectural limitation.
Revised: We use this even in single-agent games, although this is for convenience and not due to an architectural limitation.

Original: The main requirements for the MDP simulator are that it provides a reset () and a step ( action ) function.
Revised: The main requirements for the MDP simulator are that it provides a reset () and a step ( action ) function.

Original: Frontend The CrowdPlay frontend is written in React for a modern UI experience.
Revised: Frontend The CrowdPlay frontend is written in React for a modern UI experience.

Original: Its main task is to receive image observations and associated metadata (such as real-time task progress, estimated bonus payment and other statistics) and display them to the user.
Revised: Its main task is to receive image observations and associated metadata (such as real-time task progress, estimated bonus payment and other statistics) and display them to the user.

Original: Optionally users can choose the task / game they wish to play in a library-style user interface; users can be shown a post-game “high score” page which shows performance and other game statistics relative to other participants; and users can report technical problems and leave feedback.
Revised: Optionally users can choose the task / game they wish to play in a library-style user interface; users can be shown a post-game “high score” page which shows performance and other game statistics relative to other participants; and users can report technical problems and leave feedback.

Original: For multiplayer games, the frontend supports waiting rooms and can notify players who joined early once enough other players have joined to start the game.
Revised: For multiplayer games, the frontend supports waiting rooms and can notify players who joined early once enough other players have joined to start the game.

Original: Environment and task deﬁnitions CrowdPlay supports multiple environments and environment variations concurrently.
Revised: Environment and task deﬁnitions CrowdPlay supports multiple environments and environment variations concurrently.

Original: Each variation or “task” can include a different base environment (e.g. a different Atari game, or a different MDP simulator altogether), different instructions for the user, different incentives and statistics (see below), different conﬁgurations of human and AI players in multiplayer games (see below) etc.
Revised: Each variation or “task” can include a different base environment (e.g. a different Atari game, or a different MDP simulator altogether), different instructions for the user, different incentives and statistics (see below), different conﬁgurations of human and AI players in multiplayer games (see below) etc.

Original: For instance, one task could be a two-player Space Invaders game with one human player and one AI agent, with the user instructed to follow a particular behavior in game, and a minimum effort requirement and bonus payment tied to their adherence to that behavior (for deployment to paid participants); while another task could be a two-player Space Invaders game with two human players and no incentives (for deployment to unpaid participants).
Revised: For instance, one task could be a two-player Space Invaders game with one human player and one AI agent, with the user instructed to follow a particular behavior in game, and a minimum effort requirement and bonus payment tied to their adherence to that behavior (for deployment to paid participants); while another task could be a two-player Space Invaders game with two human players and no incentives (for deployment to unpaid participants).

Original: Users can be assigned to a task through a URL parameter.
Revised: Users can be assigned to a task through a URL parameter.

Original: Tasks can also be grouped into task groups.
Revised: Tasks can also be grouped into task groups.

Original: Users can then either select a task of their choice among a task group; or the backend can automatically assign a task.
Revised: Users can then either select a task of their choice among a task group; or the backend can automatically assign a task.

Original: This automatic assignment can take into account the amount of high-quality data already collected for each task, and can either aim to collected a minimum amount for each task before assigning incoming users to the next task, or to collect an equal amount of data for each task.
Revised: This automatic assignment can take into account the amount of high-quality data already collected for each task, and can either aim to collected a minimum amount for each task before assigning incoming users to the next task, or to collect an equal amount of data for each task.

Original: Quality is determined by episode and session-level realtime statistics (see below).
Revised: Quality is determined by episode and session-level realtime statistics (see below).

Original: If any tasks in a task group are multiplayer tasks, the scheduler will ﬁrst try to assign users to any alreadyinstantiated environments that are waiting for additional human players, before considering creating a new environment.
Revised: If any tasks in a task group are multiplayer tasks, the scheduler will ﬁrst try to assign users to any alreadyinstantiated environments that are waiting for additional human players, before considering creating a new environment.

Original: AI agents Because CrowdPlay uses standard MDP interfaces also used in established RL methods, it can support AI agents trained with those methods.
Revised: AI agents Because CrowdPlay uses standard MDP interfaces also used in established RL methods, it can support AI agents trained with those methods.

Original: We currently have an implementation for policies trained with RLLib, which would be easy to extend to other existing libraries like OpenAI
Revised: We currently have an implementation for policies trained with RLLib, which would be easy to extend to other existing libraries like OpenAI

Original: Baselines.
Revised: Baselines.

Original: CrowdPlay includes a custom image preprocessor that replicates the behavior of standard “DeepMind” wrappers such as those included in OpenAI Gym, but without having to wrap around the MDP environment.
Revised: CrowdPlay includes a custom image preprocessor that replicates the behavior of standard “DeepMind” wrappers such as those included in OpenAI Gym, but without having to wrap around the MDP environment.

Original: This allows us to provide AI agents with a standard “four frames stacked, each four frames apart” observation, while providing human players with an unprocessed observation every frame.
Revised: This allows us to provide AI agents with a standard “four frames stacked, each four frames apart” observation, while providing human players with an unprocessed observation every frame.

Original: Multi-player Multi-agent environments are supported with both multiple human players as well as mixed human-AI games.
Revised: Multi-player Multi-agent environments are supported with both multiple human players as well as mixed human-AI games.

Original: For human players, the frontend routes multiple users’ websocket connections to the same environment process.
Revised: For human players, the frontend routes multiple users’ websocket connections to the same environment process.

Original: AI agents are implemented as above.
Revised: AI agents are implemented as above.

Original: As a key robustness feature, AI agent can take over on-the-ﬂy if a human player disconnects, which enables the remaining human players to continue playing.
Revised: As a key robustness feature, AI agent can take over on-the-ﬂy if a human player disconnects, which enables the remaining human players to continue playing.

Original: This is important on platforms such as Amazon Mechanical
Revised: This is important on platforms such as Amazon Mechanical

Original: Turk, so that one participant disconnecting does not leave another participant unable to complete their task through no fault of their own.
Revised: Turk, so that one participant disconnecting does not leave another participant unable to complete their task through no fault of their own.

Original: In such environments, we create fallback AI policies for all agents when the environment is started, to avoid an interruption during gameplay (instantiating the policy takes a few seconds).
Revised: In such environments, we create fallback AI policies for all agents when the environment is started, to avoid an interruption during gameplay (instantiating the policy takes a few seconds).

Original: However we only start observation preprocessing and querying the AI policy for actions for each agent only if and once the human player controlling that player disconnects.
Revised: However we only start observation preprocessing and querying the AI policy for actions for each agent only if and once the human player controlling that player disconnects.

Original: Real-time statistics engine CrowdPlay can calculate and record gameplay statistics and other metadata using an extensible real-time statistics engine.
Revised: Real-time statistics engine CrowdPlay can calculate and record gameplay statistics and other metadata using an extensible real-time statistics engine.

Original: The information to be recorded can be conﬁgured for each task.
Revised: The information to be recorded can be conﬁgured for each task.

Original: CrowdPlay ships with several generic statistics collectors such as for score and active playtime, and several game-speciﬁc collectors such as the player’s x-position in Space
Revised: CrowdPlay ships with several generic statistics collectors such as for score and active playtime, and several game-speciﬁc collectors such as the player’s x-position in Space

Original: Invaders and Riverraid, and the order in which aliens have been hit in Space Invaders.
Revised: Invaders and River Raid, and the order in which aliens have been hit in Space Invaders.

Original: These can be used to determine how closely the player’s gameplay behavior matches a given target behavior.
Revised: These can be used to determine how closely the player’s gameplay behavior matches a given target behavior.

Original: In turn, CrowdPlay supports minimum effort requirements to consider a task “complete” as well as sliding bonus payments based on the statistics.
Revised: In turn, CrowdPlay supports minimum effort requirements to consider a task “complete” as well as sliding bonus payments based on the statistics.

Original: For each statistic deﬁned for a task, per-step information is saved as part of the trajectory in a compressed binary blob, while episode-level and session-level aggregate statistics are saved as structured metadata in a relational database to allow for fast ﬁltering of individual trajectories in downstream ofﬂine learning pipelines.
Revised: For each statistic deﬁned for a task, per-step information is saved as part of the trajectory in a compressed binary blob, while episode-level and session-level aggregate statistics are saved as structured metadata in a relational database to allow for fast ﬁltering of individual trajectories in downstream ofﬂine learning pipelines.

Original: Statistic collectors are implemented as functions or Python callables that take as input the trajectory information at each step, including actions, observations and emulator state.
Revised: Statistic collectors are implemented as functions or Python callables that take as input the trajectory information at each step, including actions, observations and emulator state.

Original: Because all of this information is saved, statistic collectors can also be run ofﬂine against already recorded trajectories should requirements change later.
Revised: Because all of this information is saved, statistic collectors can also be run ofﬂine against already recorded trajectories should requirements change later.

Original: Deployment and Scalability The CrowdPlay backend is fully containerized using Docker and can be deployed on any compatible platform.
Revised: Deployment and Scalability The CrowdPlay backend is fully containerized using Docker and can be deployed on any compatible platform.

Original: Within-instance scaling to multiple processor cores is handled within the backend.
Revised: Within-instance scaling to multiple processor cores is handled within the backend.

Original: CrowdPlay also includes conﬁguration options for deployment using
Revised: CrowdPlay also includes conﬁguration options for deployment using

Original: Amazon Elastic Beanstalk (EB).
Revised: Amazon Elastic Beanstalk (EB).

Original: When deploying on EB, a load-balancer can be used, which enables auto-scaling across instances (additional server instances will be started automatically depending on load).
Revised: When deploying on EB, a load-balancer can be used, which enables auto-scaling across instances (additional server instances will be started automatically depending on load).

Original: This enables CrowdPlay to scale to hundreds or thousands of simultaneous users.
Revised: This enables CrowdPlay to scale to hundreds or thousands of simultaneous users.

Original: EB can also be conﬁgured to use AWS Spot Instances for signiﬁcantly reduced operating cost.
Revised: EB can also be conﬁgured to use AWS Spot Instances for signiﬁcantly reduced operating cost.

Revised: Latency CrowdPlay records at every keypress event the sequence number of the frame shown on screen at the time of the keypress.

Revised: This allows us to measure end-to-end latency of the system as the difference between the current frame sequence number in the back end when the keypress is received there, and the sequence number of the frame shown to the user when the key was pressed.

Revised: Across our current Atari dataset, we measured a median latency of 7 frames or 116ms, and downstream learning pipelines could ﬁlter for acceptable levels of latency should this be a concern.

Original: Dataset Engine Recorded trajectories and metadata can be downloaded into a locally installable
Revised: Dataset Engine Recorded trajectories and metadata can be downloaded into a locally installable

Original: Python package providing user-friendly metadata search and trajectory loading functionality.
Revised: Python package providing user-friendly metadata search and trajectory loading functionality.

Original: Download from the backend MySQL database can be online and incremental.
Revised: Download from the backend MySQL database can be online and incremental.

Original: Metadata is saved locally in a SQLite database, and is mapped into Python classes by the dataset package.
Revised: Metadata is saved locally in a SQLite database, and is mapped into Python classes by the dataset package.

Original: This allows ﬁltering episodes by metadata using either SQL queries, or native Python techniques (e.g. list comprehension) according on user preference.
Revised: This allows ﬁltering episodes by metadata using either by the backend, or expanded into uncompressed pickle ﬁles for much faster loading at the expense of storage efﬁciency. Observation Processing Most Atari RL pipelines use some variation of “Deepmind”-style observation preprocessing as pioneered in Mnih et al. This includes taking the pixel-wise maximum of two consecutive frames, skipping several frames between observation, rescaling images to 84x84 grayscale, and stacking four such consecutive observations. Frameskipping in particular is a challenge in our context as it effectively reduces the time-resolution of the environment by a factor of four (one observation given for every step in the Atari emulator), which we do not want to do for human players, but still need to do for AI players in mixed human-AI multiplayer games.

Original: Trajectories can be stored in the bzip2-compressed format used by the backend, or expanded into uncompressed pickle ﬁles for much faster loading at the expense of storage efﬁciency.
Revised: Similarly, all these processing steps are usually implemented as environment wrappers, which aren’t directly applicable to ofﬂine data. We therefore include a custom implementation of the above.

Original: A. D ATASET D ETAILS A.2. G AME V ARIANTS
Revised: This is implemented through a custom frame buffer that replaces the usual “Deepmind” wrappers. Our implementation keeps a buffer of the latest several raw observations and can generate a maxand-skipped, rescaled, framestacked observation on the ﬂy. The implementation is veriﬁed to give the same output frame-by-frame and pixel-by-pixel as standard implementations such as those in OpenAI Gym, and can be applied post-hoc to already recorded trajectories.

Revised: This same observation processing is also used for the AI agents that we bring into mixed human-AI environments.

Revised: A. D ATASET D ETAILS A.2. G AME V ARIANTS

Original: We instructed participants to follow these ﬁve behavior types:
Revised: We instructed participants to follow these ﬁve behavior types:

Original: Left-side-only In both Space Invaders and RiverRaid, we instructed participants to score as high as possible in the game, while staying only on the left half of the game screen.
Revised: Left-side-only In both Space Invaders and River Raid, we instructed participants to score as high as possible in the game, while staying only on the left half of the game screen.

Original: Right-side-only As above, but staying on the right side of the game screen.
Revised: Right-side-only As above, but staying on the right side of the game screen.

Original: Row-by-row In Space Invaders we instructed participants to maximize game score while aiming to shoot aliens row by row from the bottom; i e., participants were asked to completely clear the bottom-most row of aliens, then the second-from-the-bottom row, etc. Outside-in In Space Invaders, we instructed participants to maximize game
Revised: Row

Original: score while aiming
Revised: -by-row

Original: to
Revised: In

Original: shoot aliens column by column from the outside in; i

Original: e., participantsshould completely clear the two outermost columns, before moving on to the next two columns, etc [Table] Beamrider [Table]
Revised: Space Invaders we instructed participants to maximize game score while aiming to

Original: [Table]

Original: [Table]

Original: Riverraid [Table] [Table] left
Revised: shoot aliens row by row from the bottom

Original: [Table] [Table] - plain [Table]
Revised: ; i.e

Original: [Table] [Table] [Table] [Table]
Revised: , participants were

Original: right [Table] [Table] [Table]
Revised: asked to

Original: [Table] Space [Table] [Table]
Revised: completely

Original: [Table] [Table] [Table] - insideout
Revised: clear the bottom-most row of aliens

Original: [Table]

Original: - left 18.

Original: - outsidein 11. - plain 10. - right 33. - rowbyrow 18.
Revised: , then the second-from-the-bottom row, etc.

Original: Space Invaders

Original: [Table]

Original: - competitive 2. - cooperative 3. Space Invaders (2P w/AI [Table]
Revised: Outside-in In Space Invaders, we instructed participants to maximize game score while aiming to shoot aliens column by column from the outside in; i.e., participants should completely clear the two outermost columns, before moving on to the next two columns, etc

Original: - competitive 45. - cooperative 9. Total 243. Inside-out In Space Invaders, we asked participants to shoot aliens column by column, from the inside out; i.e., participants should completely clear the center two columns (column 3 and 4), then the next two columns from the center (columns 2 and 5), and ﬁnally the two outermost columns ( and 6).
Revised: Inside-out In Space Invaders, we asked participants to shoot aliens column by column, from the inside out; i.e., participants should completely clear the center two columns (column 3 and 4), then the next two columns from the center (columns 2 and 5), and ﬁnally the two outermost columns ( and 6).

Original: A.2. F ULL L IST OF I NCLUDED D ATA
Revised: A.2. F ULL L IST OF I NCLUDED D ATA

Original: Table 2 shows a detailed list of available data by game, game variant, and recruitment channel.
Revised: Table 2 shows a detailed list of available data by game, game variant, and recruitment channel

Original:  A.2. D ATA E VALUATION
Revised: . A.2. D ATA E VALUATION

Original: The dataset contains rich metadata accessible through both SQL queries as well as Python classes.
Revised: The dataset contains rich metadata accessible through both SQL queries as well as Python classes.

Original: Metadata includes universal attributes like total episode score and length, but also task-speciﬁc attributes like total and fraction of aliens shot in the correct order in the above Space Invaders tasks.
Revised: Metadata includes universal attributes like total episode score and length, but also task-speciﬁc attributes like total and fraction of aliens shot in the correct order in the above Space Invaders tasks.

Original: This allows ﬁne-grained ﬁltering of available trajectories according to speciﬁed criteria.
Revised: This allows ﬁne-grained ﬁltering of available trajectories according to speciﬁed criteria.

Original: We propose that “data quality” should be thought of as multidimensional.
Revised: We propose that “data quality” should be thought of as multidimensional.

Original: For instance, we saw in our multimodal task data both trajectories with moderately high task adherence (as fraction of aliens shot in correct order) and very high overall performance and length, and vice versa trajectories of moderately high performance and very high task adherence.
Revised: For instance, we saw in our multimodal task data both trajectories with moderately high task adherence (as fraction of aliens shot in correct order) and very high overall performance and length, and vice versa trajectories of moderately high performance and very high task adherence.

Original: Our dataset allows ﬁltering for minimum levels of both attributes, as well as others.
Revised: Our dataset allows ﬁltering for minimum levels of both attributes, as well as others.

Original: Further, any metadata not saved explicitly as such could be recovered from the trajectories themselves, as the full emulator RAM is stored in addition to observations. 
Revised: Further, any metadata not saved explicitly as such could be recovered from the trajectories themselves, as the full emulator RAM is stored in addition to observations. [Table]

Revised: For the analysis performed in this paper, the t-SNE embeddings were generated using scikit-learn version 1.0.1, with PCA initialization, and all other parameters left at default.

Original: A.2. E XAMPLE P ARTICIPANT I NSTRUCTIONS
Revised: A.2. E XAMPLE P ARTICIPANT I NSTRUCTIONS

Original: Standard Gameplay For standard gameplay, we provided participants with instructions on how to play the game, technical information and platform-dependent information on payment.
Revised: Standard Gameplay For standard gameplay, we provided participants with instructions on how to play the game, technical information and platform-dependent information on payment.

Original: For instance, for Space Invaders, our instructions to participants recruited via social media read as follows:
Revised: For instance, for Space Invaders, our instructions to participants recruited via social media read as follows:

Original: In this experiment you will play the game Space Invaders.
Revised: In this experiment you will play the game Space Invaders.

Original: Your goal is to shoot as many of the alien invaders shown on the screen as possible.
Revised: Your goal is to shoot as many of the alien invaders shown on the screen as possible.

Original: You control the small cannon at the bottom of the screen.
Revised: You control the small cannon at the bottom of the screen.

Original: Use your keyboard’s arrow keys to move your cannon left and right, and use the spacebar key to ﬁre.
Revised: Use your keyboard’s arrow keys to move your cannon left and right, and use the spacebar key to ﬁre.

Original: The aliens will ﬁre at you too.
Revised: The aliens will ﬁre at you too.

Original: If you are hit by alien ﬁre three times, the game ends.
Revised: If you are hit by alien ﬁre three times, the game ends.

Original: The game will also end if the alien invaders reach the bottom of the screen.
Revised: The game will also end if the alien invaders reach the bottom of the screen.

Original: A new game will then start automatically, but you can end the experiment clicking the Finish
Revised: A new game will then start automatically, but you can end the experiment clicking the Finish

Original: Experiment button on the top right.
Revised: Experiment button on the top right.

Original: You can end the experiment at any time, but we encourage you to play at least a few games.
Revised: You can end the experiment at any time, but we encourage you to play at least a few games.

Original: For participants recruited via MTurk, the instructions read:
Revised: For participants recruited via MTurk, the instructions read:

Original: In this experiment you will play the game Space Invaders.
Revised: In this experiment you will play the game Space Invaders.

Original: Your goal is to shoot as many of the alien invaders shown on the screen as possible.
Revised: Your goal is to shoot as many of the alien invaders shown on the screen as possible.

Original: You control the small cannon at the bottom of the screen.
Revised: You control the small cannon at the bottom of the screen.

Original: Use your keyboard’s arrow keys to move your cannon left and right, and use the spacebar key to ﬁre.
Revised: Use your keyboard’s arrow keys to move your cannon left and right, and use the spacebar key to ﬁre.

Original: The aliens will ﬁre at you too.
Revised: The aliens will ﬁre at you too.

Original: If you are hit by alien ﬁre three times, the game ends.
Revised: If you are hit by alien ﬁre three times, the game ends.

Original: The game will also end if the alien invaders reach the bottom of the screen.
Revised: The game will also end if the alien invaders reach the bottom of the screen.

Original: A new game will then start automatically.
Revised: A new game will then start automatically.

Original: You must play for at least 10 minutes and show some effort in the form of a minimum score to complete this task.
Revised: You must play for at least 10 minutes and show some effort in the form of a minimum score to complete this task.

Original: Your bonus payment will depend entirely on your score.
Revised: Your bonus payment will depend entirely on your score.

Original: Multimodal Behavior For speciﬁc multimodal behavior we showed speciﬁc instructions:
Revised: Multimodal Behavior For speciﬁc multimodal behavior we showed speciﬁc instructions:

Original: In this task you will play the game Space Invaders.
Revised: In this task you will play the game Space Invaders.

Original: Please shoot aliens in order ”inside out” starting with the inner-most columns (columns 3 and 4), then the second two columns from the inside (columns 2 and 5), then ﬁnally the outermost two columns (columns 1 and 6).
Revised: Please shoot aliens in order ”inside out” starting with the inner-most columns (columns 3 and 4), then the second two columns from the inside (columns 2 and 5), then ﬁnally the outermost two columns (columns 1 and 6).

Original: Do not shoot aliens except in this order.
Revised: Do not shoot aliens except in this order.

Original: User your keyboard’s arrow keys and space bar to interact with the game.
Revised: User your keyboard’s arrow keys and space bar to interact with the game.

Original: It is up to you to follow the instructions above, please follow them carefully.
Revised: It is up to you to follow the instructions above, please follow them carefully.

Original: Multiplayer Games For multiplayer games, we provided these instructions:
Revised: Multiplayer Games For multiplayer games, we provided these instructions:

Original: In this task you will play the game Space Invaders.
Revised: In this task you will play the game Space Invaders.

Original: This game is cooperative - you and the other player both get points if either of you hits an alien.
Revised: This game is cooperative - you and the other player both get points if either of you hits an alien.

Original: Please IGNORE
Revised: Please IGNORE

Original: THE SCORE SHOWN ON THE GAME SCREEN, and try to maximise the score shown in the text box to the right of the game screen.
Revised: THE SCORE SHOWN ON THE GAME SCREEN, and try to maximise the score shown in the text box to the right of the game screen.

Original: This score is shared between you and the other player.
Revised: This score is shared between you and the other player.

Original: You may want to cooperate with the other player.
Revised: You may want to cooperate with the other player.

Original: In all cases, speciﬁc instructions were followed with these generic instructions:
Revised: In all cases, speciﬁc instructions were followed with these generic instructions:

Original: User your keyboard’s arrow keys and space bar to interact with the game
Revised: User your keyboard’s arrow keys and space bar to interact with the game.

Original: It is up to you to follow the instructions above, please follow them carefully.
Revised: It is up to you to follow the instructions above, please follow them carefully.

Original: Please do not open multiple tabs or windows of this link at the same time.
Revised: Please do not open multiple tabs or windows of this link at the same time.

Original: In a two-player task, if you are not being connected to another human player, or there appear to be issues (e.g. other player is not moving at all), please click ‘Reload’ in your browser window and tell the other player to do the same.
Revised: In a two-player task, if you are not being connected to another human player, or there appear to be issues (e.g. other player is not moving at all), please click ‘Reload’ in your browser window and tell the other player to do the same.

Original: Instructions for other games and tasks were similar.
Revised: Instructions for other games and tasks were similar.

Original: A.2.  D ATASET ERRATA
Revised: A.2. M ULTIAGENT D ATA

Revised: Game Variants The CrowdPlay Atari dataset contains data from both standard two-player Space

Revised: Invaders as well as a custom cooperative mode.

Revised: These differ in two ways: Firstly, in the standard variant there is a bonus score when the other player gets hit by alien ﬁre.

Revised: We remove this score bonus in the cooperative variant, in order to remove incentives to hurt the other player.

Revised: Secondly, we give both players the average of their clipped rewards instead of the usual rewards.

Revised: For instance if one player hits an alien and gets an unclipped reward of 15, while the other player does not get a reward in the frame, we ﬁrst clip the rewards (from 15 and 0 to 1 and 0, respectively), and then average the clipped rewards to give each agent a reward of 0.5.

Revised: In the standard non-cooperative variant, the players would receive rewards of 1 and 0, respectively.

Revised: Fallback AI Agents AI agents were also used in nominally human-human environments with

Revised: MTurk participants, in two circumstances.

Revised: One, if one human player disconnected, an AI agent took over control.

Revised: This is to allow the remaining MTurk worker to complete their task.

Revised: Similarly, if an MTurk worker was waiting for 15 minutes without a second human player connecting, a game would start with an AI agent controlling the other player.

Revised: The dataset includes a record of what type of agent was controlling each player at every frame.

Revised: AI Agent Training AI agents were trained in two-agent environments with two separate A2C policies.

Revised: We used RLLib’s implementation of A2C (Mnih et al., 2016; Liang et al., 2018) formillion timesteps, using standard Deepmind-style observation processing.

Revised: We used Ray / RLLib version 1.4.0, and used the default hyperparameters for A2C therein.

Revised: The only hyperparameters we explicitly set is the learning rate schedule: 0 . 0007 at training start, 1e-12 at training end, linear in

Revised: [Table]

Revised: between.

Revised: We trained separate agents for the standard and cooperative variants as described above.

Revised: Table 3 shows the hyperparameters used.

Revised: Following ﬁndings in (Tylkin et al., 2021)

Revised: we trained AI agents using randomized starting positions each episode for better exploration of the states likely encountered in human-AI environments

Revised: . A.2. D ATASET ERRATA

Original: In a number of space invaders right trajectories, we had initially set a requirement of at least 90% time spent on the right side of the screen.
Revised: In a number of space invaders right trajectories, we had initially set a requirement of at least 90% time spent on the right side of the screen.

Original: This proved too hard for some MTurkers, and we thus lowered the requirement to 80%.
Revised: This proved too hard for some MTurkers, and we thus lowered the requirement to 80%.

Original: This is not distinguished through a separate task ID.
Revised: This is not distinguished through a separate task ID.

Original: MTurkers were compensated for their time in these instances.
Revised: MTurkers were compensated for their time in these instances.

Original: In a small number of SI insideout email, RR left right email tasks due to an error a task adherence of 80 instead of 0.8 was formally required, and the task thus not completeable.
Revised: In a small number of SI insideout email, RR left right email tasks due to an error a task adherence of 80 instead of 0.8 was formally required, and the task thus not completeable.

Original: We later corrected the effort requirement after the fact in the metadata database, both to be able to automatically enter those who achieved the intended requirements into a rafﬂe, and to allow for ﬁltering on that metadata as it was intended.
Revised: We later corrected the effort requirement after the fact in the metadata database, both to be able to automatically enter those who achieved the intended requirements into a rafﬂe, and to allow for ﬁltering on that metadata as it was intended.

Original: This is the only instance where metadata in the database was altered post-hoc.
Revised: This is the only instance where metadata in the database was altered post-hoc.

Original: A. R ECRUITMENT AND P LATFORM I NTEGRATIONS
Revised: A. R ECRUITMENT AND P LATFORM I NTEGRATIONS

Original: We currently target and explicitly support the following platforms.
Revised: We currently target and explicitly support the following platforms.

Original: Amazon Mechanical Turk CrowdPlay can capture and store worker IDs and other metadata for payment processing.
Revised: Amazon Mechanical Turk CrowdPlay can capture and store worker IDs and other metadata for payment processing.

Original: Participants are paid a ﬁxed sum for their participation subject to minimum requirements, as well as a variable bonus payment depending on their performance.
Revised: Participants are paid a ﬁxed sum for their participation subject to minimum requirements, as well as a variable bonus payment depending on their performance.

Original: As above, these are shown to users in real-time, and stored for simple payment processing.
Revised: As above, these are shown to users in real-time, and stored for simple payment processing.

Original: In multi-player environments, AI agents can take over if one user disconnects, to allow remaining users to successfully complete their tasks.
Revised: In multi-player environments, AI agents can take over if one user disconnects, to allow remaining users to successfully complete their tasks.

Original: Participants can be assigned to tasks automatically based on previous participant’s performance in a way that aims to balance the amount of high-quality data recorded for each task against deﬁned targets.
Revised: Participants can be assigned to tasks automatically based on previous participant’s performance in a way that aims to balance the amount of high-quality data recorded for each task against deﬁned targets.

Original: Direct User Payments and Rafﬂes Our architecture is able to capture user data like email addresses to contact them for payment or to enter them into a rafﬂe.
Revised: Direct User Payments and Rafﬂes Our architecture is able to capture user data like email addresses to contact them for payment or to enter them into a rafﬂe.

Original: Task-speciﬁc minimum time and effort requirements can be enforced similarly to MTurk.
Revised: Task-speciﬁc minimum time and effort requirements can be enforced similarly to MTurk.

Original: Lab in the Wild (Reinecke & Gajos, 2015) is a platform for recruiting diverse participants for online experiments.
Revised: Lab in the Wild (Reinecke & Gajos, 2015) is a platform for recruiting diverse participants for online experiments.

Original: Participants are not monetarily compensated, but are instead motivated through gaining insights about themselves through the experiments.
Revised: Participants are not monetarily compensated, but are instead motivated through gaining insights about themselves through the experiments.

Original: Experiments are thus required to give users feedback with interesting conclusions about the user’s performance and behavior.
Revised: Experiments are thus required to give users feedback with interesting conclusions about the user’s performance and behavior.

Original: CrowdPlay supports this through its real-time statistics engine, which is used to show users a feedback page after they ﬁnish the experiment.
Revised: CrowdPlay supports this through its real-time statistics engine, which is used to show users a feedback page after they ﬁnish the experiment.

Original: Users are shown a variety of statistics about their gameplay behavior beyond score, e.g. shot accuracy, including a comparison with the experiment cohort.
Revised: Users are shown a variety of statistics about their gameplay behavior beyond score, e.g. shot accuracy, including a comparison with the experiment cohort.

Original: Figure(right) shows a screenshot of this.
Revised: Figure(right) shows a screenshot of this.

Original: Social Media Similarly to Lab in the Wild, participants recruited from social media platforms are not compensated for their time but intrinsically motivated.
Revised: Social Media Similarly to Lab in the Wild, participants recruited from social media platforms are not compensated for their time but intrinsically motivated.

Original: CrowdPlay supports user choice of individual games and tasks through a modern UI resembling a “game library”, and uses real-time statistics to restate task instructions as “challenges”.
Revised: CrowdPlay supports user choice of individual games and tasks through a modern UI resembling a “game library”, and uses real-time statistics to restate task instructions as “challenges”.

Original: Figure 5 (left) shows a screenshot of the “game library” feature.[Table] [Table]
Revised: Figure 6 (left) shows a screenshot of the “game library” feature.

Original: B B ENCHMARK E XPERIMENT D ETAILS
Revised: B B ENCHMARK E XPERIMENT D ETAILS

Original: We performed all the experiments using the open source implementations of the algorithms that are provided as part of d3rlpy 3 library.
Revised: We performed all the experiments using the open source implementations of the algorithms that are provided as part of d3rlpy 3 library, speciﬁcally the version with commit hash e89445956d8570102a8ca3b34bf0db4538b3b43e.

Original: We follow the default training parameters for all the algorithms and run each algorithm for 1M iterations before reporting the score.
Revised: We follow the default training parameters for all the algorithms and run each algorithm for 1M iterations before reporting the score.

Original: For the normalized score, we consider best score as the one obtained by a human on the particular task.
Revised: For the normalized score, we consider best score as the one obtained by a human on the particular task.

Original: Random score is the average over 5 runs of random policy for each task and we comput the normalized score as: 
Revised: Random score is the average over 5 runs of random policy for each task and we comput the normalized score as: Normalized [Equation] [Table] [Table]

Original: Normalized Score = 100 ∗
Revised: SI-OI -9. SI-row -33. SI-regular -2.

Original: [Figure]
Revised: RR-L -5. RR-regular -18. Q*Bert -0. BeamRider* -33.