Large-Scale Retrieval for Reinforcement Learning
Anonymous Author(s)
AfﬁliationAddress email
Abstract
Effective decision making involves ﬂexibly relating past experiences and relevantcontextual information to a novel situation.
In deep reinforcement learning (RL),the dominant paradigm is for an agent to amortise information that helps decisionmaking into its network weights via gradient descent on training losses.
Here,we pursue an alternative approach in which agents can utilise large-scale contextsensitive database lookups to support their parametric computations.
This allowsagents to directly learn in an end-to-end manner to utilise relevant informationto inform their outputs.
In addition, new information can be attended to by theagent, without retraining, by simply augmenting the retrieval dataset.
We studythis approach for ofﬂine RL in 9x9 Go, a challenging game for which the vastcombinatorial state space privileges generalisation over direct matching to pastexperiences.
We leverage fast, approximate nearest neighbor techniques in order toretrieve relevant data from a set of tens of millions of expert demonstration states.
Attending to this information provides a signiﬁcant boost to prediction accuracyand game-play performance over simply using these demonstrations as trainingtrajectories, providing a compelling demonstration of the value of large-scaleretrieval in ofﬂine RL agents.
Introduction
How can reinforcement learning (RL) agents leverage relevant information to inform their decisions?
Deep RL agents have typically been represented as a monolithic parametric function, trained togradually amortise useful information from experience by gradient descent.
This has been effective [25, 38], but is a slow means of integrating experience, with no straightforward way for an agentto incorporate new information without many additional gradient updates.
In addition, this requiresincreasingly massive models (see e.g. [31]) as environments become more complex – scaling is drivenby the dual role of the parametric function, which must support both computation and memorisation.
Finally, this approach has a further drawback of particular relevance in RL – the only way in whichpreviously encountered information (that is not contained in working memory) can aid decisionmaking in a novel situation is indirectly through weight changes mediated by network losses.
Thereis no end-to-end means for an agent to attend to information outside of working memory to directlyinform its actions.
While there has been a signiﬁcant amount of work focused on increasing the information available from previous experiences within an episode (e.g., recurrent networks, slot-basedmemory [19, 28]), more extensive direct use of more general forms of experience or data has beenlimited, although some recent works have begun to explore utilising inter-episodic information fromthe same agent [6, 10, 29, 34, 41].
We seek to drastically expand the scale of information that isaccessible to an agent, allowing it to attend to tens of millions of pieces of information, while learningin an end-to-end manner how to use this information for decision making.
We view this as a ﬁrst steptowards a vision in which an agent can ﬂexibly draw on diverse and large-scale information sources,including its own (inter-episodic) experiences along with experiences from humans and other agents.
In addition, given that retrieved information need not match the agent’s observation format, retrieval could enable agents to integrate information sources that have not typically been utilised such asvideos [2], text, and third-person demonstrations.
We investigate a semi-parametric agent architecture for large-scale retrieval, in which fast and efﬁcientapproximate nearest neighbor matching is used to dynamically retrieve relevant information from adataset of experience.
We evaluate this approach in an ofﬂine RL setting for an environment with acombinatorial state space – the game of 9x9 Go with ≈ 10 38 possible games, where generalisationfrom past data to novel situations is challenging.
We equip our agent with a large-scale dataset of∼ 50M Go board-state observations, ﬁnding that a retrieval-based approach utilising this data is ableto consistently and signiﬁcantly outperform a strong non-retrieval baseline.
Several key advantages of this retrieval approach are worth highlighting: Instead of having to amortiseall relevant information into its network weights, a retrieval-augmented network can utilise more of itscapacity for computation.
In addition, this semi-parametric form allows us to update the informationavailable to the agent at evaluation time without having to retrain it.
Strikingly, we ﬁnd that thisenables improvements to agent performance without further training when games played against theevaluation opponent are added to its knowledge base.
Methods
[Table]
[Table]
and associated
[Table]
[Figure]
Before introducing the details of our method,let us ﬁrst present its high-level ingredients.
We train a model-based agent [36] that predicts future policies and values conditionedon future actions in a given state.
This semiparametric model incorporates a retrieval mechanism, which allows it to utilise information froma large-scale dataset to inform its predictions.
We train the agent in a supervised ofﬂine RLsetting, which allows us to directly evaluate thedegree to which retrieval improves the qualityof the model predictions.
We subsequently evaluate the resulting model, augmented by MonteCarlo tree search, against a reference opponent.
This allows us to determine how these improvements translate to an effective acting policy forout-of-training-distribution situations.
To effectively improve model predictions usingauxiliary information, we need 1) a scalable wayto select and retrieve relevant data and 2) a robust way of leveraging that data in our model.
As with many successful attention mechanisms,we establish relevance for 1) through an innerproduct in an appropriate key-query space andselect the top N relevant results.
Choosing the relevant key-query space is critical, since performing nearest-neighbor on the raw observations will only deliver desirable results for the simplest ofdomains.
For example, in the game of Go, a single stone difference on the board can dramaticallychange the outcome and reading of the board, while seemingly different board positions might stillshare high-level characteristics (e.g. inﬂuence, partial local sequences, and life-and-death of groups).
At the scale we want to operate, learning the key & query embeddings end-to-end in order to optimizethe ﬁnal model predictions is challenging.
Instead, as in the language modelling work by Borgeaudet al.
[7], we learn an embedding function through a surrogate procedure and use the resulting frozenfunction to describe domain-relevant similarity.
Moreover, scale also constrains the nearest-neighborslookup to be approximate, since getting the true nearest-neighbors is too time-consuming at inferencetime and good approximations can be obtained.
To address 2) and make the best use of the relevant data, we provide the data associated withthe nearest-neighbor as additional input features to the parametric part of our model, so that thenetwork can learn to interpret it.
We provide some light inductive biases in the architecture to ensure permutation invariance in the neighbors and robustness to outliers and distribution shifts(see Secs.
Letting the network interpret the nearest-neighbor data is essential in largeand complex environments, such as Go, because the typically imperfect match between the currentsituation and retrieved neighbors will require different aspects of this retrieved data to be ignoredor emphasized in a highly context speciﬁc manner.
This is in contrast to episodic RL approacheslike [6, 29] which prescribe how to use the retrieved data.
Subsequent sections describe the model,retrieval process, and the full algorithm in more detail.
2.1 Retrieval-augmented agents
We consider a setting in which we wish to train a neural network model m θ on a set of environmenttrajectories τ ∈ D (for example, the dataset D π of trajectories produced by a policy π ).
For eachtimestep t of τ , the model takes an observation o t 1 and must produce predictions ˆ y of targets y t .
In our retrieval paradigm, in addition to having access to o t , the model also has access to an auxiliarydataset of knowledge or experience D r .
The auxiliary set D r could be the same as D , but it canalso contain more information, or in the most general case, information from a completely differentdistribution and in a different format.
If there is overlap between D r and D , it is important forrobustness to ensure that the model cannot retrieve the same trajectory as it is being trained on(otherwise the network will tend to overly trust information retrieved from D r ).
Common informationsources in RL are trajectories from other agents or experts (ofﬂine RL) or the agent’s own previousexperiences (episodic memory, replay buffer).
Note that, in contrast to ofﬂine RL, we do not assumethat we should directly use this auxiliary data as trajectories to train on.
We wish to use D r to inform the model predictions ˆ y t , such that ˆ y t
= m θ ( o t , D r ) .
This is challenging,as D r is typically far too large to directly be consumed as a model input.
One solution, shown in
Fig. 1, is to adopt a retrieval-based model, wherein the parametric and differentiable portion ofthe model m θ is provided with an informative subset of data { x 1 t , x 2 t , . . .
, x Nt } retrieved from D r ,conditioned on o t :
[Equation]
This approach requires a number of design choices relating to the retrieval mechanism, which weexplore further below.
2.1.1 Scalable nearest-neighbor retrieval using SCaNN
Inspired by previous work in language modelling, we chose to leverage the SCaNN architecture [11]for fast approximate nearest-neighbor retrieval.
This requires each entry
o i in the dataset D r to beassociated with a key vector k i ∈ R d , and a given observation o t to be mapped to a query vectorq t ∈ R d .
During retrieval, the squared Euclidean distances between q t and the dataset keys k i areused to determine which neighbors to retrieve – reminiscent of neural attention mechanisms.
Thisretrieval process is very efﬁcient and can be scaled to datasets with billions of items.
We will typically want to retrieve further associated meta-data, or context, for each neighbor.
Forexample, if a neighbor is part of a trajectory, we would like to retrieve information about actionchoices and their consequences.
The neighbor observation o i , together with its meta-data, forms theauxiliary input x it to the model.
The nearest-neighbor retrieval process is non-differentiable, which means that the query and keymappings cannot be trained end-to-end directly.
Instead, in this ﬁrst study, we pre-train a non-retrievalprediction network m eφ on our experience dataset D (details in Sec.
A.4).
We then use this network togenerate an embedding corresponding to a given observation o t by retrieving the network activationsfrom a speciﬁed layer of m eφ .
We use principal component analysis to compress these activations to ad dimensional vector representing this observation state.
The embedding and projection step togetherform our key (& query 3 ) network k i
=
g φ ( o i ) .
We preprocess all of the observations in D r using g φ to produce corresponding keys.
The resultingdataset of keys and observations is then used by ScaNN to retrieve neighbors given a query vector.
For
[Table] [Table]
[Figure]
(cid:5)(cid:28)(cid:21)(cid:4)(cid:26)(cid:21)(cid:18)(cid:31)(cid:13)(cid:28)(cid:25)(cid:3)(cid:11)(cid:22)(cid:26)(cid:22)(cid:20)(cid:28)(cid:17) (cid:29)(cid:21)(cid:26)(cid:18)(cid:20)(cid:24)(cid:17)(cid:22)(cid:23) ofﬂine RL training with a ﬁxed training dataset, we can also preprocess the nearest neighbor lookupsfor all training dataset observations, i.e., q t =
g φ ( o ) ∀ o t ∈ D .
However, to act during evaluation, wemust do this nearest-neighbor lookup online for each new observation.
In future experiments, weintend to incorporate end-to-end learning of the query vector [12] to improve agent performance.
Afuture online RL pipeline will also require us to dynamically retrieve neighbors during training.
2.1.2 Policy optimization with retrieval
The objective in our ofﬂine RL setting is to leverage the available data in order to optimize a policyπ for acting.
Our proposed semi-parametric model (Eqn. 1) is compatible with several methodsfor policy optimization in this ofﬂine setting.
Indeed, it can represent the actor π θ ( o t , D r ) and/orthe critic Q θ ( o t , a, D r ) in an ofﬂine actor-critic method [21], or the Q-value in a value iterationapproach [32].
Here we focus on a model-based approach, inspired by MuZero [36], where thelearned model is employed in a search procedure based on Monte-Carlo Tree Search (MCTS) [8].
This is motivated by the efﬁcacy of MuZero in the ofﬂine RL setting [37, 24].
Model-based search with retrieval Following MuZero, our prediction model m θ is conditionedon the current observation
o t but also on a sequence of future actions (cid:126)a t = a t +1 , a t +2 , . .
. , a t + K .
The model m θ is therefore redeﬁned as ˆ y t = m θ ( o t ,(cid:126)a t , x 1 t , x 2 t , . . .
, x Nt ) .
The architecture for thisretrieval prediction model is illustrated in Fig.
The model m θ can be decomposed into an encodingstep s t = f θ ( o t , x 1 t , x 2 t , . . .
, x nt ) , which incorporates the observation and neighbor information,followed by iterative action-conditioned inference s k +1 t
= h θ ( s kt , a t + k ) to produce embeddingss kt corresponding to subsequent time steps (where s 0 t =
s t ).
The model output ˆ y t is composed of K + 1 value and policy distributions for the current and next K time-steps: { ˆ v kt , ˆ π kt } k =0 ...
K .
These outputs are obtained from their respective embeddings s kt .
The target for value predictionsis the game outcome and the targets for the policy predictions are the actions taken by the experta t , a t +1 , . .
. , a t + K .
The sample loss L ( θ, ˆ y t , y t ) is obtained by summing the K + 1 individual lossterms for value and policy predictions (see detail in App. A.2).
The training procedure for thisretrieval prediction model is summarized in Alg 1.
In order to act using the trained model, online search is used to generate an improved policyπ s = Search ( m θ , D r ) .
We use MCTS with a pUCT rule for internal action selection [35, 40],carrying out n sims simulations per time step.
Model-based search is implemented as follows: afterretrieving the observation’s neighbors, the encoded state s t is computed.
Search is carried out froms t by varying the input action sequence (cid:126)a for each simulation and collecting the model outputs toupdate search statistics.
Details of this process can be found in [36].
At the end of the search, theresulting policy π s ( a | o t ) can be sampled to select the next action.
Due to the semi-parametric nature of the model supporting the search, introducing changes to D r willhave an immediate effect on the acting policy π s , even if the model parameters θ remain ﬁxed.
As weexplore in Sec. 3.4, this provide a mechanism for fast adaption to new information or experiences.
2.1.3 Using neighbor information
Several choices are possible for the network f θ that processes the observation and neighbors.
Sincethis is not the main focus of this work, we chose a straightforward approach.
We ﬁrst
computean embedding o et of the observation o t .
We then compute an embedding for each neighbor e ti =p
θ ( o et , x ti ) , using the same network p θ for all neighbors.
These streams are combined in a permutationinvariant way through a sum, and then concatenated with o e t to produce the ﬁnal embedding s t , :
[Equation]
2.2 Evaluating retrieval in the domain of Go
In order to test the utility of retrieval in RL, we wish to evaluate whether agents can effectivelygeneralise between related but distinct experiences, as opposed to simply retrieving the outcomeof a previous instance of an identical situation.
This motivates our choice of Go as a domain.
Wefocus on 9x9 Go, instead of full-scale 19x19 Go, as the less demanding computational costs of 9x9experiments enable a more thorough analysis.
Even for 9x9 Go, the state space of 10 38 possiblegames is vastly larger than the number of positions that we could hope to query.
We collected a dataset of ∼ 3.5M expert 9x9 Go self-play games from an AlphaZero-style agent [40].
We randomly subsampled 15% of the positions from these games, leaving us with ∼ 50M boardstate observations.
These, along with metadata on the game outcome, ﬁnal game board, and future actions (all encoded as input planes), form our retrieval dataset D r .
We chose to subsample aswe hypothesised that this would reduce the chance of retrieving multiple neighbors from the sametrajectory, and therefore boost the retrieved neighbor diversity.
However, we did not perform ablationson this choice.
A future solution would be to use a ﬁltering mechanism to reject neighbors fromthe same trajectory.
In this initial work, training and retrieval datasets are the same – at least duringtraining.
During training, we split D r in two halves such that each game’s observations are only inone of the datasets. We retrieve neighbors for an observation o t from the half it is not contained in.
This is simply to avoid retrieving the same position as the query.
Other ways to obtain the same effectcould be devised.
2.3 Regularisation
We wish to make our network robust to poor quality neighbors, in order to ensure that the network canperform well in settings for which there is lower overlap with the retrieval dataset than encounteredin training.
We therefore explore several techniques to improve network robustness to irrelevantneighbors.
We randomly zero-out a subset of retrieved neighbors during training ("neighbor dropout"),and/or more adversarially, randomly replace a subset of retrieved neighbors with the neighbors of adifferent observation ("neighbor randomisation").
Inspired by [10], we also explore using a loss toregularise the embedding produced by the neighbor retrieval towards the embedding produced withthe observation alone ("neighbor regularisation").
Further details are given in Sec.
A.5.
We carried out ablations of these techniques (Appendix Fig. 7), which show that neighbor randomisation is important in some contexts for maintaining performance, but that the others do not seemto have a signiﬁcant effect in our ﬁnal conﬁguration.
The results reported in this study utilise all ofthese augmentations, as during the development process they had been found to slightly beneﬁt performance.
Interestingly, as we explore in later sections, MCTS with enough simulations compensatesfor the harmful effect of low-quality neighbors, and can perform effectively without these trainingaugmentations.
Results
3.1 Qualitative examination of retrieved neighbors
[Figure]
Changing a single stone in a Go board can dramatically affect the game, and the number of Gopositions is enormous.
We ﬁrst wanted to assess whether, despite the combinatorial aspect of the domain, meaningful nearest-neighbors could be retrieved from the dataset using our learnedkeys/queries.
Examples of retrieved positions in Go are shown in Fig.
3, where we observed relevantmatches in terms of both local and global structure.
In some cases, especially in the early game, theretrieved position is an exact match even though the rest of the game (and therefore the associatednearest-neighbor meta-data) differs.
Row 1 in Fig.
3 is an example of this – in this case, the retrieveddata effectively provides sample rollouts from the query position akin to those derived from MCTS.
[Figure]
3.2 Impact of retrieval on supervised training
We next evaluated the extent to which the model can learn to exploit retrieved information forbetter model predictions and better decision-making.
First, we evaluated the impact of retrieval onsupervised learning losses.
As a baseline comparison, we trained the same network architecturesbut setting x i = 0 for all i — this has the effect of maintaining the number of parameters, ﬂops,and useful capacity in the network, while removing access to the retrieval data.
We evaluated lossesfor retrieval networks trained with different N (Fig. 4b) and model sizes (Fig.
4c, see Sec.
A.1 fordetails).
Across conditions, we consistently observed a signiﬁcant boost in test-set accuracy for allmetrics and over the course of training.
This improvement to predictions is further observed acrossgame trajectories, and is not limited, for example, to opening play positions.
One advantage of the semi-parametric approach we outlined is that we can modify the retrievaldataset and potentially see immediate effects on the prediction, without changing the parameters θ .
We ﬁrst veriﬁed this by evaluating our model when allowed access to varying fractions of the fulldataset D r (Fig. 4d).
We observed clear gains in prediction accuracy from increasing the size ofthe retrieval set (only half is used at train time).
A further important observation is that large-scaledatasets are clearly important - the evaluation metrics drop below the baseline level for a dataset D rthat is 1% the size of the full dataset.
3.3 Evaluation against a reference opponent
While the results observed on the ofﬂine dataset suggest a strong positive effect from learning toleverage the retrieval data, this is in the context of a ﬁxed test data distribution that matches theretrieval data distribution.
When deployed, games played against different opponents will likely diverge from that distribution.
We evaluated the performance of our search policy π s ( n sims = 200 )by playing against a ﬁxed reference opponent — the Pachi program [5], which can perform beyondstrong amateur level of play in 9x
Go given sufﬁcient simulation budget (we evaluate against 400ksimulations).
We observed a signiﬁcant boost to performance for retrieval-based networks over theequivalent baseline network of the same capacity (Fig. 5).
Interestingly, as shown in Appendix Fig.
8,playing using only the base policy prior ˆ π 0 shows a much smaller boost over the baseline network.
As we explore further below, the quality of retrieved neighbors available during play is much lowerthan for training - we hypothesise that the search policy is more robust to this distributional shift thanˆ π 0 .
[Figure]
Retrieval leads to a clear performance boost compared to non-retrieval baselines of the same capacity.
[Figure]
We observed a signiﬁcant distribution shift between game positions observed in play against the Pachiopponent versus positions in the retrieval dataset (Fig. 6a), as measured by the empirical distancedistribution of game observations to their approximate nearest neighbors.
Changing or augmentingthe retrieval dataset D r modiﬁes this distributional shift.
A particularly interesting modiﬁcation in thissetting is to augment the dataset with play recorded between our agents and the Pachi opponent.
Ascan be seen in Fig.
6a, augmenting the dataset with a set of ∼ 600k agent-Pachi game states increasesthe similarity between positions observed in play and their retrieved neighbors.
Strikingly, we foundthat integrating these games into our retrieval dataset leads to improvements in performance withoutfurther training for subsequent play against the Pachi opponent (play with the MCTS policy
π s isshown in Fig.
6b, play with the policy prior ˆ π 0 is shown in Appendix Fig. 9).
This highlights thepotential of a semi-parametric model to rapidly integrate recent experience without further training.
As a related intervention, we instead inputted randomly retrieved neighbors into the model atevaluation time (see Appendix Fig. 6).
This signiﬁcantly impaired performance for play using thepolicy prior ˆ π 0 , but somewhat unexpectedly, for play with the MCTS policy π s , performance did notfully regress to the level of the baseline network.
This suggests that our retrieval network is able toamortise some information from neighbors during training, leading to better performance even whenno relevant neighbors are available.
Related work
The idea of supporting decision making by directly attending to a cache of related events hasbeen visited many times in different contexts, under the names of case-based reasoning [30], nonparametric RL [16, 4, 1, 27], or episodic memory [20, 23].
The aim of some methods is to bettersupport the working memory of an agent during a given episode [26, 42] or a series of successiverelated episodes [33].
Other methods, including ours, aim to leverage a broader class of relevantexperience or persistent knowledge (including across episodes, and from other agents) to bettersupport reasoning and planning.
One differentiating factor in our work to these past approaches is thatwe do not prescribe how to process the information from the available data (e.g. through specifyingthe agent’s action-value directly in terms of previously generated value estimates [6, 13, 15, 29], or amodel from observed transitions [39]) but rather learn end-to-end how the data can support betterpredictions within the parametric model.
A recent approach by Goyal et al.
[10] has considered anattention mechanism to select where and what to use from available trajectories, but over a smallretrieval batch of data rather than the full available experience data.
Another class of method toleverage a transition dataset is to replay the data at training time in order to perform more gradientsteps per experience, this is a widespread technique in modern RL algorithms [21, 22, 25, 36] but itdoes not beneﬁt the agent at test time, requires additional learning steps to adapt to new data, anddoes not allow end-to-end learning of how to relate past experience to new situations.
Discussion
Our approach and empirical results highlight how reinforcement learning agents can beneﬁt fromdirect access to a large collection of raw interaction data at inference time, through a retrievalmechanism, in addition to their already effective parametric representation.
We showed this was thecase even 1) when the domain is large enough to require generalisation in how to interpret past data,2) when there is signiﬁcant distribution shift when acting, and 3) at a scale where only approximatenearest-neighbors can be retrieved.
We believe this already demonstrates the potential of this approachfor many possible scenarios and applications.
We show that retrieval can be effectively combined with model-based search.
We ﬁnd that thebeneﬁts from retrieval and search are synergistic, with increasing numbers of retrieved neighbors andincreasing simulations both leading to performance increases in almost all contexts we investigated.
Furthermore, empirical evidence suggests that search signiﬁcantly improves agent robustness todistributional shift as compared to playing with the policy prior.
In Appendix Fig.
10, we comparethe boost in performance as a function of parametric-model compute cost for increasing MCTS simulations versus increasing the number of retrieved neighbors processed by the model.
This provides atentative indication that retrieval is also a compute efﬁcient means of improving performance.
A key future direction is to investigate the online learning scenario in which recent experience of theagent is rapidly made available for retrieval, hence progressively growing the retrieval dataset overthe course of training.
While there are additional challenges associated with the online paradigm(e.g.
, it may be desirable to update the queries and/or keys during training [12]), the fast adaptationeffect we highlighted in this work may have even more impact there.
There are many potentially relevant sources of information beyond an agent’s own experience, orthat of other humans or agents.
For example, it has been shown that YouTube videos are usefulfor learning to play the Atari game Montezuma’s revenge [2].
Training an embedding network onsufﬁciently diverse data may enable retrieval of information from a wide range of contexts [43],including third-person demonstrations, videos and perhaps even books.
Checklist
For all authors...
(a) Do the main claims made in the abstract and introduction accurately reﬂect the
paper’scontributions and scope?
[Yes]
(b) Did you describe the limitations of your work?
[Yes]
(c) Did you discuss any potential negative societal impacts of your work?
[No]
(d) Have you read the ethics review guidelines and ensured that your paper conforms to them?
[Yes]
If you are including theoretical results...
(a) Did you state the full set of assumptions of all theoretical results?
[N/A]
No theoretical
results.
(b) Did you include complete proofs of all theoretical results?
[N/A]
If you ran experiments...
(a) Did you include the code, data, and instructions needed to reproduce the main experi- mental results (either in the supplemental material or as a URL)?
[No] Instructions and links to relevant libraries were provided.
(b) Did you specify all the training details (e.g., data splits, hyperparameters, how they were chosen)?
[Yes]
(c) Did you report error bars (e.g., with respect to the random seed after running experi- ments multiple times)?
[Yes] (d) Did you include the total amount of compute and the type of resources used (e g., typeof GPUs, internal cluster, or cloud provider)?
[Yes
If you are using existing assets (e
g , code, data, models) or curating/releasing new assets
(a) If your work uses existing assets, did you cite the creators?
[N/A]
(b) Did you mention the license of the assets?
[N/A]
(c) Did you include any new assets either in the supplemental material or as a URL?
[No]
(d) Did you discuss whether and how consent was obtained from people whose data you’re
using/curating?
[N/A]
No people were involved in the data gathering process.
(e) Did you discuss whether the data you are using/curating contains personally identiﬁable
information or offensive content?
[N/A]
If you used crowdsourcing or conducted research with human subjects...
(a) Did you include the full text of instructions given to participants and screenshots, if
applicable?
[N/A]
(b) Did you describe any potential participant risks, with links to Institutional Review
Board (IRB) approvals, if applicable?
[N/A]
(c) Did you include the estimated hourly wage paid to participants and the total amount
spent on participant compensation?
[N/A]