Meta-Reward-Net: Implicitly Differentiable Reward Learning for Preference-based Reinforcement Learning
s email
Abstract
Setting up a well-designed reward function has been challenging for many reinforcement learning applications.
Preference-based reinforcement learning (PbRL)provides a new framework that avoids reward engineering by leveraging humanpreferences (i.e., preferring apples over oranges) as the reward signal.
Therefore,improving the efficacy of data usage for preference data becomes critical.
In thiswork, we propose Meta-Reward-Net (MRN), a data-efficient PbRL framework thatincorporates bi-level optimization for both reward and policy learning.
The key ideaof MRN is to adopt the performance of the Q-function as the learning target.
Basedon this, MRN learns the Q-function and the policy in the inner level while updatingthe reward function adaptively according to the performance of the Q-function onthe preference data in the outer level.
Our experiments on locomotion tasks androbotic manipulation tasks demonstrate that MRN outperforms prior methods inthe case of little feedback and significantly improves data efficiency, achievingstate-of-the-art in preference-based RL.
Ablation studies further demonstrate that MRN learns a more accurate Q-function compared to prior work and shows obviousadvantages when only a small amount of feedback is available.
Introduction
In recent years, reinforcement learning has achieved great success in solving complex sequentialdecision-making tasks, such as video or board games [1, 2, 3, 4], autonomous driving [5], quantitativefinance [6], automation system [7, 8, 9] etc.
For common decision making tasks, the goal of the agentis to maximize the cumulative reward.
However, one central challenge to reinforcement learningis how to design reward functions.
On the one hand, the quality of the designed reward functionlargely depends on the problem solver’s understanding of task objective, operation logic and relatedbackground knowledge.
Even excellent engineers still need plenty of time to try different methods incomplex RL tasks.
On the other hand, there is a problem that the agent might hack the reward function.
In policy learning, the agent utilizes the defect of the reward function to maximize the cumulativereward instead of solving the expected task.
Besides, in human-involved scenarios, the objective ofthe agent is to maximize happiness of humans, making it hard to specify a reward function.
Previous work has provided some ideas to avoid directly constructing reward functions, such asimitation learning [10].
Although imitation learning has an excellent performance in some tasks,its performance is difficult to surpass human level.
Preference-based reinforcement learning is amore flexible and convenient alternative method.
A human expert can easily give a preference to atrajectory pair of the agent, which implies the desired behaviors, that is to say, the goal of the task.
Inpreference-based RL, the reward function is learned through preferences given by the human teacheron trajectory pairs. The feedback from humans guides the agent to achieve specified goals or learn
[Figure]
feedback from
Q-function policy evaluation
[Figure]
desired behaviors.
Recent work [11, 12, 13] shows that providing sufficient feedback achieves betterperformance under this setting.
However, preference data queried from human experts is expensive.
In reality, there is likely tobe only a small amount of data available.
With the limitation of the amount of feedback, previousmethods perform poorly or even do not work.
Meanwhile, recent work on semi-supervised learningtakes success in computer vision [14], pseudo labels can be adjusted according to the feedback fromstudent networks and further improve student’s performance.
Inspired by this, we consider utilizingthe feedback from the Q-function for reward learning via bi-level optimization.
By considering theloss calculated according to the feedback, the optimization of the reward function is aware of theaccuracy of the Q-function, which is beneficial for the learning of the Q-function and the policy.
In this work, we focus on the efficiency of feedback in the learning of preference-based RL.
We use bilevel optimization method for reward learning which contains two loops.
In the inner loop, we updatethe Q-function and the policy via the reward function, while optimizing the reward function accordingto the performance of the Q-function on the preference data in the outer loop.
Our experimentsdemonstrate that our method considerably improves feedback efficiency.
Besides, the results offurther evaluation show that MRN exceeds other methods by a large margin when a small amount offeedback is available and learns a more accurate Q-function.
In summary, the main contributions of our work are three-fold.
Firstly, we propose a new preferencebased RL algorithm Meta-Reward-Net, which utilizes bi-level optimization methods in rewardlearning.
Secondly, we show that MRN substantially improves the feedback efficiency and outperforms preference-based RL baselines on a variety of robotic manipulation tasks from Meta-world [15]and locomotion tasks from DeepMind Control Suite (DMControl)
Lastly, we demonstratethat benefiting from bi-level optimization, the advantage becomes obvious compared to PEBBLEwhen only a small amount of feedback is provided.
We also show that the feedback from the Qfunction to the reward function is beneficial for agent learning, leading to a more accurate Q-functionand a better policy.
Related Work
Reinforcement Learning.
Reinforcement learning has gradually become an effective and powerfulmethod to solve complex sequential decision-making problems.
In recent years, much prior work hasproven the capacity of reinforcement learning.
Reinforcement learning applications include videogames [1, 2, 3], robot control [18], manipulation [8, 9], board games [4], autonomous driving [5] andso on.
In addition, there are many applications of reinforcement learning in the fields of computervision [19], natural language processing [20], and recommendation system [21, 22].
In the frameworkof reinforcement learning, an agent obtains data from interaction with the environment to optimizepolicy to maximize the expected return.
In this process, the reward function plays a crucial role.
Our method differs in that we do not assume that there is a reward function from engineering.
Instead, weutilize the preferences of humans to guide the agent to learn desired behaviors.
Preference-based Reinforcement Learning.
Prior work has successfully trained the agent tocomplete specific tasks or achieve goals through the feedback from the teacher.
[23] provides ageneral learning framework of preference-based reinforcement learning.
[24] utilizes two kinds offeedback, initializes with imitation learning policy, and further improves the performance of the policywith feedback from humans.
In this work, we mainly focus on feedback efficiency in the learning ofpreference-based RL.
Much previous work [25, 26, 27, 28, 29] considered learning reward functionfrom the most informative data to be consistent with the human preferences.
Recently, severalfeedback-efficient preference-based RL algorithms have been proposed.
PEBBLE [11] combinesunsupervised pre-training and the technique of relabeling experience to improve feedback efficiency.
SURF [12] learns the reward function by semi-supervised learning and data augmentation.
RUNE [13]facilitates exploration via reward uncertainty to reduce the amount of feedback.
However, thesemethods only focus on guiding reward learning through supervised loss between human preferencesand estimated preferences.
We have a different approach in that in addition to utilizing the supervisedloss, we also consider the performance of the Q-function on a labeled dataset as feedback to assistreward learning, thus beneficial for agent learning.
By introducing extra feedback, MRN gets moretask-related information from human preferences to improve feedback efficiency remarkably.
Bi-level Optimization.
In computer vision, there are several bi-level optimization algorithms thatachieve great success.
Meta-Weight-Net [30] provides weights for samples from an unbalanceddataset in a bi-level manner, while Meta Label Correction [31] view this problem as a label correctionproblem.
Meta Pseudo Labels [14] combines Pseudo Labels methods and bi-level optimizationto generate high quality pseudo labels.
In reinforcement learning, LIIR [32] learns an intrinsicreward function for each agent to achieve better cooperation in multi-agent reinforcement learning.
CAIL [33] proposes to reweight demonstrations of different optimality in imitation learning.
Ourmethod uses a similar bi-level method for reward learning.
To the best of our knowledge, we are thefirst to introduce bi-level optimization into preference-based RL.
Preliminaries
Preference-based Reinforcement Learning.
In standard reinforcement learning framework, a finite
Markov decision process (MDP) can be presented as a tuple of ⟨S , A , R, P, γ ⟩ , which consists ofstate space S , action space A , transition function, reward function, and discount factor.
P ( s ′ | s, a )represents stochastic dynamics of the environment, which is the probability of selecting action a totransit to s ′ in a given state s .
R ( s, a ) represents the reward obtained by selecting an action a in agiven state s .
The policy π ( a | s ) is a mapping from state space to action space.
The objective of theagent is to collect trajectories from interaction with the environment to maximize the expected return.
In the general preference-based RL from [23], there is no reward function from reward engineering and a reward function estimator (cid:98) r ψ should be learned to be consistent with preferencesfrom the human expert.
Specifically, a segment σ is a sequence of states and actions which is( s t +1 , a t +1 , · · · , s t + k , a t + k ) .
Human expert provides a preference y on given two segments ( σ 0 , σ 1 )and y is the distribution over { 0 , 1 } , y ∈ { (1 , 0) , (0 , 1) , (0 . 5 , 0 . 5) } .
Following the Bradley-Terrymodel [34], a preference predictor constructed by the reward function estimate (cid:98) r ψ is formulated as:
[Equation]
(1) where σ 0 ≻ σ 1 denotes σ 0 is more consistent with the expectations of human experts comparedwith σ 1 .
The reward function learning can be solved by minimizing the cross-entropy loss betweenpredictions from preference predictors and human preferences.
[Equation]
We refer to this objective as supervised loss in the following sections.
By optimizing the rewardfunction using this loss, segments that are more in line with human preferences obtain a highercumulative reward.
Soft Actor-Critic.
SAC [35] is an off-policy algorithm based on the maximum entropy RL, whichencourages the agent to explore the environment by acting as randomly as possible.
SAC consists of soft Q-function Q θ ( s, a ) with parameters θ and policy π ϕ ( a | s ) with parameters ϕ .
Q-function withparameters θ is defined as the expectation of return:
[Equation]
where γ ∈ [0 , 1] is a discount factor.
The parameters θ of soft Q-function are trained by minimizing the soft Bellman residual:
[Equation]
where ¯ V ( s t ) =
E a t ∼ π ϕ (cid:2) Q ¯ θ ( s t , a t ) − α log π ϕ ( a t | s t ) (cid:3) , τ t = ( s t , a t , s t +1 , r t ) is the transition at timestep t , α is a learnable temperature parameter that controls the item of entropy, ¯ θ are parameters ofthe target soft Q-function, and B is replay buffer.
After the updating of the Q-function, policy π ϕ isupdated by minimizing the loss:
[Equation]
[Equation]
By performing policy evaluation and policy improvement alternately, SAC trains an agent withexcellent and stable performance.
In this work, we consider using SAC as our backbone reinforcementlearning algorithm.
Meta-Reward-Net
In this section, we formally present Meta-Reward-Net, which includes two key components, optimizing the reward function based on the performance of the Q-function in outer loop and learningthe agent in the inner loop.
In the following, we first provide a new perspective that the Q-functioncan be used to compute preference labels, then define the objective of MRN and formulate a bi-leveloptimization problem.
4.1 The Objective
The probability that segment σ 0 is preferred is proportional to the exponential return of it.
Motivatedby this, we use Q θ ( s 00 , a 00 ) and Q
θ ( s 10 , a 10 ) to respectively measure of the return of σ 0 and σ 1 sincethe Q-value equal to the expectation of segment return.
Therefore, the probability that σ 0 is preferredto σ 1 is computed through the Q-function:
[Equation]
Given human preference labels y , the performance of the Q-function is measured by the cross-entropyloss between preference predictions computed by (6) and ground-truth labels:
[Equation]
where θ ( ψ ) denotes the updating of θ depends on the reward provided by (cid:98) r ψ .
The core of MRN is utilizing the feedback from the Q-function in reward learning, which means thatthe optimization of the reward function takes the performance of current Q-function into consideration.
Based on this idea, we formulate the objective using (7) as a measurement of the Q-function.
Theobjective of MRN is to minimize the loss of Q θ on a labeled preference dataset and the Q-function istrained by minimizing the Bellman residual.
The overall objective is formulated as:
[Equation]
By formulating MRN as a bi-level optimization algorithm, this allows the reward function to providerewards that are beneficial for improving the performance of the Q-function, which further leads to abetter policy π ϕ .
4.2 Bi-level Optimization
The objective in (8) is solved by the following bi-level optimization algorithm: θ is optimized bythe reward estimation from (cid:98) r ψ in inner loop while ψ is updated according to the performance of the
Q-function on the labeled data in outer loop.
Pseudo Updating: Building Connection between θ and ψ .
To utilize the feedback information fromthe Q-function and improve the performance of Q-function, we formulate the outer loop updatingas optimizing the loss of θ on the labeled preference data with respect to ψ .
However, we can notdirectly optimize this objective since the objective is independent of ψ .
So the first step is to build aconnection between θ and ψ .
Sample a mini-batch state-action pairs from the replay buffer and usethem to query current reward function with parameters ψ ( k ) to obtain reward estimation (cid:98) r ψ ( s t , a t ) ,where k denotes current iteration step.
Then (4) becomes:
[Equation]
At each bi-level optimization step, we first pseudo update the parameters of the Q-function.
Pseudoupdating means that we do not directly perform the updating on the Q-function, but update theparameters of a copy of current Q-function by minimizing (9) with learning rate α :
[Equation]
(10) where ˆ θ ( k ) denotes the updated copy of θ ( k ) .
By performing (10), the connection between ˆ θ ( k ) andψ ( k ) is built.
Outer Loop: Optimizing ψ to Improve the Performance of Q θ on Labeled Data.
After buildingconnections through pseudo updating, the copy of Q-function with parameters ˆ θ ( k ) is tested on labeledpreference data.
The predicted preference label P θ ( x ) is computed by the Q-function using (6), wherex denotes a segment pair ( σ 0 , σ 1 ) .
We use implicit differentiation in our method.
The objectiveof outer loop is formulated in (7), and the implicit derivative of the outer loss with respect to ψ iscalculated using the chain rule:
[Equation]
where h = − α · (cid:16) ∇ θ L feedback (cid:16) ˆ θ ( k ) (cid:17)(cid:17) ⊤ ·∇ θ J Q (cid:0) θ ( k ) (cid:1) and full derivation can be found in Appendix A.(11) formulates the gradient of feedback from the Q-function and this can be done easily usingautomatic differentiation in Pytorch [36].
Since pseudo updating is performed and the connectionbetween θ and ψ is built, ψ is updated to improve the performance of the Q-function by minimizingthe cross-entropy between preference labels from Q θ and ground-truth labels:
[Equation]
(12) where β is the learning rate of the outer loop.
Inner Loop: Optimizing θ and ϕ .
In the inner loop, the objective is the same as (4) and (5) in thetraining of SAC.
To calculate the new reward (cid:98) r ψ ( s t , a t ) , use the same batch of state-action pairs inpseudo updating to query the updated reward function.
However, we do not need the connection inthe inner loop since the connection is used for outer level optimization.
With newly obtained rewardestimation, we update Q-function Q θ by minimizing (4) with learning rate α :
[Equation]
and update policy π ϕ by minimizing (5) with learning rate α :
[Equation]
Auxiliary Loss.
In addition to optimizing the loss in the outer loop, the reward function is augmentedwith a supervised loss generally used in preference-based RL, which is formulated in (2).
Ourintuition is that the optimization of the reward function needs both supervised learning and thefeedback from the Q-function, and neither of them can be removed.
On the one hand, supervised loss is necessary for it ensures the reward estimation is consistent with human preferences.
On the otherhand, the feedback loss at the outer level is beneficial because it improves the performance of the Q-function, leading to a more accurate Q-function and finally a better policy.
The full procedure of our method is detailed in Algorithm 1. Before reward learning, we first initializethe policy and replay buffer with unsupervised exploration, which is proposed in PEBBLE [11] andcan be found in Appendix B.
The reward function is updated using supervised loss per K iterations,while the bi-level optimization is performed per N iterations.
We use off-policy RL algorithm SACto collect transitions and save them in the replay buffer.
The Q-function and the policy is optimizedin each training step.
do 5: Take
∼ ∼ Store transition s
Sample minibatch { ( j ) Bj 1 ∼ B 8: if iteration % K
0 then 9:
Query a human teacher for M preference labels and store them in D 10:
Sample preference data in D 11:
Optimize (2) with respect to ψ 12: Use updated (cid:98) r ψ to relabel the replay buffer B 13: end if 14: if iteration % N == 0 then 15: Sample preference data in D 16:
Pseudo update θ using (10) 17:
Update ψ using (12) 18:
Use updated (cid:98) r to relabel the replay buffer B 19: end if 20:
Update
and ϕ using (13) and (14), respectively 21: end for Output: policy π ϕ
Experiments
In this section, our method is evaluated on a variety of robotic manipulation tasks from Metaworld [15] and locomotion tasks from DeepMind Control Suite (DMControl) [16, 17].
The tasksused in our experiments are shown in Appendix C, which are the same as the tasks used in SURF.
5.1 Setup
Baselines.
Reward-based SAC and three state-of-the-art preference-based RL algorithms are used forcomparison:
• SAC [35]: SAC is considered as the ground-truth algorithm since the agent is provided withreal reward function, which is not the case in preference-based RL.
SAC is evaluated in ourexperiments because it is the backbone RL algorithm of PEBBLE.• Preference PPO [23]: the method is a re-implementation using PPO [37].
It uses an ensemble ofreward functions and disagreement sampling for querying.• PEBBLE [11]: the method is a preference-based RL method with unsupervised exploration andreward relabeling. • SURF [12]: the method combines temporal data augmentation and pseudo labels in semisupervised learning, which is the state-of-the-art algorithm in preference-based RL.
• Meta-Reward-Net (MRN): the proposed method, which utilizes the feedback from the Q- function in reward learning through bi-level optimization
Implementation Details.
For all methods, we use unsupervised exploration proposed in PEBBLE [11].
For the sampling of queries, disagreement-based sampling is used for all preference-based
RL methods, following the setting in [23].
An ensemble of three reward functions is used andthe reward output is computed by averaging output of three reward functions.
To systematicallyevaluate the performance and speed up the training process, following the setting in PEBBLE [11]and SURF [12], we consider a script teacher that can always provide the ground-truth preferencelabel of a segment pair.
Concretely, this is implemented by comparing the ground-truth return of eachsegment, but the reward is not accessible to the agent under the setting of preference-based RL.
For the implementation of SAC, Preference PPO and PEBBLE, we use the publicly released repositoryof B-Pref [38] in our experiments.
In their implementation, Preference PPO is re-implementedusing on-policy algorithm PPO.
SURF is also implemented using their released code.
2 For SAC,
PEBBLE and SURF, the hyperparameters and network architectures we use are the same as them(e.g.
, number of network layers, learning rate, frequency of supervised reward learning).
For theamount of human’s preference feedback, we set 100 for Walker, Cheetah, Button Press and Window
Open, 700 for Quadruped, 1000 for Door Open and Drawer Open, 4000 for Sweep Into, and 10000for Hammer.
Our method is implemented by using PEBBLE as the backbone.
We use bi-level updating frequency
N = 5000 for Cheetah, Hammer, Door Open, Button Press, Drawer Open and Window Open,
N = 1000 for Walker, N = 3000 for Quadruped, and N = 10000 for Sweep Into.
For each task, we run all algorithms independently for ten times and report the average with a standarddeviation.
Tasks of Meta-world are measured on success rate while the tasks of DMControl aremeasured on ground-truth episode return.
The experiments are run on a single machine with one
NVIDIA RTX 2080 Ti GPU.
Details on hyperparameters, network architectures can be found in
Appendix C.
5.2 Results
Meta-world Tasks.
Examples of the six continuous control tasks from Meta-world are shown in
Appendix C. These tasks are selected for our experiments, including robotic manipulation skills ofvarious difficulty.
The details of the tasks can be found in Appendix C.
Figure 2 shows the training curves of MRN and the baselines on the Meta-world tasks.
In this figure,
SAC achieves the best performance in each task by using the ground-truth reward function.
Sincelittle feedback is provided, we observe that there is a gap between all preference-based RL methodsand the best performance, but MRN still exceeds the preference-based RL baselines by a large margin.
These results demonstrate that MRN considerably improves the performance when only a smallamount of feedback is available.
We also notice that Preference PPO does not work in most tasks.
DMControl Tasks.
For DMControl, three locomotion tasks in Appendix C are used for evaluation,including Walker-walk, Cheetah-run and Quadruped-walk.
These tasks encourage the agent to moveforward by providing the agent with the reward that is positively correlated with agent’s velocity.
However, the agent is not accessible to the ground-truth reward function, and all preference-based RLmethods are provided with human preference labels.
Similar to tasks of Meta-world, we only providethe baselines and MRN with few but the same amount of preference labels.
Figure 3 shows the results of five methods on DMControl tasks.
SURF achieves the same returnas PEBBLE, while MRN shows obvious advantages compared with baseline methods.
The resultsdemonstrate that MRN performs well with a small amount of feedback and improves feedbackefficiency.
We remark that MRN can be regarded as introducing feedback from the Q-function through bi-leveloptimization based on PEBBLE.
Comparing the results of MRN and PEBBLE in Figure 2 and Figure 3, we find that using bi-level optimization can significantly improve performance when only asmall amount of feedback is provided.
[Figure]
[Figure]
[Figure]
5.3 Ablation Study
Number of Human’s Feedback.
Figure 2 and 3 show that our method outperforms Preference PPO,
PEBBLE and SURF under relatively small amounts of human’s feedback.
To further analyze theperformance of different algorithms with different amounts of feedback, Extensive experiments areconducted on Walker and Door Open with varying amounts of feedback: { 100 , 400 , 1000 , 2000 } .
The results in Figure 4 suggest that our method performs better when the number of human’s feedbackis small.
As the number increases, the performance of three algorithms becomes closer.
Intuitively,this phenomenon could be considered that as feedback information becomes more sufficient, theperformance gap caused by feedback efficiency gradually disappears.
Accuracy of Q-function.
To compare the quality of Q-function trained by MRN and baselines, wethe mean squared error between ground-truth Q-values and the output of Q-function.
Each methodis evaluated on the same ten trajectories by calculating the MSE of the output of the Q-functionand the ground-truth Q-values.
We report mean and standard deviation across ten runs in Table 1.
SAC is presented as the upper bound of the quality of Q-function and MRN achieves the lowest
PEBBLE
[Figure]
+ SURF
[Figure]
MSE among three methods.
In MRN, reward learning additionally takes the feedback from the
Q-function into consideration.
So the reward provided by the reward function is not only consistentwith the ground-truth preference label but also suitable for the performance of the current Q-function.
Therefore, the Q-function learned by MRN is more accurate compared to the baselines.
[Table]
Conclusion
In this work, we propose MRN, a novel feedback-efficient preference-based RL method.
By incorporating bi-level optimization for reward and policy learning, we demonstrate MRN outperformsprior methods when a small amount of feedback is available and considerably improves the feedbackefficiency on a variety of robotic tasks.
In particular, our method exceeds the baselines by a largemargin when the amount of feedback is small.
From empirical results and analysis, we concludethat the efficiency of feedback in our method mainly benefits from: firstly, our method learns a moreaccurate Q-function.
Secondly, our method learns the Q-function and policy in the inner loop andoptimizes reward function according to the performance of Q-function on the preference data in theouter loop.
By this way, MRN successfully establishes an efficient mode of information transmission,which can extract more information.
To our knowledge, we do not see any potential negative socialinfluences of our work.
We hope our method can provide inspiration for future work and encouragepreference-based reinforcement learning to be better extended to practical applications. Checklist
For all authors... (a) Do the main claims made in the abstract and introduction accurately reflect the paper’s contributions and scope? [Yes] See abstract and Section
(b) Did you describe the limitations of your work? [Yes] See Section 6. (c) Did you discuss any potential negative societal impacts of your work? [Yes] See
Section 6.
(d) Have you read the ethics review guidelines and ensured that your paper conforms to
them?
[Yes] We have read the ethics and make sure our work conforms to them.
If you are including theoretical results...
(a) Did you state the full set of assumptions of all theoretical
(a) Did you include the code, data, and instructions needed to reproduce the main ex-
perimental results (either in the supplemental material or as a URL)? [Yes] Seeimplementation details in Section 5.1. (b) Did you specify all the training details (e.g., data splits, hyperparameters, how theywere chosen)? [Yes] See implementation details in Section 5.1 and
Appendix C.
(c) Did you report error bars (e.g., with respect to the random seed after running experiments multiple times)?
[Yes] See Figure 2, Figure 3, Figure 4 and Table 1 inSection 5.2 and Section 5.3.
(d) Did you include the total amount of compute and the type of resources used (e.g., typeof GPUs, internal cluster, or cloud provider)?
[Yes] See Section 5.1.
If you are using existing assets (e.g., code, data, models) or curating/releasing new assets...
(a) If your work uses existing assets, did you cite the creators?
[Yes] Our code refers to
the source code published by B-Pref and SURF, and their papers are cited in our paper.
See Section 5.1.
(b) Did you mention the license of the assets?
[Yes] All referenced code is open source
and we notice that B-Pref provides MIT license and mujoco provides Apache License.
As for DMControl and Meta-world, you can see references for license details.
(c) Did you include any new assets either in the supplemental material or as a URL?
[Yes]
We release the link of our source code in Appendix C.
(d) Did you discuss whether and how consent was obtained from people whose data you’re
using/curating? [Yes] MIT license permits us to copy and modify the source code of B-Pref. We also get the permission of the author of SURF.
(e) Did you discuss whether the data you are using/curating contains personally identifiable
information or offensive content?
[N/A]
If you used crowdsourcing or conducted research with human subjects... (a) Did you include the full text of instructions given to participants and screenshots, if applicable? [N/A]
(b) Did you describe any potential participant risks, with links to Institutional Review
Board (IRB) approvals, if applicable?
[N/A] (c) Did you include the estimated hourly wage paid to participants and the total amount spent on participant compensation? [N/A]