TTOpt: A Maximum Volume Quantized Tensor Train-based Optimization and its Application to Reinforcement Learning
Abstract
We present a novel procedure for optimization based on the combination of efficient quantized tensor train representation and a generalized maximum matrix volume principle.
We demonstrate the applicability of the new Tensor Train Optimizer (TTOpt) method for various tasks, ranging from minimization of multidimensional functions to reinforcement learning.
Our algorithm compares favorably to popular gradient-free methods and outperforms them by the number of function evaluations or execution time, often by a significant margin
Introduction
In recent years learning-based algorithms achieved impressive results in various applications, ranging from image and text analysis and generation [52] to sequential decision making and control [40] and even quantum physics simulations [49].
The vital part of every learning-based algorithm is an optimization procedure, e.g., Stochastic Gradient Descent.
In many situations, however, the problem-specific target function is not differentiable, too complex, or its gradients are not helpful due to the non-convex nature of the problem [30, 2, 58].
The examples include hyper-parameter selection during the training of neural models, policy optimization in reinforcement learning (RL), training neural networks with discrete (quantized) weights [57] or with non-differentiable loss functions [20].
In all these contexts, efficient direct gradient-free optimization procedures are highly needed.
Recently, [53] showed that an essential class of gradient-free methods, namely the evolutionary strategies (ES) [22, 26], are competitive in reinforcement learning problems.
In RL, the goal is to find the agent’s action distribution π (the policy), maximizing some cumulative reward function J .
The policy is usually parameterized with a set of parameters θ .
It follows that the reward is a function of the parameters of the policy: J ( π ( θ ))
=
J ( θ ) .
The idea of [53] and similar works [36, 15, 10, 11] is to directly optimize the cumulative reward function J ( θ ) =
J ( θ 1 , θ 2 , . .
, θ d ) with respect to the parameters of the policy.
We pursued a similar approach to transform a traditional Markov
Decision Process into an optimization problem (we provide the details in Appendix B.3).
Although the agents trained with ES often demonstrate more rich behavior and better generalization compared to traditional gradient-based policy optimization, the convergence of ES is often slow [11].
As an alternative to previous works, we present a tensor-based 6 gradient-free optimization approach and apply it to advanced continuous control RL benchmarks.
The algorithm that we called TensorTrain (TT) Optimizer ( TTOpt ), works for multivariable functions with discrete parameters by reformulating the optimization problem in terms of tensor networks.
Consider a function J ( θ ) :
N d −→ R of a d -dimensional argument θ , where each entry θ k of the vector θ takes a value in the discrete set { ω i } Ni =1 .
The function J may be viewed as an implicitly defined d -dimensional tensor J .
Each entry in J is a value of J for some argument.
Maximizing J is equivalent to finding the sets of indices { θ ( m ) max } Mm =1 of maximal entries { j ( m ) max } Mm =1 of J .
By using only a tiny fraction of adaptively selected tensor elements (e.g., a small number of function evaluations), our method builds a representation of J in the TT-format and finds a set of the largest elements.
Although the algorithm works only with functions of discrete arguments, the grid size for each parameter can be huge thanks to the efficiency of the TT-format.
Fine discretization makes it possible to almost reach a continuous limit and obtain large precision with TTOpt.
The strength of our approach is, however, the direct handling of discrete (quantized) parameters.
Contributions.
We propose an efficient gradient-free optimization algorithm for multivariable functions based on the low-rank TT-format and the generalized maximum matrix volume principle 7 .
We demonstrate that our approach is competitive with a set of popular gradient-free methods for optimizing benchmark functions and neural network-based agents in RL problems.
We empirically show that agents with discrete (heavily quantized) weights perform well in continuous control tasks.
Our algorithm can directly train quantized neural networks, producing policies suitable for low-power devices.
Optimization with tensor train
In this section, we introduce a novel optimization algorithm.
We show how to represent the optimization problems in the discrete domain efficiently and then formulate optimization as a sampling of the objective function guided by the maximum volume principle.
2.1 Discrete formulation of optimization problems
We first need to transfer the problem to the discrete domain to apply our method.
It may seem that discretizing an optimization problem will make it harder.
However, it will allow us to use powerful techniques for tensor network representation to motivate the algorithm.
For each continuous parameter θ k ( k = 1 , 2 , . . .
, d ) of the objective function J ( θ ) we introduce a grid { θ ( n k ) k } N k n k =1 .
At each point ( θ ( n 1 ), θ ( n 2 ), . . .
, θ ( n d ) d ) of this grid with index ( n 1 , n 2 , . . . , n d ) the objective function takes a value
J ( θ ( n 1 ), θ ( n 2 ), . . .
, θ ( n d ) d ) ≡ J [ n 1 , n 2 , . . . , n d ] .
We thus can regard the objective function as an implicit d -dimensional tensor J with sizes of the modes N 1 , N 2 , . .
. , N d .
Finding the maximum of the function J ( θ ) translates into finding the maximal element of the tensor J in the discrete setting.
Notice that the number of elements of J equals: |J | = N 1 · N 2 · . . .
· N d ∼ (max 1 ≤ k ≤ d N k ) d .
The size of J is exponential in the number of dimensions d .
This tensor can not be evaluated or stored for sufficiently large d .
Fortunately, efficient approximations were developed to work with multidimensional arrays in recent years.
Notable formats include Tensor Train (TT) [47, 46, 50, 44, 63], Tensor Chain/Tensor Ring [29, 66] and Hierarchical Tucker [21].
We use the most studied TT-format [13], but the extensions of our method to other tensor decompositions are possible.
2.2 Tensor Train decomposition
Definition 2.1.
A tensor J ∈ R N 1 × N 2 ×···× N d is said to be in the TT-format [46] if its elements are represented by the following expression
[Equation]
where n k = 1 , 2 , . . . , N k for k = 1 , 2 , . .
, d .
In TT-format the d -dimensional tensor J is approximated as a product of three-dimensional tensors
G k ∈ R R k − 1 × N k × R k , called TT-cores.
The sizes of the internal indices R 0 , R 1 , · · · , R d (with convention R 0 = R d = 1 ) are known as TT-ranks.
These ranks control the accuracy of the approximation.
The storage of the TT-cores, G 1 , G 2 , . .
. , G d , requires at most d · max 1 ≤ k ≤ d N k · (max 0 ≤ k ≤ d R k
[Equation]
memory cells, and hence the TT-approximation is free from the curse of dimensionality 8 if the
TT-ranks are bounded.
The basic linear algebra operations (such as finding a norm, differentiation, integration, and others) can also be implemented in the TT-format with polynomial complexity in dimensionality and mode size.
Building TT-approximation.
Several efficient schemes were proposed to find TT-approximation if all or some of the elements of the initial tensor are known or may be generated by the function’s call.
Examples include TT-SVD [47, 46], TT-ALS [27] and TT-CAM [48] (Cross Approximation Method in the TT-format).
We build upon the TT-CAM but modify it not to compute the approximation for the entire tensor, but rather to find a small subset of its maximal entries.
The original algorithm builds a TT-approximation by adaptively requesting elements of the input tensor.
As we will show below, these elements with high probability will have large absolute values.
Based on this observation, we formulate a robust optimization algorithm for multivariate functions (either discrete or continuous).
To simplify the understanding, we outline the approach for the two-dimensional case, and after that, we describe our gradient-free optimization method for the multidimensional case.
2.3 Maximal element in a matrix
The Cross Approximation Method (CAM) for matrices [18, 9, 1] is a well-established algorithm for building a rank- R approximation ˜
J of an implicitly given matrix J :
[Equation]
where J C consists of R columns of J , J R is composed of R rows of J , and ˆ J is a submatrix at their intersection.
Such approximation (also called cross or skeleton decomposition) may be built iteratively using a well known alternating directions method and a maximum volume ( maxvol ) algorithm 9 [18], as we will sketch below.
Intuition behind TTOpt.
The main interest in optimization problems is not the approximation (2) itself, but the following property of the resulting maximum volume submatrix ˆ J .
[18] proved that if ˆ
J is an R × R submatrix of maximal volume (in selected rows and columns)
then the maximal (by modulus) element ˆ J max ∈
ˆ J bounds the absolute maximal element J max in the full matrix J :
[Equation]
This statement is evident for R = 1 , and for the case R > 1 it gives an upper bound for the element.
By using elementwise transformations of J , this upper bound can be used to obtain a sequence that converges to the global optimum.
The main idea of the maxvol -based methods is that it is easier to find a submatrix with a large volume rather than the element with the largest absolute value.
Moreover, our numerical experiments show that this bound is pessimistic, and in practice, the maximal-volume submatrix contains the element which is very close to the optimal one.
TTOpt algorithm for matrices.
The idea of the TTOpt algorithm for matrices is to iteratively search for the maximal volume submatrices in the column and row space of the implicitly giveninput matrix J ∈ R N 1 × N 2 .
After T iterations a series of “intersection” matrices { ˆ J ( t ) }
Tt =1 ∈ R R × R is produced.
The maximal element is searched in these small submatrices.
We schematically represent the TTOpt algorithm in Figure and a description is given below:
At
the
initial stage
, we set the expected rank of the approximation, R , and select R ran-
dom
columns I ( C, 1) .
We
then generate the corresponding column submatrix
J (1) C = J
, I ( C, 1) ] ∈ R N 1 × R .
Using the maxvol algorithm
we find the maximal-volume submatrix ˆ J (1) ∈ R R × R in J (1) C and store its row indices in the list
I ( R, 1) .
The indices I ( R, 1) are used to generate a row submatrix J (2) R =
J [ I ( R, 1) , :] ∈ R R × N 2 .
Then, using the maxvol algorithm, we find the maximal-volume submatrix ˆ J (2) in the matrix J (2) R and store the corresponding column indices in the list
I ( C, 2) .
We generate the related columns J (3) C =
J [: , I ( C, 2) ] , apply again the maxvol algorithm to the column submatrix, and iterate the process until convergence.
The approximate value of the maximum modulus element of the matrix J is found as
[Equation]
2.4 Optimization in the multidimensional case
As we explained previously, the target tensor, J , is defined implicitly, e.g., by a multivariable function
J .
We propose a novel method to find the optimum in this implicit tensor.
We outline the approach below and provide detailed algorithms in Appendix A.2.
As shown in Figure 2, we begin by considering the first unfolding 11 J 1 ∈ R N 1 × N 2 ...N d of the tensor
J and select R 1 random columns I ( C ) 1 .
Precisely, I ( C )here is a list of R 1 random multi-indices of size d − 1 , which specify positions along modes k = 2 to k = d .
We then generate the submatrix
J ( C )∈ R N 1 × R 1 for all positions along the first mode (shown in green in Figure 2).
Like in matrix case, we apply the maxvol algorithm to find the maximal-volume submatrix ˆ J 1 ∈ R R 1 × R 1 and store the corresponding indices of R 1 rows in the list
I ( R ) 1 .
In contrast with matrix case, we cannot generate the row submatrix J ( R )∈ R R 1 × N 2 N 3 ...N d for the selected row indices I ( R ) 1 , since it contains an exponential number of elements.
The following trick is used instead.
We consider the implicit matrix J ( R )and reshape it to a new matrix J 2 ∈
R R 1 N 2 × N 3 ...N d .
We sample R 2 random columns
I ( C )in the matrix J 2 and generate the entire small submatrix J ( C )
[Equation]
Next we find the maximal-volume submatrix ˆ J 2 ∈ R R 2 × Rin J ( C )and store the corresponding R 2 row multi-indices in the list
I ( R ) 2 .
The resulting submatrix
J ( R )∈ R R 2 × N 3 N 4 ...
N d is then transformed (without explicitly evaluating its elements) into the matrix J 3 ∈ R R 2 N 3 × N 4 ...N d .
We continue the described operations, called sweeps, until the last mode of the initial tensor, J , is reached.
After that, we repeat the process in the opposite direction, sampling now the row indices instead of the column indices.
These sequences of forward and backward sweeps are continued until the algorithm converges to some row and column indices for all unfolding matrices 12 or until the user-specified limit on the number of requests to the objective function, J , is exceeded.
Finally, after
T sweeps, the approximate value of the maximum modulo element can be found by the formula (4), as in the two-dimensional case.
Note that currently there is no analog of Eq.
(3) in the multidimensional case, and hence there are no formal guarantees of the convergence of the sweeps to the global minimum, nor the rate of this convergence.
The only guarantee is that the result will monotonically improve with iterations.
2.5 Complexity of the algorithm
It can be easily shown that the described algorithm requires to evaluate only
O
[Equation]
elements of the implicit tensor in one sweep.
Thus, with a total number of sweeps T , we will have
[Equation]
calls to the objective function J .
In practice, it turns out to be more convenient to limit the maximum number of function calls, M , according to the computational budget.
If the time of a single call to J is significant, then the effort spent on the algorithm’s operation will be negligible.
Otherwise, the bottleneck will be the calculation of the maximal-volume submatrices by the maxvol algorithm.
Taking into account the estimate of the maxvol complexity, given in
Appendix A.1, it can be shown that in this case the complexity of our algorithm is
[Equation]
2.6 Implementation details
For the effective implementation of the TTOpt algorithm, the following important points should be taken into account (see also the detailed pseudocode in Appendix A.2).
Stability.
Submatrices J ( C ) k
(or J ( k ) C and J ( k ) R for the two-dimensional case) that arise during the iterations may degenerate, and in this case it is impossible to apply the maxvol algorithm.
To solve this problem, we first calculate the QR decomposition for these matrices and then apply the maxvol to the corresponding Q factors 13 .
Rank selection.
We do not know in advance the exact ranks R 1 , R 2 , . . .
, R d of the unfolding matrices (or rank R of the matrix J for the two-dimensional case).
Therefore, instead of the maxvol algorithm, we use its modification, i.e., the rect_maxvol algorithm 14 [39], within the framework of which several (“most important”) rows are added to the maximal-volume submatrix.
In this case, we have ˆ J ( C )∈
[Equation]
R
[Equation]
R
[Equation]
R
[Equation]
R 2 , etc.
Note that the final approximation obtained in this case may have an overestimated rank, which, if necessary, can be reduced by appropriate rounding, for example, by truncated SVD decomposition.
Mapping function.
Maximal-volume submatrices contain the maximum modulus element but not the minimum or maximum element of the tensor (i.e., the sign is not taken into account).
We introduce a dynamic mapping function to find the global minimum (or maximum).
Instead of J at each step of the algorithm, we evaluate 15 :
[Equation]
where J min is the current best approximation for the minimum element of the tensor.
Quantization.
To reach high accuracy, we often need fine grids.
In the case when the sizes of the tensor modes N 1 , N 2 , . . .
, N d are large, the sizes of unfolding matrices become large, which leads to a significant increase in computational complexity of the maxvol algorithm.
To solve this problem, we apply additional compression based on quantization of the tensor modes [45].
Assume without the loss of generality that the size of each mode is N k = P q ( k = 1 , 2 , . . .
, d ; P ≥ 2 ; q ≥ 2 ).
Then we can reshape the original d -dimensional tensor J ∈ R N 1 × N 2 ×···× N d into the tensor ˜ J ∈ R P × P ×···× P of a higher dimension d · q , but with smaller modes of size P .
The TTOpt algorithm can be applied for this “long” tensor instead of the original one 16 .
We found that this idea significantly boosts the accuracy of our algorithm and reduces the complexity and execution time.
Typically, P is taken as small as possible, e.g. P = 2 .
Experiments
To demonstrate the advantage of the proposed optimization method, we tested TTOpt on several numerical problems.
First, we consider analytical benchmark functions and then the practically
) is most suitable.
[Table]
D
[Table]
significant problem of optimizing the parameters of the RL agent.
We select GA (Genetic Algorithm [26, 60]), openES (basic version of OpenAI Evolution Strategies [53]) and cmaES (the
Covariance Matrix Adaptation Evolution Strategy [22]) as baselines for both experiments.
Additionally we used DE (Differential Evolution [59]), NB (NoisyBandit method from Nevergrad [5]) and PSO (Particle Swarm Optimization [24, 33]) for benchmark functions 17 .
We also compared the proposed approach with gradient-based methods applied for all benchmark functions 18 : BFGS (
–Fletcher–Goldfarb–Shanno algorithm), L-BFGS (Limited-memory BFGS), CG (Conjugate Gradient algorithm),
NCG (Newton CG algorithm), Newton (Newton Exact algorithm), TR
NCG (Trust-Region NCG algorithm) and TR (Trust-Region Exact algorithm).
According to our approach, the TTOpt solver has the following configurable parameters: a and b are lower and upper grid bounds (for simplicity, we use the same value for all dimensions); R is a rank (for simplicity, we use the same value for all unfolding matrices); P is a submode size (mode size of the quantized tensor; for simplicity, we use the same value for all dimensions); q is the number of submodes in the quantized tensor (each mode of the original tensor has size N = P q ); M is a limit on the number of requests to the objective function.
[Table]
3.1 Benchmark functions minimization
To analyze the effectiveness of the TTOpt
[Table]
we applied it to 10 -dimensional benchmark functions with known global minimums; see Table 1 in Appendix B.1 with the list of functions and their properties (note that some of the considered benchmarks are multimodal non-separable functions).
Also, in
Appendix B.1, we present a more detailed study of the TTOpt solver and the dependence of the accuracy on the value of its parameters ( R , q and M ).
In all experiments with baselines ( GA , openES , cmaES , DE , NB , PSO , BFGS , L-BFGS , CG ,
NCG , Newton , TR NCG and TR ), we used default parameter values.
In Appendix B.2 we also present the additional experiments with Bayesian Optimization [37].
For TTOpt we selected rank
R = 4 , submode size P = 2 and the number of submodes q = 25 .
For all methods, a limit on the number of requests to the objective function is chosen as M = 10 5 .
All calculations are performed on a standard laptop.
The results are demonstrated in Table 1.
For each method we list the absolute deviation of the result ˆ J min from the global minimum J min , i.e., ϵ =
| ˆ J min − J min | .
We also present the total running time, τ .
Compared to other benchmarks, TTOpt is consistently fast, accurate and avoids random failures to converge seen in other algorithms.
Additionally, TTOpt turns out one of the fastest gradient-free algorithms ( GA , openES , cmaES , DE , NB , PSO ), despite a simple Python implementation.
One of the advantages of the proposed TTOpt approach is the possibility of its application to essentially multidimensional functions.
In Table 2 we present the result of TTOpt for the F benchmark function of various dimensionality (results for other benchmarks are in Appendix B.1).
Note that as a limit on the number of requests to the objective function we chose M = 10 4 · d , and the values of the remaining parameters were chosen the same as above.
As can be seen, even for 500 -dimensional functions, the TTOpt method gives a fairly accurate result.
3.2 Application of TTOpt to Reinforcement Learning
We used several continuous RL tasks implemented in Mujoco [61] and OpenAI-GYM [8]: Swimmerv
[16], LunarLanderContinuous-v2, InvertedPendulum-v2 and HalfCheetah-v3 [64].
In all experiments, the policy π is represented by a neural network with three hidden layers and with tanh and
ReLU activations.
Each layer is a convolution layer.
See additional details about hyperparameters in
Table 6 in Appendix B.
We discretize (quantize) the values of agent’s weights.
The TTOpt method is used to optimize discrete agent’s weights in order to maximize the cumulative reward of the episode.
This corresponds to on-policy learning.
[Figure]
HalfCheetah-v
[Figure]
HalfCheetah-v
[Figure]
To properly compare TTOpt with other methods, we propose modified evolutionary baselines that enforce constrained parameter domain.
We adapt penalty term and projection techniques from [31, 6] to introduce constraints, see Appendix B.4 for details.
First, we run benchmarks with small mode size N = 3 1 with lower and upper grid bounds ± 1 .
Another series of experiments was done with finer mode of size N = 2 q with the same bounds.
These experiments model the case of neural networks with discrete (quantized) weights which use q -bits quantization.
Finally, we provide the results of using TTOpt as a fine-tuning procedure for linear policies from [36].
We present characteristic training curves based on the number of environment interactions and execution time for the HalfCheetah-v3 experiment ( N = 3 ) in Figure 3.
Training curves for other environments can be found in Appendix B.5, in Figure 3 for N = 3 and in Figure 4 for N = 256 .
TTOpt consistently outperforms all other baselines on the coarse grid with mode size N = 3 .
Our method is still best for finer grids with mode size N = 256 on InvertedPendulum-v2 and
LunarLanderContinuous-v2, and second-best on HalfCheetah-v3.
Moreover, TTOpt has significantly lower execution time compared to evolutionary baselines.
Another interesting observation is that the training curves of TTOpt have low dispersion, e.g., the algorithm performs more consistently than the baselines (see Appendix B.5).
Table 3 summarizes the experiments for the coarse and fine grids.
Results for fine-tuning of linear policies are presented in Table 5 in Appendix B.5.
We also did rank and reward dependency study in Appendix B.
Related work
In the case of high dimensional optimization, evolutionary strategies (ES) [56, 42] are one of the most advanced methods of black-box optimization.
This approach aims to optimize the parameters of the search distribution, typically a multidimensional Gaussian, to maximize the objective function.
Finite difference schemes are commonly used to approximate gradients of the parameters of the search distribution.
Numerous works proposed techniques to improve the convergence of ES [42].
[65] proposed to use second-order natural gradient of [12] to generate updates, while [22] suggested to include the history of recent updates to generate next ones.
[35] presented the concept of surrogate gradients for faster convergence.
Another series of works aimed to reduce the high sampling complexity of ES.
In [11] the authors described how to use active subspaces [14] to reduce the number of objective function calls dynamically.
Plenty of the already mentioned works in gradient-free optimization specifically applied these methods to RL tasks [53, 10, 11, 36, 15, 23].
Overall, the performance of ES-based methods is comparable to conventional policy gradients, especially if the number of model parameters is small [54, 55].
Another advantage of ES over policy gradient is that it produces more robust and diverse policies [32] by eliminating the problem of delayed rewards and short length time horizons.
Finally, evolutionary approaches are suitable for the problems with non-Markovian properties [23].
Other metaheuristic [25, 38] and classical optimization [51] techniques are also studied within RL scope.
The examples include simulated annealing [4], particle swarm optimization [24, 33] and even classical Nelder-Mead algorithm
These methods, however, are not tested on common RL task sets.
Several other works combined evolutionary methods with RL to achieve better performance in complex scenarios [28, 17], e.g. AlphaStar [62, 3].
Finally, low-rank tensor approximations have been applied to RL problems in settings different from ours, including multi-agent scenarios [34], and improving dynamic programming approaches [19, 7].
The idea of our method is quite different from the presented works, especially in the RL area.
Conclusion
We proposed a new discrete optimization method based on quantized tensor-train representation and maximal volume principle.
We demonstrate its performance for analytical benchmark functions and reinforcement learning problems.
Our algorithm is more efficient under a fixed computational budget than baselines, especially on discrete domains.
Moreover, the execution time for TTOpt is lower by a significant margin compared with other baselines.
Finally, we show that the agents with discrete parameters in RL can be as efficient as their continuous parameter versions.
This observation supports the broad adoption of quantization in machine learning.
We hope that our approach will serve as a bridge between continuous and discrete optimization methods.
Acknowledgments and Disclosure of Funding
The work was supported by Ministry of Science and Higher Education grant No. 075-10-2021-068.
Checklist
For all authors... (a) Do the main claims made in the abstract and introduction accurately reflect the paper’s contributions and scope?
[Yes] (b) Did you describe the limitations of your work?
[Yes] (c) Did you discuss any potential negative societal impacts of your work?
[N/A]
The work has no any societal impacts, since its about optimization methods.
(d) Have you read the ethics review guidelines and ensured that your paper conforms to them?
[Yes] 2.
If you are including theoretical results... (a) Did you state the full set of assumptions of all theoretical results?
[Yes] (b) Did you include complete proofs of all theoretical results?
[N/A] 3.
If you ran experiments... (a) Did you include the code, data, and instructions needed to reproduce the main experimental results (either in the supplemental material or as a URL)?
[Yes] The code will be provided in supplemental material and as github repository after decision (b) Did you specify all the training details (e.g., data splits, hyperparameters, how they were chosen)?
[Yes] (c) Did you report error bars (e.g., with respect to the random seed after running experiments multiple times)?
[Yes] We used a typical shade(mean-std) plots that used in reinforcement learning experiments (d) Did you include the total amount of compute and the type of resources used (e.g., type of GPUs, internal cluster, or cloud provider)?
[Yes] 4.
If you are using existing assets (e.g., code, data, models) or curating/releasing new assets... (a) If your work uses existing assets, did you cite the creators?
[Yes] .
We mentioned all references and source-codes for gradient-free baselines.
(b) Did you mention the license of the assets?
[N/A] (c) Did you include any new assets either in the supplemental material or as a URL?
[N/A] (d) Did you discuss whether
and how consent was obtained from people whose data you’re using/curating?
[N/A] (e) Did you discuss whether the data you are using/curating contains personally identifiable information or offensive content?
[N/A] 5.
If you used crowdsourcing or conducted research with human subjects... (a) Did you include the full text of instructions given to participants and screenshots, if applicable?
[N/A] (b) Did you describe any potential participant risks, with links to Institutional Review Board (IRB) approvals, if applicable?
[N/A] (c) Did you include the estimated hourly wage paid to participants and the total amount spent on participant compensation?
[N/A]