Paper: Cascading Reinforcement Learning (ICLR-2024)

Reviewer comment (Novelty):
The significance of the contribution is not substantial enough to justify acceptance into the venue in my opinion. One of the main contributions is providing an algorithm that does not scale in complexity (both sample or computational) with the space of all possible rankings. I believe we have well-known recipes for how such algorithms can be formulated since the papers on Cascading Bandits from 2015 (Kveton et. al, Combes et al.). Providing a solution scaling with the number of items is expected in my opinion and not a surprising contribution (we know that estimating individual CTRs is enough to construct optimal rankings, we don't need to keep statistics on each individual ranking). This also leads me to believe the baseline used in the numerical experiments is weak.

Author response:
Thank you for giving us an opportunity to interact with you!

**1. Significance of Our Contribution**

Indeed it is already known that one only needs to evaluate each item, instead of each ranking, in classic cascading (combinatorial) bandits [Kveton et al. 2015]. However, the objective function in cascading bandits
$$
\max_{A \in \mathcal{A}} \quad f(A,q) = \sum_{i=1}^{|A|} \prod_{j=1}^{i-1} \Big( 1-q(A(j)) \Big) q(A(i)) = 1- \prod_{i=1}^{|A|} \Big( 1-q(A(i)) \Big) . \quad Eq. (i)
$$
only involves **one attribute** of items, i.e., the attraction probability $q(a)$. This objective function completely depends on $q(a)$, which is obvious to compute. They only need to select $m$ items with the maximum attraction probabilities $q(a)$.

However, when generalizing to RL, the objective function at each step $h$ is
$$
\max_{A \in \mathcal{A}} f(A,q,w) :=  \sum_{i=1}^{|A|} \prod_{j=1}^{i-1} \Big( 1-q(A(j)) \Big) q(A(i)) w(A(i)) , \quad Eq. (ii) 
$$ 
which involves **two attributes** of items, i.e., the attraction probability $q(a)$ and the expected current and future reward $w(a)$ (here $w(a)=r(s,a)+p(\cdot|s,a)^\top V^*_{h+1}$ given the state $s$ and step $h$).
**Eq. (ii) itself is a complex combinatorial optimization problem** even when one only evaluates the attributes of individual items and ignores the recursion over step $h$, and there is **no obvious monotone rule** with respect to $q(a)$, $w(a)$ or $q(a)*w(a)$.

To solve Eq. (ii), we design an efficient computational oracle BestPerm, using a novel dynamic programming rule based on the cascading feature of $q(a)$ and $w(a)$:
$$
F[i][k] = \max \\{ F[i+1][k],\  q(a_i)  w(a_i) +(1-q(a_i)) F[i+1][k-1] \\} ,\quad 1\leq i \leq J-1,\ 1 \leq k \leq \min\\{m,J-i+1\\} , \quad Eq. (iv)
$$
where 

$
F[i][k]:=\max_{(a'_1,\dots,a'_k) \subseteq (a_i,\dots,a_J)} f((a'_1,\dots,a'_k),q,w)
$

denotes the optimal objective value achieved by selecting $k$ items from items $a_i,\dots,a_J$ which are sorted in descending order of $w(a)$.
The intuition behind our dynamic programming rule is that: for a sorted item sequence $a_i,a_{i+1},\dots,a_J$, if we do not select $a_{i}$, then the optimal objective value is equal to that achieved by selecting $k$ items from $a_{i+1},\dots,a_J$, i.e., $F[i+1][k]$. Otherwise, if we select $a_{i}$, then the optimal objective value is equal to $(1-q(a_i))$ multiplying the optimal value achieved by selecting $k-1$ items from $a_{i+1},\dots,a_J$, i.e., $(1-q(a_i)) F[i+1][k-1]$, due to the cascading feature of the objective function.

We note that the "dynamic programming" we mentioned in our contribution statements refers to the dynamic programming Eq. (iv) that we design to solve the combinatorial optimization Eq. (ii), which iterates over the indices of items $1\leq i \leq J-1$ and the number of selected items $1 \leq k \leq \min\\{m,J-i+1\\}$. **We were not referring to the well-known MDP dynamic programming that iterates over step $h=H,H-1,\dots,1$.**

In other words, our dynamic programming rule (Eq. (iv)) and oracle BestPerm provide an efficient computational solution to the combinatorial optimization Eq. (ii), which one needs to solve at each step $h$ in cascading RL even when only using the attributes of individual items. This is a significant contribution and novel to the combinatorial bandit/RL literature.

**2. Counter-example for the Naive Algorithm**

 A naive algorithm that first sorts items by $q(a)*w(a)$ and further sorts them by $w(a)$ cannot find the optimal ranking.

 Consider a counter-example as follows:
 There are 10 items $a_1,a_2,\dots,a_{10}$, and we want to find the optimal ranking with the maximum cardinality $m=3$. The attraction probabilities $q(a)$, weights $w(a)$ and their products $q(a)*w(a)$ of items are shown in the following table.

|   |  $a_1$ |  $a_2$ | $a_3$  | $a_4$  | $a_5$  | $a_6$  | $a_7$  |  $a_8$ | $a_9$  | $a_{10}$  |
| ------------ | ------------ | ------------ | ------------ | ------------ | ------------ | ------------ | ------------ | ------------ | ------------ | ------------ |
|  $q(a)$ | 0.1  |  0.2 |  0.3 |  0.4 | 0.5  | 0.6  | 0.7  | 0.8  | 0.9  | 1  |
|  $w(a)$ |  1 |  0.9 |  0.8 | 0.7  | 0.6  |  0.5 |  0.4 | 0.3  |  0.2 | 0.1  |
| $q(a)*w(a)$  |  0.1 | 0.18| 0.24| 0.28| 0.3 | 0.3 | 0.28| 0.24| 0.18| 0.1 |

The naive algorithm the reviewer mentioned will output $(a_4,a_5,a_6)$, while our computational oracle BestPerm will output $(a_3,a_4,a_5)$ according to our dynamic programming rule Eq. (iv).

The objective values $f(A,q,w) :=  \sum_{i=1}^{|A|} \prod_{j=1}^{i-1} ( 1-q(A(j)) ) q(A(i)) w(A(i))$ of $(a_4,a_5,a_6)$ and $(a_3,a_4,a_5)$ are

$$
f((a_4,a_5,a_6),q,w)=0.28+(1-0.4)*0.3
$$

$$
\qquad\qquad\qquad\qquad\qquad +(1-0.4)*(1-0.5)*0.3=0.55 ,
$$

$$
f((a_3,a_4,a_5),q,w)=0.24+(1-0.3)*0.28
$$

$$
\qquad\qquad\qquad\qquad\qquad +(1-0.3)*(1-0.4)*0.3=0.562 > f((a_4,a_5,a_6),q,w) .
$$

This demonstrates that the output of the naive algorithm $(a_4,a_5,a_6)$ is not the optimal solution.

Reviewer comment (Novelty):
The contribution related to the generalization to MDPs is not opening a significant amount of new doors. In my opinion, this generalization does not provide further theoretical insights and has limited additional practical relevance.

Author response:
## Rebuttal 1 (ID: rc6MDLxCzV)
Thank you very much for your time and effort in reviewing our paper! We have incorporated your comments in the revised version, and highlighted our revision in *red* font.


**1. Contribution of our Generalization to Cascading MDPs**

We emphasize that our generalization to cascading MDPs is very different from prior cascading bandit works [Kveton et al. 2015; Combes et al. 2015], since we face a unique challenge on **how to efficiently compute the optimal permutation**.
To tackle this challenge, we make use of special properties of the objective function for cascading MDPs, and develop a novel and efficient oracle BestPerm using a dynamic programming approach for combinatorial optimization. 

Specifically, prior cascading bandit works [Kveton et al. 2015; Combes et al. 2015] consider finding the optimal subset of items as follows:
$$
\max_{A \in \mathcal{A}} \quad f(A) = \sum_{i=1}^{|A|} \prod_{j=1}^{i-1} \Big( 1-q(A(j)) \Big) q(A(i)) = 1- \prod_{i=1}^{|A|} \Big( 1-q(A(i)) \Big) . \quad Eq. (i)
$$
This objective function only depends on the attraction probabilities of items $q(A(i))$. Thus, prior works **only need to select $m$ items with the maximum attraction probabilities**, which do not involve computational difficulty.

In contrast, our cascading MDP problem considers  finding the optimal ordered subset of items as follows (for step $h=H,H-1,\dots,1$):
$$
\max_{A \in \mathcal{A}} \quad Q^{*}_h(s,A) 
$$

$$
\ \qquad = \sum_{i=1}^{|A|} \prod_{j=1}^{i-1} \Big( 1-q(s,A(j)) \Big) q(s,A(i)) \Big( r(s,A(i))+p(\cdot|s,A(i))^\top V^{*}_{h+1} \Big) .
$$ 

Our objective function depends on both the attraction probabilities $q(s,A(i))$ and the expected future rewards $r(s,A(i))+p(\cdot|s,A(i))^\top V^{*}_{h+1}$. 

This requires us to **solve a combinatorial optimization problem** for each $s \in \mathcal{S}$ as follows:
$$
\max_{A \in \mathcal{A}} \quad f(A,q,w) =  \sum_{i=1}^{|A|} \prod_{j=1}^{i-1} \Big( 1-q(A(j)) \Big) q(A(i)) w(A(i)) , \quad Eq. (ii)
$$
where $q(a):=q(s,a)$ and $w(a):=r(s,a)+p(\cdot|s,a)^\top V^{*}_{h+1}$ 

(in the online RL process, we can have the estimates of $r(s,a)$, $p(s'|s,a)$ and $V^*_{h+1}(s')$). 
This optimization involves two attributes of items $q(a)$ and $w(a)$, instead of just depending on $q(a)$, which is far more challenging to efficiently compute than Eq. (i).

To overcome this computational difficulty, we first analyze the special properties of $f(A,q,w)$, including the fact that sorting items in descending order of $w(a)$ gives the optimal permutation (Section 4.1). Then, based on these properties, we design an efficient oracle BestPerm to solve Eq. (ii), building upon **a novel dynamic programming approach** for finding the best subset from sorted items (Section 4.2):
$$
F[i][k] = \max \\{ F[i+1][k],\  q(a_i)  w(a_i) +(1-u(a_i)) F[i+1][k-1] \\} ,\quad 1\leq i \leq J-1,\ 1 \leq k \leq \min\\{m,J-i+1\\} .
$$
Here $F[i][k]:=\max_{(a'_1,\dots,a'_k) \subseteq (a_i,\dots,a_J)} f((a'_1,\dots,a'_k),q,w)$ denotes the optimal objective value that can be achieved by selecting $k$ items from items $a_i,\dots,a_J$ which are sorted in descending order of $w(a)$. Furthermore, we also provide rigorous analyses for the properties of $f(A,q,w)$ and the correctness of oracle BestPerm (Lemmas 1 and 2).

To sum up, our generalization to cascading MDPs has the following novelties: (1) This generalization introduces a significant computational challenge on how to efficiently find the optimal permutation. (2) To resolve this challenge, we analyze the special properties of the objective function for cascading MDPs, and design an efficient oracle BestPerm based on a novel dynamic programming. 
Rigorous proofs are provided for the properties of the objective function and the correctness of oracle BestPerm. 
(3) Building upon oracle BestPerm and the carefully-designed exploration bonuses for estimates, we develop provably computationally-efficient and sample-efficient algorithms for cascading RL.

Reviewer comment (Theory):
I am not fully convinced of the practicality of this model. Firstly, historical user behavior can be modeled as part of the context in contextual bandit frameworks. Moreover, "artificially" creating states appears fairly cumbersome compared to the contextual bandit structure and it is also not entirely clear how such states would be defined in practice. However, I could be convinced otherwise.

Author response:
Thank you very much for your appreciation for our work and raising an insightful future direction! We will further explore how to design sensible states for better practicability in our future work.

Reviewer comment (Novelty):
Could you further explain why contextual approaches are insufficient and what the merit of the RL formulation is in contrast to contextual bandits or recommendation approaches based on context trees? In the RL framework it is important what states the user transitions to, as some states may have the potential to yield higher rewards than others. Do you think that this is realistic in practice?

Author response:
Thank you very much for your time in reviewing our paper and your positive comments! We have revised our paper according to your suggestions. Our revision is highlighted in *orange* font.

**1. Merit of the RL Formulation Compared to Contextual Bandits**

We agree with the reviewer that contextual bandits can also be used to formulate historical user behaviors as part of contexts.
We think that RL is more suitable for long-term reward maximization, since it considers state transition and potential future rewards in decision making.

For example, consider a video recommendation scenario. The recommendation system encounters a user who is interested in funny videos and TV series.
Given this context, contextual bandits mainly recommend videos which can maximize the reward in the current context. Hence, contextual bandits may recommend funny videos which usually give higher instantaneous utilities.
In contrast, RL recommends videos which maximize the expected cumulative (long-term) reward, which also considers the potential rewards generated by future states. Thus, in this case, RL may recommend TV series videos. Although TV series videos may have lower instantaneous utilities, once the user is attracted to some of them, the user can enter a high-reward successor state, i.e., he/she is obsessed with it and keeps watching subsequent videos of this TV series. Therefore, using the RL formulation can obtain higher rewards in the long term.

Reviewer comment (Novelty):
It would be great if you could go into a bit more detail in your related work section and more clearly highlight the differences to prior models. For example, in contrast to the classical cascading bandits, the order of items also matters in your case as you usually would like the item with largest reward in state s to be clicked.

Author response:
Thank you very much for your time in reviewing our paper and your positive comments! We have revised our paper according to your suggestions. Our revision is highlighted in *orange* font.

**2. Related Works**

We detail the differences of our cascading RL model from prior works below.

[Kveton et al. 2015; Cheung et al. 2019; Zhong et al. 2021; Vial et al. 2022] study the cascading bandit model. In their model, there is no state (context), and the reward generated by each item (if clicked) is one. Thus, the order of selected items does not matter, and they only need to select $m$ items with the maximum attraction probabilities.
[Zong et al. 2016; Li \& Zhang 2018] consider the  contextual cascading bandit problem. In their problem, the agent first observes the context (i.e., the feature vector of each item) at each timestep, and the attraction probability of each item is the inner-product of its feature vector and a global parameter. The order of selected items still does not matter, and they need to select $m$ items with the maximum attraction probabilities in the current context.
[Li et al. 2016] investigate contextual cascading bandits with position discount factors and a general reward function. In their model, the order of selected items matters, and they assume to have access to an oracle that can output the optimal ordered subset of items for the general reward function.

Different from the above works, our cascading RL formulation further considers state transition, and the attraction probabilities and rewards of items depend on states. Thus, we need to put the items with higher rewards in the current state in the front. In addition, we require to both maximize the attraction probabilities of selected items, and optimize the potential rewards from future states. Furthermore, instead of assuming access to an oracle, we design an efficient oracle to find the optimal ordered subset of items.

Reviewer comment (Writing):
In the 4th contribution, I found the statement about "$\varepsilon$ sufficiently large" slightly confusing. It only becomes clearer later in Theorem 2 when the full bound is stated. Maybe there is a way to state this less confusingly in the introduction.

Author response:
Thank you very much for your time in reviewing our paper and your positive comments! We have revised our paper according to your suggestions. Our revision is highlighted in *orange* font.

**3. Replies to Minor Things**

Thank you for pointing out the unclear sentence and typo!

We have revised the statement in the fourth contribution to "CascadingBPI is
optimal up to a factor of $\tilde{O}(H)$ when $\varepsilon < \frac{H}{S^2}$" to avoid confusion, and fixed the typo.

Reviewer comment (Writing):
Typo in the 3rd paragraph of Section 7: "Regarding the best policy identification objective" instead of "Regarding the regret minimization objective".

Author response:
Thank you very much for your time in reviewing our paper and your positive comments! We have revised our paper according to your suggestions. Our revision is highlighted in *orange* font.

**3. Replies to Minor Things**

Thank you for pointing out the unclear sentence and typo!

We have revised the statement in the fourth contribution to "CascadingBPI is
optimal up to a factor of $\tilde{O}(H)$ when $\varepsilon < \frac{H}{S^2}$" to avoid confusion, and fixed the typo.

Reviewer comment (Presentation):
The readability of the algorithms can be improved. e.g., Update rule of $\bar{q}^k$ and $\underbar{q}^k$ in Line 6 of Algorithm 2, Line break in Line 8 of Algorithm 2, etc.

Author response:
Thank you for your valuable suggestions! We have revised Algorithm 2 according to your comments to improve readability in our revision.

Reviewer comment (Theory):
Where does the gap $\sqrt{H}$ appear? Can you elaborate on this? Any possible direction to carve off this given that the tabular RL methods with a single action can achieve minimax optimality?

Author response:
**4. The Gap $\sqrt{H}$**

The gap $\sqrt{H}$ appears because we construct the exploration bonuses for $q(s,a)$ and $p(\cdot|s,a)^\top V^*_{h+1}$ separately, which leads to an additional $\sqrt{H}$ factor when summing up $b^{k,q}(s,a)$ and $b^{k,pV}(s,a)$ in regret analysis.

Below we elaborate this in detail.  In regret analysis, using value difference lemma, the regret decomposition contains the following term
$$ 
\sum_{k=1}^{K} \sum_{h=1}^{H} \mathbb{E} \Bigg[ 
\sum_{i=1}^{|A_h|} \prod_{j=1}^{i-1} (1-q(s_h,A_h(j))) \bigg( \bar{q}^k(s_h,A_h(i)) \left( \hat{p}^k(\cdot|s_h,A_h(i))^\top \bar{V}^k_{h+1} + b^{k,pV}(s_h,A_h(i)) \right)- q(s_h,A_h(i)) p(\cdot|s_h,A_h(i))^\top \bar{V}^k_{h+1} \bigg) \Bigg]
$$
$$
= \sum_{k=1}^{K} \sum_{h=1}^{H} \mathbb{E} \Bigg[ \sum_{i=1}^{|A_h|} \prod_{j=1}^{i-1} (1-q(s_h,A_h(j))) \bigg( \Big( q(s_h,A_h(i))+2b^{k,q}(s_h,A_h(i)) \Big)  \Big( \hat{p}^k(\cdot|s_h,A_h(i))^\top \bar{V}^k_{h+1} + b^{k,pV}(s_h,A_h(i)) \Big) - q(s_h,A_h(i)) p(\cdot|s_h,A_h(i))^\top \bar{V}^k_{h+1} \bigg) \Bigg] \quad Eq. (i)
$$
$$
\leq \sum_{k=1}^{K} \sum_{h=1}^{H} \mathbb{E} \Bigg[ \sum_{i=1}^{|A_h|} \prod_{j=1}^{i-1} (1-q(s_h,A_h(j))) \bigg( q(s_h,A_h(i)) \Big( \hat{p}^k(\cdot|s_h,A_h(i))^\top \bar{V}^k_{h+1} + b^{k,pV}(s_h,A_h(i)) - p(\cdot|s_h,A_h(i))^\top \bar{V}^k_{h+1} \Big) + 4Hb^{k,q}(s_h,A_h(i)) \bigg) \quad Eq. (ii) \Bigg]
$$
Since we construct exploration bonuses for $q(s,a)$ and $p(\cdot|s,a)^\top V^*_{h+1}$ separately, in Eq. (i) above, the regret contains a product $(q+b^{k,q})(\hat{p}^\top \bar{V}+b^{k,pV}-p^\top V)$, while the regret for classic RL only contains the term $\hat{p}^\top \bar{V}+b^{k,pV}-p^\top V$. To handle this product, we bound it by $q(\hat{p}^\top \bar{V}+b^{k,pV}-p^\top V)+b^{k,q}(\hat{p}^\top \bar{V}+b^{k,pV}-p^\top V)=q(\hat{p}^\top \bar{V}+b^{k,pV}-p^\top V)+O(H b^{k,q})$. Here the $O(H b^{k,q})$ term directly leads to a $\tilde{O}(H\sqrt{HK})$ regret.

A straightforward idea to improve the $\sqrt{H}$ gap is to combine the attraction probability $q$ and transition distribution $p$ as an integrated transition distribution $\tilde{p}(s'|s,A)$ for each item list $A$, and construct an overall exploration bonus for $\tilde{p}(s'|s,A)$. However, this strategy forces us to maintain the exploration bonus for each $A \in \mathcal{A}$, which will incur an additional dependency on $\mathcal{A}$ in the regret bound and is computationally inefficient.

We think that a more fine-grained analysis for bounding $b^{k,q}(\hat{p}^\top \bar{V}+b^{k,pV}-p^\top V)$ or a more advancing bonus construction approach may be helpful to close the $\sqrt{H}$ gap. We plan to further investigate it in our future work.

Reviewer comment (Experiments):
Currently the experiment results are very limited. In particular, this paper has only demonstrated experiment results in small-scale synthetic problems. I think experiment results on large-scale practical problems will further strengthen this paper.

Author response:
**2. Experiment**

Following the reviewer's suggestion, we conduct experiments on a larger-scale **real-world** dataset MovieLens [Harper \& Konstan 2015] (also used in prior works [Zong et al. 2016; Vial et al. 2022]) with more baselines. We present preliminary experimental results here (due to time limit), and will certainly include more results for larger numbers of states and items in our revision.

The MovieLens dataset contains 25 million ratings on a 5-star scale for 62000 movies by 162000 users. 
We regard each user as a state, and each movie as an item. For each user-movie pair, we scale the rating to [0,1] and regard it as the attraction probability.
The reward of each user-movie pair is set to 1. For each user-movie pair which has a rating no lower than 4.5 stars, we set the transition probability to this state (user) itself as 0.9, and that to other states (users) as $\frac{0.9}{S-1}$. For each user-movie pair which has a rating lower than 4.5 stars, we set the transition probability to all states (users) as $\frac{1}{S}$. 
We use a subset of data from MovieLens, and set $\delta=0.005$, $K=100000$, $H=3$, $m=3$, $S=20$, $N \in \\{10,15,20,25,30,40\\}$ (the number of items) and $|\mathcal{A}|=\sum_{\tilde{m}=1}^{m} \binom{N}{\tilde{m}} \tilde{m}! \in \\{820,2955,7240,14425,25260,60880\\}$ (the number of item lists). 

We compare our algorithm CascadingVI with three baselines, i.e., CascadingVI-Oracle, CascadingVI-Bonus and AdaptVI. Specifically, CascadingVI-Oracle is an ablated version of CascadingVI which replaces the efficient oracle BestPerm by a naive exhaustive search. CascadingVI-Bonus is an ablated variant of CascadingVI which replaces the variance-aware exploration bonus $b^{k,q}$ by a variance-unaware bonus. 
AdaptVI adapts the classic RL algorithm [Zanette
\& Brunskill 2019] to the combinatorial action space, which maintains the estimates for all $(s,A)$ rather than $(s,a)$.
The following table shows the cumulative regrets and running times of these algorithms. (CascadingVI-Oracle, which does not use our efficient computational oracle, is slow and still running in some instances with large $N$. We will update its results once it finishes.)

| Regret; Running time  | CascadingVI (ours)  | CascadingVI-Oracle  | CascadingVI-Bonus  |  AdaptVI |
| ------------ | ------------ | ------------ | ------------ | ------------ |
| $N=10, \mid \mathcal{A} \mid =820$  | 2781.48; 1194.29s  | 4097.56; 21809.87s  | 8503.45; 1099.32s  | 29874.84; 28066.90s  |
| $N=15, \mid \mathcal{A} \mid =2955$  | 4630.11; 2291.00s  | 6485.36; 71190.59s  | 13436.38; 2144.96s  | 30694.40; 53214.95s  |
|  $N=20, \mid \mathcal{A} \mid =7240$ | 6446.51; 2770.94s  | 9950.22; 145794.64s  | 17472.21; 2731.21s  | 34761.91; 81135.42s |
|  $N=25, \mid \mathcal{A} \mid =14425$ | 8283.60; 5216.00s   | 13170.39; 196125.72s  |  21868.54; 5066.27s | 42805.06; 95235.55s  |
|  $N=30, \mid \mathcal{A} \mid =25260$ | 11412.85; 7334.13s  |  Still running | 27738.31; 7256.94s  | 52505.85; 188589.60s  |
|  $N=40, \mid \mathcal{A} \mid =60880$ | 16825.65; 18628.83s  |  Still running | 37606.95; 17829.40s  |  56045.01; 319837.83s |

As shown in the above table, our algorithm CascadingVI achieves the lowest regret and a fast running time. CascadingVI-Oracle has a comparative regret performance to CascadingVI, but suffers a much higher running time, which demonstrates the power of our oracle BestPerm in computation.
CascadingVI-Bonus attains a similar running time as CascadingVI, but has a worse regret (CascadingVI-Bonus looks slightly faster because computing a variance-unaware exploration bonus is simpler). This corroborates the effectiveness of our variance-aware exploration bonus in enhancing sample efficiency.
AdaptVI suffers a very high regret and running time, since it learns the information of all permutations independently and its statistical and computational complexities depend on $|\mathcal{A}|$.

We have also presented the figures of these experimental results in our revised version.