Paper: Robust Reinforcement Learning for Portfolio Management via Competition and Cooperation Strategies (ICLR-2024)

Reviewer comment (Writing):
The paper proposes a new multi-agent formulation but fails to properly define many important concepts in the approach, such as the definition of the state and action spaces.

Author response:
The action space is [-1,1], and each model will generate an action array with a length of the number of stocks. Each element represents the number of stocks sold or bought for this stock, and the positive sign represents the purchase and the negative sign represents the sale. In the actual processing, it will be multiplied by 1000 to restore the true value range.

Reviewer comment (Writing):
It is unclear how the RL actions translate to the decisions in the portfolio optimization problem.

Author response:
The action space is [-1,1], and each model will generate an action array with a length of the number of stocks. Each element represents the number of stocks sold or bought for this stock, and the positive sign represents the purchase and the negative sign represents the sale. In the actual processing, it will be multiplied by 1000 to restore the true value range.

Reviewer comment (Experiments):
The authors only compared their method with the performances of standard single-agent RL algorithms despite their formulation being multi-agent, which raises questions about the fairness of such comparisons.

Author response:
A2C, PPO, and DDPG are three single agents, and it is not appropriate to compare them with multi-agent.

Reviewer comment (Reproducibility):
Regarding the statement "*This opponent is generated by the CCRRL model based on the output of the previous step.*": How is this done? Does the opponent use the same model?

Author response:
Opponent is created using S1-S4, and its significance lies in using the results of the previous step to guide the direction of action generation in this step.

Reviewer comment (Writing):
The paper is not well-written as a lot of ideas have not been well-explained. For example, what is the reward? Why one has to use a modification function? Why is the adversarial action taken like in S1 and S2? Why is the cooperative action taken like in S3, and S4?

Author response:
In the new version, we have made improvements to the literature review and model mechanism explanation sections, and changes have been marked in red in the main text.

In the model, the states include the current total funds, the closing prices of each stock, and the current holdings of each stock.

The reward calculation first determines the total funds before and after the market opens and closes, takes the difference between them, and then subtracts the transaction fees to obtain the reward for the day. Finally, the reward value is multiplied by 0.001 to narrow its range.

In the model, after the action is generated, we apply strategies S1-S4 to process the action.

Reviewer comment (Reproducibility):
How each of the S1, S2, S3, and S4 are appended with the action?

Author response:
In the new version, we have made improvements to the literature review and model mechanism explanation sections, and changes have been marked in red in the main text.

In the model, the states include the current total funds, the closing prices of each stock, and the current holdings of each stock.

The reward calculation first determines the total funds before and after the market opens and closes, takes the difference between them, and then subtracts the transaction fees to obtain the reward for the day. Finally, the reward value is multiplied by 0.001 to narrow its range.

In the model, after the action is generated, we apply strategies S1-S4 to process the action.

Reviewer comment (Theory):
The action generation process appears to be an additional function on top of learned actions, which is not theoretically justified in any way.

Author response:
The motivation of the modify function is to consider the reward of the previous action and adjust the action of this step accordingly. If the reward of the previous step is negative, it indicates that the previous action was not good and needs to be avoided in order to achieve robustness and improve reward.

Reviewer comment (Theory):
What is the motivation for the action modifications?

Author response:
The motivation of the modify function is to consider the reward of the previous action and adjust the action of this step accordingly. If the reward of the previous step is negative, it indicates that the previous action was not good and needs to be avoided in order to achieve robustness and improve reward.

Reviewer comment (Writing):
A majority of this paper is not well-written. The authors should revise both the format and the content of this paper.

Author response:
In the new version, we have made improvements to the literature review and model mechanism explanation sections, and changes have been marked in red in the main text.

Reviewer comment (Writing):
Notations like $reward$ to represent the reward in previous step is wired.

Author response:
In our model, we avoid the previous step's reward having too much influence on the current step's action. Because the result of the previous step only has a reference effect, it is not possible to copy it completely.

Reviewer comment (Reproducibility):
What is the form of the `modify` function?

Author response:
The function of Modify can be expressed as:
\begin{equation}
	modify(reward) = \begin{cases}
		0 & \text{if } reward = 0 \\
		10^n \times reward & \text{if } |reward| < 1\\
		reward & \text{if } |reward|>1
	\end{cases}
	\label{equation2}
\end{equation}

Reviewer comment (Theory):
How does the proposed method, Robust Reinforcement Learning via Multi-Agent System (CCRRL), correlate with the multi-agent system? The developed strategies are not agent-level but permutation along the action domain.

Author response:
The subjects of cooperation and competition are the four models trained with s1-s4. There is no data interaction between them, except that in each round, the model with the best reward in the simulation is selected to provide actions for training ddpg. However, they share the same environment and there is indirect data interaction. The system in this case can be regarded as a distributed agent system.

Reviewer comment (Experiments):
This paper in fact proposes a multi-agent reinforcement learning method for portfolio optimization problem. However, no MARL methods are discussed and compared with the proposed method.

Author response:
Comment 2: We conducted a search and found no open-source projects in the current literature that use MARL to solve dynamic portfolio optimization problems.

Reviewer comment (Theory):
The author claims that they introduce both competition and cooperation strategies in the proposed method. However, this will lead to unstationary learning environment, which makes the method hard to be converged. In fact, most MARL methods work only for cooperative cases.

Author response:
Comment 1: Because the four models do not affect the environment simultaneously, they can solve the problem of unstable environments. However, it is unclear whether multiple agents that do not affect the environment simultaneously are truly multi-agent environments.

Reviewer comment (Experiments):
In the experiment, no MARL methods are used as baselines.

Author response:
Comment 2: We conducted a search and found no open-source projects in the current literature that use MARL to solve dynamic portfolio optimization problems.

Reviewer comment (Writing):
The paper does not clearly define what type of behaviors it expects to generate, and why these behaviors would be useful for the stock portfolio. Are they like brokers hired by the owner? What does it mean for two brokers of an owner to compete or to collaborate? Aren't they sharing all the information?

Author response:
In the new version, we have made improvements to the literature review and model mechanism explanation sections, and changes have been marked in red in the main text.

The subjects of cooperation and competition are the four models trained with s1-s4. There is no data interaction between them, except that each round selects the model with the best reward in the simulation to provide actions for training ddpg. However, they share an environment and there is indirect data interaction.