Paper: SocioDojo: Building Lifelong Analytical Agents with Real-world Text and Time Series (ICLR-2024)

Reviewer comment (Experiments):
Since GPT-3.5-Turbo was used as foundation models with a non-zero (0.2) temperature, the results should not be fully deterministic, while the paper may miss some studies on randomness.

Author response:
We choose a non-zero setting as 0 temperature may restrict the model performance. While the chosen temperature of 0.2 (on a scale of 0 to 2) is relatively low and aligns with recommendations for deterministic tasks, we recognize the value of further exploring how varying temperatures might affect outcomes. We are currently conducting additional studies on this aspect and plan to include these findings in future updates of our work.

Reviewer comment (Experiments):
The setting of "Forbid day trading", that an asset cannot be sold within 5 days of purchase to avoid profiting from short-term patterns, might be overly strict.

Author response:
The stringent "Forbid day trading" rule was designed to elevate the cost of decision-making, thereby promoting depth and consideration in the agents' analyses. This aligns with our goal of fostering high-quality, thoughtful decision-making. Additionally, our use of daily close price data, which are public while minute-level data are usually commercial, necessitates such a restriction to counterbalance the potential information advantage that agents might have due to data delays. We have added Appendix D to examine the implications of utilizing close price data.

Reviewer comment (Experiments):
When multiple news articles sent out on the same day, did the order of news matter? i.e., to which extend the agent is permutation invariant?

Author response:
In SocioDojo, each news piece is timestamped to maintain chronological order, mirroring real-world scenarios. Overlaps in timestamps are rare, but even if they occur, they accurately reflect real-life information flows, ensuring that the agent's response is realistically grounded.

Reviewer comment (Evaluation):
The proposed “return” of “hyperportfolio”, as some kind of evaluation metric, is very hard to interpret. While it seems that it is related to how accurately the agent is able to predict the future of various time series, it’s hard to make sense of the numerical value, especially when various time series are fused together.

Author response:
We have handled your concern on time series like GDP in **Section 3.2.2:** ***"Since some assets are almost ever-growing, such as GDP, which leads to risk-free returns, we wish all returns to come with risk, thus requiring good analysis to achieve a better risk-return trade-off."*** and further detailed in **Appendix C Robustness Analysis of SocioDojo**. We introduced mechanisms like overnight rate and explored the proper settings by experiment to eliminate "risk-free" returns for such time series to ensure every positive return can reflect an improvement in societal understanding and decision-making. We agree that developing more intuitive evaluation metrics for agents is an essential aspect of our future work, but also challenging akin to evaluating a fund in the real-world financial context.

Reviewer comment (Evaluation):
Instead of expected return, modern portfolio theory typically tries to maximize risk-adjusted return. Is there any specific reason that the authors do not consider the standard deviation?

Author response:
Our initial focus on expected return was for simplicity and intuitive comparison. Acknowledging your valid point, we have now included Sharpe Ratio calculations in Table 3 to provide a more comprehensive risk-adjusted performance assessment.

Reviewer comment (Presentation):
In Table 1, it is shown that there would be 9 research papers per day. Is this an average statistic or subscribed information that will constantly have 9 papers?

Author response:
The figure quoted is an average. We have updated the table to clarify this by changing the label to “Avg. Per Day” to prevent confusion.

Reviewer comment (Experiments):
For some of the time series, e.g. GDP as an economic time series, we would have existing forecasting beforehand. They might be part of the report, and the model can directly use these numbers, which are typically fairly accurate. In this case, how do you confirm that the agent utilizes the comprehensive information from different sources to make the “investment” instead of directly taking those numbers?

Author response:
We view the use of publicly available forecasts as a legitimate strategy, akin to what a human analyst might employ. In addition, the beforehand forecastings are not always reliable and can vary a lot even for famous models like GDPNow. For example, the Fed Banks’ GDP Estimates Vary Widely for the third quarter this year reported by Barron's. Thus, it is still a challenge to figure out the right prediction even provided with the forecasted values even for human experts, and requires the ability to integrate and verify diverse information sources.

Reviewer comment (Evaluation):
Financial time series is typically the most noisy one compared with other time series, which should also be the least predictable. However, it is shown that the financial time series achieves the highest return. How do authors interpret such results?

Author response:
We delve into this in the newly added Appendix D. Firstly, it cannot be directly interpreted as a real financial return, as it does not fully follow the real market rules like trading hours, and we use daily close price instead of real-time data for public distribution as real-time data demands commercial access. However, the agent may have an information advantage since the close prices are delayed and do not reflect the latest news thus the agents are traded at a discounted price. We analyze the impact of using non-real-time data in Appendix D.1 by adjusting the financial returns with the minute-level data, which shows an overall decrement in the returns while still preserving the comparative advantages among agents. Secondly, the agent is still skewed by the information sources that frequently report big tech and medical companies, which leads to a portfolio similar to the case of the FANG+ index that tracks large trending companies. This strategy performs well in the experiment time span. We analyzed it in Appendix D.2.

Reviewer comment (Presentation):
Typos: equation 3.1.2, Table 3.2.1 and Figure 3.2.1 are not pointing to the correct place and are not indexed correctly. Table 1 in the Knowledge Base part should be Table 2.

Author response:
Thank you for pointing out these errors. We have corrected the mentioned typos and references, and have conducted a thorough review of the paper to ensure accuracy in all citations and references.