Paper: AutoCast++: Enhancing World Event Prediction with Zero-shot Ranking-based Context Retrieval (ICLR-2024)

Reviewer comment (Writing):
The writing of the paper needs to be further improved and all the symbols need to be interpreted.

Author response:
Thank you for your comment. We have updated the notation explanations in the PDF to enhance clarity.

Reviewer comment (Writing):
The work focuses on a single textual modality, so why mention multiple data sources, multiple modalities in the introduction section?

Author response:
Thank you for your feedback. In response, we have adjusted our text to more explicitly state our objective of exploring data modalities that are more complex than traditional structured time-series data. The revised sentence is presented below for reference:

*However, as the demand for more accurate forecasts in diverse domains has grown, the need to integrate data from **beyond the structured time-series modality** has become apparent.*

Reviewer comment (Writing):
N_q is a subset of D. Why are the elements in N_q n and not d?

Author response:
Thank you for pointing out this inconsistency. We have updated our notation to use $n$ instead of $d$ to represent each news article, with $n$ serving as an abbreviation for 'news'. Our text has been revised accordingly to reflect this change.

Reviewer comment (Experiments):
The dataset used is too small and results on more as well as larger datasets are needed to be validated.

Author response:
Thank you for your feedback. The Autocast dataset [1] comprises approximately 5700 questions, including both training and testing sets. In comparison, the most similar dataset, ForecastQA [2], contains slightly over 10,000 questions in total. However, we contend that the quantity of questions should not be the primary criterion for assessing the actual scope of such datasets.

While Autocast has fewer questions, it offers a significantly larger context retrieval space. ForecastQA's questions are limited to the time span from January 11, 2019, to November 12, 2019, all within the year 2019. In contrast, Autocast's questions span from 2016 to 2022, necessitating the retrieval and reader module to potentially handle more intricate information that encompasses greater long-term unpredictability.

Furthermore, as highlighted in the Autocast paper [1], the closest benchmark dataset, ForecastQA, has certain limitations in its data. We quote from [1] as follows:

> ForecastQA's questions were authored by crowdworkers without any forecasting experience. Consequently, these questions often lack coherence or clarity due to the absence of additional context, as exemplified by queries like "To how many people will the Representative of an internet speak to by September 2019?" or "In July 2019, will an article say there were no volunteers in 2016?"

We believe that exploring our methodology on additional datasets is a promising avenue for future research, and we are committed to pursuing this in our future work.

Ref:\
[1] Forecasting future world events with neural networks. NeurIPS 2022.\
[2] FORECASTQA: A Question Answering Challenge for Event Forecasting with Temporal Text Data. ACL 2021.

Reviewer comment (Experiments):
In ablation study, no specific experiment has been conducted to demonstrate the isolated effectiveness of the Alignment Loss component.

Author response:
We are thankful for the reviewer's valuable feedback.
>**Q2. The isolated effectiveness of the Alignment Loss component.**

Please kindly refer to Table 3 in our paper, where we present ablation studies focused on the impact of disabling the alignment loss in our model. 

Additionally, in this rebuttal, we trained the FiD Static baseline with additional alignment loss. The summarized results of these experiments, specifically for the 0.2B model sizes, are detailed in the table below.

| Model                             | Retriever (GPT)   | Retriever (non-GPT) | Reader (GPT)  | Reader (non-GPT)  | Reader (non-GPT) | T/F$\\uparrow$ | MCQ$\\uparrow$ | Num$\\downarrow$ |
|-----------------------------------|-------------------|---------------------|---------------|-------------------|------------------|----------------|----------------|------------------|
|                                   | Relevance Re-rank | Recency Re-rank     | Summarization | Numerical Binning | Alignment Loss   |                |                |                  |
|             FiD Static            | No                | No                  | No            | No                | No               | 62.0           | 29.6           | 24.5             |
|       FiD Static + alignment      | No                | No                  | No            | No                |          **Yes** | 62.5           | 31.2           | 23.5             |
| FiD Static + GPT3| **Yes**           | No                  | **Yes**       | No                |               No | 65.5           | 42.1           | 21.3             |
| Autocast++ without alignment loss | Yes               | Yes                 | Yes           | Yes               | **No**           | 66.7           | 43.5           | 19.9             |
| Autocast++ (ours)                 | Yes               | Yes                 | Yes           | Yes               | Yes              | 66.7           | 43.8           | 19.8             |

This alignment loss is beneficial for the plain FiD Static baseline, which lacks any LLM-enhanced components. However, in the full Autocast++ model, where relevance re-ranking and text summarization are employed, the alignment loss appears to have a diminished role in enhancing performance.

In our additional experiment `FiD Static + GPT3`, we demonstrate that the baseline performance can be significantly enhanced by using the pretrained GPT3-enabled techniques. We acknowledge that the effect of alignment loss is relatively minor when compared to using pretrained LLM in the retriever and reader.