Paper: Abstractive Summarization through the PRISM of Decoding Strategies (ICLR-2024)

Reviewer comment (Presentation):
Because of the density of information and the lack of differentiability in the chosen colors for different decoding strategies, Figure 3 is very hard to follow. For example, Greedy Search and Beam Sampling are both presented with "reddish" color, and the reviewer finds it very difficult to identify which is which in the plot. Additional comments on the figure are presented as questions in the "Questions" section.

Author response:
Thank you for bringing this concern to our attention. In response to your observation, we revised the colors related to decoding strategies, contributing to a more accessible interpretation of our findings. We updated every figure accordingly. Additionally, we employed distinct marks (circles and triangles) to visually contrast deterministic and stochastic decoding strategies.

Reviewer comment (Presentation):
Because of the lack of interpretability of Figure 3 and lack of tabulated data for the results, it is hard for the reviewer to confirm the quantitative differences referred to in the Results section (e.g., +48% UNR avg. score mentioned between short and long summaries).

Author response:
We included tabular data in Appendix G (Table 11), thus providing a detailed breakdown of Figure 3 and ensuring the reproducibility of the reported findings. Please note that the exact metric scores will be openly released in our PRISM dataset.

Reviewer comment (Evaluation):
Contrary to the discussion on the relative advantage of certain decoding strategies over others in section 5.1, the impression of the reviewer from Figure 8 is that  
(i) sampling, top-k, and top-p sampling clearly have larger sample variance than all other strategies, even after hyperparameter tuning, on five out of six datasets;  
(ii) the behavior of all decoding methods changes significantly on Multi-LS dataset, showing larger scale of variation in average score and much smaller sample-specific variance;  
(iii) apart from the three sampling strategies mentioned in (i), the remaining methods on all datasets except Multi-LS seem to be consistently on par with each other as evaluated by most of the evaluation metrics.  
These observations may contradict the quantitative findings in section 5.1, and there seems to lack specific discussion on those arguably more obvious patterns than the ones presented in Figure 3 and discussed in the text.

Author response:
We appreciate your suggestion not to overlook more evident patterns. We agree with your observations, although we found no contradictions with the quantitative findings previously reported in Section 5.1. Following your comments, we meticulously double-checked all the results outlined in the main paper and Appendix. We confirmed the content of each figure and table, except Figure 3, where we found erroneous scores. After correction, we reviewed the quantitative findings listed in Section 5, adding the suggested ones if not already present. More generally, we provided a more detailed discussion of results, enriching the answer to every research question.

Reviewer comment (Evaluation):
The generalizability of the findings is limited. While the lack of a universal strategy for optimal decoding is substantively supported by the results, the paper seems to be lacking in identifying other general trends or patterns in the combination of research parameters (decoding strategy, task, model, hyperparameters). This limits the value of the results to being observational rather than inferential.

Author response:
We are dedicated to providing insights that go beyond simple observations. In line with this commitment, we revised and expanded our analysis, introducing per-dataset rankings (Appendix H) and new findings (Section 5). Combined with result verification, we identified prevalent performance patterns and established a set of recommendations to better assist practitioners in real-world summarization use cases. Following your suggestions, we attached a visual and easy-to-follow guideline to the conclusion (Appendix L), driving the selection of valuable decoding spaces (i.e., strategy + hyperparams) depending on the optimization target and AS type.

Reviewer comment (Evaluation):
The statement **"Elevating Fluency undermines Recall, Precision, and Factuality."** in Section 5.2 is not rigorous based on the results. What the authors have observed is a negative correlation between fluency and other quality factors as evaluated by humans, but the phrasing of the finding turns such a correlation into a misleading causal statement.

Author response:
Upon careful review, we concur with your observation that our statement could be misconstrued as implying a causal relationship. We did not intend to assert a causal link and we appreciate your diligence in bringing this to our attention. We revised the finding description by using the "negative correlation" keyphrase.

Reviewer comment (Evaluation):
It is not rigorous to draw conclusions on "factuality", "factual flaws", or "hallucination" solely based on BARTScore (e.g., Section 5.1, **stochastic vs deterministic**), which is at best a proxy for those quality measures. The paper should at the very least state in the main text that the studied automatic metrics are "proxies" of the quality dimensions of interest.

Author response:
We added a cautionary note at the beginning of Section 5, explicitly stating that the studied automatic metrics are "proxies" for the quality dimensions of interest. We underlined this concept in other parts of the paper, including Table 3 and Appendix L.

Reviewer comment (Presentation):
Typo in Table 4: shouldn't UNR be "Unique N-gram Ratio", not "unigram n-gram ratio"?

Author response:
We amended the reported typographical errors and revised the entire paper for improved fluency and clarity.

Reviewer comment (Evaluation):
While the experiments are comprehensive, I find that the paper lacked overall recommendations that practitioners can follow in a real-world setting (which I believe is the main purpose of the paper). For example, if one has a new summarization use case, how does the paper help them decide which model and which decoding strategies (+ hyperparams) to use? The paper does provide a list of findings, however a set of recommendations as part of the conclusion would greatly improve the paper's impact.

Author response:
- We acknowledge your concerns. In response, we revised and expanded our analysis, introducing per-dataset rankings (Appendix H) and new findings (Section 5). We identified prevalent performance patterns and established a set of recommendations to better assist practitioners in real-world summarization use cases. Following your suggestions, we attached a visual and easy-to-follow guideline to the conclusion (Appendix L), driving the selection of valuable decoding spaces (i.e., strategy + hyperparams) depending on the optimization target and AS type.

Reviewer comment (Experiments):
The models are unfortunately limited to 400-500M parameters. This is significantly small if compared with LLMs, thus it is uncertain whether the results shown in this paper transfer to these large models. It could have helped to see models of varying sizes (perhaps one small-scale and one XL-scale) to show that the results actually hold even when the model size is different.

Author response:
- We recognize your point. We did not experiment with LLMs and tiny models. As explicitly outlined in the abstract, introduction, related works, method, and Appendix N, the focal point of our study was the evaluation of models predominantly utilized by the AS community, as opposed to those that may rise to prominence in the future. It is important to note that, at the beginning of this project, open-source LLMs were not as widespread as they are today. Our meticulous model selection process involved identifying cutting-edge solutions within each AS family (short, long, multi-document). The chosen models exhibit an encoder-decoder architecture and fall within the range of a million parameters or barely surpass 10B---a lower boundary often cited when discussing LLMs.  To further elucidate our decision-making, we refined and expanded our bibliometric analysis within the SCOPUS database.  In Appendix B, we introduced a chart depicting the intersection of AS and LLMs in the literature. The analysis unveiled that a mere 15.8\% of conference papers from 2020 onward explicitly integrate LLMs and AS in their title, abstract, and keywords. While an extension of our study to include LLMs and tiny LMs would be commendable, focusing solely on one model for these families would be unsound, with the risk of including biased non-representative decoding-level conclusions. Consequently, it is imperative to encompass several new models. In this sense, we emphasize that the completion of our study necessitated 73 cumulative days of computation using NVIDIA 3090 RTX Turbo GPUs (Section 4.1). In particular, incorporating experiments with LLMs would entail a substantial additional commitment of time and resources. Our empirical tests revealed that transitioning from 440M models to 7B models (Bloom, Falcon, LLaMA-2) results in an average increase in decoding time per instance by a factor of x150. It becomes crucial to strike a balance between the depth of analysis and the practical constraints. As such, we believe that this extension warrants a dedicated effort and a separate study.

Reviewer comment (Experiments):
How do these results transfer at scale (smaller or larger)?

Author response:
- We recognize your point. We did not experiment with LLMs and tiny models. As explicitly outlined in the abstract, introduction, related works, method, and Appendix N, the focal point of our study was the evaluation of models predominantly utilized by the AS community, as opposed to those that may rise to prominence in the future. It is important to note that, at the beginning of this project, open-source LLMs were not as widespread as they are today. Our meticulous model selection process involved identifying cutting-edge solutions within each AS family (short, long, multi-document). The chosen models exhibit an encoder-decoder architecture and fall within the range of a million parameters or barely surpass 10B---a lower boundary often cited when discussing LLMs.  To further elucidate our decision-making, we refined and expanded our bibliometric analysis within the SCOPUS database.  In Appendix B, we introduced a chart depicting the intersection of AS and LLMs in the literature. The analysis unveiled that a mere 15.8\% of conference papers from 2020 onward explicitly integrate LLMs and AS in their title, abstract, and keywords. While an extension of our study to include LLMs and tiny LMs would be commendable, focusing solely on one model for these families would be unsound, with the risk of including biased non-representative decoding-level conclusions. Consequently, it is imperative to encompass several new models. In this sense, we emphasize that the completion of our study necessitated 73 cumulative days of computation using NVIDIA 3090 RTX Turbo GPUs (Section 4.1). In particular, incorporating experiments with LLMs would entail a substantial additional commitment of time and resources. Our empirical tests revealed that transitioning from 440M models to 7B models (Bloom, Falcon, LLaMA-2) results in an average increase in decoding time per instance by a factor of x150. It becomes crucial to strike a balance between the depth of analysis and the practical constraints. As such, we believe that this extension warrants a dedicated effort and a separate study.