Paper: BooookScore: A systematic exploration of book-length summarization in the era of LLMs (ICLR-2024)

Reviewer comment (Evaluation):
The evaluation of human annotations focuses only on precision and does not investigate recall. However, the authors are open about this limitation.

Author response:
In order to measure recall, we would require ground truth annotations with all possible coherence errors in the summaries. However, prior studies (SNaC and Scarecrow) have discussed that this is challenging for subjective fine-grained evaluation tasks where annotators generally display high precision but low recall.

One solution could be to collect annotations from multiple annotations per summary, and consider their union as the exhaustive set of errors. Scarecrow and SNaC employ 10 and 3 annotators per summary respectively, but such a scheme would be very costly for our setting, as annotating 100 summaries already costs $1.5K. Annotation precision is more economical to evaluate, while also serving as an informative metric. We have updated Section 3 to make it clearer. We believe future work in human evaluation that makes recall-centric annotation more tractable is a promising area that could complement the precision-centric measurement we propose here.

Reviewer comment (Experiments):
Some of the examples of types of errors shown in Table 1 seem to be of questionable quality; for example, the first example about the "mysterious man" might as well be unanswered in the book (although I do not know the contents of the book in question). This may be a side effect of offering a monetary reward based in part on the number of annotations (cf. Appendix G).

Author response:
This is a valid concern, and we appreciate the reviewer for pointing it out. We acknowledge that sometimes annotators or GPT-4 may ask questions that are also unanswered in the book, and there is no way to confirm without reading the source text. Using the reviewer’s example, if the identity of the mysterious man is not revealed in the book, the summary could say something like “A mysterious man whose identity is never revealed.” Since our focus is on coherence, the nature of our evaluation protocol is to have readers naturally raise questions as they read through a summary. Checking whether a question can be answered by the source text and finding answers to these questions would fall under faithfulness evaluation, which we leave to future work.

We appreciate the reviewer for going through the appendix. For the first evaluation task where we collected span-question pairs, we actually paid $15 USD per summary regardless of how many annotations were provided as mentioned in Section 3.

Appendix G (now J) is for the second human evaluation where we validated annotations collected from the first one. For this evaluation, we paid based on how many annotations from the first task an annotator had to annotate, which also did not incentivize them in any way. The annotators were informed of the second evaluation task a few weeks after they completed the first one, so payment for the second task could not have biased annotators to create more annotations for the first. We understand that this may be unclear, and have made clarifications in Appendix J.

Reviewer comment (Evaluation):
The sentence-level score may disproportionately favor summaries that contain a large number of (short) sentences. Furthermore, the evaluation of different models does not further investigate whether some of the score differences can be explained by the different length of the summaries (e.g., a shorter summary may be more prone to omissions).

Author response:
We understand this concern. In practice, we do not observe significant differences in sentence length. We computed the average number of tokens per sentence for the 100 summaries collected under each configuration (10 in total). The minimum is 21, maximum is 28.3, and standard deviation is 2.5. Future work may decompose the summary into atomic units (like FactScore) before running fine-grained evaluation. However, this decomposition may be hard to automate.

This is a valid concern. We have added a plot of BOOOOKSCORE vs. summary length in Appendix F, which shows no correlation between the two.

Reviewer comment (Experiments):
Table 2: Why is there no entry for LLaMA2 in the incremental update section?

Author response:
This is explained in Section 5, we could not get LLaMA 2 to perform incremental updating.

Reviewer comment (Writing):
Page 7, Section 5, "Incremental..." paragraph - shouldn't "lower" by "higher"?

Author response:
That is correct, we have changed it in the paper.

Reviewer comment (Evaluation):
Page 8, top paragraph: "Highly extractive" seems to contradict the large proportion of novel trigrams, which would be more indicative of an abstractive summary.

Author response:
“Highly extractive” is relative to other models. We have updated the paper with clarification.

Reviewer comment (Evaluation):
The faithfulness evaluation could have been done by asking annotators to assess each fact and search for it inside the book to verify its factuality.

Author response:
We thank the reviewer for proposing a possible experimental setup for evaluating faithfulness. Coincidentally, we did try the task ourselves using the exact setup as described in the reviewer’s comments, and found it incredibly challenging. Some facts may span multiple pages, even chapters, of a book, and verifying a single fact could take 10-20 minutes, let alone a full summary’s worth. Considering the amount of time, money, and energy it would take to complete a full-scale human evaluation of faithfulness, we decided to stick to coherence and acknowledge that our work is complementary in nature. Prior work like LongEval and FactScore have established guidelines and methods for evaluating faithfulness, and our evaluation protocols can always be adapted to or combined with these work for faithfulness evaluation. We focus on coherence as it emerges as a noteworthy problem on its own when it comes to evaluating very long summaries of book-length documents.

Reviewer comment (Experiments):
How did you chose the number of `O` in `BOOOOKSCORE` ?

Author response:
There are 4 O’s in BOOOOKSCORE because we have four authors on the paper!