Paper: Disco-Bench: A Context-Aware Evaluation Benchmark for Language Modelling (ICLR-2024)

Reviewer comment (Experiments):
For almost every dataset, there is an existing dataset on that task and this paper proposes a new version of the task, without providing the advantages of their new task or enough details to compare them. That issue is particularly fraught for ZPR, where the benchmark proposes to replace a manually annotated dataset with an automatically generated one. The same issue arises with a lack of reasons provided for using this dataset in comparison to existing long-form Chinese benchmark (Guan et al. 2022).

Author response:
1. **About replacing a manually annotated dataset with an automatically generated one.** First, the newly introduced 8.1K testing set remains manually annotated and spans five distinct text domains  (but Zhang et al., 2019b only contains 1K single domain testing set). Second, we introduced an alternative, automatically generated training set, which significantly addresses the issue of long-standing data scarcity (but Zhang et al., 2019b only contains 5K human-annotated training set). Such kind of auto-annotated ZP training datasets have been thoroughly validated across various tasks, including ZPR, MT, question answering, and summarization tasks [1-3]. For future work, there is the flexibility to opt for manually annotated, automatically annotated, or a hybrid dataset for fine-tuning. Our approach simply offers more options in this regard.

Reviewer comment (Experiments):
A number of these datasets dismiss relevant smaller high-quality datasets in exchange for automatically generated tasks, particularly in the context of the ZPR data. But there's simply no real experimental analysis that actually establishes the validity of making those judgement calls, or the quality of their automatically bootstrapped datasets.

Author response:
1. **About replacing a manually annotated dataset with an automatically generated one.** First, the newly introduced 8.1K testing set remains manually annotated and spans five distinct text domains  (but Zhang et al., 2019b only contains 1K single domain testing set). Second, we introduced an alternative, automatically generated training set, which significantly addresses the issue of long-standing data scarcity (but Zhang et al., 2019b only contains 5K human-annotated training set). Such kind of auto-annotated ZP training datasets have been thoroughly validated across various tasks, including ZPR, MT, question answering, and summarization tasks [1-3]. For future work, there is the flexibility to opt for manually annotated, automatically annotated, or a hybrid dataset for fine-tuning. Our approach simply offers more options in this regard.

Reviewer comment (Evaluation):
I appreciate that the IAA scores are provided for ZP recovery, but I'm confused about the numbers: the ZPR has kappa in the 0.9+ (implying a very clear-cut task), but the best model performance of 34.3 in this data -- compared to 46.81 F1 model performance for other ZP resolution models from years ago, which are presumably the same task (Song et al. (2020) -- https://aclanthology.org/2020.acl-main.482.pdf). Do the authors have a suggestion for the reason for the dramatic difference in behavior across different Chinese ZP recovery tasks? Is the domain chosen particularly hard?

Author response:
2. **High IAA scores yet low model performance & different scores in Song et al. (2020).** Professionally trained annotators (we employed university professors specializing in linguistics and translation studies), having received thorough annotation training, are indeed capable of achieving a high level of consistency in their annotations. While such omissions do not pose a problem for humans, as they can easily recall missing pronouns from the context, the task remains a significant challenge for models. This discrepancy is precisely why we focus on this task in the benchmark. Song et al. (2020) only use BaiduZhidao QA forum training and testing set (Zhang et al., 2019b), however, we report socres averaged on large five domains (movie subtitle, QA forum, government news, web fiction, and personal profile).

Reviewer comment (Presentation):
What are the stars to the left of each figure?

Author response:
3. **Difficulty stars in Figure 1~3.** As stated in figure captions and first graph of each subsetion in Section 2, the stars denote difficulty levels of different tasks. These levels are determined based on key factors: 1) the length of contexts required to find cues, and 2) the extent of linguistic knowledge needed for reasoning. We acknowledge that, according to current automatic evaluation methods, model performance on ZPR tasks is generally lower than on MRC tasks. To avoid overclaim, in the revised version, we could present the issue of difficulty levels as a more open question for further exploration and discussion.

Reviewer comment (Writing):
Similarly, they claim the NT task contains "richer linguistic phenomena, especially in discourse.", but again provide no detail: could they explain what they mean?

Author response:
The perspective that literary texts encompass more complex linguistic and cultural knowledge compared to non-literary ones has been substantiated both qualitatively and quantitatively in previous studies [1-2]. This is the primary reason for our decision to construct 9 benchmark tasks focused on the literary domain.

[1] Towards a literary machine translation: The role of referential cohesion. Rob et al. NAACL 2012. 

[2] Neural poetry translation. Marjan et al. NAACL 2018.

Reviewer comment (Reproducibility):
The authors claim that their "Novel Translation" dataset took 5,134 chapters in 152 books (Table 3 suggests it's 1.4 million units -- I assume sentences?) and they manually aligned it. Does that mean that the authors did millions of alignment annotations? If not, could they clarify their methodology?

Author response:
Yes, tthis dataset comprises 1.4 million sentence pairs, sourced from 5,134 chapters across 152 books.. A significant financial investment was made to annotate this dataset thoroughly. We will make this data available to promote the advancement of research in this field. For details regarding copyright issues, please refer to the response to Reviewer iRfK Q1. For information on our annotation methods, see the response to Reviewer maWf Q3.

Reviewer comment (Evaluation):
For pre-trained models evaluation, the benchmark mixes up the Chinese language capabilities of the models under evaluation and the capabilities of such models with discourse phenomena. A model that excels at handling discourse phenomena but is limited with Chinese (e.g., due to lack of Chinese pre-training data) will still perform quite poorly on all tasks of the benchmark.

Author response:
**Q1. The benchmark mixes up the Chinese language capabilities.**

We acknowledge the valid concerns you have raised regarding our dataset design. Our considerations in constructing this dataset are based on the following aspects:
1. **Universality of Discourse Knowledge**: Discourse knowledge often shares commonalities across various languages. This implies that merely increasing the data volume for a single language does not necessarily enhance the model's ability to process discourse in that specific language. Instead, the understanding and handling of discourse are largely cross-linguistic. Therefore, our dataset is designed to assess the model's general capability in processing discourse-level information, rather than being limited to a particular language.
2. **Importance of Cross-Lingual Capabilities**: In the era of large language models (LLMs), cross-lingual capabilities become increasingly crucial, especially for languages with rich resources like Chinese and English. Our dataset aims to evaluate and foster the development of such abilities, which are essential for building more universally applicable and adaptable LLMs.
3. **Diversity of the Dataset**: Both Evaluation Benchmark and Diagnostic Test are designed to encompass both Chinese and English instead of Chinese only. 

We believe these considerations are crucial for advancing the field and appreciate the opportunity to clarify our approach.

Reviewer comment (Evaluation):
The paper does not provide any qualitative analysis that confirms the conclusions of the comparison results. Also, some of the results in Table 5 need to be explained or at least justified with a reasonable intuition. For example, why does Disco-Bench Pretraining significantly hurst ellipsis handling in the TC task? Also, Disco-Bench pretraining hurts most phenomena in the MRC task. The paper justifies that with "understanding tasks are mostly classification tasks, whose signals may not be sufficient to guide models to learn discourse information" which does not make sense since the continual pre-training phase is independent of the task!

Author response:
**Q2. About qualitative analysis.**

Due to page limitation, we did not include a detailed qualitative analysis. However, we recognize the importance of this aspect and will add details in the revised version. For example,
1. **Why does Disco-Bench Pretraining significantly hurst ellipsis in TC.** It has been observed that the finetuned model may develop a preference for generating complete sentences with no grammatical omissions, rather than preserving ellipses that require reader inference. This finetuning bias towards producing syntactically complete structures could potentially impact the model's ability to effectively handle ellipsis phenomena.
2. **Disco-Bench pretraining hurts Con., Ref., Sub. phenomena in MRC.** In MRC task, such phenomena have a more direct impact on understanding than conjunction, reference, and substitution. For instance, the use of repetition and synonyms can more directly aid the model in capturing the main themes and key information of a text, while ellipsis might play a crucial role in understanding the implied meanings within the text. In the finetuning phase, the primary goal is to optimize the model's performance for a specific task. Consequently, the model may adjust its focus to better align with the demands of the task.

Reviewer comment (Experiments):
In section 2.1, how was the ground truth constructed for the Speaker Identification task?

Author response:
**Q3. About more details.**

1. **We will add all annotation details in Appendix in the revised version.** Please see Reviewer iRfK Q2  for general annotation process. Specifically, we employed a hybrid method of automatic annotation and manual verification for both the Speaker Identification Task and the Alignment in Novel Translation Task. Specifically, 1) SI, an automated process was used to extract all entities and pronouns from the text based on the context. These extracted elements were then presented to human annotators who selected the correct speaker for each dialogue segment. 2) NT, an automatic alignment tool was initially used to roughly align sentences. This preliminary alignment was subsequently refined through manual corrections.

Reviewer comment (Experiments):
In section 2.2., how were 45k chapters manually aligned at both the document and sentence-level? What is the definition of a document in the Novel translation task?

Author response:
**Q3. About more details.**

1. **We will add all annotation details in Appendix in the revised version.** Please see Reviewer iRfK Q2  for general annotation process. Specifically, we employed a hybrid method of automatic annotation and manual verification for both the Speaker Identification Task and the Alignment in Novel Translation Task. Specifically, 1) SI, an automated process was used to extract all entities and pronouns from the text based on the context. These extracted elements were then presented to human annotators who selected the correct speaker for each dialogue segment. 2) NT, an automatic alignment tool was initially used to roughly align sentences. This preliminary alignment was subsequently refined through manual corrections.

Reviewer comment (Reproducibility):
In Table 4, how were the English-only models (e.g. BERT and RoBERTa) fine-tuned for the tasks in Chinese?

Author response:
**Q3. About more details.**

2. **How were the English-only models fine-tuned for the tasks in Chinese?** We have listed all details of pretrained in Appendix A.3. For example, we used BERT-Chinese developped by Hugginface team [1] and chinese-roberta-wwm-ext by Cui et al. [2]. 

[1] https://huggingface.co/bert-base-chinese. 

[2] https://huggingface.co/hfl/chinese-roberta-wwm-ext/tree/main.

Reviewer comment (Reproducibility):
The paper does not discuss copyright and ethical issues for any of the datasets. The source descriptions are also rather vague (e.g., "we crawl 45,134 chapters in 152 books from web fiction websites" needs more explicit statements of these websites – perhaps in appendix/supplementary material).

Author response:
**Q1. About copyright and ethical issues for any of the datasets.**

Our datasets consist of three types of sources: novel/book, dianji/poetry, composition. We will add the details in appendix:

1. **novel/book**: we are the copyright owners of these web fictions and books. We make this data available to promote the advancement of research in this field. 
2. **dianji/poetry**: dianji refers to the Chinese texts which originated before the imperial unification by 221 BC [1] and poetry mainly includes Shi, Ci, Qu, Fu created by 1300 AD [2]. Thus, these are public domain works which are beyond the copyright protection period.
3. **composition**: compositions are derived from the reading comprehension section of the Chinese National College Entrance Examination. According to local copyright law, such examination content belongs to "documents of an administrative nature," and therefore is not subject to copyright protection.

[1] https://en.wikipedia.org/wiki/Chinese_classics

[2] https://en.wikipedia.org/wiki/Classical_Chinese_poetry

Reviewer comment (Reproducibility):
Although some quality assurance (IAA/checks) are reported (Table 2), there are unclarities: Were the SI/ZPR/MRC data fully doubly annotated? Who were the annotators? Were there any annotation guidelines, well-defined procedures?

Author response:
**Q2. About annotation details of the SI/ZPR/MRC/Diagnostic Test.**

The common anntation process for all datasets is: 

A one-week trial annotation phase was conducted for bidding on an enterprise-level annotation platform. The authors answered questions posted by the annotators of these companies and updated annotation guidelines accordingly. The Q&A history is recorded and updated in the formal annotation phase. After evaluating the trial annotations based on accuracy and consistency, we selected the best candidate company with their professional annotators. We ensured that none of the annotators had conflicts of interest, and their annotations were routinely cross-checked for consistency. In terms of compensation, annotators received a fair pay, which aligns closely with the mean hourly wages in U.S.. 

Some details such as annotator reuqirement varied from nature of different dataset:
1. **SI/MRC**: we engaged native Chinese speakers who possess at least a Bachelor's degree. They should have a strong ability of the language  proficiency, contextual understanding and analytical skills.
2. **ZPR/diagnostic test**: We employ experts in both source and target languages with Ph.D. degrees in translation and linguistics. These background is essential for accurately labelling nuanced linguistic elements.
Besides, an exhaustive IRB review was undertaken and finalized before the project's onset. We will add above information and detailed background of each annotator in appendix.

Reviewer comment (Experiments):
Related to the above, the description of the "diagnostic test data" is also rather terse, and insufficient.

Author response:
**Q2. About annotation details of the SI/ZPR/MRC/Diagnostic Test.**

The common anntation process for all datasets is: 

A one-week trial annotation phase was conducted for bidding on an enterprise-level annotation platform. The authors answered questions posted by the annotators of these companies and updated annotation guidelines accordingly. The Q&A history is recorded and updated in the formal annotation phase. After evaluating the trial annotations based on accuracy and consistency, we selected the best candidate company with their professional annotators. We ensured that none of the annotators had conflicts of interest, and their annotations were routinely cross-checked for consistency. In terms of compensation, annotators received a fair pay, which aligns closely with the mean hourly wages in U.S.. 

Some details such as annotator reuqirement varied from nature of different dataset:
1. **SI/MRC**: we engaged native Chinese speakers who possess at least a Bachelor's degree. They should have a strong ability of the language  proficiency, contextual understanding and analytical skills.
2. **ZPR/diagnostic test**: We employ experts in both source and target languages with Ph.D. degrees in translation and linguistics. These background is essential for accurately labelling nuanced linguistic elements.
Besides, an exhaustive IRB review was undertaken and finalized before the project's onset. We will add above information and detailed background of each annotator in appendix.

Reviewer comment (Writing):
Typo/language issue (abstract): "We totally evaluate" → "We evaluate".

Author response:
**Q3. About typo/reference issues.**

Accordingly, all typos and missing references will be corrected/citated in our revised version.

Reviewer comment (Writing):
Typo/language issue (Fig. 1 caption): "propertie" → "property".

Author response:
**Q3. About typo/reference issues.**

Accordingly, all typos and missing references will be corrected/citated in our revised version.

Reviewer comment (Presentation):
Table 1: better use full names of the tasks in column 1; there seems to be enough space.

Author response:
**Q3. About typo/reference issues.**

Accordingly, all typos and missing references will be corrected/citated in our revised version.

Reviewer comment (Writing):
Page 3 paragraph on ZPR: ", while ZPR considers ..." → ", ZPR considers ...".

Author response:
**Q3. About typo/reference issues.**

Accordingly, all typos and missing references will be corrected/citated in our revised version.

Reviewer comment (Reproducibility):
For corpora and tools referred to everywhere, prefer citing the papers describing them, rather than providing (non-permanent) URL references.

Author response:
**Q3. About typo/reference issues.**

Accordingly, all typos and missing references will be corrected/citated in our revised version.

Reviewer comment (Writing):
Page 6, first paragraph of section 3: "may be not sufficient" → "may not be sufficient".

Author response:
**Q3. About typo/reference issues.**

Accordingly, all typos and missing references will be corrected/citated in our revised version.

Reviewer comment (Writing):
Table 3: "Incorrect: She think the Qingshuang is funny." has a likely typo in the example: "think" should be "thinks". However, this may also point to a systematic error in the corpus. If so, it needs to be corrected.

Author response:
**Q3. About typo/reference issues.**

Accordingly, all typos and missing references will be corrected/citated in our revised version.