Paper: GoLLIE: Annotation Guidelines improve Zero-Shot Information-Extraction (ICLR-2024)

Reviewer comment (Novelty):
The paper omits key references from the era before LLMs that discuss label descriptions and verbalization. Examples include:  
- Zero-Shot Relation Extraction via Reading Comprehension  
- An Empirical Study on Multiple Information Sources for Zero-Shot Fine-Grained Entity Typing  
- Label Verbalization and Entailment for Effective Zero- and Few-Shot Relation Extraction

Author response:
Regarding the second weakness, we will happily add the missing references (see paper changes). In fact, one of them was already in the paper. Thank you for your suggestions.

Reviewer comment (Writing):
Use ``’’ instead of ‘’’’.

Author response:
Thank you for your recommendations, we will update the paper accordingly.

Reviewer comment (Reproducibility):
The paper doesn't detail the human (or domain expert) effort required to craft the prompts. Given the variety of labels and tasks covered in the study, it would be beneficial to track this for subsequent research.

Author response:
Regarding your second question, the human effort involved was very little in the case of freely available guidelines. We just needed to copy and paste, and if they were very long and detailed (the case of ACE and TACRED for example), simplify. When no guidelines are provided for a given dataset, we either reuse some templates from previous works (for WikiEvents for example) or follow other dataset styles to generate very simple guidelines. The human effort here was also very little. The more important point is that no domain expert was required. We have added a new section in the updated paper (Appendix F) with a more detailed disclosure of the human effort involved to generate the inputs.

Reviewer comment (Novelty):
The paper's novelty appears somewhat limited as similar approaches have been explored in past research. [1] proposes a zero-shot entity typing approach that utilizes the type description available from Wikipedia to build a distributed semantic representation of the types.

Author response:
Thank you for the reference. The way Obeidat et al. (2019) incorporates the label descriptions into the model differs substantially from the current use of LMs. In addition, we believe that our work presents other significant contributions to the field, such as generalization to multiple tasks. We also present an in-depth study of the cases and reasons in which GoLLIE succeeds and fails at following guidelines, which we hope would be of great relevance for future works in the field. We have updated the related work section to add this reference.

Reviewer comment (Experiments):
The paper omits experiments examining the sensitivity of prompts, leading to concerns about the method's stability. The model's performance under varying definitions, code structures, and code comment styles remains unexplored, making its robustness questionable.

Author response:
For the fourth weakness, under this scenario, the prompt variations are very restricted. All the inputs should follow the template illustrated in Figure 2.  However, we do expect the users to test different class definitions, or guidelines styles. The purpose of the evaluation on un-seen tasks is designed to evaluate how the model performs in this scenario. Each dataset used in the zero-shot evaluation has its own definitions written in its own styles, which allows us to indirectly verify the robustness of our method.

Reviewer comment (Experiments):
The paper lacks experiments analyzing the impact of training dataset diversity on the LLM's ability to follow unseen guidelines. It remains unclear how much data is required to achieve satisfactory performance, leaving questions about the scalability and efficiency of the approach.

Author response:
Regarding the last weakness, we plan to explore how the model scales with data in the future. However, the efficiency of the approach is proved, as GoLLIE 7B outperforms 11B Instruct-UIE on zero-shot IE tasks with much less training data: 9 IE datasets on our side vs 1.8k FLAN tasks and 32 IE datasets for Instruct-UIE.

Reviewer comment (Experiments):
Datasets differ in scale, and a naive mixture could introduce imbalance. How did the authors handle the sampling process for the datasets used in this study? Were experiments conducted to assess the effects of different sampling techniques on the results? This aspect wasn't addressed in the paper and could provide clarity on the method's robustness across varied data distributions.

Author response:
* __Sampling strategy:__  The only training dataset that has a size that significantly differs from the others is OntoNotes. This dataset is generated semi-automatically and is orders of magnitude larger than the other ones. Therefore, we sampled 30.000 random examples from the training set. The imbalance on dataset size for the rest of the datasets is small. As it can be seen in the supervised evaluation result, our model does not overfit to a single dataset/task. Therefore, we do not consider it necessary to implement any additional sampling strategies.

Reviewer comment (Experiments):
For datasets with a large number of labels, say in the order of hundreds, the input length for your method could become substantially lengthy, leading to efficiency concerns. In such complex scenarios with multiple labels, how does the inclusion of guidelines impact the performance? This wasn't touched upon in the paper but would provide deeper insights into the method's scalability and effectiveness in real-world applications.

Author response:
* __Scalability with Numerous Labels:__ The inclusion of guidelines in datasets with hundreds of labels will make inputs extremely long, and they would not fit in the context size of current LLMs. This is an already known constraint in LLMs, and recently a lot of research effort is being directed to algorithms that efficiently increase the context window size. We expect that in the future, LLMs will have a context window large enough to fit  not only more labels, but also more detailed guidelines as well. For now, the problem could be solved by batching the labels into multiple input examples. That is, instead of prompting the model with, for example, 100 labels in a single input, it is possible to prompt the model with 10 inputs that incorporate 10 labels each and combine all the outputs into a single one. In any case, we will mention in the paper that this is a limitation for our model. We have added a discussion about this limitation to the updated paper (Appendix E).

Reviewer comment (Experiments):
The authors mentioned employing QLoRA for training, citing its superior performance over fine-tuning the entire model on zero-shot tasks and faster training speed. However, the paper lacks empirical evidence to support this. What are the relative impacts of full-parameter SFT and techniques like LoRA and QLoRA on zero-shot IE performance?

Author response:
* __QLoRa vs Full-Model finetuning:__ we did provide information about our preliminary analysis in Appendix C.3 (also referred to in section 4.2 “Training details”). We have tested multiple hyperparameter configuration for Full-Model finetuning (we have tested the same hyperparameters used by previous work) and none of it was successful. This doesn’t mean that such a configuration doesn’t exist. What we claim in our paper is that LoRA achieves very good performance and requires much less compute resources than Full-Fine Tuning. Which is supported by previous research and demonstrated in Appendix C.3. We will clarify this in the paper.

Reviewer comment (Experiments):
Many of the state-of-the-art methods used for comparison in the paper, such as UIE, appear outdated. Why weren't more recent and potentially stronger methods like USM, InstructUIE, and GPT4 considered for benchmarking?

Author response:
* __Benchmark selection:__ Instruct-UIE is our main baseline (See Section 5.2), which is  the current state-of-the-art Information Extraction model in zero-shot settings. In the Instruct-UIE paper, they already demonstrate that their method is superior to USM, so we didn’t use it in our evaluation (Although we discuss USM in the Related Work Section). Regarding closed-source models such as GPT-4, our objective is to test the hypothesis that training Large Language Models (LLMs) to adhere to annotation guidelines can enhance their zero-shot performance in Information Extraction tasks. Validating this hypothesis involves building a baseline not trained to follow annotation guidelines and performing multiple ablation studies. This is not possible with closed-source models such as GPT-4. Moreover, we know almost nothing from the underlying model(s) powering this commercial product or the data used for training. On top of that, this product is regularly updated, and previous versions are not accessible anymore, therefore, experiments with GPT-4 are not reproducible. Consequently, evaluating GPT-4, or any other closed-source modes, would not add any additional scientific insight in this context of this paper.

Reviewer comment (Experiments):
The motivation behind the code style and its effects are unclear. Note that incorporating the annotation guidelines in a natural language style rather than the code style is also feasible. The LLMs pretrained on codes may lack natural language understanding capabilities that are crucial to conducting IE tasks. This may result in the low performance of the baseline. The advantages may lie in the capability of generating outputs following code grammar, which mitigates the need for parsing outputs. The authors should justify the necessity of the code style and its underlying effects compared to the natural language style.

Author response:
The use of code (more precisely Python syntax) as an underlying format for the input and output was primarily motivated by previous research works [1, 2]. Prompting IE tasks requires some kind of structured format for the input and output. As you mention, we could also have prompted the model with more natural style language. However, we decided to use a format that is familiar for both the human and the model, in order to represent the input-output. Previous work has demonstrated that LLMs trained on code are capable of performing  IE [1, 2]. Furthermore, most LLMs fine-tuned for code generation, including Codellama, are fine-tuned on top of a general purpose LLM and retain most of its natural language understanding capabilities. Additionally, our method outperforms Instruct-UIE [3] on zero-shot, a model prompted with natural language.

That being said, using code-style format is not strictly our contribution; so we did not dig deeper on the matter. It is more of a convenience for easily parsing the input and output, than a decision that contributes to the model performance. We believe that, since fine-tuning is involved, LLMs will easily adapt to any input structure, that being code-style format, natural language instructions or any other alternative. While performing this evaluation would have great interest, we currently lack the human and computer resources for such experiments. 

Regarding the “This may result in the low performance of the baseline.”, both the baseline and GoLLIE share the same conditions, except for the mentioned contributions.

Reviewer comment (Experiments):
The ablation studies of class order shuffling and guideline paraphrasing are missing.

Author response:
In the last update of the paper we have added the missing ablation studies. As expected, the shuffle and paraphrase did not have a significant effect.

Reviewer comment (Experiments):
How many samples are used to supervise finetuning LLaMA?

Author response:
Regarding your question about how many samples we have used to train the model, the models were trained for around 15k steps (3 epochs of our entire dataset with a batch size of 32). Note that for a single dataset a single sample might appear several times as we generate different inputs per task. We have updated the paper to include this information in Appendix D.2.

Reviewer comment (Experiments):
The guidelines used in this paper are specifically the basic definitions of entity/relation/event types, which should have been provided to LLMs. However, the full annotation guidelines usually cover many edge cases and may contain tens of pages. Have the authors considered incorporating more fine-grained guidelines?

Author response:
Regarding your question about the full annotation guidelines, it is something that we definitely want to try. However those guidelines are very long, and considering the amount of labels we have per example, we found them intractable with our current models/infrastructure setup. With the recent efficient methods and models for larger sequences we believe we could soon explore the use of more detailed and longer guidelines in our experiments. However, it is important to note that in the paper we use some of the more basic guidelines to illustrate our method, since we consider that they were the easiest to understand. We have included a new figure (Figure 8, in Appendix E) that shows the guideline for the entity Point in the HarveyNer dataset. This guideline is very descriptive including edge cases and exceptions. All the guidelines we use will be made publicly available and we will add a link to them in the paper.

Reviewer comment (Experiments):
My main concern is as follows. Although the authors try to split the datasets for training and testing based on domains. I believe that there is still a large overlap among their output spaces. For example, RAMS, which is considered as one of the training tasks, contains role label *transporter*, *vehicle*, and *place*, while WikiEvents, which is consider as one of the testing tasks, also contains those role labels. In NER tasks, I believe the same situation happens as well. This makes the setting not *truly* zero-shot because the model has already learned those concepts. The model can get improvements just because including more *in-domain* supervised training signals.

Author response:
About the zero-shot evaluation: In order to compare GoLLIE with previous work, we decided to use as zero-shot datasets the same datasets that have been used for this purpose in previous works, such as a InstructUIE [1], USM [2] or UniNER [3] among others. These methods use an even larger set of pre-training datasets for their models, so more entities in the zero-shot datasets overlap with the ones in the training dataset. Thus, even with less overlap,, GoLLIE outperforms all of them. It is also noteworthy that, while some datasets share similar entities, the annotation guidelines often vary between them. For example, ACE05  annotates pronouns as persons, while, CoNLL03 does not. In any case, we agree with your comment: as we also commented in the paper (first paragraph of Section 5.2), such overlap could make both the baseline and CoLLIE perform better in the evaluation. We believe that understanding the model performance with totally unseen labels is very important. Therefore, we have updated the paper to include an analysis regarding seen and unseen label evaluation in Section 5.2. This analysis is further extended in the Appendix B. As a short summary, the baseline model achieves significantly lower F1 scores on unseen labels than on seen labels. In contrast, GoLLIE demonstrates a more robust performance, showing a smaller gap in F1 scores between the seen and unseen labels.