Paper: UniversalNER: Targeted Distillation from Large Language Models for Open Named Entity Recognition (ICLR-2024)

Reviewer comment (Experiments):
It's not clear why ChatGPT is doing worse even though you are distilling from it. You write: for "ChatGPT (gpt-3.5-turbo-0301). We use the prompting template in Ye et al. (2023) for NER." For fairness, it also makes sense to use your UniversalNER prompting template with ChatGPT. Otherwise we don't know what the cause is for you beating ChatGPT while distilling from it. Please add this experiment.

Author response:
### 1. Comparison with ChatGPT's Performance
In knowledge distillation, it is not uncommon that the student model may outperform the teacher model, given that the teacher model is more general, whereas the student model just needs to focus on the specific tasks it is fine-tuned for (open NER in this case). While individual self-supervised examples may be noisy, with a large number of them, the student model may learn about the underlying cleaner patterns. 

For instance, consider a recent paper on knowledge distillation for drug adverse event extraction, where the student model surpasses the GPT3.5 teacher by over 6 absolute points and nearly matches the supervised performance:

https://arxiv.org/abs/2307.06439  
Distilling Large Language Models for Biomedical Knowledge Extraction: A Case Study on Adverse Drug Events.

Re *using the UniversalNER prompting template with ChatGPT*: We experimented with various prompting techniques for ChatGPT, selecting the most effective one to attain the strongest results. We did try the UniversalNER prompting template on ChatGPT, but it didn't perform as well, resulting in a drop of more than 4 absolute points in F1 compared to the best prompt. We ultimately used the prompt template from [Ye et al., 2023](https://arxiv.org/abs/2303.10420), which yielded the strongest ChatGPT baseline among all prompt templates we compared.

Reviewer comment (Experiments):
The section "Recognition of diverse entity types" does not "effectively demonstrate our model’s capacity to capture entities of various types." It's just a single picked example. Please consider doing a manual evaluation on 100 samples to actually demonstrate what you claim here.

Author response:
### 3. Recognition of diverse entity types
Thank you for your suggestion. We conducted a manual evaluation on 100 samples, which indeed confirms the UniversalNER model's ability to recognize diverse entity types across different domains. 

We uploaded these 100 samples to an [anonymous folder](https://drive.google.com/file/d/1ONV_2Ih4JdFgHAE-C2pqCJ30qRRw15ai/view?usp=drive_link) and also provided five samples in a separate [response](https://openreview.net/forum?id=r65xfUb76p&noteId=gYXa3fR4ex) (due to the space limit) for a quick demonstration.

Reviewer comment (Writing):
In section 3.2, you write "Then, for each entity type that appears in the output, we transform it into a natural language query “What describes it?” - To be clear, here you don't ask the model to output the entity types, but you did in 3.1?

Author response:
### 1. Question on Entity Type Transformation:
In Section 3.1 of our paper, our goal is to construct a diverse dataset comprising various entity types. Given that these types are identified by ChatGPT and are not known in advance, it is crucial for ChatGPT to output both the entities and their corresponding types. Conversely, in Section 3.2, the focus shifts to instruction tuning using the dataset created in Section 3.1. At this stage, as the entity types are already known and explicitly included in the query (as in "*What describes $t$* in the text??", where $t$ represents the entity type), there is no need for the model to output the entity type again. The model's task here is to respond to the query with appropriate entities related to the specified type, not to identify or output the type itself. This distinction in the model's output requirements between Sections 3.1 and 3.2 is an intentional design choice that aligns with the specific goals of each stage of our methodology.

Reviewer comment (Experiments):
In sec 4, Dataset processing you write "This is because some entity types (e.g., ELSE) are not coming from consistent sources across the different datasets." Shouldn't this be addressed by the "Dataset-specific Instruct Tuning Template"?

Author response:
### 2. Question on ELSE:
Thanks for the question. The issue with ELSE and other excluded types is that they serve as catch-all types in some datasets and have no clear definition. ChatGPT won’t be able to create appropriate examples for them due to the lack of definition of these ambiguous types. Note that even if we add the dataset name, ChatGPT still won’t be able to generate appropriate examples, as it’s not privy to the idiosyncrasies of the given dataset.

The dataset-specific instruction-tuning template just adds the dataset name in the prompt, which proves beneficial when supervised examples are available. This enables the model to discern fine-grained differences for the same entity type across various datasets. In the supervised setting (e.g., Table 2), we did include ELSE types. However, this won’t help the zero-shot setting, when we are using ChatGPT to generate labels.

Reviewer comment (Experiments):
In figure 6, the biggest gains are for the most famous datasets. Can you try to check if these datasets have somehow been included in LLaMA's pre-training dataset?

Author response:
### 3. Question on Figure 6
 The main purpose of Figure 6 is to demonstrate the impact of employing dataset-specific instruction tuning templates on the performance, particularly in relation to label overlap. In this figure, we compare two variants of our UniNER model, both derived from LLaMA and trained on the exact same datasets. The only difference between these two models lies in using dataset-specific templates or not. Our analysis, as illustrated in Figure 6, suggests that the substantial improvement in performance for well-known datasets can be attributed to the effective reduction of label overlap through the application of dataset-specific templates. We believe that the observed improvements are not indicative of data contamination.

Reviewer comment (Experiments):
Lack of some supervised baselines such as [a]. The “BERT” column in Table 2 did not represent the SoTA supervised baselines. Even so, the presented results in Table 2 show that the UniNER with such a large scale of parameters did not improve significantly and are not comparable with supervised small models in some dataset. Therefore, it is doubtful to use prompt-based method with LLM to solve NER problems. Moreover, the training cost are not presented explicitly. The motivation should be clarified carefully.

Author response:
### 1. Re Supervised Baselines
Thank you for suggesting an additional supervised baseline. We will include it and add discussion in the final version. However, we do want to point out that the suggested baseline from ([Liu et al. 2022](https://arxiv.org/abs/2210.14698)) was *only evaluated on CoNLL and ACE-05, which are classic closed NER datasets*. In contrast, our evaluation in Table 2 is based on a much broader collection of datasets introduced by the latest InstructUIE paper ([Wang et al. 2023a](https://arxiv.org/abs/2304.08085)), which we compared against and demonstrated a substantial performance gain.

In sum, the supervised results in the suggested [Liu et al. 2022](https://arxiv.org/abs/2210.14698) paper were attained on much easier datasets with substantially narrower scope. They are not comparable to Table 2, which reported results on much more challenging datasets. UniNER attains the state-of-the-art accuracy and represents a large step forward compared to prior state of the art InstructUIE.

In general, it’s also important to emphasize that the true power for UniNER lies in **open** NER in the **zero-shot setting**, where no manual annotation or supervised labels are available. E.g., see Figure 5. This open NER setting is an important research frontier, as many real-world scenarios require recognition capabilities for a large number of entity types, for which supervised learning simply can not scale. E.g., in biomedicine, there are tens of thousands of diagnostic types alone, not to mention other medical concepts.

There may exist supervised baselines for a tiny fraction of the entity types, where there are prior efforts in annotating labeled data for supervised learning. However, for the vast majority of entity types in real-world applications, it’s simply infeasible to train supervised NER systems covering all of them, which would incur prohibitive cost in annotating sufficient number of labeled examples for each entity type. We agree that UniversalNER is far from perfect for many entity types, but as a zero-shot open NER system, it attains the state-of-the-art performance, and represents a big leap forward compared to existing generalist NER systems.

Similarly, it’s important to note that any training cost in data generation and distillation is dwarfed by the efforts to train supervised NER systems for all possible entity types.

Reviewer comment (Reproducibility):
The proposed method could be very unstable as the dataset construction includes several uncontrol factors. Thus the reproduction of the results could be problematic. I suggest the authors should claim the related limitations clearly.

Author response:
### 2. Re Instability
To address potential instability in our data construction process, we set the `temperature` parameter to 0 during the data generation phase with GPT-3.5-turbo. Our initial exploration confirmed that this adjustment (i.e., `temp=0`) ensured consistent and reproducible outputs for our dataset. This approach mitigates the variability often associated with generative models and enhances the stability of our dataset, providing a solid foundation for our experiments.

We understand the importance of reproducibility in scientific research and are committed to facilitating it for our work. To this end, we plan to release the pretraining data and the necessary scripts for reproducing pretraining and inference, enabling other researchers to independently verify the effectiveness of UniversalNER. Additionally, our repeated re-training of the model has consistently reproduced the reported results. We believe these measures effectively address the reviewer's concerns regarding stability and reproducibility, and we hope this clarification demonstrates our commitment to maintaining the highest standards of scientific rigor in our work.

Reviewer comment (Reproducibility):
The collecting of the passage-query pairs of instruction tuning is not detailed enough for reproduction.

Author response:
### 3. Re Collection of Passage-Query Pairs
Regarding the question about the collection of passage-query pairs, we followed a standard practice to ensure a balanced representation of datasets. Specifically, for each dataset containing more than 200,000 passage-query pairs, we subsampled 200,000 pairs to include in the instruction-tuning set. This approach prevents larger datasets from dominating the learning process. We will release our subsampled datasets to facilitate reproduction and ensure transparency.