Paper: PromptNER : Prompting For FewShot Named Entity Recognition (ICLR-2024)

Reviewer comment (Novelty):
Concerning the introduced PromptNER methodology, I didn't discern any noteworthy technical novelty. In essence, the idea that the paper alludes to strikes me as rather naive. Almost every LLM-based NER initiative could effortlessly incorporate the strategy of embedding entity definitions within prompt texts, facilitating the LLM's comprehension. In a nutshell, I'm unconvinced that this paper heralds any significant fresh technological insights.

Author response:
We thank the reviewer for sharing their comments and concerns. We believe that the simplicity of the PromptNER approach is something that enables it to perform well in diverse scenarios without the need for much modification. To the best of our knowledge, we are the first to present a system that uses a prompting template to perform end-to-end Named Entity Recognition in a FewShot setting without also requiring some form of prefix tuning or parameter updates. It is certainly true that the improvements of PromptNER come from using more powerful LLMs like GPT4. The scaling experiments also confirm that this improvement does not occur when using smaller models. However, in exchange for requiring a much larger model, this method now requires significantly less NER data that matches the target distribution. As for the rationale for the selecting of evaluation benchmarks and methods, we made an effort to test a wide range of 6 different NER benchmarks (ConLL, Genia, FewNERD, CrossNER, FabNER, TweetNER), in an effort to get standard FewShot NER benchmarks (ConLL), NER benchmarks in specialized domains (Genia, CrossNER, FabNER, TweetNER) as well as NER benchmarks which have contradictory information between train and test domains (FewNERD), we compare against 3-7 other methods from the past 2 years and make sure we include the method that reports the highest performance on each dataset. For the FabNER and TweetNER tasks, we selected the best methods from the ConLL dataset results to manually compute the results.

Reviewer comment (Experiments):
There's a palpable omission of numerous standard few-shot NER benchmarks, such as WNUT-2017, MIT-Movie and MIT-Restaurant, JNLPBA, among others. Moreover, there's a conspicuous absence of comparative analysis with many of the state-of-the-art few-shot NER models.

Author response:
We thank the reviewer for sharing their comments and concerns. We believe that the simplicity of the PromptNER approach is something that enables it to perform well in diverse scenarios without the need for much modification. To the best of our knowledge, we are the first to present a system that uses a prompting template to perform end-to-end Named Entity Recognition in a FewShot setting without also requiring some form of prefix tuning or parameter updates. It is certainly true that the improvements of PromptNER come from using more powerful LLMs like GPT4. The scaling experiments also confirm that this improvement does not occur when using smaller models. However, in exchange for requiring a much larger model, this method now requires significantly less NER data that matches the target distribution. As for the rationale for the selecting of evaluation benchmarks and methods, we made an effort to test a wide range of 6 different NER benchmarks (ConLL, Genia, FewNERD, CrossNER, FabNER, TweetNER), in an effort to get standard FewShot NER benchmarks (ConLL), NER benchmarks in specialized domains (Genia, CrossNER, FabNER, TweetNER) as well as NER benchmarks which have contradictory information between train and test domains (FewNERD), we compare against 3-7 other methods from the past 2 years and make sure we include the method that reports the highest performance on each dataset. For the FabNER and TweetNER tasks, we selected the best methods from the ConLL dataset results to manually compute the results.

Reviewer comment (Novelty):
Given the stated weaknesses, how do the authors justify the claimed novelty of the PromptNER methodology, especially in light of its evident simplicity and seemingly naive approach?

Author response:
We thank the reviewer for sharing their comments and concerns. We believe that the simplicity of the PromptNER approach is something that enables it to perform well in diverse scenarios without the need for much modification. To the best of our knowledge, we are the first to present a system that uses a prompting template to perform end-to-end Named Entity Recognition in a FewShot setting without also requiring some form of prefix tuning or parameter updates. It is certainly true that the improvements of PromptNER come from using more powerful LLMs like GPT4. The scaling experiments also confirm that this improvement does not occur when using smaller models. However, in exchange for requiring a much larger model, this method now requires significantly less NER data that matches the target distribution. As for the rationale for the selecting of evaluation benchmarks and methods, we made an effort to test a wide range of 6 different NER benchmarks (ConLL, Genia, FewNERD, CrossNER, FabNER, TweetNER), in an effort to get standard FewShot NER benchmarks (ConLL), NER benchmarks in specialized domains (Genia, CrossNER, FabNER, TweetNER) as well as NER benchmarks which have contradictory information between train and test domains (FewNERD), we compare against 3-7 other methods from the past 2 years and make sure we include the method that reports the highest performance on each dataset. For the FabNER and TweetNER tasks, we selected the best methods from the ConLL dataset results to manually compute the results.

Reviewer comment (Experiments):
For the Cross Domain NER tasks, as Table 2 shows, only 2 examples are used in prompting and the F1 scores for AI and Sciences are not the highest. Why can't we add more examples (e.g. up to 200) to improve performance?

Author response:
We thank the reviewer for their comments. To address their questions, we were limited by the context window length and so could place only a few examples in the prompt. We did not explore the idea of fine-tuning GPT, this is because we wanted to keep the method a FewShot one that uses fewer than 5 examples.

Reviewer comment (Experiments):
Did we also consider fine-tune GPT to see if we can get the higher F1 scores?

Author response:
We thank the reviewer for their comments. To address their questions, we were limited by the context window length and so could place only a few examples in the prompt. We did not explore the idea of fine-tuning GPT, this is because we wanted to keep the method a FewShot one that uses fewer than 5 examples.