What Makes Good In-Context Examples for GPT -3? (2022)

Abstract: GPT-3 has attracted lots of attention due to its superior performance across a wide range of NLP tasks, especially with its in-context learning abilities. Despite its success, we found that the empirical results of GPT-3 depend heavily on the choice of in-context examples. In this work, we investigate whether there are more effective strategies for judiciously selecting in-context examples (relative to random sampling) that better leverage GPT-3’s in-context learning capabilities. Inspired by the recent success of leveraging a retrieval module to augment neural networks, we propose to retrieve examples that are semantically-similar to a test query sample to formulate its corresponding prompt. Intuitively, the examples selected with such a strategy may serve as more informative inputs to unleash GPT-3’s power of text generation. We evaluate the proposed approach on several natural language understanding and generation benchmarks, where the retrieval-based prompt selection approach consistently outperforms the random selection baseline. Moreover, it is observed that the sentence encoders fine-tuned on task-related datasets yield even more helpful retrieval results. Notably, significant gains are observed on tasks such as table-to-text generation (44.3% on the ToTTo dataset) and open-domain question answering (45.5% on the NQ dataset).

## experimental setup we apply our proposed method to the following three tasks: sentiment analysis, table-to-text generation, and question answering. dataset split setups and prompt templates are shown in ## conclusion this work presented a first step towards investigating the sensitivity of gpt-3 to in-context examples. to this end, we proposed kate, a non-parametric selection approach that retrieves in-context examples according to their semantic similarity to the test samples. on several natural language understanding and generation tasks, the proposed method improves gpt-3's performance, over the random sampling baseline, by a significant margin. particularly, kate enables gpt-3 to achieve performance comparable to a fine-tuned t5 model on the tableto-text generation task and outperforms t5 on the qa task. moreover, we found that fine-tuning the sentence embeddings for retrieval on task-related datasets gave rise to further empirical gains. detailed analysis was conducted to explore the robustness of kate to different hyperprameters, such as the number of in-context examples, examples' order, etc. one limitation we notice is that despite the improved performance on sentiment analysis, gpt-3 still lags behind the fine-tuned t5 model by a small margin. this suggests that our proposed method is more suitable and effective on long text generation tasks. we hope this work could provide insights for better understanding the behaviors of gpt-3 and represents a helpful step towards further improving its in-context learning capabilities. our work has suggested researchers should focus more on investigating the in-context learning of pretrained models. one potential future direction is for researchers to scale-down the sizes of pre-trained models to find a balance between model performance and model size. once a smaller model is obtained with comparable performance (enhanced by kate), our proposed method can become more widely accessible to individual researchers.