Exploring the Curious Case of Code Prompts (2023)

Abstract: Recent work has shown that prompting language models with code-like representations of natural language leads to performance improvements on structured reasoning tasks. However, such tasks comprise only a small subset of all natural language tasks. In our work, we seek to answer whether or not code-prompting is the preferred way of interacting with language models in general. We compare code and text prompts across three popular GPT models (davinci, code-davinci-002, and text-davinci-002) on a broader selection of tasks (e.g., QA, sentiment, summarization) and find that with few exceptions, code prompts do not consistently outperform text prompts. Furthermore, we show that the style of code prompt has a large effect on performance for some (but not all) tasks and that fine-tuning on text instructions leads to better relative performance of code prompts.

## introduction recent work has shown that pre-training language models (lms) on a mixture of text and program code (e.g., python or javascript) makes them more capable of reasoning over natural language @xcite . such program-trained language models (plms) significantly outperform text-only lms on tasks such as math problems and tracking shuffled objects despite such tasks lacking any explicit code formulae @xcite . furthermore, prompting such plms with codelike structures (e.g., python, json, pddl) instead of text has been shown to lead to performance improvements on structured common sense reasoning @xcite , event argument extraction @xcite , knowledge graph construction @xcite , story understanding @xcite , and causal reasoning @xcite . ## results what is the best type of code prompt? we compare performance across the four code prompt types from section 2 on all 12 tasks using code-davinci-002 and report our results in figure 3 . we find that no single type of code prompt performs significantly better than the others across all tasks and that the relative difference in performance between code prompts also varies significantly across tasks. for example, on imdb and squad all code prompts have roughly even performance while for tasks such as wikihow-temporal and winogrande we see a near 14% accuracy dif-you are trying to {goal}. you need to do two things: (a) {step0} (b) {step1} the first thing to do is {first} @xmath0, predict the correct order to do the steps to achieve the goal" @xmath1, predict the correct order to do the steps to achieve the goal" @xmath2 ## conclusion in this work we investigate whether or not there exists a systematic performance difference between prompting plms with code or with text. we confirm that there are indeed tasks for which code prompting is significantly more effective than text prompting and that this finding holds across different types of models. however, for most tasks, we find that text prompting is still the best method for eliciting few-shot generalization from plms. given this result it seems reasonable to attempt to predict which tasks will benefit from code prompts and which tasks will not. however, we show that making such predictions based on simple heuristics such as domain and task category is difficult and that the larger trends remain unclear. future work should seek to investigate the core mechanism behind what makes code prompting effective for certain tasks. finally, concurrent to our work, a new line of research has emerged wherein models generate code and execute that code to produce valid output @xcite @xcite . future work should consider whether or not the tasks that benefit from executable code prompts and non-executable code prompts have any overlap.