Prompting with Pseudo-Code Instructions (2023)

Abstract: Prompting with natural language instructions has recently emerged as a popular method of harnessing the capabilities of large language models (LLM). Given the inherent ambiguity present in natural language, it is intuitive to consider the possible advantages of prompting with less ambiguous prompt styles, like pseudo-code. In this paper, we explore if prompting via pseudo-code instructions helps improve the performance of pre-trained language models. We manually create a dataset of pseudo-code prompts for 132 different tasks spanning classification, QA, and generative language tasks, sourced from the Super-NaturalInstructions dataset. Using these prompts along with their counterparts in natural language, we study their performance on two LLM families - BLOOM, CodeGen. Our experiments show that using pseudo-code instructions leads to better results, with an average increase (absolute) of 7-16 points in F1 scores for classification tasks and an improvement (relative) of 12-38% in aggregate ROUGE-L scores across all tasks. We include detailed ablation studies which indicate that code comments, docstrings, and the structural clues encoded in pseudo-code all contribute towards the improvement in performance. To the best of our knowledge, our work is the first to demonstrate how pseudo-code prompts can be helpful in improving the performance of pre-trained LMs.

## introduction prompting with natural language instructions has recently emerged as a popular method of harnessing the capabilities of large language models. in addition to fine-tuning, models are often fine-tuned using instructions on a large collection of datasets listing 1 an example pseudo-code instruction for the task from @xcite . a successful model is expected to use the provided pseudo-code instructions and output responses to a pool of evaluation instances. 1 def generate_sentiment(sentence: str) -> str: 2 """for the given sentence, the task is to 3 predict the sentiment. for positive 4 sentiment return "positive" else return 5 "negative". to help improve the ability of lms to follow instructions and performance on unseen tasks @xcite . however, natural language instructions can be ambiguous and under-specified, and therefore have multiple interpretations -including detailed instructions may not always be beneficial, as it can add to the complexity of reasoning for models. this has led to the growing body of work around 'prompt-engineering' where specialized prompting strategies are developed for different domains and task types @xcite @xcite @xcite . in addition, inference-time prompting strategies that specifically aid multi-step reasoning have also been found to be helpful -e.g: the inclusion of chainof-thought reasoning in few-shot settings results in improved performance over standard prompts @xcite , the infamous "let's think stepby-step"-prompt for boosting 0-shot performance @xcite . input: q, k, and v : input matrices. ## related work finetuning large language models on instruction datasets can enhance their performance and even their ability to generalize to unseen tasks @xcite . many aspects of instruction finetuning such as the number of tasks, model size, and finetuning on chain-of-thought data have been found to be useful @xcite . consequently, significant efforts have been invested in manually creating instruction datasets, as well as using existing generative models to train and evaluate language models @xcite @xcite . the instructions available in instruction tuning datasets are mostly in natural language, but have been applied for both natural language tasks and programming tasks. but alternatives to natural language instructions such as programming language code, pseudo-code, symbols (maccartney and manning, 2007) etc. have not been thoroughly explored even for programming tasks. compared to natural language, code or pseudo-code has less ambiguity due to its inherent nature of using functions or steps that contribute towards accomplishing a task. this makes them a natural choice for specifying instructions. recently, few works (marvinai; @xcite have explored code and pseudocode as inputs. unlike contemporaneous work by @xcite we find that pseudo-code instructions indeed provide better performance over nl instructions on a wide variety of tasks. ## dataset the super-naturalinstructions dataset @xcite comprises 1, 616 diverse nlp tasks, and each task contains the task instruction, positive/negative examples, and instances. we sampled a mixture of 132 tasks that did not require multilingual capabilities and re-wrote instructions for a subset of this dataset using python constructs. note that we borrow python constructs only to express our prompts in pseudo-code and our prompts do not result in executable python code. further, we do not include any additional steps/instructions that were not present in the original natural language instructions. all task instructions follow the schema as described in listing 1. the schema consists of the following elements. function prototype: this defines the prototype of the main pseudo-code function. the function names are descriptive and summarize the task to be performed. they also include all variables passed as input along with their data types and return type. we follow the pep 8 foot_2 style guidelines for writing the pseudo-code and use strongly typed prototypes. we avoid declaring global variables whenever possible and pass them as arguments to a method. to the extent possible, we also avoid the use of classes and enumerations. line number 1 in listing 1 provides an example function prototype for a sentiment classification task. ## evaluation in order to study if instruction specification via pseudo-code results in improved performance over baseline nl english instructions, we choose to experiment with bloom @xcite , code-gen @xcite ) models. our choice of models is motivated by the fact that these models have not been instruction-fine-tuned on the natural instructions dataset. in addition, they have both been trained on code and natural language data. the bloom models are trained on the roots corpus @xcite consisting of 46 natural and 13 programming languages. on the other hand, the codegen models are trained on the pile corpus @xcite ), google's publicly available bigquery and bigpython datasets @xcite . the bloom models have been trained on a mixture of natural language and code simultaneously. as for the codegen models we utilize, they were initially trained on natural language and subsequently received additional ## conclusion and future work in this paper we presented our work on prompting with pseudo-code instructions. we created a collection of pseudo-code instructions comprising of 132 nlp tasks from the super-naturalinstructions dataset @xcite . we evaluated the performance of the following families of models -codegen and bloom at different model sizes and found that prompting all models with pseudo-code instructions results in significant gains as compared to prompting with nl instructions. our work opens up multiple directions of future work. it is interesting to observe that not only do pseudo-code instructions help when used with code models, they also work better on models designed for natural language tasks. in addition, the fact that code mod-els used in our experiments perform better than nl models, even when prompted with natural language instructions, suggests that it could be useful to explore instruction tuning of code models instead of pure nl models for nl applications. based on the findings of this paper it may also be useful to consider the effects of instruction fine-tuning with pseudo-code instructions as opposed to nl instructions. another aspect worth studying is how traditional chain-of-thought may compare with pseudo-code prompts -how would reasoning enabled by pseudocode instructions compare with chain-of-thought reasoning with and without fine-tuning? further, pseudo-code instructions may not only be used as direct inputs to a model, but they could also be used to create intermediate responses that a model needs to generate prior to returning a response.