Less is More: Summary of Long Instructions is Better for Program Synthesis (2022)

Abstract: Despite the success of large pre-trained language models (LMs) such as Codex, they show below-par performance on the larger and more complicated programming related questions. We show that LMs benefit from the summarized version of complicated questions. Our findings show that superfluous information often present in problem description such as human characters, background stories, and names (which are included to help humans in understanding a task) does not help models in understanding a task. To this extent, we create a meta-dataset from the frequently used APPS dataset and the newly created CodeContests dataset for the program synthesis task. Our meta-dataset consists of human and synthesized summaries of the long and complicated programming questions. Experimental results on Codex show that our proposed approach outperforms baseline by 8.13% on the APPS dataset and 11.88% on the CodeContests dataset on an average in terms of strict accuracy. Our analysis shows that summaries significantly improve performance for introductory (9.86%) and interview (11.48%) related programming questions. However, it shows improvement by a small margin ( 2%) for competitive programming questions, implying the scope for future research direction.

## introduction recently, large pre-trained lms have been proven pivotal in programming-related tasks @xcite @xcite @xcite 2 . program synthesis aims to generate a code given the natural language description of a problem. programming requirements in these problems vary in terms of complexity from a 3-5 line simple function to multiple functions that use advanced data structures. ## experimental setup all the experiments are performed using the davincicodex @xcite model provided through openai foot_3 . at inference time, we use a modified version of the evaluation code foot_4 provided by @xcite . this evaluation code has four different outputs for each test case: experiments to show effectiveness of the proposed approach, we have performed three different experiments using human generated summaries: motivation behind ewpr and bwpr if a summary caused every test case to perform worse then it's likely the crowd worker produced a faulty summary. to mitigate the effect of outliers in the dataset, we use the ewpr method to remove such problems. another hypothesis is that every problem benefits from some level of summarization (i.e., basic or expert). to measure this, we use the bwpr method. from ## conclusion this paper introduces a summarization-based approach for efficient program synthesis. experimental results show that the proposed approach improves the performance of the codex model by on average ∼ 8% across various levels of programming questions provided by the apps and ∼ 11% on the codecontests. further, this paper proposes a meta-dataset consisting of ∼ 450 human-generated basic and expert-level summaries as well as ∼ 8k synthetically generated summaries by gpt-3 and studio21; this can be helpful for future research on writing better instructions for the program synthesis. we show that program synthesis models benefit from concise prompts, hence, we believe that less number of high-quality instances are better than more low-quality data instances. future extensions the decomposition of prompts has been shown to improve accuracy @xcite ; splitting up the summarization task the resulting summary can potentially result in higher accuracy for the codex model in future. additionally, the pegasus model could be used in conjunction with other models to perform the detailed algorithm outlined in appendix n.