A rch C ode: Incorporating Software Requirements in Code Generation with Large Language Models (2024)

Abstract: This paper aims to extend the code generation capability of large language models (LLMs) to automatically manage comprehensive software requirements from given textual descriptions. Such requirements include both functional (i.e. achieving expected behavior for inputs) and non-functional (e.g., time/space performance, robustness, maintainability) requirements. However, textual descriptions can either express requirements verbosely or may even omit some of them. We introduce ARCHCODE, a novel framework that leverages in-context learning to organize requirements observed in descriptions and to extrapolate unexpressed requirements from them. ARCHCODE generates requirements from given descriptions, conditioning them to produce code snippets and test cases. Each test case is tailored to one of the requirements, allowing for the ranking of code snippets based on the compliance of their execution results with the requirements. Public benchmarks show that ARCHCODE enhances to satisfy functional requirements, significantly improving Pass@k scores.Furthermore, we introduce HumanEval-NFR, the first evaluation of LLMs’ non-functional requirements in code generation, demonstrating ARCHCODE’s superiority over baseline methods. The implementation of ARCHCODE and the HumanEval-NFR benchmark are both publicly accessible.

## introduction recent advancements in large language models (llms) have significantly improved code generation capabilities @xcite @xcite . although the primary goal for llms in this domain is to generate functionally correct code based on textual descriptions @xcite @xcite , real-world software development encompasses more than just functionality. figure 1 : the archcode framework infers software requirements of correct code solution for a given textual description, then conditions them to generate code, as well as test cases for verification. in software engineering, software requirements provide a detailed framework describing what a software system is intended to achieve @xcite , divided into two categories (glinz, 2007): despite the critical role of software requirements, considering these criteria has not been studied actively in previous code generation works, merely assert x_or_y(13, 77, 2) == 77, 'failed to return the value of x for a prime number.' assert x_or_y(24, 8, 9) == 9, 'failed to return the value of y for a non-prime number.' assert x_or_y(-7, 77, -5) == -5, 'failed to handle a negative input number.' assert x_or_y(0, 77, 0) == 0, 'failed to handle a zero input number.' assert x_or_y(2**31-1, 34, 0) == 34, 'failed to handle large input size.' assert not x_or_y @xcite , 'failed to handle a non-integer input number.' assert complexityvisitor.from_code('''def x_or_y …''').@xmath0, 'failed to have a cyclomatic complexity less than or equal to 5 by radon.' ## related work despite the fact that llms recently have shown impressive capabilities in code generation, the majority of evaluations have focused solely on functional requirements @xcite @xcite @xcite . solely targeting functional requirements early research such as feng et al. (2020), chen et al. (2021), brown et al. (2020), and li et al. ( 2022 ) directly generates code from natural language descriptions, which may not fully capture all software requirements due to their vagueness or imperfections. later studies @xcite @xcite have targeted to better capture functional requirements by generating code-like outlines via in-context learning (icl; @xcite @xcite . more recent methods enhance fr satisfaction through self-verification of the generated code: on one hand, code filtering utilizes 'over-generatethen-filter' strategies, where the filtering can be achieved either by predicting functional correctness without code execution @xcite @xcite , or execution with given @xcite or generated test cases @xcite self-planning @xcite ) ✓ ✗ brainstorm @xcite ✓ ✗ algo @xcite ✓ ✗ coderanker @xcite ✓ ✗ lever @xcite ✓ ✗ coder-reviewer @xcite ✓ ✗ alphacode @xcite ✓ ✗ mbr-exec @xcite ✓ ✗ codet @xcite ✓ ✗ reflexion @xcite ✓ ✗ mpsc @xcite ✓ ✗ wizardcoder @xcite ✓ △ pie @xcite ✗ △ titanfuzz @xcite ✗ △ fuzz4all @xcite ✗ △ archcode (this work) ✓ ✓ on the other hand, code refinement iteratively reflects and refines the generated code to improve its functionality via execution results of current version of code with generated test cases @xcite . targeting narrow scope of non-functional requirements while much research covers frs, few studies have addressed specific attributes of non-functional requirements (nfrs) such as reliability and robustness @xcite , or time/space performance @xcite . ## the archcode framework we propose archcode, a novel code generation framework that employs in-context learning (icl) to llms, incorporating software requirements in code generation. as shown in figure 3 , arch-code delineates software requirements from textual descriptions, generates code, then verifies it using custom test cases. 2 formally, given a problem space p and a code space c, the code generation task is to build a function f : p ñ c that maps each textual problem description p p p into its corresponding code implementation c p c. archcode decomposes f by f " g ˝f . g : p ñ p ˆr maps problems to problem-requirements pairs, and f : p ˆr ñ c generates code from the problemrequirements pairs, where r is a space of software requirements. the test case generation function h : p ñ t is also decomposed by archcode into h " g ˝h, where t is the space of test cases and h : p ˆr ñ t . ## experiments we evaluate archcode's effectiveness using three benchmarks, categorized into two types: 1) a novel benchmark for assessing both fr and nfr satisfaction; 2) two public benchmarks aimed at fr evaluation, facilitating comparison of arch-code with existing baselines. for the former, we introduce humaneval-nfr for comprehensive nfr assessment, overcoming the conventional focus on fr alone. for the latter, we explore two code modalities: 1) function-level and 2) competitionlevel code generation.