Revisiting the Impact of Pursuing Modularity for Code Generation (2024)

Abstract: Modular programming, which aims to construct the final program by integrating smaller, independent building blocks, has been regarded as a desirable practice in software development. However, with the rise of recent code generation agents built upon large language models (LLMs), a question emerges: is this traditional practice equally effective for these new tools? In this work, we assess the impact of modularity in code generation by introducing a novel metric for its quantitative measurement. Surprisingly, unlike conventional wisdom on the topic, we find that modularity is not a core factor for improving the performance of code generation models. We also explore potential explanations for why LLMs do not exhibit a preference for modular code compared to non-modular code.

## introduction with recent advances in the capabilities of large language models (llms; openai, 2024; gemini team, 2024; inter alia), their application areas have expanded beyond simple text-based tasks. among these, coding assistants are becoming practically essential for programmers, enhancing their efficiency through tasks such as natural language to code (nl2code) generation. similar to other use cases of llms, coding assistants are typically utilized in zero-or few-shot manners. the problem is that as the length of code is usually much longer than that of a sentence, the number of code examples available for each run is strictly limited. furthermore, the same functionality can be represented with different forms of code, making it challenging for users to select a proper example for a target task. the diversity of code formats also poses challenges in fine-tuning setups, as constructing an appropriate training dataset becomes non-trivial. it is thus important to understand what characteristics of the code provided to the agents contribute to their final performance of such models. among the many possible properties that influence the characteristics of code snippets, this work investigates the impact of code modularity on the performance of llms for nl2code generation. modular programming, the practice of building software with independent components, has long been considered a cornerstone of good software development. while this paradigm facilitates desirable properties of code for human programmers, such as reusability, readability, and maintainability, it remains an open question whether it offers the same level of effectiveness for llms. notably, jain et al. (2024) argued that leveraging a set of modular functions can improve code generation accuracy for both in-context learning (icl) and fine-tuning. as it is not trivial to guarantee the modularity of each code snippet, the authors asked gpt-3.5-turbo 1 to convert an existing code snippet into a more modular one, while ensuring its functional correctness. however, we claim that their report warrants revisiting for two reasons. first, since llms are notorious for their verbosity, it is unclear whether the conversion process aimed solely for modularity or accidentally introduced unexpected side effects. second, the lack of a formally defined quantitative method for estimating modularity hinders more extensive analyses related to the problem. in this paper, we (re-)investigate the effectiveness of pursuing modularity in nl2code generation. we aim to push the boundaries of previous work by (1) introducing a novel metric that quantifies the modularity of a code snippet using numeric values. based on the metric, we (2) classify code snippets as modular or non-modular without relying on llms, and evaluate how each category contributes to performance. @xcite moreover, beyond previous work, we (3) conduct experiments on models with parameters exceeding 7b (i.e., 33b and 34b) to investigate the impact of model scale. figure 1 illustrates the core research question of this work. in experiments, we discover that contrary to conventional wisdom in the literature, the modularity of a code example may not be the crucial factor for performance. we also explore potential explanations for why llms do not exhibit a preference for modular code compared to non-modular one. ## quantitative definition of modularity to assess the impact of code modularity, the first essential step is to develop a method that provides a measurable score for code modularity. while the previous study (jain et al., 2024) bypassed this vital step, 3 we present a reasonable metric for estimating code modularity, which is challenging due to the inherent subjectivity of the concept itself. inspired by the software engineering literature, we employ the concept of cyclomatic complexity (cc) (mccabe, 1976) to determine the ideal number of modules, m * , for a given code snippet. cc counts the number of independent execution 1 https://platform.openai.com/docs/models/gpt-3-5-turbo 2 note that this was infeasible in the previous study (jain et al., 2024) as there was no clear standard for determining whether each code snippet is modular or not. @xcite the authors instead utilized llms to transform all code snippets into supposedly modular ones. paths in the control-flow graph (cfg) of the target code, where the cfg is a graph representation of all potential paths that a program might follow during execution. cc can also be calculated as e -n + 2, where e and n correspond to the number of edges and nodes in the cfg. @xcite the cc values are computed at either the whole code level (total cc; cc total ) or the function level (meaning the average cc across all functions in the code; cc avg ). a high cc value generally indicates a complex code structure. it functions as a guideline for code decomposition, suggesting that a function whose cc is exceeding a certain threshold value τ , e.g., 5 (mccabe, 1976) or 10 (mcconnell, 2004), might benefit from being broken down into smaller subfunctions. based on the concept, we assume that the average cc of an ideal modular code example, denoted by cc avg * , should be equal to the threshold τ . @xcite in other words, ideally, every function within a modular code snippet is expected to have a cc value of τ . following the intuition, we define m * , the number of ideal modules, as follows: finally, we define the modularity score, dubbed mos, as follows: in appendix a, we show that mos is effective not only in capturing the structural properties of a code snippet but also in revealing the frequency of interactions between functions within the code. ## four code categories by modularity with a way to quantify code modularity, we can now classify a code dataset into two categoriesmodular and non-modular (= singular). we further leverage prior research by including llm-based code transformations and their corresponding manually recovered counterparts for controlled experiments. this allows us to create four distinct clusters of code separated by their modularity levels. @xcite modular code (mc) is a collection of code snippets with high mos among solutions for each problem in a dataset. ## experimental setups we explore the impact of modularity by comparing how the four code collections, categorized by their modularity levels, affect performance. we first focus on the case of utilizing code llms with few-shot in-context learning. we leverage two-shot demonstrations (providing two code examples) unless otherwise specified. @xcite in addition, we explore the scenario of fine-tuning llms with datasets that have varying levels of modularity. datasets we employ two nl2code generation datasets-apps (hendrycks et al., 2021) and code-contests (li et al., 2022). @xcite they are based on competitive programming contests and provide a set of different solutions for each problem. @xcite in this study, we focus our evaluation on python. ## main results in in we thus argue that the previously reported effectiveness of modularity on performance was likely due to unforeseen consequences of the transformation process, rather than the modularity itself. on the other hand, the performance of llms fine-tuned on mc and sc from codecontests is reported in ## analysis 6.1 correlation study we conduct an extra experiment to dive deeper into the modularity-performance relationship. specifically, given 100 code samples used as demonstrations, @xcite we compute the pearson and spearman correlations between their modularity (mos) and resulting performance (pass@1). for simplicity, we perform one-shot icl on codecontests. experimental results are presented in all the examples are either mc or sc type. ## conclusion in this work, we propose a metric, called mos, for quantifying the modularity of code snippets and evaluate its impact on performance. our evaluation reveals no significant correlation, or even a possible weak negative correlation, between modularity and performance. this suggests that factors influencing the usefulness of code examples may differ between human and llm perspectives. exploring the influence of other code properties beyond modularity is a promising direction for future work.