C ode SCM : Causal Analysis for Multi-Modal Code Generation (2025)

Abstract: In this paper, we propose CodeSCM, a Structural Causal Model (SCM) for analyzing multi-modal code generation using large language models (LLMs). By applying interventions to CodeSCM, we measure the causal effects of different prompt modalities, such as natural language, code, and input-output examples, on the model. CodeSCM introduces latent mediator variables to separate the code and natural language semantics of a multi-modal code generation prompt. Using the principles of Causal Mediation Analysis on these mediators we quantify direct effects representing the model’s spurious leanings. We find that, in addition to natural language instructions, input-output examples significantly influence code generation.

## introduction modern large language models (llms) have shown remarkable effectiveness in code reasoning tasks, particularly code generation @xcite @xcite . this task involves generating code that meets specific multi-modal requirements, constrained by natural language instructions, code snippets, and input-output (i/o) example pairs @xcite . additionally, some multi-modal prompt components contain information from both code and natural language modalities @xcite , such as function signatures and variable names, where code structure and natural language coexist. this enriched coding context, combining programming and natural language semantics, helps llms better understand both the semantics and syntactic requirements of the desired code. prior research has shown the effectiveness of prompt tuning in improving generation performance (brown et al., 2020; liu et al., 2021; wei * these authors contributed equally to this work. @xcite . these works have shown that multimodal prompts can be highly sensitive, where small adjustments might result in drastically different responses from the model @xcite @xcite . however, the interactions between the multi-modal components of code generation prompts and their direct or indirect effects on the generated code are still not well understood. in this paper, we systematically explore these complex multi-modal effects using a causal approach. we propose a novel causal framework, codescm, to measure the causal effects of different modalities in the prompt on the performance of code generation llms. codescm defines a structural causal model @xcite , shown in figure 1 , where each modal component of the prompt is treated as an independent variable that causally affects the code generated by the model. to account for similar natural language and code semantics of different surface forms, we introduce two latent mediator variables to capture code semantics and natural language semantics of the input prompt, mimicking a human mental model for generating correct code snippets from a multi-modal input problem. specifically, we make four key contributions in this paper: (i) we introduce codescm, a novel framework for causal inference in multi-modal code generation tasks, enhancing interpretability and causal understanding of codellms. while codescm is designed for the code generation task in this paper, it can be extended to other modalities, tasks, and transformations. (ii) using codescm, we define the total effects of the modalities on code generation, highlighting that input-output example pairs and natural language code components, like function headers, are significant modal components alongside natural language instructions. we also observe benchmark memorization in llms like gpt-4t with our total effect analysis; (iii) through targeted interventions on codescm, we measure the direct effects of each modality, representing spurious model correlations, and show that simple semantics-preserving transformations to input-output example pairs lead to a significant drop in accuracy; and (iv) following the asymmetric causal effects of modalities, we examine the effect of multi-modal code-specific pretraining on the embedding space, which shows that codellm codellama can align different prompt modalities better than llama-2 in the embedding space. our code is available on github foot_0 . ## conclusion we propose codescm, a structural causal model for analyzing multi-modal code generation using llms. our analysis revealed that input-output examples and natural language code components significantly influence model generation. additionally, our interventions show that semantics-preserving changes can impact accuracy and can also lead to fewer hallucinations in some cases.