R EPRESENTATIONS OF COMPUTER PROGRAMS IN THE HUMAN BRAIN
Anonymous authors
Paper under double-blind review
A BSTRACT
We present the ﬁrst study relating representations of computer programs generated by unsupervised machine learning (ML) models and representations of computer programs in the human brain.
We analyze recordings—brain representations— from functional magnetic resonance imaging (fMRI) studies of people comprehending Python code.
We discover brain representations, in different and speciﬁc regions of the brain, that encode static and dynamic properties of code such as abstract syntax tree (AST)-related information and runtime information.
We also map brain representations to representations of a suite of ML models that vary in their complexity.
We ﬁnd that the Multiple Demand system, a system of brain regions previously shown to respond to code, contains information about multiple speciﬁc code properties, as well as machine learned representations of code.
We make all the corresponding code, data, and analysis publicly available.
I NTRODUCTION
Unsupervised machine learning (ML) models trained on computer code are increasingly being used in software engineering workﬂows (Allamanis et al., 2018a).
Dubbed code models , they have been shown to perform well on tasks like code summarization (Alon et al., 2019), detecting variable misuse (Bichsel et al., 2016), and more recently, code auto-completion (Chen et al., 2021).
These models process large corpora of code to produce ML model representations of computer programs.
These representations likely encode syntax and semantics of programs (Bichsel et al., 2016; Allamanis et al., 2018b; Srikant et al., 2021).
A question of interest, as yet uninvestigated, is whether these ML model representations faithfully reﬂect brain representations of computer code during comprehension.
One way to address this question is by comparing representations from ML models to the code-related information encoded in the human brain.
Yamins et al.
(2013) ﬁrst showed how information encoded in our visual system resembles what convolutional neural networks learn when trained to recognize images.
Could it then be possible that common code properties and semantics of programs are also reﬂected in brain activity patterns?
More directly, do the current model architecture choices used to reason about code faithfully capture what is encoded by our brains?
These questions form the focus of this work.
A few prior works have investigated the neural bases of programming (Siegmund et al., 2017;
Floyd et al., 2017; Peitek et al., 2018; Castelhano et al., 2019; Huang et al., 2019; Krueger et al., 2020; Prat et al., 2020; Ivanova et al., 2020;
Liu et al., 2020; Ikutani et al., 2021; Peitek et al., 2021).
These works use analysis techniques like functional magnetic resonance imaging (fMRI) and electroencephalography (EEG) to investigate brain regions involved in coding tasks like code comprehension, code writing, and data structure manipulation.
The primary quest of these works though, has been to locate physical regions in the brain involved in these activities, with the goal of determining whether code-related activity joins other activities supported by those brain regions.
It remains unclear what speciﬁc code-related information these regions encode.
Do these regions encode speciﬁc syntactic or semantic code properties?
Further, are different regions involved in encoding different such properties?
Likewise, how brain activity data— brain representations —of code relate to what
ML models of code learn is an open question.
To date, no systematic method exists to generate the evidence from which to draw answers.
In this work, we study the task of code comprehension, and utilize the fMRI recordings dataset from
Ivanova et al.
(2020) to investigate human brain representations of code.
We establish two major directions of investigation—the probing of brain region representations for speciﬁc code properties, and an analysis of the mapping of these representations onto various code models with dif- fering model complexity.
The latter can potentially inform us of properties encoded in continu- ous representations learned by these models which need not be contained in the set of properties we evaluate in the former.
We learn afﬁne maps from brain representations to predict code representations of different code models, and similarly predict code properties which summarize the syntactic and semantic behavior of programs.
We investigate the effects that brain regions, the nature of code properties, and the complexity of the code models have on the accuracies with which we decode brain representations.
We ﬁnd that we can decode code-related information from brain representations, and can map representations of code models from brain representations.
We ﬁnd the Multiple Demand system, a system of brain regions, to consistently encode speciﬁc code properties and map with machine learned representations of code.
We provide an open-source framework to replicate our experiments, and we release our data and analysis publicly.
Link - https://github.com/anonmyous-author/anonymous-code .
The framework allows both brain data and program metrics to be easily added to extend the current set of experiments.
This should enable authors from other neuroimaging studies or code model developers to collaborate and analyze data across these works, which will also help amortize the high costs of carrying out such experiments.
R ELATED W ORK
Of the prior works that have investigated the neural bases of programming through fMRI and EEG techniques (Siegmund et al.,
; Floyd et al., 2017; Peitek et al., 2018; Castelhano et al., 2019;
Huang et al., 2019; Krueger et al., 2020; Ivanova et al., 2020; Liu et al., 2020; Ikutani et al., 2021;
Peitek et al., 2021) and through behavioral studies (Prat et al., 2020; Casalnuovo et al., 2020; Crichton et al., 2021), the following probe brain recordings for program properties encoded in them.
Floyd et al.
(2017) learn a linear model to successfully classify whether an observed brain activity corresponds to reading code or reading text.
Ikutani et al.
(2021) study expert programmers and show that it is possible to classify code into the four problem categories–math, search, sort, and string from the brain activations corresponding to the code.
Similarly, Liu et al.
(2020) classify whether a brain signal corresponds to code implementing an if condition or not.
Peitek et al.
(2021) analyze correlations between brain recordings of participants reading code and a set of code complexity metrics.
In testing for code properties, our work uses a similar methodology (a linear model trained on fMRI data), but we evaluate a larger set of static and dynamic code properties.
We also systematically test whether key programming constructs like control ﬂow and data operations are encoded by these brain representations.
A systematic analysis of multiple such properties has not been performed in any of these works.
Further, we perform these tests in those brain regions identiﬁed by Liu et al.
(2020) and Ivanova et al.
as being responsive speciﬁcally to code comprehension.
This gives us a ﬁner insight into how different regions encode these properties.
In addition to these tests, we study representations generated by a suite of ML models with varying model complexity and compare those representations to brain representations.
Brain representations have also been studied in domains like natural language, vision, and motor control.
Among related works in natural language, a domain that resembles programming languages,
Mitchell et al.
; Pallier et al.
; Brennan & Pylkk ¨ anen (2017); Jain & Huth (2018);
Gauthier & Levy (2019); Schwartz et al.
; Wang et al.
; Schrimpf et al.
Cao et al.
(2021) have studied brain representations of words and sentences by relating them to representations produced by language models.
While the broader tools we use to investigate these representations, like multi-voxel pattern analysis (MVPA), are similar to some of these prior works, our focus is on properties speciﬁc to code and not natural language.
B ACKGROUND
We provide a brief background on fMRI signals as a proxy for brain representations and describe the brain systems that we probe in this work.
Measuring brain activity with fMRI.
Functional magnetic resonance imaging (fMRI) is a brain imaging technique used to measure brain activity in speciﬁc brain regions.
When a brain region is active, blood ﬂows into the region to aid its processing.
An MRI machine measures this change in blood ﬂow, and reports BOLD (blood oxygen level dependent) values sampled at the machine’s frequency (Glover, 2011, usually 2 seconds).
The smallest unit of brain tissue for which BOLD signal is recorded is called a voxel (an equivalent of a 3D pixel); it comprises several cubic millimeters of brain tissue.
For our analyses, we select subsets of voxels belonging to speciﬁc brain systems.
Following common practice, the parameters of a general linear model, ﬁt to time-varying
BOLD values, are used as a measure of the overall activation in each voxel in response to a given input.
It is the values of these parameters that, according the neuroscience community, are brain representations.
Brain systems.
A system of brain regions can span different areas of the brain but behaves as a holistic unit, showing similar patterns of engagement across a given cognitive task.
We probe the following systems of regions in our work.
See Appendix A for a detailed description of these systems along with relevant references.
• Multiple Demand (MD) system.
This system of regions is known to engage in cognitively demanding, domain agnostic tasks like general problem solving, logic, and spatial memory tasks.
Liu et al.
and Ivanova et al.
(2020) reported that this system is active during code comprehension.
• Language system (LS).
This system responds during language production and comprehension across modalities (speech, text) and languages (across 10 language families, including American sign language).
Siegmund et al.
(2014; 2017) reported activity in regions resembling the LS during code comprehension, whereas Floyd et al.
; Liu et al.
Krueger et al.
(2020) concluded that brain regions processing language and code are distinct.
Ivanova et al.
(2020) reported moderate levels of activity within the LS in response to Python, but not to code in a visual programming language, ScratchJr.
We also probe brain representations in two brain systems responsible for perception.
• Visual system(Vis).
These regions respond primarily to visual inputs.
We expect activity in this system to reﬂect low-level visual properties of the code ( e.g. code length and indentation).
• Auditory system.
These regions respond primarily to auditory inputs.
We do not expect activity in this system to reﬂect code-related properties.
B RAIN AND M ODEL R EPRESENTATIONS
We describe in this section the method we follow to gather representations of code in the brain (Section 4.1), evaluate the different code properties they encode (Section 4.2), and how we compare brain representations to those generated by ML models (Section 4.3).
4. B RAIN REPRESENTATIONS AND DECODING
We provide a summary of how we process activation signals in the brain elicited by code comprehension, to probe whether they encode any speciﬁc code properties.
We provide details in Appendix B.
Dataset.
We use the publicly available brain recordings released as part of the study by Ivanova et al.
It contains brain recordings of 24 participants reading 72 programs from a set ofunique Python programs.
The 72 programs were presented in 12 blocks of 6 programs each.
These programs were 3-10 lines in length and contained simple Python constructs, such as lists, for loops and if statements.
A whole program was presented at once, and the task required participants to read the code and mentally compute the expected output, press a button when done, and select one of four choices presented to them which matched their calculated output.
From dataset to brain representations.
The original dataset contains 3D images of the brain of each participant.
Each voxel value in these images is an estimate of the response strength in this voxel when a particular code (or sentence) problem is presented.
To determine which brain systems contain information about particular code properties, we focus our analyses to four systems – MD,
Language, Visual, and Auditory (Section 3).
A vector of voxels’ activation values in each brain system is then taken to constitute that system’s representation of a computer program and serves as an input to all our analyses.
See Appendix B for details on data processing and voxel selection.
Analyzing brain representations of code.
We probe brain representations from each participant separately.
We do not average data across participants since the regions which respond to any task (comprehending code in our case) need not align anatomically.
For each of two experiments— decoding different code properties, and mapping to code model representations—we train ridge regression/classiﬁcation models which take as input normalized brain representations per participant.
We hence learn 24 different regression models, for each code property or code model (one per participant), and then report the mean performance of these models across participants.
This procedure is also referred to as multi-voxel pattern analysis (MVPA) (Norman et al., 2006).
Lin- ear models are conventionally preferred for probes into brain representations since there has been evidence supporting the idea that other brain areas linearly map information from such brain representations (Kamitani & Tong, 2005; Kriegeskorte, 2011).
We choose a linear model primarily to control for over-ﬁtting in light of the relatively small dataset.
A remark on data scarcity.
We train a linear regression/classiﬁcation model with L2-regularization for every participant on unique cross-validated leave-one-run-out folds of the 72 programs they attempted (or 48 programs when sentences are removed).
On the order of 1000 voxels were selected from each brain region responding to any given program per participant.
As a baseline for the model predictions, we use the accuracy of a null permutation distribution generated from samplingrandom assignments of the labels.
4. C ODE PROPERTIES
We attempt to decode the following code properties from brain representations.
• Code vs. sentences .
The dataset provides a control condition where code problems are described in English sentences (referred to as sentences ).
See Figure B, Appendix B for an example.
We classify these two stimuli categories from brain data.
• Variable language .
We classify programs containing meaningful variables names from brain representations.
The dataset has half its programs with variables in English, and the other half in Japanese (written in English characters), which do not convey any contextual meaning.
• Control ﬂow .
We predict whether a program contains a loop ( for loop), a branch ( if condition), or has sequential instructions without any control branching.
The dataset contains an equal number of programs with each of the three conditions.
• Data type .
We predict whether a program contains string operations or numeric operations in them.
Half the programs in the dataset exclusively have string operations within them, while the other half are exclusively numeric.
• Static analysis .
We decode static properties of a program.
Speciﬁcally, we predict token count (number of tokens in the program) and node count (number of AST nodes).
We also consider cyclomatic complexity , and Halstead difﬁculty (details in Appendix B), which have been used by software engineering practitioners to quantify the complexity of code, and to quantify the difﬁculty a human would experience when comprehending code respectively.
We defer predicting other advanced static analysis metrics such as tracking abstract interpretation joins, data ﬂow analysis-related metrics, etc . to future work.
• Dynamic analysis .
We decode information about a code’s execution behavior.
We evaluate two properties– runtime steps (number of instructions executed in the program) and bytecode ops (number of bytecode operations executed in running the program).
We expect these properties to be correlated with the number of mental instructions needed to arrive at the output.
Program length as a potential confound.
Program length is a potential low-level code feature which can explain many of the properties we investigate in this work.
To verify this, we measured the inter-correlations of these properties, and their correlation to the number of tokens in the program (program length) (see Appendix G).
We expectedly found the four static analysis properties were highly correlated to each other and to bytecode ops .
We hence use one representative metric each from the two categories of properties for the rest of our analysis– token count for static analysis , and runtime steps for dynamic analysis .
Importantly, the other properties we examine cannot be explained by program length alone, and we hence disregarded program length as a confound in our experiments.
The brain representations (Section 4.1) are mapped to each of the code properties by training a ridge regression or classiﬁcation model each for every participant-property pair.
To evaluate model performance, we use classiﬁcation accuracy when the predicted values are categorical ( e.g. string vs. numeric data types), and the Pearson correlation coefﬁcient when the predicted values are continuous ( e.g. number of runtime steps).
We choose Pearson correlation over RMSE, the canonical distance metric for continuous values, for its simplicity and interpretability.
See Appendix I for the RMSE results, which are similar to the correlation results.
When testing for the signiﬁcance of these predictions, we perform false discovery rate (FDR) correction for the number of brain systems tested and the number of properties tested.
See Appendix B for a detailed description of model hyper-parameters and cross-validation settings employed.
4. M ODEL REPRESENTATIONS AND DECODING .
We evaluate a bench of unsupervised language models, spanning from count-based language models to transformer neural networks (Vaswani et al., 2017).
These models were all trained on large ( ∼ 1M programs)
Python datasets (Husain et al., 2019; Puri et al., 2021).
We use the output of the trained encoders (raw logits) in each of the neural network models as representations of the code input to the model.
We vary the general complexity of these models to test whether that variation is meaningful in establishing the quality of brain to model ﬁts.
Model complexity here is the number of a model’s learnable parameters.
We evaluate the following models, ordered by their increasing model complexity: simple frequency-based language models— bag-of-words , TF-IDF ; auto-encoder based unsupervised models— seq2seq (Sutskever et al., 2014), CodeTransformer (Z ¨ ugner et al., 2021),
CodeBERTa (HuggingFace, 2020); auto-regressive model with a model complexity similar to that of transformers and CodeBERTa — XLNet (Yang et al., 2019).
We compare the results of the above models against an aggressive baseline (relative to the nulldistribution labeling baseline) provided by using unique Gaussian-distributed random vectors for the token embeddings in a vocabulary, and returning the sum of these token embeddings across a program.
The resultant embedding is not transformed by any model or any weights–it instead serves as a proxy for the tokens that appear in the program.
The brain representations (Section 4.1) are mapped to code representations by training another set of ridge regression models to learn an afﬁne map, and a ranked accuracy metric is used to compare outputs.
Ranked accuracy scores are commonly used in information retrieval where several elements in a range are similar to the correct one.
In our setting, the score assigns full credit to getting the exact stimulus program at the top of the ranking, and partial credit if the top-ranked program is close to the target.
When reporting result signiﬁcance, we perform false discovery rate (FDR) correction for the number of brain systems and the number of models.
See Appendix B for a detailed discussion of the implemented code models and the corresponding metrics.
E XPERIMENTS & R ESULTS
Our experiments address two research questions:
• Experiment 1.
How well do the different brain systems encode speciﬁc code properties?
• Experiment 2.
How do the brain representations of code correspond to the representations learned by computational language models of code?
E XPERIMENT 1 - H OW
WELL DO THE DIFFERENT BRAIN SYSTEMS ENCODE SPECIFIC CODE PROPERTIES ?
Here, we analyze the classiﬁcation models and regressions trained on brain representations to predict each of the code properties described in Section 4.2.
The results of our analyses are summarized in
Figure 3.
The classiﬁcation and regression tests are marked on the x-axis of the left and right subplots respectively; the classiﬁcation accuracy or Pearson correlation for each of the tasks is marked on the y-axes.
We plot dynamic and static properties separately from the others because their baselines are different due to a difference in the similarity metric (classiﬁcation accuracy vs. Pearson correlation).
Auditory and Visual systems.
The Auditory system serves as a negative control for the other systems.
Given that the code comprehension task we use is visual, we do not expect to decode any meaningful information from it.
We ﬁnd that, out of 6 tested properties, only code vs sentences can be decoded from the Auditory system, and even then the decoding performance is much lower than in other systems.
The Visual system also serves as a control for low-level visual properties of the code (as opposed to abstract semantic or syntactic features).
Given the visual attributes of a program such as the layout and indentation of the code, the length of the program, and the presence of letters and alphabets in the programs (Park et al., 2012; Roux et al., 2008; Polk et al., 2002), this system might reﬂect at least some of the properties we evaluate.
This is indeed what we observe: 4 out of 6 tested properties can be decoded from the visual system above-chance.
The MD and the Language systems yield the following speciﬁc observations.
In a follow-up analysis (Analysis 3), we investigate the effect of the Visual system in the decoding performance of the MD and the
Language systems.
Analysis 1 - How accurately are different properties predicted by MD and LS?
We analyze how accurately each of MD and LS predict all the properties we evaluate.
For ease of viewing, we re-plot in Figure 5 the results from a selection of code properties for just the LS and MD from Figure 3.
We plot the z-scores to normalize the different scales used in the two sets of code properties shown in Figure 3.
• Code vs. sentences Code and sentence properties are decoded most accurately among all the properties we test.
The MD, LS, and Visual system decode with ∼ 90% accuracy.
The high accuracy from MD validates claims from previous works suggesting the involvement of these regions in code comprehension.
• Variable language
This requires distinguishing between sets of nearly identical programs, with the only variation being in the language of the variable names (English vs. Japanese).
We hence expect the Language system will most accurately encode this information, since it is sensitive to the presence of new words.
However, no such effect is observed–no brain system shows any signiﬁcant decoding.
This unexpected ﬁnding is however consistent with Ivanova et al.
(2020), who show a lack of any signiﬁcant difference in the aggregate neural activity between the two variations.
Both suggest that these brain systems do not seem to rely on variable names as a meaningful feature, consistent with the fact that a change in variable names does not alter the expected output of the program
• Control ﬂow and data type Both control ﬂow and data type tasks are most accurately predicted by the MD, followed by the LS.
In the case of control ﬂow , the more accurate predictions by the MD can possibly be explained by the larger number of mental operations needed for programs with loops when compared to programs with one or no branches.
The difference seen in data type prediction is unclear.
While programs with string operations contain more literals and words from the English alphabet, we do not see any preference in the LS being able to decode this property more accurately
• Static and Dynamic analysis
We observe the LS predicts static analysis properties with the highest accuracy.
This is expected, since activity in the LS is known to be sensitive to the length of any text being read (independent of any speciﬁc program instructions in our case).
For the dynamic analysis property runtime steps , we see the MD system has the highest accuracy.
This again is reﬂective of the workings of the MD system, which is sensitive to working memory tasks, and the number of steps executed by a program matches the number of mental calculations performed by a person tracing through the program to calculate its output.
Analysis 2 - How accurately do MD and LS encode different properties?
We use t-tests to examine whether for a given property, any one brain system decodes it signiﬁcantly more than another.
We ﬁnd no differences between the MD system and the Language system for any properties (Table 8, Appendix E).
We additionally test if any brain system has a preference for a speciﬁc code property over another.
For instance, is the evidence seen in Figure 5, of MD more accurately decoding dynamic analysis properties than static analysis properties, statistically significant?
We do not ﬁnd any signiﬁcant differences (Table 9, Appendix E).
While neither MD nor LS preferentially encode any of the code properties we explore in this work, we do ﬁnd that the two brain systems individually encode these properties, which is a new result.
In the following experiment, we evaluate whether there exist other properties which continuous representations of programs, obtained from different code models, encode which are also encoded in brain representations.
Analysis 3 - The effect of Vis on predictions from MD, LS.
The decoding performance of the
Visual system is comparable to that of the MD and the Language systems (Figure 5).
It is then possible that the MD and LS encode the same properties which the Visual system encodes (low-level features, all which are potentially related to program length).
While an inter-correlation analysis of the different code properties revealed that they are not highly correlated to program length (see
Section 4.2), we conﬁrm this further through an ablation analysis.
For each brain system R i , we train two models–one which decodes from R i , and another which decodes from R i + Visual system.
If the difference in the prediction accuracies between the two models is signiﬁcant, we conclude that R i encodes at least some information which is orthogonal to the information encoded by the Visual system.
For control ﬂow , data type , and code vs sentences , we ﬁnd the MD to encode orthogonal information than the Visual system.
For control ﬂow speciﬁcally, the LS also encodes differently than Vis, suggesting that the results from Experiment 1 cannot be explained by the MD and LS encoding low-level code properties alone.
Detailed results are tabulated in Table 15, Appendix F.
5. E XPERIMENT 2 - H OW DO THE BRAIN REPRESENTATIONS OF CODE CORRESPOND TO THE REPRESENTATIONS LEARNED BY COMPUTATIONAL LANGUAGE MODELS OF CODE ?
Here, we train ridge regression models with brain representations of programs from a speciﬁc brain system to predict code model representations of the same programs.
The full set of results from this experiment are summarized in Figure 4.
Auditory and Visual systems.
Similar to Experiment 1, the Auditory system performs as expected, exhibiting the lowest decoding performance across all code models.
The Visual system exhibits the second lowest performance across code models with the exception of CodeBERTa to which it maps more accurately to than LS, but not MD.
However, this result is not statistically signiﬁcant (Table 10, Appendix E).
Analysis 1 - How well do brain representations in MD and LS map to code model representations?
To ease our analysis of the MD and LS systems, we re-plot in Figure 6 the decoding accuracies of just these two brain systems from Figure 4.
We ﬁnd that the MD system and
Language system map to all the models in our suite signiﬁcantly more accurately than the null permutation baseline (baseline accuracy of 50% in Figures 4 and 6).
Further, from Figure 6, we see the MD maps more accurately than the LS across all code models.
We test the signiﬁcance of the differences between the two systems and ﬁnd that the MD rank accuracy is higher than
LS for CodeBERTa , seq2seq , and TF-IDF , while in the rest of the models, the differences are not signiﬁcant (Table 10, Appendix E).
We discuss the implication of these results in Section 6.
Analysis 2 - The effect of model complexity on brain-code model mapping.
We investigate the impact model complexity has on the quality of the mapping between brain and code representations.
Model complexity here is the number of a model’s learnable parameters.
In analysis 1 above, we saw that the MD system mapped to the representations of CodeBERTa , seq2seq , and TF-IDF signiﬁcantly more accurately than the Language system.
These three models vary substantially in their complexity, thus suggesting the lack of any relationship between model complexity and brain systems.
We also compare each of these code models against the Random embedding model.
This baseline model assigns a random vector of numbers to every unique token that appears in the training vocabulary, and sums the vectors of all the tokens that appear in a program.
By its design, this model encodes only the presence of speciﬁc tokens in programs.
The mapping accuracies of random embeddings in the MD and the Language systems are marked in blue and green horizontal lines in
Figure 6.
We ﬁnd that the Multiple Demand system maps to all the models more accurately than the Random embedding .
This is not seen in the LS, which conﬁrms the lack of variabilility of rank accuracies observed in the LS.
In a set of pairwise signiﬁcance tests, we ﬁnd that the MD maps to CodeBERTa signiﬁcantly more accurately than Random embedding while other systems are more accurate numerically, but not signiﬁcantly (Table 11, Appendix E).
We discuss its implications in the following section.
D ISCUSSION
We investigate what aspects of computer programs are encoded by the brain.
In Experiment 1, we investigate whether a speciﬁc set of code properties, which encode various dimensions of a program, are encoded in different brain regions.
In Experiment 2, by comparing brain representations to code model representations, we investigate if there exist other code properties learned by code models but not contained in the set of code properties we investigate in Experiment 1 which are encoded by the brain.
This additionally informs us whether code models learn and encode those aspects of code which humans seem to encode when comprehending programs.
Key among our contributions is demonstrating it is possible to predict a range of code properties from brain system activity (representations).
Control ﬂow and data type information is encoded in the Multiple Demand system, which the ablation analysis suggests is not merely a consequence of low-level features of the code.
Runtime information of programs is also encoded most strongly in MD, while code properties related to the length of a program are weakly encoded in Language system.
The Multiple Demand system encoding both static and runtime code properties, along with distinguishing features like operation types and control ﬂow suggests that the it is responsible both for parsing code, and simulating it and calculating its outcome.
Future work should test a larger set of programs that vary in the complexity of the properties they possess.
Another key contribution is demonstrating that it is possible to map brain representations to representations learned by code models.
Although our decoding accuracies are relatively modest, they are comparable with similar decoding results from the domain of language (Pereira et al., 2018;
Wang et al., 2020; Schrimpf et al., 2020).
Moreover, unlike most prior decoding works in cogni- tive neuroscience, our decoders are trained on individual trials (rather than on averages of multiple recorded responses), a task that is difﬁcult due to inherent noise present in fMRI data.
Thus, our result demonstrates the feasibility of investigating the speciﬁcs of code comprehension in the brain without the need to present the same program multiple times to a participant.
We learn three pieces of information from our decoding analysis using language models.
(a) It is possible to map various language models from brain activation data.
(b) The quality of this mapping does not seem to be correlated with the complexity of these models.
(c) The quality of this mapping varies between MD and LS signiﬁcantly for three models - codeBERTa, seq2seq, and TF-IDF.
We infer the following from these results -
• In predicting continuous representations of programs, afforded by these different models, we do ﬁnd signiﬁcant differences in predictions from the MD when compared to LS.
We did not see such a consistent difference for any of the properties we evaluated in Experiment 1.
This suggests that the brain activation data encodes more than just the code properties we evaluated; these other properties are encoded by the different language models.
• We found it surprising that the neural information from MD and LS maps to some complex models like seq2seq and code transformers almost as well as to a combination of random token embeddings.
One plausible explanation for this is that the information we are attempting to decode from MD and LS captures mostly token-level information, as against richer structural information about the programs.
The program stimuli are simple enough to allow the different properties evaluated in our work (control ﬂow, data types, etc.) to be discerned from token level information alone (as validated in Experiment 1), which is likely why the random embeddings model is also able to predict these properties very well (see Table 4, Appendix C.2).
• This suggests that the information being decoded from brain activations in these two regions is driven at the least by token-level information in the programs.
We cannot come to this conclusion from the code properties investigated in Experiment 1 alone. We note that neither
does this imply that the code models do not encode anything more sophisticated than just token information, nor does it imply that the brain does not encode anything more sophisticated.
CodeBERTa alone, for which the response is signiﬁcantly greater than random embeddings, provides evidence that the MD system encodes more than just token information.
This will need more investigation.
Given the current data and the number of observations we have, we can only conclude that tokens do get encoded well in both - brain activations and code models.
This is a new ﬁnding on what aspects of code comprehension get encoded by our brains which can be successfully read out using a linear model.
Limitations.
The average program in any software project exhibits non-trivial control and data dependencies, object manipulation, function calls, types, and state changes, which our dataset does not possess.
However, the programming tasks in Ivanova et al.
(2020) are short snippets of procedural code with limited program properties.
Responses to longer, more realistic programs should be studied on a larger number of participants in the future, building on the understanding of simpler snippets provided by our work.
Further, while there are equally important aspects to programming like designing solutions, selecting appropriate data structures, and writing programs, we have chosen to study a very speciﬁc activity related to programming—comprehending programs.
Broader impact.
Our ﬁndings have the potential to improve our understanding of the organization of the human brain, which can in turn lead to the design of better code models.
In computer vision, recent results by Tschopp et al.
(2018) show how deep network architectures that mimic the visual system in fruit ﬂies exhibit superior image classiﬁcation rates on image recognition tasks.
One immediate outcome is reconsidering the current design of ML models of code.
Extant code model architectures do not explicitly model the Multiple Demand system in any way–they only model syntactic information and infer dependency information from program ASTs.
Taking inspiration from the role of the Multiple Demand system we identiﬁed in this work, modeling both static and dynamic information should be explored.
This work could perhaps also enhance code prosthetics –artiﬁcial interfaces in the body that can help the physically challenged engage with programming environments.
Such systems generally rely on brain decoders–models that convert brain activity data to electrical impulses modulating external devices.
An open challenge is to improve them.
See the discussion in Nuyujukian et al.
and
Andersen et al.
for details.
Finally, our aspiration is that the framework we release in this work will bring together the neuroscience and machine learning communities to better understand the cognitive and neural bases of programming.
R EPRODUCABILITY STATEMENT
All the results, and corresponding tables, plots, and intermediate results we introduce in this paper are fully reproducible.
We have made the source code to our work publicly available through this anonymized code repository - https://github.com/anonmyous-author/ anonymous-code/
The repository contains detailed instructions for setting up the source code and running the two main experiments we introduce in this work.
Further, a copy of the dataset released by Ivanova et al.
Ivanova et al.
(2020) is also available for in the repository, along with the source code.
All the intermediate results which we use to arrive at our conclusions are available in Appendix A through Appendix I.
A B RAIN REGIONS
Regions of Interest (ROIs).
We investigate the representations of code in well-studied systems of brain regions–the MD system, Language system, Visual system, and the Auditory system (details below).
A region (also referred to as parcel ) here denotes a contiguous chunk of brain mass involved in a cognitive task.
A system of regions (also referred to as a network ) can comprise multiple disjoint regions that exhibit shared activity patterns across a range of tasks.
For many cognitive tasks, a region marks only the approximate location of voxel populations involved in a cognitive task.
Functional ROIs (fROIs) are voxels within these broad regions that respond most strongly to a given task (language, working memory, etc .).
The use of fROIs enables accounting for the exact anatomical locations of these task-sensitive voxels, which vary across individuals.
Multiple Demand (MD) system.
Located in the prefrontal and parietal areas of the brain, this system of regions is active in a host of tasks requiring working memory and general problem solving skills, including math and logic (Duncan, 2010; Fedorenko et al., 2013; Amalric & Dehaene, 2019).
Language system.
These regions have been identiﬁed to respond to both comprehension and production of language across modalities (written, speech, sign language), respond to typologically diverse languages ( > 50 languages, from across 10 language families), form a functionally integrated system, reliably and robustly track linguistic stimuli, and have been shown to be causally important for language (Clark & Cummings, 2003; Fedorenko et al., 2010; Blank & Fedorenko, 2017; Ayyash et al., 2021).
Visual system.
These regions in the occipital lobe of the brain respond to visual input, ranging from low-level features like lines and edges, through intermediate categories like shapes, letters, and numbers, through higher order structures like faces and scenes.
(Hubel & Wiesel, 1959; Polk et al., 2002; Epstein & Kanwisher, 1998; Kanwisher et al., 1997).
Auditory system.
The auditory system is located in the superior temporal region of the brain.
This region uniquely encodes pitch, speech, and music, but is not involved in high-level language comprehension and production (Norman-Haignere et al., 2015; 2019).
In our experiments pertaining to programming language comprehension, we use the activity seen in the auditory system as a negative control.
B M ETHOD - D ETAILS
Selecting brain representations.
For each trial in the fMRI experiment, stimulus responses in each voxel were extracted from the parameters of a General Linear Model (GLM) ﬁt to the timevarying BOLD signal.
The predictors for the GLM included trial ID (equivalent to problem ID), run number, and motion regressors.
The voxels were then ﬁltered using gray-matter masking and (for MD and the Language systems) network localization.
Although fMRI measurements return whole brain responses, only a thin layer of cortex dubbed gray matter contains BOLD signal of interest to these analyses.
Gray matter voxels were selected using a Bayesian segmentation of the anatomical brain image into standard tissue types, and then returning the set of indices where the posterior gray matter probability exceeds 0.70 (Ashburner et al., 1999).
Next, these sets of voxels were ﬁltered separately for each of the brain regions outlined in Appendix A. For the visual and auditory networks, primary sensory areas were identiﬁed using an anatomical atlas (Rolls et al., 2020).
For the MD and the Language systems, voxels were functionally localized as those containing the top 10% of responses to their respective functional localizer tasks, as described in Ivanova et al.
See Fedorenko et al.
(2010) for a discussion of the functional localization approach as it pertains to the language network.
GLM modeling and gray matter segmentation was performed using SPM12; functional localization was performed using the toolbox released by Ivanova et al.
Once voxel responses within each brain region were extracted for each trial of each run, some additional preprocessing was required before ﬁnalizing the brain representations, and passing them to downstream models.
Due to differences in MRI sensitivity across runs, each of theruns of 6 programs for a given subject appeared with nonuniform mean and variance.
In order to normalize these signals, so as to leverage data across a participant’s entire scanning session, brain representations were z-transformed within each run to achieve a common scale.
In order to avoid data-leakage from this preprocessing step, all downstream analyses were cross-validated via a leaveone-run-out approach, such that no intra-run rescaling could be used for prediction.
MVPA cross-validation and hyperparameters.
For each brain system and each code property or code model, we run a separate MVPA analysis.
For each subject, we train a separate ridge regression model on each cross-validated leave-one-run-out fold, and then predict the remaining targets using the brain representations from the left-out run.
We use L 2 regularization to control for model complexity, and employ a leave-one-out cross validation scheme to select λ.
We score resulting predictions using a series of different metrics to maximize interpretability.
For the categorical code properties, we report classiﬁcation accuracy of model estimates.
For the continuously scaled code properties, we report Pearson correlation between the model estimates and true targets.
For the model representations, we report rank accuracy between model representation predictions and the true model representations of left-out samples.
We calculate rank accuracy as the percentile of a predicted and true target pairing within the full set of possible pairings sorted by a Euclidean distance metric.
Rank accuracy was chosen for this experiment as it returns a more interpretable 50% baseline through which to evaluate the brain to model mappings.
Following metric calculation on each cross-validation fold, we take the mean of those estimates to report an out-of-sample score over the entire dataset for each subject.
We then take the mean of those scores across subjects to derive an overall performance measure.
As a baseline, we repeat the above process in its entirety 1000 times for each MVPA analysis, to provide a null permutation distribution against which to compare scores.
These null distributions are ﬁt with a univariate Gaussian and used to calculate z-statistics and p-values for MVPA scores.
All statistical tests were corrected for false discovery rate.
The code to run our MVPA analysis is available in https://github.com/anonmyous-author/anonymous-code .
Code properties.
The code properties we consider are the following
– • Token count.
The total number of tokens that appear in the program.
This is a sanity check to measure brain activity against, since it is natural to expect the activity in the brain to be correlated to the length of a program, and in general, the amount of content being comprehended.
By their design, the dataset had program lengths with a small standard deviation.
• Node count.
The total number of AST nodes that appear in a program.
Similar to the number of tokens, we verify if brain activity correlates to a proxy for the amount of syntactic content in it.
• Runtime steps.
We execute the programs and measure the number of instructions the program steps through.
While two programs can have the same program length, the number of instructions executed can differ ( e.g. for 1:10 vs. for 1:50 ).
We measure if any brain regions capture the number of mental operations needed to compute the output of a program.
• Bytecode ops.
We execute the programs and count the number of bytecode operations performed.
This metric should mimic the number of mental operations performed needed to compute the output of a program.
• Cyclomatic complexity.
This metric (McCabe, 1976) is used to measure the general complexity of software systems.
It is deﬁned as a function of the number of nodes, edges, and the connected components in a program’s control ﬂow graph.
While its efﬁcacy as a metric to measure software complexity has been contested (Shepperd, 1988), we include it to see if brain activity is correlated to any explicit syntactic constructs of a program.
• Halstead difﬁculty .
This metric (Halstead, 1977) was deﬁned to measure how difﬁcult any piece of software would be for a programmer to comprehend or write.
It is deﬁned as a function of the number of tokens, operations, vocabulary that appears in a program.
The inter-correlations between these metrics have been tabulated in Table 17, Appendix G.
Code models - Model details.
We evaluate the following models in this work–
• bag-of-words , TF-IDF .
These are count-based language models.
They predict the likelihood of a token appearing in a program based on vocabulary statistics and frequencies.
• seq2seq (Sutskever et al., 2014), CodeTransformer (Z¨ugner et al., 2021), CodeBERTa (HuggingFace, 2020).
We evaluate these three autoencoder (AE) models with increasing model complexity.
AE based pretraining reconstructs the original program from corrupted input.
• XLNet
(Yang et al., 2019).
We evaluate an auto-regressive (AR) model with a model complexity similar to that of transformers and CodeBERTa .
AR language modeling estimates the probability distribution of a text corpus in one direction–either forward or backward, while AE models capture dependencies in both directions.
We use an AR model to mimic a top-down, single pass comprehension style by humans.
• Random embedding .
We compare the results of the above models against an aggressive baseline provided by using unique Gaussian-distributed random vectors for the token embeddings in a vocabulary, and returning the sum of these token embeddings across a program.
The resultant embedding is not transformed by any model or any weights–it instead serves as a proxy for the tokens that appear in the program.
This sets a higher bar than the null-distribution labeling baseline (Section 5.2).
Code models - Conﬁguration details.
In setting up our code model bench, we aimed to select a collection of models ranging in their complexity, namely bag-of-words , TF-IDF (Term frequencyInverse document frequency), seq2seq , XLNet , CodeTransformer , and CodeBERTa , as well as a Random embedding embedding model.
• The bag-of-words , TF-IDF , seq2seq , and Random embedding models all use the same custom tokenizer deﬁned by the current authors, whereas XLNet , CodeTransformer , and CodeBERTa use tokenizers deﬁned by the original training authors (Z¨ugner et al., 2021; HuggingFace, 2020).
• The tokenizer set up for the models trained by the current authors establishes a unique token in the vocabulary for each Python keyword, each Python builtin function, one token for all numeric types, one token for all string types, and N tokens for each of the N variables in a given program.
• In the case of the Random embedding baseline, this tokenizer was used to map each token in a piece of source code to an index in the vocabulary, which in turn was used to index a Gaussian distributed random matrix ( D = 128 ).
The sum of these random token projections was calculated to return a unique embedding for each program.
• For bag-of-words and TF-IDF , the validation split of the code-search-net Python dataset (Husain et al., 2019) was used to enumerate vocabulary and token occurrence statistics.
These data were then used to transform each program to a vector where each dimension represented the raw ( bagof-words ) or document-weighted ( TF-IDF ) count of the unique items in the vocabulary.
• If a given set of programs viewed by a subject never included a speciﬁc token, leading that column to equal the zero vector, then those dimensions would be ﬁltered out.
• We used the dataset released by Project Codenet (Puri et al., 2021) to pretrain a seq2seq model (Sutskever et al., 2014).
The dataset contains Python programs that are solutions to olympiadstyle programming problems in data structures and algorithms.
We trained a seq2seq model with
a GRU unit for 15 epochs, with a dropout probability of 0 . 2 , dot product attention, and a maximum sentence length of 500 .
• For XLNet , CodeTransformer , and CodeBERTa , each program was passed through that model’s tokenizer and model pipeline as deﬁned by the original authors.
The representations of each program were extracted from the pretrained encoders.
XLNet and CodeTransformer were originally trained on the Python subset of the code-search-net dataset, whereas CodeBERTa was originally trained on the full code-search-net dataset, which also incorporates go , java , javascript , php , and ruby .
C A DDITIONAL R ESULTS C. E XPERIMENT
Using representations from localized brain regions
[Table]
we attempt to decode static and dynamic properties of comprehended code
[Table]
and learn maps to code model representations of that same code.
We report decoding performance of each brain region to the original Ivanova et al.
(2020) code properties in Table 1, the complexity-related code properties in Table 2, and the code model mappings in Table 3.
We ﬁnd that the MD system and the Language system decode all code properties and code models except variable language signiﬁcantly above baseline, as established through a null permutation test.
The Visual system decodes 4 of the 6 code properties and 6 of the 7 code models, whereas the Auditory system only decodes the code vs sentences property.
[Table]
[Table] [Table]
We additionally evaluate whether code models contain linearly decodable information about the code properties we explore in this work (Section 4.2).
The models decode with near perfect accuracy on most classiﬁcation tasks (avg.
accuracy = 0 .
96 ± 0 . 01 ), and with high correlations on the regression tasks (avg.
r = 0 .
89 ± 0 . 02 ).
This is expected for the classiﬁcation tasks, since the dataset contains tokens which help with perfectly separable decision boundaries.
For example, for the control ﬂow code property, the dataset contains programs with if , for , or neither, but never both.
These results suggest that the code models we evaluate faithfully encode the set of properties we evaluate them on
[Table]
D A NALYSIS OF V ARIANCE
We run a series of one-way ANOVA statistical tests to assess whether decoding performance varies across brain regions for each code property and code model, and whether decoding performance varies across code models for each brain region.
We report these results in Tables 5,
[Table]
and 7.
We ﬁnd that decoding performance varies across brain region for all code properties and code models, except the variable language property and the Random embedding model
[Table]
and decoding performance varies across code models for the MD system and the Visual system
[Table] [Table]
[Table]
E P AIRWISE A NALYSIS
To extend the ﬁndings from the ANOVA analyses towards speciﬁc pairwise comparisons between brain regions and code models
[Table]
we compare scores between brain regions for each code property or code model
[Table]
and compare scores between code models and a subset of code properties for each brain region, using two-sample two-tail t-tests with unequal variance.
We report all pairwise brain region comparisons in Tables 8 and 10, and report all pairwise code property and model comparisons in Tables 9 and 11.
To highlight a few key results from code properties, we ﬁnd that the MD system decodes runtime steps and control ﬂow signiﬁcantly better than the Visual system (Table 8).
Additionally, the Language system decodes static analysis and control ﬂow signiﬁcantly better than the Visual system.
Moving onto code models, we ﬁnd that the MD system decodes CodeBERTa , seq2seq
[Table]
and TF-IDF signiﬁcantly more accurately than the Language system
[Table]
and additionally decodes CodeTransformer , seq2seq , TF-IDF
[Table]
and bag-of-words signiﬁcantly more accurately than the
Visual system (Table 10).
Finally, an investigation into model complexity reveals that CodeBERTa is signiﬁcantly more accurately decoded from the MD system than the Random embedding model
[Table] [Table]
[Table] [Table]
F A BLATION ANALYSIS : S CORES AND SIGNIFICANCE
In order to investigate whether each of the brain regions included contribute unique variance towards the decoding tasks, we combine brain representations from different regions in paired combinations and effects on downstream decoding performance across all experiments.
We ﬁnd that the addition of MD or LS to Visual system, relative to Visual system alone, improves downstream decoding of code vs sentences .
The same effect is observed for the combination of MD and LS compared to either alone.
These data suggest that the MD system and the Language system encode unique variance relevant to the decoding of code vs sentences , and this is above and beyond information encoded in the Visual system or each other individually.
Additionally, we observe that the addition of MD to Visual system, relative to Visual system alone, improves downstream decoding of control ﬂow and data type .
This same effect is observed for the addition of LS to Visual system for control ﬂow .
These data suggest that the MD system and the Language system encode unique variance above and beyond information encoded in the Visual system for the control ﬂow decoding task, and the MD system encodes unique variance above and beyond information encoded in the Visual system for the data type decoding task.
[Table]
[Table] [Table]
[Table]
G A LL CODE PROPERTIES : S CORES AND C ORRELATIONS
We analyzed a series of code properties as part of the static and dynamic code analysis.
As several of these properties ( token count , node count , Halstead difﬁculty , cyclomatic complexity
[Table]
and bytecode ops ) were revealed to be highly correlated in a post-hoc analysis, we report only one measure for this subset in Experiment 1, token count , but include all scores here for completeness (Table 16).
We also report the correlation matrix between all code properties that led us to select token count as the representative property for this subset (Table 17)
[Table] [Table]
H S ENSITIVITY OF BRAIN MAPPING TO MODEL OUTPUT DIMENSIONS
Of potential interest to the decoding framework is not just the complexity of code models, but their intrinsic dimensionality as well.
In order to investigate to what extent brain model mappings are robust to changes in code model dimensionality, and to assess which model representations are most sensitive to compression and expansion, we rerun our current decoding framework while controlling for embedding size.
For each brain network to code model mapping task, prior to the MVPA analysis, we control for the dimensionality of the code model embedding via projection through a Gaussian random matrix R d 1 × d 2 drawn from N (0 , 1 /d 2 ) where d 1 is the original code model embedding dimensionality and d 2 is the desired dimensionality.
We observe that models of lower complexity (e.g., bag-of-words , TF-IDF , seq2seq , XLNet ) appear relatively robust to compression, whereas the most complex models (e.g., CodeTransformer ,
CodeBERTa ) gain considerable performance from higher dimensional expression, and suffer considerably when constrained.
This indicates that higher dimensions of the encoder in complex models encode relevant neural information.
Additionally, these effects appear to be most pronounced when decoding from the brain region whose representations yield the strongest mappings, the MD system.
We note here however that we could be observing an interaction effect between model complexity and dimensionality output in these results.
Since we cannot fully control for the complexity of these models (by making them all ‘equally complex’), this experiment alone cannot drive deﬁnitive conclusions.
These results instead constitute a preliminary exploration into the effects of code model dimensionality on brain to model representation mappings, and suggest an avenue for future investigation.
I R OBUSTNESS OF R ESULTS TO R EGRESSION M ETRIC
For the decoding analysis of the continuous valued dynamic analysis and static analysis properties
[Table]
it is reasonable to ask why the Pearson correlation metric was chosen as opposed to RMSE , as is typically customary for regression tasks.
While we present the results using the Pearson correlation metric in the core results for interpretability via the zero-baselne, here we conﬁrm that the use of an RMSE metric leads to the same conclusions.
As we see here, the MD, LS, and Visual system decode the dynamic analysis property
[Table]
and MD and LS decode the static analysis property with signiﬁcantly lower RMSE than the null permutation baseline.
These results precisely conﬁrm and mirror the patterns observed in Figure
[Table]