A st BERT : Enabling Language Model for Financial Code Understanding with Abstract Syntax Trees (2022)

Abstract: Using the pre-trained language models to understand source codes has attracted increasing attention from financial institutions owing to the great potential to uncover financial risks. However, there are several challenges in applying these language models to solve programming language related problems directly. For instance, the shift of domain knowledge between natural language (NL) and programming language (PL) requires understanding the semantic and syntactic information from the data from different perspectives. To this end, we propose the AstBERT model, a pre-trained PL model aiming to better understand the financial codes using the abstract syntax tree (AST). Specifically, we collect a sheer number of source codes (both Java and Python) from the Alipay code repository and incorporate both syntactic and semantic code knowledge into our model through the help of code parsers, in which AST information of the source codes can be interpreted and integrated. We evaluate the performance of the proposed model on three tasks, including code question answering, code clone detection and code refinement. Experiment results show that our AstBERT achieves promising performance on three different downstream tasks.

## introduction programming language and source code analysis using deep learning methods have received increasing attention in recent years. using pre-trained model such as such as bert @xcite , albert @xcite receive a great success on different nlp tasks. inspired by that, some researchers attempt to apply this technique to comprehend source codes. for instance, codebert @xcite ) is a pre-trained model using six different programming languages from github, demonstrating a good performance comparing with different embedding techniques. ## related work in this part, we describe existing pre-trained models and datasets in code language interpretation in detail. ## astbert in this part, we describe the details about astbert. ## experiments we test the performance of our proposed model on different code understating tasks using the different released test datasets. we also look into the ablation studies. ## conclusion in this paper, we propose astbert, a simple and effective way to enable pre-trained language model for financial code understanding by integrating semantic information from the abstract syntax tree (ast). in order to encode the structural information, astbert uses a designated ast-segment and ast-position in the embedding layer to make model incorporate such ast information. following that, we propose the ast-mask-self-attention to limit the region when calculating attention scores, preventing the input from deviating from its original meaning. we conduct three different code understanding related tasks to evaluate the performance of the astbert. the experiment results show that astbert outperforms baseline models on both code question answering and clone detection. for code refinement task, the model achieves state-of-the-art performance on bfp_small dataset and competitive performance on bfp_medium dataset.