We present CoTexT, a  transformer-based architecture encoder-decoder pre-trained model that learns the representative context between natural language (NL) and programming language (PL) through multi-task learning. CoTexT is pre-trained , in self-supervised fashion, based on large programming language corpus to learn general-purpose understanding and code-text generation supporting downstream NL-PL task such as code summarizing/documentation, code generation, defect detection, code debugging, etc . We train CoTexT on different combination of available PL corpus including both "bimodal" and "unimodal" data where the former is the combinations of both natural texts and their corresponding code snippets in an input sequence and the latter is merely code snippets. We evaluate multi-task learning CoTexT on different generation and classification tasks on CodeXGLUE and it achieves state-of-the-art on all downstream tasks .
We present CoTexT, a pre-trained, transformer-based encoder-decoder model that learns the representative context between natural language (NL) and programming language (PL) . Using self-supervision, CoTexT is pre-trained on large programming language corpora to learn a general understanding of language and code. CoTexT supports downstream NL-PL tasks such as code summarizing/documentation, code generation, defect detection, and code debugging . We train CoTexT on different combinations of available PL corpus including both "bimodal" and "unimodal" data . Here, bimodal data is the combination of text and corresponding code snippets , whereas unimodal data is merely code snippets. We first evaluate CoTexT with multi-task learning : we perform Code Summarization on 6 different programming languages and Code Refinement on both small and medium size featured in the CodeXGLUE dataset. We further conduct extensive experiments to investigate CoTexT on other tasks within the CodeXGlue dataset, including Code Generation and Defect Detection. We consistently achieve SOTA results in these tasks, demonstrating the versatility of our models .
A
pre-trained,
D
architecture
D
pre-trained
R
through multi-task learning.
. Using self-supervision,
D
, in self-supervised fashion, based
R
corpus to learn general-purpose understanding and code-text generation supporting
corpora to learn a general understanding of language and code. CoTexT supports
R
task
tasks
R
code debugging, etc
and code debugging
R
combination
combinations
R
where the former is the combinations of both natural texts and their
. Here, bimodal data is the combination of text and
R
in an input sequence and the latter
, whereas unimodal data
R
evaluate
first evaluate CoTexT with
R
CoTexT on different generation and classification tasks on CodeXGLUE and it achieves state-of-the-art on all downstream tasks
: we perform Code Summarization on 6 different programming languages and Code Refinement on both small and medium size featured in the CodeXGLUE dataset. We further conduct extensive experiments to investigate CoTexT on other tasks within the CodeXGlue dataset, including Code Generation and Defect Detection. We consistently achieve SOTA results in these tasks, demonstrating the versatility of our models