update model
Browse files
architectures/incoder.txt
CHANGED
@@ -2,10 +2,13 @@
|
|
2 |
|
3 |
|Model | # parameters |
|
4 |
| - | - |
|
|
|
5 |
| Decoder |6.7B |
|
6 |
|
7 |
|
8 |
[Causal Masking objective](https://arxiv.org/abs/2201.07520) is a hybrid approach of Causal and Masked language models, "it combines the benefit of per-token generation with optional bi-directionality specifically tailored to prompting".
|
9 |
-
During the training of InCoder, spans of code were randomly masked and moved to the end of each file, which allows for bidirectional context. Figure 1
|
10 |
|
11 |
-
So in addition to program synthesis (via left-to-right generation), InCoder can also perform editing (via infilling). The model gives promising results in some zero-shot code infilling tasks such as type prediction, variable re-naming and comment generation.
|
|
|
|
|
|
2 |
|
3 |
|Model | # parameters |
|
4 |
| - | - |
|
5 |
+
| Decoder |1.3B |
|
6 |
| Decoder |6.7B |
|
7 |
|
8 |
|
9 |
[Causal Masking objective](https://arxiv.org/abs/2201.07520) is a hybrid approach of Causal and Masked language models, "it combines the benefit of per-token generation with optional bi-directionality specifically tailored to prompting".
|
10 |
+
During the training of InCoder, spans of code were randomly masked and moved to the end of each file, which allows for bidirectional context. Figure 1 illustrates the training process.
|
11 |
|
12 |
+
So in addition to program synthesis (via left-to-right generation), InCoder can also perform editing (via infilling). The model gives promising results in some zero-shot code infilling tasks such as type prediction, variable re-naming and comment generation.
|
13 |
+
|
14 |
+
In the code generation demo we use InCoder 1.3B.
|