galtimur commited on
Commit
fe82027
β€’
1 Parent(s): ec7e6c9

Update src/tasks_content.py

Browse files
Files changed (1) hide show
  1. src/tasks_content.py +19 -1
src/tasks_content.py CHANGED
@@ -22,7 +22,25 @@ TASKS_DESCRIPTIONS = {
22
  For further details on the dataset and the baselines from 🏟️ Long Code Arena Team, refer to `library_based_code_generation` folder in [our baselines repository](https://github.com/JetBrains-Research/lca-baselines) or to our preprint (TODO).
23
  """,
24
 
25
- "ci_builds_repair": "cool description for Bug Localization on Build Logs task",
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
26
 
27
  "project_code_completion": """# Project-Level Code Completion\n
28
 
 
22
  For further details on the dataset and the baselines from 🏟️ Long Code Arena Team, refer to `library_based_code_generation` folder in [our baselines repository](https://github.com/JetBrains-Research/lca-baselines) or to our preprint (TODO).
23
  """,
24
 
25
+ "ci_builds_repair": """# CI Builds Repair\n
26
+
27
+ Our CI Builds Repair benchmark πŸ€— [JetBrains-Research/lca-project-level-code-completion](https://huggingface.co/datasets/JetBrains-Research/lca-project-level-code-completion) includes four datasets:
28
+ * `small-context`: 144 data points,
29
+ * `medium-context`: 224 data points,
30
+ * `large-context`: 270 data points,
31
+ * `huge-context`: 296 data points.
32
+
33
+ We use standard Exact Match (EM) metric for one-line code completion.
34
+ We evaluate Exact Match for different line categories:
35
+ * *infile* – functions and classes are from the completion file;
36
+ * *inproject* – functions and files are from the repository snapshot;
37
+ * *committed* – functions and classes are from the files that were added on the completion file commit;
38
+ * *common* – functions and classes with common names, e.g., `main`, `get`, etc.;
39
+ * *non-informative* – short/long lines, import/print lines, or comment lines;
40
+ * *random* – lines that doesn't fit to any of previous categories.
41
+
42
+ For further details on the dataset and the baselines from 🏟️ Long Code Arena Team, refer to `code_completion` folder in [our baselines repository](https://github.com/JetBrains-Research/lca-baselines) or to our preprint (TODO).
43
+ """,
44
 
45
  "project_code_completion": """# Project-Level Code Completion\n
46