lysandre HF staff lvwerra HF staff commited on
Commit
73bf59e
1 Parent(s): e8002b4

add coding question question (#2)

Browse files

- add coding question question (589fd5f90cc687951c9410dd18682e0baf88e72c)
- update reference (db46314c779bf0bc756f7021f6609083f0b5c735)


Co-authored-by: Leandro von Werra <lvwerra@users.noreply.huggingface.co>

Files changed (1) hide show
  1. app.py +10 -1
app.py CHANGED
@@ -64,11 +64,20 @@ References
64
  - [5] - `joblib`: https://joblib.readthedocs.io/en/latest/
65
  """
66
 
 
 
 
 
 
 
 
 
 
67
  internships = {
68
  'Accelerate': default_question,
69
  'Diffusion distillation': default_question,
70
  'Skops & Scikit-Learn': skops_question,
71
- "Code Generation": default_question,
72
  "Document AI Democratization": default_question,
73
  "Evaluate": default_question,
74
  "ASR": default_question,
 
64
  - [5] - `joblib`: https://joblib.readthedocs.io/en/latest/
65
  """
66
 
67
+ code_question = """
68
+ You are probing your code generation model on a program synthesis benchmark and
69
+ 1 out of 4 the candidate solutions produced by your model pass the unit tests of a coding challenge.
70
+ <br/>
71
+ <br/>
72
+ What’s the pass@2 metric (in percent) as introduced in the
73
+ <a href="https://arxiv.org/abs/2107.03374" target="_blank"><code>Codex paper</a></code> (see section 2.1)?
74
+ """
75
+
76
  internships = {
77
  'Accelerate': default_question,
78
  'Diffusion distillation': default_question,
79
  'Skops & Scikit-Learn': skops_question,
80
+ "Code Generation": code_question,
81
  "Document AI Democratization": default_question,
82
  "Evaluate": default_question,
83
  "ASR": default_question,