Spaces:
Running
on
CPU Upgrade
Running
on
CPU Upgrade
Plan for supporting for custom prompt?
#8
by
kiyoonyoo
- opened
Hi,
Thank you for creating this benchmark.
Currently, both Eleuther AI's Language Model Evaluation Harness and this benchmark do not support using custom prompts for evaluation. According to this issue, Eleuther AI has plans to support custom prompts so one can use the prompts that the model was trained.
Do you have plans for adding this feature to this benchmark as well?
Thanks!
Hello,
Thank you for opening the issue! Currently, we do not have plan to support custom prompts. But we'll let you know if we decide to support it in the future!
BearSean
changed discussion status to
closed