Evaluate on CodeSearchNet
I evaluate the gte-base model on Ruby of CodeSearchNet. But my results are inconsistent with the paper's results.
My result is as follow:
Much lower than the result in the paper: 76.1%.
I use CodeSearchNet from https://github.com/github/CodeSearchNet.
Do you have any idea about the difference?
I figure out this problem. The model is evaluated on a new dataset, not the original CodeSearchNet. Maybe the new dataset should not be called CodeSearchNet any more. This name is very misleading.
I figure out this problem. The model is evaluated on a new dataset, not the original CodeSearchNet. Maybe the new dataset should not be called CodeSearchNet any more. This name is very misleading.
Thank you for your interest in the GTE model. Indeed, there are serveral versions of the CodeSearchNet dataset. If you have any other questions, please feel free to contact us.