Repository: stanford-crfm/helm
Issue: Problems and questions on the reproducibility of experiments
Root cause: Documentation Deficiency

Hi, i'm trying to reproduce some of yours evaluation experiments. In particular i'm doing a **robustness evaluation on the task of Sentiment Analysis** on the *IMDB* dataset. As you said i'm using the IMDB dataset and the contrast dataset (488 samples each). I'm testing the ***gpt-j-6b model***.

I'm trying to replicate the evaluation by writing the code to do it by myself. However, i have so questions about it. 

Since i'm using the transformers library from huggingface, i'm loading the model as a **Text generetor**, but i did't get if you use a few-shot situation or you are using a zero-shot prompting. And if you are using a few-shot prompring technique, how many prompt examples? Are you defining the task for the model before the evaluation?

Can you provide some details in order to let me able to reproduce the evaluation by coding the evaluation framework for the robustness on sentiment analysis of gpt-j?

Could you clarify if you're trying to reproduce the results (1) by reimplementing the evaluations in your own scripts / framework, or (2) by re-running evaluation the HELM framework?

If 1, for the prompt format, you can look at the runs list for imdb on gpt-j-6b [here](https://crfm.stanford.edu/helm/classic/v0.4.0/#/runs?q=imdb%3A.*gpt-j-6b). You can click on the run, then click *Request Details* which will give you the exact prompt format for the requests.

If 2, could you let me know if you intend to use local inference of the Hugging Face model?

Hi! 
Thanks for answer.

I'm trying to reproduce the evaluation of the gpt-j-6b model by myself without the use of your library. So i'm on the situation (1). 
In particular i'm trying to reproduce the robustness evaluation on the sentiment analysis task. 

What i infer from the link is that you used a 5-shots input context without any other prompt tuning. Is it correct? 

But i have one another question, did you use the vanilla model from eleuther AI or a different fine tuned version? 

Is this from huggingface https://huggingface.co/EleutherAI/gpt-j-6b or a different one ?

Great, thanks for the clarification.

> What i infer from the link is that you used a 5-shots input context without any other prompt tuning. Is it correct?

That is correct.

> But i have one another question, did you use the vanilla model from eleuther AI or a different fine tuned version?
> Is this from huggingface https://huggingface.co/EleutherAI/gpt-j-6b or a different one ?

We used the base model (which you linked to), not a fine-tuned version.



Really thanks for the reply!
I'm sorry to bother you but i need this clarification because your research it's really interesting and i want to emulate your results in order to get a better understand of the topic.

I have only one last doubt.

In order to evaluate the robustness of gpt-j-6b on the sentiment analysis task did you used this contrast set https://github.com/allenai/contrast-sets/tree/main/IMDb/data ?
Because here, i have only 488 samples if i use the ***test set***. I'm asking that because in that way i've used 488 samples for the performance on the clear set (the same 488) and 488 for the robustness evaluation on the contrast set.
It seems that you have at least 1000 samples, is it true? Where can i find your dataset?

On those 488 contrast samples i got an accuracy of 0.7479 and on your experiments you have an accuracy near to 0.90, how is it possible?
In the clear samples (just the same 488 in a non-contrast way) you have an accuracy of 0.939 and i have an accuracy of 0.84.
As you said i've used 5-shots prompting

This is the repo for my experiments replication if you need it: https://github.com/CristianCosci/LLMs_Robustness_evaluation_on_Sentiment_Analysis

Thank you so much for the time you dedicated to me