Repository: stanford-crfm/helm
Issue: helm-create-plots errors
Root cause: Validation Gap

Hello, 

When I try to run `helm-create-plots --suite v1` outside the presentation directory, i.e, outside `helm/src/helm/benchmark/presentation`; I get following error:

`FileNotFoundError: [Errno 2] No such file or directory: '/home/helm/benchmark_output/runs/v1/groups/the_pile.json'`

Is this expected behavior? In the docs, I did not find any details in regards to running the command in a specific directory. 

The problem here is that we try to create plots for _every_ group, even for groups that are not missing to the current suite. We should change this to just skip creating plots for groups that don't exist in the current suite.

Hello, could I work on this? 

For the current release, is there a quick way to make sure we run all the groups? like a list of task we have to put into the run_specs? 

It would be super helpful to have this fixed~

@shalakasatheesh Sure, I assigned the issue to you for now. Let me know if you have any updates.

@gxxu-ml Sorry, I spent some time looking into this and I didn't find any quick workarounds.

cc @dtsip 

Good catch @shalakasatheesh ! Yes, we should just check whether the corresponding group exists for each plot we are making.

(my guess is that the issue you are getting with `the_pile` is because there is a plot visualizing performance as a function of The Pile perplexity.)

@shalakasatheesh wonder if you have figured out a configuration or a place to change to address the issue? This feels like a very general problem that people run into ~~

Hello, @yifanmai thank you. While working on this I noticed a couple of things and I would like some feedback on it:

1.  @dtsip was right, there is a plot visualization which is a function of The Pile perplexity. How would you like me to go forward with this? Shall I keep the visualization w.r.t. The Pile perplexity as a separate function? Should the user parse an argument for this visualization?

2. While running the  following configuration,

 `entries: [
   {description: "mmlu:subject=anatomy,model=huggingface/gpt2", priority: 1},
   {description: "mmlu:subject=philosophy,model=huggingface/gpt2", priority: 1},
 ]
` 
the meta data for the model 'huggingface/gpt2' can't be found, resulting in the following error:
```

Traceback (most recent call last):
  File "/home/venv/bin/helm-create-plots", line 8, in <module>
    sys.exit(main())
  File "/home/repos/helm/src/helm/benchmark/presentation/create_plots.py", line 595, in main
    plotter.create_all_plots()
  File "/home/repos/helm/src/helm/benchmark/presentation/create_plots.py", line 567, in create_all_plots
    self.create_all_accuracy_v_model_property_plots()
  File "/home/repos/helm/src/helm/benchmark/presentation/create_plots.py", line 350, in create_all_accuracy_v_model_property_plots
    self.create_accuracy_v_model_property_plot(
  File "/home/repos/helm/src/helm/benchmark/presentation/create_plots.py", line 287, in create_accuracy_v_model_property_plot
    key = model_name_to_property(model_name)
  File "/home/repos/helm/src/helm/benchmark/presentation/create_plots.py", line 335, in get_model_release_date
    release_date = self.model_metadata[model_name].release_date
KeyError: 'huggingface/gpt2'
```

I'm not sure how to go about solving this. Your feedback would be much appreciated :)  

@gxxu-ml unfortunately not yet. Maybe tweaking create_plots.py could help you find a quick fix?


Thanks for the investigation @shalakasatheesh ! This is basically #1146 (adding GPT-2 to `schema.yaml`). I can fix this next Monday (or you are welcome to send a PR earlier, if you like).

Any update on this?

@yifanmai @shalakasatheesh Is there any update on this issue?


Could someone please, provide a workaround? As simple as "here is 'the_pile.json' file, drop it in the directory to avoid the error. 

My suggestion is to skip the plotting step until we get it working.

@percyliang I think we should drop plotting from the tutorial documentation because it just doesn't work for most people.

cc @dtsip 

It is too late to remove plotting – people saw it and they want it.
Please, just make it possible to display benchmarks scores in a table in a human-readable format.

From: Yifan Mai ***@***.***>
Reply-To: stanford-crfm/helm ***@***.***>
Date: Tuesday, July 25, 2023 at 4:09 PM
To: stanford-crfm/helm ***@***.***>
Cc: "Ermolin, Sergey" ***@***.***>, Comment ***@***.***>
Subject: Re: [stanford-crfm/helm] helm-create-plots errors (Issue #1441)


My suggestion is to skip the plotting step until we get it working.

@percyliang<https://github.com/percyliang> I think we should drop plotting from the tutorial documentation because it just doesn't work for most people.

—
Reply to this email directly, view it on GitHub<https://github.com/stanford-crfm/helm/issues/1441#issuecomment-1650685933>, or unsubscribe<https://github.com/notifications/unsubscribe-auth/AF7BVAKHPLJPPI266HHJJVDXSBGZFANCNFSM6AAAAAAWONHXTM>.
You are receiving this because you commented.Message ID: ***@***.***>


My timeline for fixing plots is late Q3 to early Q4. If you need this earlier, feel free to submit a PR. I'm happy to help with review or pointers.

Plotting is now considered deprecated, so I will close this issue. Perhaps we can revisit this in the future.