Repository: stanford-crfm/helm
Issue: big_bench doesn't show results
Root cause: Validation Gap

When I use the above conf, 
![企业微信截图_169321280721](https://github.com/stanford-crfm/helm/assets/43968255/c19daebb-8a6d-4ae6-a1c4-03c7e30438ca)
the following error will appear. 
![image](https://github.com/stanford-crfm/helm/assets/43968255/e599a3e7-328c-4dc3-b550-833d95f44705)
When I use the following conf, no error will be reported, but they have no results, which can only be seen in Raw runs
![企业微信截图_16932124621887](https://github.com/stanford-crfm/helm/assets/43968255/6f718d39-101d-4db3-8ef2-dee96c634c72)
![企业微信截图_16932124714536](https://github.com/stanford-crfm/helm/assets/43968255/9dd3757b-b232-4b9b-b0e3-4e110d9df271)



Error comes from `HTTPModelClient.decode()` which is commented out [here](https://github.com/stanford-crfm/helm/blob/4810bd56e752b952bf1e85fb2963504e40f9e527/src/helm/proxy/clients/http_model_client.py#L120-L140)

@drisspg @msaroufim should the model server also expose a `decode()` endpoint? If so, can we uncomment this block?

The alternative is to ask the user to use a HELM local tokenizer instead, then the user just needs to configure the tokenizer name that corresponds to their base model... I'm making some progress on getting configurable tokenizers to work, and will probably have it ready in a couple of weeks.

@yifanmai Then why is the results not displayed?

The results are not displayed because there are no results to display. The error above prevented the run from completing and producing any output.

I am unable to repro the error,  Indeed decode is not currently exported. I will uncomment and fix up the toy submission but want something to iterate with

```
"big_bench:model=neurips/local,task=auto_debugging,subtask=", priority: 1}]
```

Should this be enough with 
`helm-run --conf-paths run_specs.conf --suite v1 --max-eval-instances 100 -n 16  `
?


@drisspg Thanks for your reply, you can use the configuration of src/helm/benchmark/presentation/run_specs_big_bench_lite.conf this should be all tasks for big_bench

@drisspg Hello, I see that you have submitted the PR and it says that you have succeeded. I would like to ask if your result interface will now display the big_bench results, just like the other tasks in the picture below. 
![image](https://github.com/stanford-crfm/helm/assets/43968255/37e88db8-8f50-4911-b3b4-a4c34dd17334)
I used your code, but there is still no result here. display of big_bench


Did you by chance grab the newest version of the toy submission: https://github.com/llm-efficiency-challenge/neurips_llm_efficiency_challenge/commit/21a3c07bac9fd5d4fd777697e6229ed2845ad44d

Yes



发自我的iPhone


------------------ Original ------------------
From: Driss Guessous ***@***.***&gt;
Date: Fri,Sep 15,2023 1:16 AM
To: stanford-crfm/helm ***@***.***&gt;
Cc: MrJungle1 ***@***.***&gt;, Author ***@***.***&gt;
Subject: Re: [stanford-crfm/helm] big_bench doesn't show results (Issue #1813)

So you have a display on your side？



发自我的iPhone


------------------ Original ------------------
From: Driss Guessous ***@***.***&gt;
Date: Fri,Sep 15,2023 1:16 AM
To: stanford-crfm/helm ***@***.***&gt;
Cc: MrJungle1 ***@***.***&gt;, Author ***@***.***&gt;
Subject: Re: [stanford-crfm/helm] big_bench doesn't show results (Issue #1813)





 
Did you by chance grab the newest version of the toy submission: ***@***.***
 
—
Reply to this email directly, view it on GitHub, or unsubscribe.
You are receiving this because you authored the thread.Message ID: ***@***.***&gt;

I am also not seeing any display on my end, but when enabling auto_debugging I see decode requests successfully being run. @yifanmai  Any thoughts?

> I am also not seeing any display on my end, but when enabling auto_debugging I see decode requests successfully being run. @yifanmai Any thoughts?

 Is there any updates about this issue? 


@yifanmai @percyliang any thoughts on this thread?

I also have decode request but not able to see big bench results. Do I need to update my helm version?

Thanks for the ping, I will investigate shortly.

Could someone help me with reproducing this issue? In particular, it would be good if you could upload a `.tar.gz` / `.zip` file of your `benchmark_output/runs/v1` folder to a public URL, and also let me know the githash from `git status` (or the version of HELM, if you are installing from PyPI).



> Could someone help me with reproducing this issue? In particular, it would be good if you could upload a `.tar.gz` / `.zip` file of your `benchmark_output/runs/v1` folder to a public URL, and also let me know the githash from `git status` (or the version of HELM, if you are installing from PyPI).

I'm facing the same issue, here you go:

`crfm-helm==0.3.0`

[v1.zip](https://github.com/stanford-crfm/helm/files/13296918/v1.zip)


Thanks @RawthiL, I'll take a look and get back to you in a couple days.

Hi, apologies for the slow response. I'm closing this issue due to staleness; if anyone is still actively affected, please reopen this issue.