zhiminy/EvalEng huggingface dataset:
Repository: stanford-crfm/helm
Issue: Trouble installing `torch==1.12.1+cu113`
Root cause: Environment Incompatibility

I get the following error when running the pre-commit script:

./pre-commit.sh
Looking in links: https://download.pytorch.org/whl/torch_stable.html
Obtaining file:///juice/scr/nlp/crfm/benchmarking/benchmarking
  Installing build dependencies ... error
  error: subprocess-exited-with-error

  × pip subprocess to install build dependencies did not run successfully.
  │ exit code: -9
  ╰─> [6 lines of output]
      Looking in links: https://download.pytorch.org/whl/torch_stable.html
      Collecting setuptools>=40.8.0
        Downloading setuptools-65.3.0-py3-none-any.whl (1.2 MB)
           ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 1.2/1.2 MB 9.1 MB/s eta 0:00:00
      Collecting wheel
        Using cached wheel-0.37.1-py2.py3-none-any.whl (35 kB)
      [end of output]

  note: This error originates from a subprocess, and is likely not a problem with pip.
error: subprocess-exited-with-error

× pip subprocess to install build dependencies did not run successfully.
│ exit code: -9
╰─> See above for output.

note: This error originates from a subprocess, and is likely not a problem with pip.
WARNING: You are using pip version 22.0.4; however, version 22.2.2 is available.
You should consider upgrading via the '/juice/scr/nlp/crfm/benchmarking/benchmarking/venv/bin/python -m pip install --upgrade pip' command.

@teetone What's the best way for me to reproduce this error? I tried running this on two different machines (one at Columbia and one at Stanford) and in both cases the pre-commit script runs to completion without any errors.

@fladhak @teetone @percyliang Is this resolved based on the Slack conversations? And should we make a note to have a clean conda install or so on to avoid the issues encountered in the README?

Yes, I think this is resolved. Percy was able to build without any errors. I think the issue is due to conflicts with previous packages in venv. Deleting the venv folder and re-installing the codebase should fix the issues.

kk can you make a note in the readme and then close this? Thanks!

Sure!
Repository: stanford-crfm/helm
Issue: Inspect `pytest` warnings
GitHub labels: p3

Running `venv/bin/pytest` returns 19 warnings - it may be good to ensure that we are aware of these / address them if necessary.

Priority: possibly a `P2`?

Flagging as P3 for now, indeed would be good, but I think the highest priority is getting stuff out that are necessary. And if we have time, we do this as a sanity check for completeness.
Repository: stanford-crfm/helm
Issue: Resolve two exact match issue
Root cause: Documentation Deficiency

Currently, we have two metrics called exact match:
1. https://github.com/stanford-crfm/benchmarking/blob/main/src/benchmark/metrics/basic_metrics.py#L716 - This metric is used for multiple choice separated, and computed by default. It is the correct metric, but I think the name "Exact match" is wrong, since this is checking whether the highest probability the model assigns to any of the answer choices belongs to a correct answer choice. 
2. https://github.com/stanford-crfm/benchmarking/blob/main/src/benchmark/metrics/basic_metrics.py#L133 - This is what I would call the correct exact match metric; it checks if the generated completion matches the reference verbatim. This is never computed by default but should be included in most of the completion-based discriminative scenarios (and is) like MMLU. 

- [ ] Rename the first metric. My proposal is "correctness". Also update the schema, and change existing metrics with display names of "correctness" to mention code (i.e. the APPS/HumanEval metrics). 
- [ ] Verify there is no downstream impact on calibration. @AnanyaKumar
Repository: stanford-crfm/helm
Issue: support adding users / editing quotas
Root cause: Unimplemented Feature Gap

Create the following functions (with corresponding REST endpoints) for managing accounts:
- `Service.create_account(auth)`: creates a new account with a random API key and returns that `Account`
- `Service.get_accounts(auth)`: returns a list of `Account`
- `Service.update_account(auth, account)`: updates the account given by `account.api_key` with the fields in `account`.

Note that these methods should look at `auth` and the corresponding `is_admin` and make sure that the following permissions are enforced:
- If `is_admin = True`, then anything is possible EXCEPT the `used` field in any `Usage` object (because this is changing, we don't want to override it).
- Otherwise, one can only view their account and edit their account (EXCEPT `api_key`, `usages`).
Any disallowed changes are just discarded silently.  This allows the client to have a simple uniform interface for updating an `Account`.
Repository: stanford-crfm/helm
Issue: Replace "canonical" perturbation with composition of benign perturbations
Root cause: Unimplemented Feature Gap

After discussing with @percyliang, we decided to go with a simpler "canonical" perturbation. Namely, instead of using typos and synonyms which can be somewhat unnatural, we will create a simpler perturbation by composing some of the existing "benign" perturbations (e.g., misspellings, spacing, lowercasing).

Sounds good to me; I do like the idea of having some more aggressive and less aggressive variants (because it could provide opportunities for more nuanced/differential findings, e.g. most models are robust to slight issues but not others). But beyond that, I think this is a good change given what we looked at during meeting and we should make sure it lands before do **The Big Run (TM)**
Repository: stanford-crfm/helm
Issue: GHA is having trouble downloading ICE tokenizer model
Root cause: Environment Incompatibility

Disabled the test for now:

https://github.com/stanford-crfm/benchmarking/commit/8d4be3118385a94cf3d8a00b4c21e74e51888de2

Should download from https://cloud.tsinghua.edu.cn/seafhttp/files/e6bfdb59-90cd-4815-8790-a738573502e1/ice_text.model

Might be related: https://github.com/actions/download-artifact/issues/140

This test was re-enabled in GHA in #1834.
Repository: stanford-crfm/helm
Issue: `code:model=code,dataset=humaneval` is failing
Root cause: Validation Gap

  Error when running code:model=code,dataset=humaneval:
Traceback (most recent call last):
  File "/juice/scr/nlp/crfm/benchmarking/benchmarking/src/benchmark/presentation/present.py", line 98, in run
    new_run_specs = run_benchmarking(
  File "/juice/scr/nlp/crfm/benchmarking/benchmarking/src/benchmark/run.py", line 60, in run_benchmarking
    runner.run_all()
  File "/juice/scr/nlp/crfm/benchmarking/benchmarking/src/benchmark/runner.py", line 90, in run_all
    self.run_one(run_spec)
  File "/juice/scr/nlp/crfm/benchmarking/benchmarking/src/benchmark/runner.py", line 129, in run_one
    metric_result: MetricResult = metric.evaluate(
  File "/juice/scr/nlp/crfm/benchmarking/benchmarking/src/benchmark/metrics/metric.py", line 150, in evaluate
    results: List[List[Stat]] = parallel_map(
  File "/juice/scr/nlp/crfm/benchmarking/benchmarking/src/common/general.py", line 183, in parallel_map
    results: List = list(tqdm(executor.map(process, items), total=len(items)))
  File "/u/nlp/anaconda/main/anaconda3/envs/crfm_benchmarking/lib/python3.8/site-packages/tqdm/std.py", line 1195, in __iter__
    for obj in iterable:
  File "/u/nlp/anaconda/main/anaconda3/envs/crfm_benchmarking/lib/python3.8/concurrent/futures/_base.py", line 619, in result_iterator
    yield fs.pop().result()
  File "/u/nlp/anaconda/main/anaconda3/envs/crfm_benchmarking/lib/python3.8/concurrent/futures/_base.py", line 437, in result
    return self.__get_result()
  File "/u/nlp/anaconda/main/anaconda3/envs/crfm_benchmarking/lib/python3.8/concurrent/futures/_base.py", line 389, in __get_result
    raise self._exception
  File "/u/nlp/anaconda/main/anaconda3/envs/crfm_benchmarking/lib/python3.8/concurrent/futures/thread.py", line 57, in run
    result = self.fn(*self.args, **self.kwargs)
  File "/juice/scr/nlp/crfm/benchmarking/benchmarking/src/benchmark/metrics/metric.py", line 79, in process
    self.metric.evaluate_generation(
  File "/juice/scr/nlp/crfm/benchmarking/benchmarking/src/benchmark/metrics/code_metrics.py", line 74, in evaluate_generation
    scores = code_metrics_helper.run_test(root=root, test=completion)  # type: ignore
  File "/juice/scr/nlp/crfm/benchmarking/benchmarking/src/benchmark/metrics/code_metrics_helper.py", line 91, in run_test
    input_output_fname = os.path.join(root, "input_output.json")
  File "/u/nlp/anaconda/main/anaconda3/envs/crfm_benchmarking/lib/python3.8/posixpath.py", line 76, in join
    a = os.fspath(a)
TypeError: expected str, bytes or os.PathLike object, not NoneType
Repository: stanford-crfm/helm
Issue: Renew Perspective API key by 7/30/2022
Root cause: Others

The [current API key](https://console.cloud.google.com/apis/api/commentanalyzer.googleapis.com/overview?authuser=1&project=hai-gcp-models) we are using in production was created with the `hai-gcp-models` account and allows 200 queries per second:


"Thanks for sharing how you're using Perspective and contacting us about a quota increase. We reviewed your request and are happy to grant your project 200 queries per second until **7/30/2022**."


When renewing, fill out the [form]( https://docs.google.com/forms/d/e/1FAIpQLSdhBBnVVVbXSElby-jhNnEj-Zwpt5toQSCFsJerGfpXW66CuQ/viewform) with your Stanford email address. For GCP ID, put down `hai-gcp-models`.

As of August 3, 2022:

![Screen Shot 2022-08-03 at 5 30 33 PM](https://user-images.githubusercontent.com/16793796/182737905-e6954dce-2251-45fd-b65f-784ce091319d.png)

12,000 requests / minute = 200 requests / second

Do we need another issue to remind us about the next time we have to renew?

> Do we need another issue to remind us about the next time we have to renew?

This issue is just to increase the rate limit. The key itself will still be valid. I think the increased rate limit will last until October. I'm not sure if we will need it past October.
Repository: stanford-crfm/helm
Issue: Occasional ConnectionError: nodename nor servname provided, or not known
Root cause: Resource Mishandling

A re-run usually solves the issue.
Command: venv/bin/benchmark-run -r mmlu:subject=philosophy
![image](https://user-images.githubusercontent.com/42804517/150281621-bd70109e-6733-467c-84f7-6b2f35e96ce6.png)


@YianZhang, is this still happening for you? I haven't seen it for some time now.

@teetone I haven't seen it for some time either!

@teetone Oh no I just experienced this again a minute ago!

Don't see this anymore.
Repository: stanford-crfm/helm
Issue: sort table rows by accuracy
Root cause: Unimplemented Feature Gap

Update: click on a column to sort by it in the frontend.

All code changes should be made in `summarize.py`.

1) Add `lower_is_better` to the `Metric` schema (true for edit distance, false for exact_match; default is false).

2) Make this show up in the group tables (append ⇧ to the metric display names).

3) In tables for each group where each row is a model, sort by the first metric.

Can we close?

Maybe not yet? There are two things I want to do:
- sort latex tables according to paper order
- add a button to sort tables by any column in the frontend (probably p2-3)

@dtsip what's the status on this?

I will finish the paper order part today! The frontend is a bit lower on my list

The paper order is added in #989 

What's the status?

We are currently ordering models according to the fixed order we have in the paper (alphabetical by org and roughly reverse size within that), so this part is done. (We can always switch to a performance based ranking.)

One nice feature would be to be able to sort table by clicking on column names in the frontend. I want to take a stab at this but definitely lower priority (p3?).

I think we can keep the default order as the alphabetical order.  Would be good to bold the top numbers though.

Let's keep this issue about sorting by rows, which is not urgent.  Separately, I do think it'd be good to indicate lower_is_better on the table and bolding columns would be nice.

#1096 adds the lower_is_better arrows so this should be visible when we summarize next. Let me try to add bold in the next frontend PR.

@yifanmai will take over the frontend

Right now, some columns shouldn't be bolded if lower_is_better is None.

With https://github.com/stanford-crfm/helm/pull/1240, each header field has a `lower_is_better` that is None when sorting makes no sense and True/False otherwise, depending on the directionality of the metric.

Leftover things to do:

- Use lower_is_better field instead of string matching
- Larger sort click target
- Visual indication of column is used for sorting
- Add sorting to URL

We should add sorting to URL before the release because we want to point people to the sorted version.  Either that or we just make sorting by the first column the default.

I think this is just frontend that @yifanmai is handling for now? Reassigned but lmk if I missed something.

Closing for now, since the URL is more complicated.
Repository: stanford-crfm/helm
Issue: Stereotypical associations don't get visualized
Root cause: Validation Gap

Should be fixed by #838. 

@dilarasoylu Closing, but please check this is now fixed, thanks!

@dilarasoylu @ryanachi 

Looks like they computed: https://crfm-models.stanford.edu/static/benchmarking.html?runs=undefined&runSpec=.*cnn.*model%3Dopenai.*#metrics

But they don't show up on website: 
https://crfm-models.stanford.edu/static/benchmarking.html?group=summarization_cnndm

Presumably some issue with schema?

@dilarasoylu @ryanachi 
Looks like they are computed but the metric is named improperly when computed (there's a missing comma):
https://crfm-models.stanford.edu/static/benchmarking.html?runs=undefined&runSpec=summarization_cnndm%3Atemperature%3D0.3%2Cdevice%3Dcpu%2Cmodel%3Dai21_j1-jumbo#metrics

I am a bit confused, since it looks like it should be included in https://github.com/stanford-crfm/benchmarking/blob/main/src/benchmark/metrics/bias_metrics.py#L98. 

At any rate, can you fix this/figure out what is wrong? Seems like it should only take a few minutes.

Hi @rishibommasani, yes, the fix from earlier included the comma fix. Do you know if these results came in after we merged the fix? If not, the metric will update once we re-run.

Could you run some scenario that involves these metrics locally, visualize the results locally, and post a screenshot that shows the comma is fixed on the website? Seems like it is fixed, but want to be certain
Repository: stanford-crfm/helm
Issue: Note why our truthfulqa results are different from prior work
Root cause: Documentation Deficiency

Mention the fancy prompting/etc. See https://docs.google.com/document/d/1XaZJzhiGidInTC5LKYfQO9wYyFIrgmn30x1YNH9_rYE/edit?disco=AAAAdy8bV48

Should just be 1-2 sentences. 

resolved - 0-shot vs 5-shot, threw in ablation just to confirm, but more importantly discussed with owain
Repository: stanford-crfm/helm
Issue: Consider setting `num_outputs=5` for Cohere + RealToxicityPrompts
Root cause: Configuration Error

Cohere only allows up to num_outputs=5. See if numbers change drastically.

Are we able to even run RealToxicityPrompts on Cohere because of https://docs.cohere.ai/usage-guidelines/?

I think we could (we could also do the thing we did with mt-nlg of simulating more completions ourselves), though it is a bit tricky for comparing the numbers to other models as a result.

Let's wait until we get a response to Percy's email/filling out the form, and assuming they even grant us access to the stress-testing (which hopefully means they waive the filter), we can decide?

I tried different `num_outputs` or `n` with `openai/davinci`:

| Run            | # of eval instances | n | Expected max toxicity (toxic) | Expected max toxicity (non-toxic) | Toxicity probability (toxic) |Toxicity probability (non-toxic) |
| ----------- | ----------- |  ----------- | ----------- | ---------- |  ----------- | ----------- |
| From paper | All | 25 | 0.75 | 0.52 | 0.87 | 0.50 | 
| CRFM (1/2022) | 100 | 25 | 0.69 | 0.47 | 0.83 | 0.46 | 
| CRFM (v4) | 1000 | 25 | 0.58 | 0.39 | 0.64 | 0.29 | 
| CRFM (v4) | 1000 | 16 | 0.52 | 0.35 | 0.50 | 0.24 |
| CRFM (v4) | 1000 | 5 | 0.38 | 0.24 | 0.31 | 0.12 | 

I think the reason why we can't reproduce the original numbers is that Perspective can change over time. From https://support.perspectiveapi.com/s/about-the-api-faqs, "...you may see that a specific score changed due to an update. Note that we are not able to notify users each time an update is released."

"Toxicity probability" is the probability of generating at least one toxic completion over `n` generations. Setting `n` to such a high number (e.g., 25) might make it seem like the models are more toxic than they are. Maybe we should also compute the average toxicity score. What do you think @rishibommasani?

Ya, the toxicity probability thing is from the realtoxicityprompts paper which I agree makes it artificially seem like the model is very toxic. 

I think we should depict the n somewhere, so it is at least clear (e.g. at least 1 toxic in 25 is clearly different from at least 1 toxic in 5). 

average toxicity score seems good for me; I think the issue is the toxicity classifier itself is poorly calibrated from what i was told, so thresholding it at 0.5 might be better and we should then maybe just report fraction of instances where it is >0.5?
Repository: stanford-crfm/helm
Issue: update `contamination.yaml`
Root cause: Unimplemented Feature Gap

Mark any (scenario group, model) as contaminated as follows:
- weak: overlap based on n-gram counts
- medium: model's training data contains ~i.i.d. copy of the scenario group (e.g., train split of a dataset)
- strong: model literally contains the test data

This is an easy change, but will do after Sept 15. My focus until then is actually getting the paper to be able to be sent out to the most favorable external party (e.g. Jacob S) by then, and then will cycle back to some of the pending issues wrt frontend/documentation/etc. 

I think this is done.
Repository: stanford-crfm/helm
Issue: Many duplicates of same metric on results page of grouped run_specs
Root cause: Algorithmic Error

https://crfm-models.stanford.edu/static/benchmarking.html?runSpec=synthetic_efficiency%3Atokenizer%3Dai21_j1%2Cnum_prompt_tokens%3D1536%2Cnum_output_tokens%3D1%2Crandom%3DNone%2Cmodel%3Dai21_j1-jumbo%7Csynthetic_efficiency%3Atokenizer%3Dai21_j1%2Cnum_prompt_tokens%3D1536%2Cnum_output_tokens%3D2%2Crandom%3DNone%2Cmodel%3Dai21_j1-jumbo%7Csynthetic_efficiency%3Atokenizer%3Dai21_j1%2Cnum_prompt_tokens%3D1536%2Cnum_output_tokens%3D4%2Crandom%3DNone%2Cmodel%3Dai21_j1-jumbo%7Csynthetic_efficiency%3Atokenizer%3Dai21_j1%2Cnum_prompt_tokens%3D1536%2Cnum_output_tokens%3D8%2Crandom%3DNone%2Cmodel%3Dai21_j1-jumbo%7Csynthetic_efficiency%3Atokenizer%3Dai21_j1%2Cnum_prompt_tokens%3D1536%2Cnum_output_tokens%3D16%2Crandom%3DNone%2Cmodel%3Dai21_j1-jumbo%7Csynthetic_efficiency%3Atokenizer%3Dai21_j1%2Cnum_prompt_tokens%3D1536%2Cnum_output_tokens%3D32%2Crandom%3DNone%2Cmodel%3Dai21_j1-jumbo%7Csynthetic_efficiency%3Atokenizer%3Dai21_j1%2Cnum_prompt_tokens%3D1536%2Cnum_output_tokens%3D64%2Crandom%3DNone%2Cmodel%3Dai21_j1-jumbo

![image](https://user-images.githubusercontent.com/2724038/189504689-e780ab68-794e-4159-8826-8c63225f3ea7.png)


This happens when you try to visualize multiple models at once.
Repository: stanford-crfm/helm
Issue: ECE metric not showing up for tasks where adapter method = generation (e.g., boolq)
Root cause: Unimplemented Feature Gap


To add a bit here, there are some tasks with adapter method = generation where calibration does not make sense (e.g. text summarization). 

So a necessary property  is that we should compute `exact_match` as one of the metrics for the scenario (which we do for boolq, for example, but do not do for summarization), then we probably should be able to measure calibration (but currently don't because we only measure calibration when a `multiple_choice` adapter is used). 


I guess what makes this trickier is I don't believe computing `exact_match` is sufficient since, as I understand, the calibration metrics need a finite and known space of choices, since this is used to normalize the model probability (though maybe this not strictly needed? I guess interpreting the models raw probability of the completion might not be good though, but should presumably always be an optinon?) And not every scenario that uses `generation` and computes `exact_match` has an obvious finite/known label space, e.g. **DataImputation**, where the model generates the answer to a question about the missing entry in a data table (see https://crfm-models.stanford.edu/static/benchmarking.html?runs=undefined&runSpec=entity_data_imputation%3Adataset%3DBuy%2Cmodel%3Dai21_j1-jumbo for example). 
Unlike BoolQ, there is no obvious/predetermined label space (afaik), but the primary metric is (quasi) exact match. 


One option is we could annotate somewhere whether a scenario is amenable to calibration, with MC always being amenable (and so we always measure calibration, which is already implemented) and generation only amenable when specified. 
Or we could try to have a more general approach for measuring calibration (or maybe I have misunderstood what we need to measure calibration in the generation settings, so please correct me) that will work without this annotation.



I think all you need for ECE (top-1 calibration) is meaningfulness of `exact_match` and the ability to compute the probability over the answer (which is fine for generation).

#890 completes this!
Repository: stanford-crfm/helm
Issue: TruthfulQA results comparison
GitHub labels: p1

Paper

Could you update the entries here with results from prior work that evaluates on TruthfulQA, specifically under conditions that best match our evaluation?
https://docs.google.com/spreadsheets/d/1jpm0Cy0r5Yk_l9o9kwVpxbSfdKrx6ZcQ7iQfs7QYPIc/edit#gid=1233800927


Once done, can you compare to our results
https://crfm-models.stanford.edu/static/benchmarking.html?group=truthful_qa
and further update the spreadsheet with colors for the cell text:
Bold: Our results match (+/- 1, maybe up to 2 pts) prior work for the same model
Pink and Bold: Our results are worse than prior work for the same model
Blue and Bold: Our results are better than prior work for the same model

Seems like Instruct-Davinci is very good, seemingly more so than indicated in
https://www.lesswrong.com/posts/yYkrbS5iAwdEQyynW/how-do-new-models-from-openai-deepmind-and-anthropic-perform


E.g. if the metric we measure is the one reported in this figure, it seems like it utterly crushes/invalidates this linear projection? (Which we should note/make as an important point if true in the paper)

What's the status here?

This is addressed, what remains is just discussion in the paper. We have discussed with the dataset author (Owain Evans) as well to provide further confidence in what we are doing.
Repository: stanford-crfm/helm
Issue: allow API to return embeddings
Root cause: Unimplemented Feature Gap

- See which providers have embedding APIs.
- Modify `Request` to have an `include_embeddings` flag or something.
- Modify `RequestResult` to have an embeddings.
- Change the `openai_client.py` etc. to generate the proper results.

This is supported (but only by OpenAI).
Repository: stanford-crfm/helm
Issue: Support fine-tuning for OpenAI
Root cause: Unimplemented Feature Gap

https://beta.openai.com/docs/guides/fine-tuning

I don't think _creating_ fine-tuned OpenAI models should be done through HELM. We will have support for _evaluating_ OpenAI fine-tuned models very soon, as part of supporting OpenAI models with arbitrary configurations (see #1415)