zhiminy/EvalEng huggingface dataset:
Repository: stanford-crfm/helm
Issue: Preserve metric fields when computing `perplexity`, `bits_per_byte`, and `logprob_per_byte`
Root cause: Validation Gap

## Purpose

This is a follow up to #597. We should ensure that all the fields are passed properly to the newly created metric names [here](https://github.com/stanford-crfm/benchmarking/blob/main/src/benchmark/metric.py#L165).
Repository: stanford-crfm/helm
Issue: Preserve all metric fields when computing the trial stats
Root cause: Architectural Constraint

## Purpose

This is a follow up to #597. We want to ensure that all the properties of an instance (not just the split) are passed to the metric in the evaluate function of metric.py at this [line](https://github.com/stanford-crfm/benchmarking/blob/dbf3355af89c48996db63f6905deb5beada51cf6/src/benchmark/metric.py#L116). In case the fields of the `MetricName` class change again, we should not count on manually assigning the field values.

(This issue is addressed in #680)
Repository: stanford-crfm/helm
Issue: Code to plot figures for paper
Root cause: Unimplemented Feature Gap

1. Decide what figures we want. This will largely be based on structural concepts we want to show, which are agnostic to seeing the results (e.g. we want to know the relationship between acc. and robustness independent of how it looks).
2. Write code to plot these figures.

@dtsip Should we close this based on #772? I think 1 + 2 will be addressed (I guess the act of copying tables over to paper won't, but I think that is fine), and 3+4 are probably a different matter. Or we can keep open, just flagging, up to you

Might as well keep this open instead of opening a new one for plots, but feel free to do whatever you think is best (as long as something remains open 🙃 )

Sg, I guess there is a bunch of paper stuff that just isn't tracked by issues for the codebase, but ya if we need plotting code we as may as well keep open. Will just rename the issue

@dtsip @deepakn94 How is stuff going on this front?

Can we close this now since we have plots now?  I think we should close this and then open up more specific issues for various plots.

That's fine by me. I was planning to integrate the plot code nicely into the codebase, and then close this issue. This is not P1 though.

Ok, would be nice to lean up the code, but can do it after release (moving to P3).
Repository: stanford-crfm/helm
Issue: Automatic faithfulness eval
Root cause: Unimplemented Feature Gap

1. We should run automatic faithfulness evaluations ASAP now that we have results from several models.
2. We should integrate the metrics code in the codebase and not have it be special-cased in anyway. For the purposes of running, it is fine for it be in runs that require GPUs, and Tony can handle this aspect of running, but beyond that it should be identical to any other metric. 
3. While on the topic, we should clean up the TODOs in https://github.com/stanford-crfm/benchmarking/blob/main/src/benchmark/run_specs.py#L1309 related to summarization.

@fladhak Let me know if this makes sense and what the timeline is for finishing this, thanks!

Also pinging @esdurmus and @Tiiiger so they are aware, as this will be a prerequisite to lauching human eval.
Repository: stanford-crfm/helm
Issue: Faithfulness metrics - GPUs
Root cause: Unimplemented Feature Gap

List `RunSpecs` that need GPUs at `src/benchmark/presentation/run_specs_gpu.conf`. I will try run those separately.

@fladhak I also noted this in #747, but remember to add a conf like Tony mentions with just the summarization scenarios with the `device=gpu` set as the argument, and then we can keep the existing `run_specs.conf` the same.
Repository: stanford-crfm/helm
Issue: Fix and reenable`test_ai21_window_service.py`
Root cause: Unimplemented Feature Gap

Remove pickle file and create a file with constants and expected values

@YianZhang Instead of mocking, a cleaner and easier way is to populate the cache with actual requests/responses before running the test. I recently did this for Cohere: https://github.com/stanford-crfm/benchmarking/pull/799/files#diff-0979f22ceab7d85df0a6f4cdd8bb3f4be04d41053baa43239eb0761a5d8e710f.

@YianZhang What is the status on this?

@YianZhang @teetone Is this an issue/can we close or get a status update?

Moving to p3 seems it doesn't seem urgent for release.

We no longer need `AI21WindowServiceTest` since we are switching from the remote tokenization service to a local tokenizer. The test will be removed by #2893.

Test has been removed.
Repository: stanford-crfm/helm
Issue: Update frontend schema fields
GitHub labels: p1

## Purpose

Make the following changes to the front-end schema:

- [x] Create new `Field` subclasses for `Metric`s, `Perturbation`s, `TableSetting`s etc. instead of overloading the `values` field.
- [x] For a given group, `k` shouldn't be set to one value for all the stats. Instead, it should depend on the stat name. We should specify the default `k` along with `stat names`.
- [x] Group names shouldn't include spaces.
- [x] Update `BiasMetric` / `BBQMetric` stat names.
- [x] Rename `stat_group`s.
Repository: stanford-crfm/helm
Issue: Anthropic + RealToxicityPrompts failing
Root cause: External Dependency Breakage

![Screen Shot 2022-08-09 at 3 59 38 PM](https://user-images.githubusercontent.com/16793796/183776197-09991574-ba79-4537-a0c2-59e21a11a9d1.png)
Repository: stanford-crfm/helm
Issue: add The Pile for measuring perplexity
Root cause: Unimplemented Feature Gap

Add `ThePileScenario` which allows you to sub select different eval sets.  Do what Gopher did.  Make sure we're not downloading more than needed.

Replicate numbers for Jurassic.
https://uploads-ssl.webflow.com/60fd4503684b466578c0d307/61138924626a6981ee09caf6_jurassic_tech_paper.pdf
Repository: stanford-crfm/helm
Issue: MT-NLG GPT-2 tokenizer doesn't encode `
` to a single token
Root cause: Validation Gap

Many of the language modeling requests fail because they exceed the max context length. We need a workaround.

In mt_nlg_window_service.py:

```python
    @property
    def prefix_token(self) -> str:
        """TODO: add explanation why we picked this value for the prefix token."""
        return "TODO"
```

@teetone It seems I can't run anything with MT-NLG. I ran `venv/bin/benchmark-run -r mmlu:subject=philosophy -m 2` and got this error: `http.client.RemoteDisconnected: Remote end closed connection without response`. Maybe there is another bug with the model?
Repository: stanford-crfm/helm
Issue: Add Twitter AAE
Root cause: Unimplemented Feature Gap

Used in the Gopher paper for measuring perplexity.

January 25 update: The LM pipeline has been implemented in #79. TODO: Implement the data filtering process that extracts tweets of different demographics from the raw data.

@YianZhang why was this issue reopened? Should we close it?

@teetone It was reopened because the current implementation of TwitterAAE is not complete.

@YianZhang Can you update the issue description with what's left?

> @YianZhang Can you update the issue description with what's left?

@teetone Done!

> > @YianZhang Can you update the issue description with what's left?
> 
> @teetone Done!

Thank you!

@YianZhang What's the status of this issue? Are you actively working on this?

@teetone Rishi, Percy, and I have reached out to Brendan about replication of the dataset and we are still waiting for the response.

Update:  @YianZhang is checking this.
Repository: stanford-crfm/helm
Issue: ModuleNotFoundError: No module named 'benchmark' / Reenable tests in test_remote_service.py
Root cause: Configuration Error

It passes locally for now, but not sure why it started falling suddenly in GHA starting August 11, 2022:

https://github.com/stanford-crfm/benchmarking/pull/730/files

Installing the dependencies worked, but cannot find the module?

```
0s
Run venv/bin/benchmark-present --skip-instances --models-to-run openai/davinci --exit-on-error
Traceback (most recent call last):
  File "venv/bin/benchmark-present", line 5, in <module>
    from benchmark.presentation.present import main
ModuleNotFoundError: No module named 'benchmark'
```

@dtsip and I spent sometime working on this yesterday with not much luck. It seems like the `benchmarking` directory is not added to the `venv` Python path.

Yes, no idea why this is happening.

For now, I added `sys.path.append("/path/to/benchmarking/src")` in `venv/bin/benchmark-run` which is not great. Ideally this should be handled by the `pip install -e .`

(also, not sure if it relevant, but at the same time, I started getting weird errors after the run 
```
Fatal error condition occurred in /opt/vcpkg/buildtrees/aws-c-io/src/9e6648842a-364b708815.clean/source/event_loop.c:72: aws_thread_launch(&cleanup_thread, s_event_loop_destroy_async_thread_fn, el_group, &thread_options) == AWS_OP_SUCCESS
Exiting Application
#######################################################################
Stack trace:
########################################################################
/sailhome/tsipras/benchmarking/venv/lib/python3.8/site-packages/pyarrow/libarrow.so.900(+0x200af06) [0x7f62f327ff06]
...
/sailhome/tsipras/benchmarking/venv/bin/python(_start+0x2e) [0x5fa5ce]
Aborted
```

> Yes, no idea why this is happening.
> 
> For now, I added `sys.path.append("/path/to/benchmarking/src")` in `venv/bin/benchmark-run` which is not great. Ideally this should be handled by the `pip install -e .`
> 
> (also, not sure if it relevant, but at the same time, I started getting weird errors after the run
> 
> ```
> Fatal error condition occurred in /opt/vcpkg/buildtrees/aws-c-io/src/9e6648842a-364b708815.clean/source/event_loop.c:72: aws_thread_launch(&cleanup_thread, s_event_loop_destroy_async_thread_fn, el_group, &thread_options) == AWS_OP_SUCCESS
> Exiting Application
> #######################################################################
> Stack trace:
> ########################################################################
> /sailhome/tsipras/benchmarking/venv/lib/python3.8/site-packages/pyarrow/libarrow.so.900(+0x200af06) [0x7f62f327ff06]
> ...
> /sailhome/tsipras/benchmarking/venv/bin/python(_start+0x2e) [0x5fa5ce]
> Aborted
> ```

I did another fresh install on my laptop and was able to run `venv/bin/benchmark-run`. I'm not sure why the cluster and GHA pipeline are different.

(I can reproduce this error on my local machine - I'm not sure what changed from before)

+1 to Dilara; I'm running into the same error on my local machine.
Repository: stanford-crfm/helm
Issue: add BIG-bench tasks as a scenario
Root cause: Unimplemented Feature Gap

We can create a `BIGBenchScenario` which specifies the name of a BIG-bench task with any `task.json`.  The `run_specs.py` would have to also read the `task.json` and read out the prompt (hopefully this is not too slow).

We could add all of BIG-bench Lite. or identify a few tasks that are complementary to the ones we already have.

@percyliang, I can take this one. I've been looking into BIG-bench and interested in working on this.
Repository: stanford-crfm/helm
Issue: Accuracy calculation for MULTIPLE_CHOICE_SEPARATE_* is incorrect when there are ties
Root cause: Algorithmic Error

The accuracy calculation in https://github.com/stanford-crfm/benchmarking/blob/main/src/benchmark/basic_metrics.py#L664 is incorrect when there are ties. For example, in binary classification if my model's predicted probabilities are [0.5, 0.5], then this code will say we got the example correct, regardless of what the answer is, because the calculation max(reference_scores) == max(answer_scores) will always return True. This is admittedly a low probability event, so it's minor, but we should probably fix it just in case!

What is the behavior we want? That the model is wrong? Or that we pick a random number from 1-k where the model selects the kth option amidst the argmax set as its prediction? 

I think it should pick one of them (but ideally deterministically), e.g., pick the smallest label if there are ties.

Why not award credit 1/(# items with in the argmax), so it's like expected accuracy?

That's a good idea too! I'm just thinking that it might complicate calculating and defining some downstream metrics (like calibration), so was leaning towards picking a deterministic prediction with an arbitrary tie-breaker, but I agree that your method is a bit more principled

ya, my intuition was the thing percy said is the best in isolation but would be unideal without more general support, in that it might disturb downstream dependencies like calibration (and doesn't seem worth building support at this moment; I think we might want support if we end up having weighted references for big-bench, but that's not a top priority). 

So I think just having some way to break ties (why is strict determinism important here?) would be good. Also imo it is the most realistic (i.e. you should not need to assume downstream that some interface built on an FM can always accommodate multiple argmax-s as the prediction). 

I was thinking of using the fractional metric just to compute exact_match.  Then we deterministically pick a single prediction which we can use downstream for other metrics such as calibration...  Do you think that'd be confusing?

This was fixed by #1494.
Repository: stanford-crfm/helm
Issue: `fits_tokens_within_context_window` (language modeling) loops infinitely for `together/yalm`
Root cause: Algorithmic Error

Gets stuck for certain prompts in the following RunSpecs:

```
"the_pile:model=full_functionality_text,subset=Github": {status: "READY", priority: 2}
"the_pile:model=full_functionality_text,subset=PubMed Central": {status: "READY", priority: 2}
"the_pile:model=full_functionality_text,subset=Wikipedia (en)": {status: "READY", priority: 2}
```

@YianZhang, do you mind taking a look? I debugged a bit, and I think it's because `conditioning_tokens` is empty, and we stopped truncating the prompt, so it never exits here: https://github.com/stanford-crfm/benchmarking/blob/main/src/benchmark/adapter.py#L691.

Loops infinitely here: https://github.com/stanford-crfm/benchmarking/blob/main/src/benchmark/adapter.py#L650

@teetone What was the command that caused this error?

> @teetone What was the command that caused this error?

@YianZhang Should be able to reproduce by running one of the RunSpecs in the issue description above.

@teetone and I found that the problem happens when the input contains long Chinese texts. YaLM and its tokenizer do not handle Chinese (and many other languages) properly so encoding then decoding a string might make it much longer. This prevents us from truncating texts correctly.

e.g. 
With OpenAITokenizer:
```
>> tokenizer.decode(tok.encode("行星运转"))
'行星运转'
```

With YaLM's tokenizer:
```
>>> tokenizer.decode(tok.tokenize("行星运转"))
'行<0xE6><0x98><0x9F><0xE8><0xBF><0x90><0xE8><0xBD><0xAC>'
```

Since the model itself only supports a few languages by design, we might consider not running it on some multi-lingual scenarios/subsets in the first place.
Repository: stanford-crfm/helm
Issue: Truncation failed for GLM + NaturalQA
Root cause: Algorithmic Error

Error when running natural_qa:model=text,mode=openbook-longans,data_augmentation=canonical:
Traceback (most recent call last):
  File "/juice/scr/nlp/crfm/benchmarking/benchmarking/src/benchmark/presentation/present.py", line 126, in run
    new_run_specs = run_benchmarking(
  File "/juice/scr/nlp/crfm/benchmarking/benchmarking/src/benchmark/run.py", line 75, in run_benchmarking
    runner.run_all()
  File "/juice/scr/nlp/crfm/benchmarking/benchmarking/src/benchmark/runner.py", line 89, in run_all
    self.run_one(run_spec)
  File "/juice/scr/nlp/crfm/benchmarking/benchmarking/src/benchmark/runner.py", line 110, in run_one
    scenario_state: ScenarioState = adapter.adapt(instances)
  File "/juice/scr/nlp/crfm/benchmarking/benchmarking/src/common/hierarchical_logger.py", line 104, in wrapper
    return fn(*args, **kwargs)
  File "/juice/scr/nlp/crfm/benchmarking/benchmarking/src/benchmark/adapter.py", line 340, in adapt
    prompt = self.construct_prompt(
  File "/juice/scr/nlp/crfm/benchmarking/benchmarking/src/benchmark/adapter.py", line 608, in construct_prompt
    prompt = self.window_service.truncate_from_right(prompt, self.adapter_spec.max_tokens)
  File "/juice/scr/nlp/crfm/benchmarking/benchmarking/src/benchmark/window_services/local_window_service.py", line 95, in truncate_from_right
    assert num_tokens <= max_length, f"Truncation failed ({num_tokens} > {max_length}). Input text: {text}"
AssertionError: Truncation failed (1752 > 1749). Input text: Passage: 

Overall No.
Saga No.
English translation[a]
Original Japanese title
Original airdate[27]
American airdate


77
1
"Let's Do It, Zen-Oh Sama! The All-Universe Martial Arts Tournament!!"
"Yarou ze Zenō-sama! Uchū ichi Budō-kai!!" (やろうぜ全王様! 宇宙一武道会!!) 
February 5, 2017
TBA
....
Repository: stanford-crfm/helm
Issue: Metadata on models (known train-test contamination, reference scores to compare against)
Root cause: Unimplemented Feature Gap

For each model we benchmark, drawn for the list below, I will:
**Model list:**

- [ ] **Contamination.** Identify any scenarios that are explicitly disclosed as contaminated in their work. As the operational contamination condition, I will take any mention of training on the scenario as a sign of contamination (i.e. I won't distinguish only training on the training set vs. not, or only training on inputs vs inputs + labels, etc.). I will also include some metadata if its not too cumbersome about the nature of the overlap, but the top line item will be a binary marker of overlap
- [ ] **Overlap in results.** Identify scenarios in the overlap between their evaluation and our benchmark. Provide reference results for these. With these in hand, we can then verify our results match (I won't do this for right now). Since the evaluation conditions in prior work likely deviate from ours, I will also ideally provide some annotation if it proves to be not too cumbersome.

A corollary is I plan to the second thing in full thoroughness (i.e. identify the full overlap, not just 2 or so datasets), because this will mean we can explicitly say what the preceding coverage is of these datasets (and hence what was missing). I.e. we will go from a sparse matrix of scenarios x models where we had results to a complete matrix (and of course this is only for accuracy since none of these works will have looked at any other metric for these scenarios). So we will get a concrete quantitative measure of what has changed, which will be a nice top-level thing in the paper.
Repository: stanford-crfm/helm
Issue: [HOLD] Ensure `k` is set properly for all the metrics
Root cause: Validation Gap

`metric.py` is adding the necessary `MetricName` context when custom metrics implement one of the following functions: `evaluate_generation`, `evaluate_references`, and `evaluate_instances`. That being said, it expects the metric themselves to handle setting `k` (CC @dtsip). In the long run, I think it would be nicer to make our metric interface easier to work with and abstract `k` away... In the short run, we should double check that all the calls to `MetricName` sets `k` properly. If this is not the case, we should ensure that there is a post processing function that replaces the `k` for the metric name object that's created without a `k`. Here is a checklist:

```
# Committed
[ ] basic_metrics.py
[ ] bbq_metrics.py
[ ] bias_metrics.py
[ ] code_metrics.py
[ ] copyright_metrics.py
[ ] disinformation_metrics.py
[ ] multiple_requests_metrics.py
[ ] numeracy_metrics.py
[ ] tokens_metrics.py
[ ] toxicity_metrics.py

# In a PR
[ ] summarization_metrics.py
```

Relevant to @dtsip @rishibommasani @percyliang 

Subsumed by #754
Repository: stanford-crfm/helm
Issue: [HOLD] Code metrics should set `k`
Root cause: Validation Gap

Hi @lxuechen and @q-hwang,

Creating this issue as part of #748 (CC @dtsip).

Metrics that use `request_results.completions` should also pass an integer value to `k` parameter of the `MetricName` class. This specifies the number of completions we have considered per request when computing a metric. From what I can tell, this is not the case for `code_metrics.py` [[relevant line](https://github.com/stanford-crfm/benchmarking/blob/92ffd093c4cc9ad540ee08bb5bf6afdb64fe2cf3/src/benchmark/code_metrics.py#L79)].

Does it make sense to compute the `APPSMetric` using different `k` values? If so, which of these would you like to have as the main code metric `k` value visualized in the paper / website?

(Actually, we may want to hold on this change, I will follow up)
Repository: stanford-crfm/helm
Issue: [HOLD] Copyright metrics should set `k`
Root cause: Configuration Error

Hiya @lxuechen,

Similar to #751, we need to set `k` for the `CopyrightMetric` [here](https://github.com/stanford-crfm/benchmarking/blob/92ffd093c4cc9ad540ee08bb5bf6afdb64fe2cf3/src/benchmark/copyright_metrics.py#L122) as it seems to be using completions.

(CC @dtsip)

(Actually we may want to hold on this change, apologies for the notification, I will follow up)
Repository: stanford-crfm/helm
Issue: Create a PyPI package
Root cause: Unimplemented Feature Gap

So users can just do `pip install crfm-benchmarking` and run benchmarking without downloading the code.

@teetone Would be good to have this before launch so people can quickly try things out.

@teetone Will this take time/should we get started on this?

Things to do:

- [x] Move everything into a top level module (e.g. `crfm_benchmarking`) to avoid namespace conflicts
- [x] Prefix command line commands (e.g. `crfm-run-proxy`) to avoid command name conflicts
- [x] Add LICENSE file
- [x] Switch `setup.py` to use only core requirements
- [x] Build and test wheel
- [x] Add project description / README
- [x] Remove Stanford specific instructions

The first two should probably be done after the runs, and will also cause every other branch / PR to have merge conflicts, unfortunately.

Some of these things are done at least, could you update and perhaps break this issue up into multiple ones?

Ran into some transitive issues in calibration: https://github.com/p-lambda/verified_calibration/issues/11
Repository: stanford-crfm/helm
Issue: Modify frontend to reflect calibration
Root cause: Unimplemented Feature Gap

Similar to #768. I imagine some of this is already there. 

The key thing will be that we want this only for the scenarios where calibration is defined. And we will need to decide on the default calibration metric we are visualizing. 

Sounds good!

I think this will be addressed through #776 and the followup PR. 

@AnanyaKumar For the purposes of visualization, we will want 1 calibration metric. Which one should we use as this default?

Let's use ece (will be renamed to ece_10_bin) as per discussion with Percy!
Repository: stanford-crfm/helm
Issue: Support `CohereWindowService` and add character count metric (see pricing)
Root cause: Unimplemented Feature Gap

Follow up to https://github.com/stanford-crfm/benchmarking/pull/791

Generate API: https://docs.cohere.ai/generate-reference
Tokenization: https://docs.cohere.ai/bpe-tokens-wiki/
Pricing: https://cohere.ai/pricing
Repository: stanford-crfm/helm
Issue: Estimate cost for AI21 models
Root cause: Unimplemented Feature Gap

New pricing model: https://studio.ai21.com/pricing

I did a dry run with P1 + P2 RunSpecs and estimated:

```
For 1 random seed:

ai21/j1-large:
cost = pricing.price_per_1k_token * (num_tokens / 1000) + pricing.price_per_request * num_request
$370.26 = $0.03 * (11010522 / 1000) + $0.0003 * 133156

ai21/j1-grande:
cost = pricing.price_per_1k_token * (num_tokens / 1000) + pricing.price_per_request * num_request
$987.37 = $0.08 * (11010522 / 1000) + $0.0008 * 133156

ai21/j1-jumbo:
cost = pricing.price_per_1k_token * (num_tokens / 1000) + pricing.price_per_request * num_request
$3418.41 = $0.25 * (11010522 / 1000) + $0.005 * 133156

Total: $4776.04
```