zhiminy/EvalEng huggingface dataset:
Repository: stanford-crfm/helm
Issue: Process `Killed` when writing out to scenario.json for MS MARCO
Root cause: Resource Mishandling

I believe it's because the MSMARCO instances have many references. 


Error: 

```
    Generated 5 stats.
    Writing 3200 characters to benchmark_output/runs/msmarco/msmarco:track=regular,valid_topk=30,model=together_opt-66b,data_augmentation=canonical/run_spec.json
    Writing 38928517 characters to benchmark_output/runs/msmarco/msmarco:track=regular,valid_topk=30,model=together_opt-66b,data_augmentation=canonical/scenario.json
Killed
```



Is this still happening and what's the impact of us getting results?

> Is this still happening and what's the impact of us getting results?

This is still happening. The temporary workaround is to just run with more memory. I think in the future we might want to generate multiple smaller files.
Repository: stanford-crfm/helm
Issue: Support change temperature for sampling
Root cause: Unimplemented Feature Gap

Support varying temperature for sampling as token-step/time-step increase. Useful for building stronger data extraction attacks as described in [Carlini et al](https://arxiv.org/abs/2012.07805). 

@lxuechen Do we still need this?

I think yes, ideally. This enables the stronger attacks. Not of high priority; I may be able to take a stab at this next quarter. 

This is complicated/specialized enough given the APIs that we have that I think it probably doesn't make sense to support in the general codebase.
Repository: stanford-crfm/helm
Issue: when click 'Prediction', should show the prompt
Root cause: Unimplemented Feature Gap

Now that the benchmarking pages are hosted statically, the 'Prediction' links are broken.  When we click on it, we could either open a new page or expand into a box that shows the prompt, decoding parameters, and the response.
![image](https://user-images.githubusercontent.com/435384/196592829-88c8c858-dba5-462e-bea3-071baf9af90e.png)


Will do this week as part of the #1024 refactor.
Repository: stanford-crfm/helm
Issue: Split out CRFM API proxy client into its own library
Root cause: Unimplemented Feature Gap

Users should be able to pip install a library and get access to the CRFM API proxy.

Also, the client code is currently entangled with the server code; it would be nice to decouple them in the long term.

Also, we should probably raise an exception with a helpful user message for models that we don't currently support in the proxy.

Are we still keeping one codebase, just two different pip packages?  Do we need to do this before release since it's sort of a structural thing that might be annoying to do later?

I think this can happen post launch (it only affects API end users).

closing as duplicate of #1327 which also has a proposal.
Repository: stanford-crfm/helm
Issue: `benchmark.metrics.code_metrics.APPSMetric` is throwing segfaults
Root cause: Algorithmic Error

For `code-cushman-001` and `code-davinci-002`:

```
  File "<string>", line 41 in paint
  File "<string>", line 41 in paint
  File "<string>", line 41 in paint
  File "<string>", line 41 in paint
  File "<string>", line 41 in paint
  File "<string>", line 41 in paint
  File "<string>", line 41 in paint
  File "<string>", line 41 in paint
  File "<string>", line 41 in paint
  File "<string>", line 41 in paint
  ...
Segmentation fault
```

Is this still happening and what's the impact in terms of us getting results?

I figured out a fix just now. I should be able to send in a PR within an hour. 

Edit: PR here https://github.com/stanford-crfm/helm/pull/1054
Repository: stanford-crfm/helm
Issue: Add a guide on how to implement scenarios and metrics to the README
Root cause: Documentation Deficiency

Add it under https://github.com/stanford-crfm/benchmarking#code-structure

Happy to take this one :)

@ezelikman What's the status of this?

I ended up holding off on this since some of the structural details changed (e.g. how we were handling the metrics, how we were handling the splits, the addition of the run config file, etc) and the guide I had written became outdated. Things seem pretty stable now though, so I could give it another shot as a pre-release step.

@ezelikman Seems like things are stable enough now, so could you add now? Thanks!!!

Pinged on Slack, just a reminder to resolve this @ezelikman. 

Here's a PR! https://github.com/stanford-crfm/benchmarking/pull/719
Repository: stanford-crfm/helm
Issue: run T5, T0++, UL2 and GLM models with <br> rather than ###
Root cause: Configuration Error

Currently, the results are with '###' separator.  Rerun everything with the `<br>` (this will be the default after #1039 ) to make everything more consistent.

Decided to use ### everywhere, since `<br>` doesn't work as well anecdotally.
Repository: stanford-crfm/helm
Issue: Benchmark new models
Root cause: Unimplemented Feature Gap

New models to try out:
- [ ] Flan-T5 
- [ ] BLOOMZ
- [ ] AlexaTM
- [ ] New Cohere model
- [ ] New AI21 model
- [ ] OPT 8-bit
- [ ] GLM 8-bit

Wishlist:
- [ ] PaLM (and its many variants)
- [ ] GLaM
- [ ] LaMDA
- [ ] Gopher
- [ ] Chinchilla
- [ ] RETRO
- [ ] Google, Meta MoE models

Close this in favor of individual issues.
Repository: stanford-crfm/helm
Issue: Installation fails due to typing
Root cause: Environment Incompatibility

On a fresh install on Ubuntu 20.04, I get:
```
$ ./pre-commit-venv.sh
...
      self._abc_registry = extra._abc_registry
  AttributeError: type object 'Callable' has no attribute '_abc_registry'
```

To workaround for now:
```
$ pip unstall typing
$ pip install -e .
```

I couldn't reproduce on a fresh install:

```
docker run -it --rm ubuntu:20.04
# In docker
apt-get update
apt-get install git python3 python3-pip wget unzip
git clone https://github.com/stanford-crfm/helm.git
cd helm
./pre-commit-venv.sh
venv/bin/pytest
```

This works fine. But if I install specific versions of pip, it breaks:

```
venv/bin/pip install pip==20.1
./pre-commit-venv.sh
```

The bad range of `pip` versions is 20.1 to 21.0.1 inclusive. Version 21.1 is OK. Looks like it's [this bug](https://github.com/pypa/pip/issues/8272). We should just upgrade pip in the `pre-commit.sh`.
Repository: stanford-crfm/helm
Issue: Add MethodRunExpander for `run_specs_extra.conf`
Root cause: Unimplemented Feature Gap


This issue is a bit complicated and requires some refactoring because we're not just changing a single value in the `AdapterSpec`. I included the method in the `RunSpec` name for now.

RunExpanders are pretty general and can modify the `RunSpec` arbitrarily...can we do that?

> RunExpanders are pretty general and can modify the `RunSpec` arbitrarily...can we do that?

The underlying AdapterSpecs are so different, and the joint one takes in `instructions`, `input_noun`, and `output_noun`:

<img width="909" alt="Screen Shot 2022-10-24 at 3 04 01 PM" src="https://user-images.githubusercontent.com/16793796/197639089-70151fc4-c943-4066-b899-fbab650b653f.png">



Can't we call these functions from the RunExpander?

This doesn't really type-check because not all adapters work with all run spec functions. Currently this is implemented by passing a method argument to the run spec function, which seems fine.
Repository: stanford-crfm/helm
Issue: Allow authenticated users who used their quota to make cached requests
Root cause: Unimplemented Feature Gap

`check_can_use` currently prohibits this:

```python
    def make_request(self, auth: Authentication, request: Request) -> RequestResult:
        """Actually make a request to an API."""
        # TODO: try to invoke the API even if we're not authenticated, and if
        #       it turns out the results are cached, then we can just hand back the results.
        #       https://github.com/stanford-crfm/benchmarking/issues/56

        self.accounts.authenticate(auth)
        model_group = get_model_group(request.model)
        # Make sure we can use
        self.accounts.check_can_use(auth.api_key, model_group)

        # Use!
        request_result: RequestResult = self.client.make_request(request)

        # Only deduct if not cached
        if not request_result.cached:
            # Count the number of tokens used
            count: int = self.token_counter.count_tokens(request, request_result.completions)
            hlog(f"{request.model_organization}: {count} tokens")
            self.accounts.use(auth.api_key, model_group, count)

        return request_result
```

Haven't gotten any user requests for this (and also we may be getting out of the playground API business). Closing.
Repository: stanford-crfm/helm
Issue: download predictions + data 
Root cause: Documentation Deficiency

> We've been mainly manually looking at your results website and predictions (Which is great by the way). Is there some place we can get this also as raw data so we can more automatically analyze it (For example to check the biases to specific choices on the predictions)? We found the json containing the question instances but not the results or the actual inputs to the model (i.e. after constructing the full prompt with the few-shot examples)

@teetone @yifanmai Thoughts? Could be good to have for release (or shortly thereafter), since seems useful for people who want to do meta-analysis. 

@yifanmai Document this or figure out what is needed, since the JSON files should be availab.e

This is available as 'Full JSON'.  Could make this friendlier.
Repository: stanford-crfm/helm
Issue: URL is too long for all models links
Root cause: Architectural Constraint

For the website served by Apache, the current all models links exceed the [Apache limit](https://stackoverflow.com/questions/1289585/what-is-apaches-maximum-url-length), resulting in a Request-URI Too Large error.

Possible options:

- Increase the limit in the Apache configuration
- Use GitHub pages instead
- Instead of all models, pick the first K models and display those instead
- Allow the user to pick which models to display, up to some limit
- Remove the link entirely

Also noting that displaying that many models will result in massive JSON data download anyway (#1024).

Probably having URLs this large is a bad sign, so I think we should probably allow the users to pick a subset of the models and view them.
Repository: stanford-crfm/helm
Issue: `get_platt_scaler` throws `ValueError: Input X contains NaN.`
Root cause: Validation Gap

```
  File "/juice2/scr2/nlp/crfm/benchmarking/benchmarking/src/benchmark/metrics/basic_metrics.py", line 790, in compute_calibration_metrics
    platt_scaler = cal.get_platt_scaler(np.array(max_probs), np.array(correct))
  File "/u/nlp/anaconda/main/anaconda3/envs/crfm_benchmarking/lib/python3.8/site-packages/calibration/utils.py", line 463, in get_platt_scaler
    clf.fit(model_probs, labels)
  File "/u/nlp/anaconda/main/anaconda3/envs/crfm_benchmarking/lib/python3.8/site-packages/sklearn/linear_model/_logistic.py", line 1138, in fit
    X, y = self._validate_data(
  File "/u/nlp/anaconda/main/anaconda3/envs/crfm_benchmarking/lib/python3.8/site-packages/sklearn/base.py", line 596, in _validate_data
    X, y = check_X_y(X, y, **check_params)
  File "/u/nlp/anaconda/main/anaconda3/envs/crfm_benchmarking/lib/python3.8/site-packages/sklearn/utils/validation.py", line 1074, in check_X_y
    X = check_array(
  File "/u/nlp/anaconda/main/anaconda3/envs/crfm_benchmarking/lib/python3.8/site-packages/sklearn/utils/validation.py", line 899, in check_array
    _assert_all_finite(
  File "/u/nlp/anaconda/main/anaconda3/envs/crfm_benchmarking/lib/python3.8/site-packages/sklearn/utils/validation.py", line 146, in _assert_all_finite
    raise ValueError(msg_err)
ValueError: Input X contains NaN.
LogisticRegression does not accept missing values encoded as NaN natively. For supervised learning, you might want to consider sklearn.ensemble.HistGradientBoostingClassifier and Regressor which accept missing values encoded as NaNs natively. Alternatively, it is possible to preprocess the data, for instance by using an imputer transformer in a pipeline or drop samples with missing values. See https://scikit-learn.org/stable/modules/impute.html You can find a list of all estimators that handle NaN values at the following page: https://scikit-learn.org/stable/modules/impute.html#estimators-that-handle-nan-values
```
Repository: stanford-crfm/helm
Issue: Support running on arbitrary HuggingFace models and revisions
Root cause: Unimplemented Feature Gap

I want to be able to benchmark Levanter checkpoints from HuggingFace hub. Also, I would like to be able to do this _without_ having to modify Helm code i.e. I should be able to `pip install crfm-helm` and import some stuff and run it.

cc @teetone @dlwh 

The API I am envisioning is something like this:

    register_model(
        Model(
            group="huggingface",
            creator_organization="Stanford CRFM",
            name="huggingface/battlestar-gpt2-small-x49-checkpoint-300000",
            display_name="Stanford CRFM GPT-2",
            description="Stanford CRFM GPT-2",
            tags=[TEXT_MODEL_TAG, FULL_FUNCTIONALITY_TEXT_MODEL_TAG, GPT2_TOKENIZER_TAG],
        )
    )
    register_window_service_for_model(
        "huggingface/battlestar-gpt2-small-x49-checkpoint-300000",
        GPT2WindowService
    )
    register_huggingface_model(
        "battlestar-gpt2-small-x49-checkpoint-300000",
        HuggingFaceModelConfig("stanford-crfm/battlestar-gpt2-small-x49", "checkpoint-300000")
    )

    run_benchmarking(
        run_spec_descriptions=["pubmed_qa:prompt_answer_choices=true,model=huggingface/battlestar-gpt2-small-x49-checkpoint-300000"],
        auth=Authentication(""),
        url="",
        local=True,
        local_path="prod_env",
        num_threads=4,
        output_path="benchmark_output",
        suite="v1",
        dry_run=False,
        skip_instances=False,
        max_eval_instances=10)


@dlwh points out that it would be nice if we could just provide the huggingface model name and revision via a command-line flag, and not have to write code at all.

with some work I'm pretty sure you could extract all that meta information from the repository itself, or at least best-guess
Repository: stanford-crfm/helm
Issue: Support aggregating quotas and tokens used by groups
GitHub labels: p3

framework

The possible command can be

`proxy-cli aggregate -g cs324 ...`

We can also support `--all`, which shows quotas and token usage for all the groups.

IMHO the CLI should just output some machine readable format, then one can just write a `bash` or `jq` one-liner to get aggregations. Closing.
Repository: stanford-crfm/helm
Issue: Unescaped HTML
Root cause: Validation Gap

For the prompting ablations, we have an input prefix that corresponds to `<input>`, but is rendered as an actual HTML text field

![image](https://user-images.githubusercontent.com/29578854/201790375-0839c125-18ee-4acd-8a0f-0037da0f1d63.png)

(this is not currently visible on the live website since I have not yet merged the schema fields that are necessary to display these ablations)

How do I reproduce this?

Oh, it now shows up on the website: https://nlp.stanford.edu/helm/current/?suite=v8&group=ablation_prompts

It's caused by runs with `prompt=input_output_html` in `run_specs_extra.conf` from what I understand.


Sounds good. We just need to escape this. And there are various other places that should be escaped too.

I think by default, HTML should be escaped in all table cells, except markdown.
Repository: stanford-crfm/helm
Issue: BoolQ: Figure out the right number for max_eval_instances
Root cause: Configuration Error

We might want to subsample instead of running on all samples.

BoolQ - 3270 samples. For contrast sets, there are 340 samples.

Blocked by: https://github.com/stanford-crfm/benchmarking/issues/94

We can close this, right?
Repository: stanford-crfm/helm
Issue: Perspective API throwing an error even after truncating the text
Root cause: Validation Gap

          Error was thrown when making a request to Perspective API: <HttpError 400 when requesting https://commentanalyzer.googleapis.com/v1alpha1/comments:analyze?key=AIzaSyBtJGGh_vnH_8dR_gWKrE58jjID_LdCVew&alt=json returned "Comment text was too many bytes. Value (23368) exceeded limit (20480).". Details: "Comment text was too many bytes. Value (23368) exceeded limit (20480).">
Repository: stanford-crfm/helm
Issue: Make it easy to add new scenarios
Root cause: Unimplemented Feature Gap

We should support adding new scenarios without having to clone and modify the repository. I think we should provide a registration API for this, similar to the code snippet in #1103.

We should also write tutorials for adding new scenarios.

We did something very similar in #1436. Basically, we need to add a command line flag that allows the user to register a run spec generator under some name. The run specs themselves contain ObjectSpecs for the scenario and metrics, so those shouldn't require registration.

@percyliang I'd like up up-prioritize and spend some time working on this in April.

I think we should making it easy for researchers to plug in their own scenario. My experience has been that researchers who want to run custom scenarios are blocked for a long time because currently code review and benchmarking resources are limited (i.e. it's only me), so I'd like to make this more self-serve.

With `GrammarScenario` (or `BigBenchScenario`), they can just create a yaml/JSON file with a bunch of instances...so that seems pretty easy?

The delta is that it would be nice to allow users to:

- Plug in scenarios _with_ only the pip install workflow _without_ having to clone the repo.
- Avoid materializing the data into YAML / JSON in a specific format; in particular, Hugging Face datasets seems like a pretty popular format

Added proposal for this at #1487.

Closing as duplicate of #1487.
Repository: stanford-crfm/helm
Issue: Use MkDocs autodoc to generate documentation
GitHub labels: p1

In particular, we want to generate documentation for built-in scenarios, models, adapters, etc. Auto-generating the documentation will make it easier to keep things in sync.

Ended up using MkDocs instead of Sphinx because:

- Native support for Markdown, which we already use (Sphinx prefers ReStructuredText and needs an extension for Markdown)
- Better developer ergonomics (e.g. hot reloading server)

Running into a weird issue where using [Google style docstrings with mkdocstrings]() causes it to spuriously create sections where it shouldn't e.g.

```
Example:

    This should be code, not a section
```

The workaround is to delete the trailing colon.

Here's what the undesired section rendering looks like:

![bad_section](https://user-images.githubusercontent.com/185227/203143841-cbf5d9c6-235d-4b2d-ad03-3f6513e5e6dc.png)

I deleted the colons for now. I'll add them back after figuring out how to disable the section rendering (maybe there's a config flag somewhere).

The regular expression that creates the undesired sections / admonitions is [here](https://github.com/mkdocstrings/griffe/blob/a386a5c530c8c16bbc10a3c771f058b8910180ee/src/griffe/docstrings/google.py#L64). As far as I can tell, there is no way to disable it. The workarounds are to avoid the regex e.g. by adding `**bold**` markdown or other punctuation to the header.

We want dataclass documentation as well, as discussed in #1117

As noted in #1209, we need to add more schemas for dataclasses e.g. `MetricName`, `Run`, `AdapterSpec`, `ScenarioState`, etc.

MkDocs is used now so I will close this issue. There's more documentation to write, but that should be in different issues.
Repository: stanford-crfm/helm
Issue: Add documentation to ReadTheDocs

We already do this for Mistral, so we should be consistent and do this too for HELM.

(Mistral does read the docs. I haven't done it yet for levanter)

On Tue, Nov 22, 2022, 11:55 AM Yifan Mai ***@***.***> wrote:

> Closed #1164 <https://github.com/stanford-crfm/helm/issues/1164> as
> completed via #1180 <https://github.com/stanford-crfm/helm/pull/1180>.
>
> —
> Reply to this email directly, view it on GitHub
> <https://github.com/stanford-crfm/helm/issues/1164#event-7871227542>, or
> unsubscribe
> <https://github.com/notifications/unsubscribe-auth/AAACLIIHM5JGYQVCP3SVN63WJUQJRANCNFSM6AAAAAASDVYZFA>
> .
> You are receiving this because you are subscribed to this thread.Message
> ID: ***@***.***>
>
Repository: stanford-crfm/helm
Issue: BBQ run was not included in Figure 40
GitHub labels: bug

p1

We have BBQ results on the website: https://crfm.stanford.edu/helm/v1.0/?runs=1&runSpec=bbq%3Asubject%3Dall%2Cmethod%3Dmultiple_choice_joint%2Cmodel%3Danthropic_stanford-online-all-v4-s3

but it was not included in Figure 40 of the paper https://arxiv.org/abs/2211.09110.

For v1.0:

Harms metrics:  https://crfm.stanford.edu/helm/v1.0/?suite=v1.0&group=harms#BBQ%20metrics
JSON file: benchmark_output/runs/v1.1/groups/json/harms_bbq_metrics.json

I'm having trouble reproducing this locally. BBQ results for all models show up:
<img width="756" alt="Screen Shot 2022-12-06 at 11 12 56 PM" src="https://user-images.githubusercontent.com/16793796/206112623-946e4582-8da1-4c4a-9e7e-af2a6b563277.png">




When generating plots, I use the tables from the Results section of the website. Taking a closer look, I see that Anthropic (+others) were missing from [v1.0](https://crfm.stanford.edu/helm/v1.0/?group=harms#BBQ%20metrics) but are actually there for [v1.1](https://crfm.stanford.edu/helm/v1.1/?group=harms#BBQ%20metrics).

So I can just regenerate the plots using the v1.1 data and this should be fixed.

> When generating plots, I use the tables from the Results section of the website. Taking a closer look, I see that Anthropic (+others) were missing from [v1.0](https://crfm.stanford.edu/helm/v1.0/?group=harms#BBQ%20metrics) but are actually there for [v1.1](https://crfm.stanford.edu/helm/v1.1/?group=harms#BBQ%20metrics).
> 
> So I can just regenerate the plots using the v1.1 data and this should be fixed.

I think the results are showing up because I reran with this fix: https://github.com/stanford-crfm/helm/pull/1233, but results are now missing in additional experiments: https://crfm.stanford.edu/helm/v1.1/?group=ablation_multiple_choice

Ah, I see, so the way to reproduce this is to just summarize `v1.1` with `main`?

This gives me the following which has Anthropic but is missing BLOOM (even though we have a corresponding run from what I can tell). I can dig into this...
<img width="781" alt="image" src="https://user-images.githubusercontent.com/29578854/207697670-16737a86-81ba-470d-9fc2-fcebb00a87db.png">


> Ah, I see, so the way to reproduce this is to just summarize `v1.1` with `main`?
> 
> This gives me the following which has Anthropic but is missing BLOOM (even though we have a corresponding run from what I can tell). I can dig into this... <img alt="image" width="781" src="https://user-images.githubusercontent.com/29578854/207697670-16737a86-81ba-470d-9fc2-fcebb00a87db.png">

That's correct. I debugged a bit, and I think it's because we also run these in additional experiments: https://github.com/stanford-crfm/helm/blob/main/src/helm/benchmark/presentation/run_specs_extra.conf#L265. The group is `ablation_multiple_choice` which changes the visibility in this table.

Fixed: https://crfm.stanford.edu/helm/v0.2.0/?group=bbq. Thanks for your help @dtsip!
Repository: stanford-crfm/helm
Issue: AlexaTM 20B is available via SageMaker (model and window service)
Root cause: Unimplemented Feature Gap

https://github.com/amazon-science/alexa-teacher-models

How to deploy and query: https://github.com/aws/amazon-sagemaker-examples/blob/main/introduction_to_amazon_algorithms/jumpstart_alexatm20b/Amazon_Jumpstart_AlexaTM_20B.ipynb

Very old, also we don't currently support seq2seq. Perhaps consider evaluating Titan instead.