zhiminy/EvalEng huggingface dataset:
Repository: stanford-crfm/helm
Issue: Add CodeGeeX 13B (model and window service)
Root cause: Unimplemented Feature Gap

CodeGeeX is an autoregressive Transformer language model with 13B parameters, which is trained on public GitHub repositories with 20+ programming languages. It supports both multilingual code generation and crosslingual code translation.


https://github.com/THUDM/CodeGeeX

Closing due to (1) staleness and (2) lack of code evals in Lite. We could consider evaluating [CodeGeeX4](https://github.com/THUDM/CodeGeeX4) in the future.
Repository: stanford-crfm/helm
Issue: Add GLM (10B)
Root cause: Unimplemented Feature Gap

https://huggingface.co/BAAI/glm-10b

Model page for GLM (10B) no longer exists on Hugging Face. We could consider adding [GLM v4](https://huggingface.co/THUDM/glm-4-9b) though.
Repository: stanford-crfm/helm
Issue: Deeper analysis of calibration
Root cause: Unimplemented Feature Gap

Did we get a chance to pin down more about calibration post Platt scaling? E.g., top level findings we discussed like (need to confirm more widely):
- Calibration of models is pretty bad, at least with generic adaptation methods
- Platt scaling can fix calibration but need a different Platt constant for each scenario which is challenging in a few shot setting
- The same model can be underconfident for some tasks but overconfident for others
Also - it looks like the post platt scaling ECE isn’t listed as a metric e.g., on https://crfm.stanford.edu/helm/v1.0/ - so we should add it at some point!

Closing for now
Repository: stanford-crfm/helm
Issue: pre-commit.sh doesn't work on Mac OS
Root cause: Environment Incompatibility

Report from Joel:

> When I was trying to setup helm to get a feel for it (running ./pre-commit.sh) on my MacBook Pro (13.0.1) with conda I get the following error:
> ERROR: Could not find a version that satisfies the requirement torch==1.12.1+cu113 (from versions: 1.4.0, 1.5.0, 1.5.1, 1.6.0, 1.7.0, 1.7.1, 1.8.0, 1.8.1, 1.9.0, 1.9.1, 1.10.0, 1.10.1, 1.10.2, 1.11.0, 1.12.0, 1.12.1, 1.13.0)

The installation command doesn't work because the package is Linux-specific.

Specific CUDA version needed for summarization metrics.  Would be nice to make things more modular, where we don't need all libraries for all metrics to be loaded to run anything.

Solution for now: remove it.  If running summarization, do these things.
Repository: stanford-crfm/helm
Issue: Investigate high client error rate
GitHub labels: p2

Client error rate (which includes 404s) seems suspiciously high

![Screenshot_20221123_101133](https://user-images.githubusercontent.com/185227/203618910-aa958266-6d1a-4380-8f43-2b978e1551cd.png)


This is still happening. I enabled [access logs](https://cloud.google.com/storage/docs/access-logs) on GCS and will take a look once logs are available.

The error is `FAILED_PRECONDITION`. Looking at the [GCS documentation on request preconditions](https://cloud.google.com/storage/docs/request-preconditions), we see that cache hits i.e. `304 Not Modified` is counted as a precondition failed error... even though it's not really an error. Looking at the logs, I confirmed that we have a large number of 304 responses. So this is not an issue.
Repository: stanford-crfm/helm
Issue: Support RWKV-4 RNN
Root cause: Unimplemented Feature Gap

Hi I am the author of RWKV-4 (a pure RNN, 100% attention-free), with up to 7B params checkpoints trained on the Pile.

https://github.com/BlinkDL/RWKV-LM

They are quite competitive. Let me know if there is anything I can help to evaluate them :)

Note: the GPT mode will be faster for most benchmarks where the prompt is long and the generation is short.

Hi @BlinkDL, we usually prefer to add models that are supported Hugging Face `transformers`, LiteLLM, vLLM, cloud inference or a inference service, as these are much easier to add to HELM. Unfortunately I don't think RWKV-4 supports any of these. However, feel free to reopen this issue if you have any other suggestions, or if you have other models you'd like to add.
Repository: stanford-crfm/helm
Issue: handle multiple_choice_separate adapter for bidirectional models
Root cause: Unimplemented Feature Gap

Currently, HellaSwag, etc. relies on multiple_choice_separate adapter, where we form one prompt (question, answer) for each answer and compute its probability.  However, for bidirectional models such as GLM, UL2, etc., we have two adaptation strategies:
1) Assess the probability of (question, answer) under the decoder - p(question, answer)
2) Assess the probability of answer under the decoder, but encode the question - p(answer | question)

2) should be stronger, but let's benchmark and find out.

To implement 2), we could add a `encoder_decoder_split_index: Optional[int]`, which tells us which part of the prompt is to to be consumed by the encoder and which is to be consumed by the decoder.

Don't think we want to support bidirectional models since all the newer APIs only support left to right.
Repository: stanford-crfm/helm
Issue: Discrepancy between the UI and Python interface
Root cause: Interface Contract Mismatch

`Request(prompt="The quick brown fox jumped over the lazy ", temperature=0)`  returns:

`completions=[Sequence(text='\t\t\t\t\tdog.\n\n\t\t\t\t \t\t\t\t \t\t\t\t\tThe`

but in the[ UI](https://crfm-models.stanford.edu/static/index.html?prompt=The%20quick%20brown%20fox%20jumped%20over%20the%20lazy%20&settings=temperature%3A%20%24%7Btemp%7D%0Amodel%3A%20openai%2Fdavinci&environments=temp%3A%20%5B0%2C%200.2%2C%200.4%5D) we trim the tabs:

![Screen Shot 2022-02-11 at 10 06 08 AM](https://user-images.githubusercontent.com/16793796/153645595-9aaa0f63-a63b-41e1-be06-a4c6c0fc6a49.png)


I guess we can format using monowidth and then allow mousing over tabs too...this is a bit trickier with newlines...

This should just require putting `white-space: pre-wrap` in the CSS, as is already done elsewhere in the codebase. ([MDN doc](https://developer.mozilla.org/en-US/docs/Web/CSS/white-space))
Repository: stanford-crfm/helm
Issue: Metric num_prompt_tokens occurs more than once
Root cause: Validation Gap

This frontend JavaScript error happens e.g. when visiting https://crfm.stanford.edu/helm/v1.0/?runs=1&runSpec=boolq%3Amodel%3Dai21_j1-jumbo%2Cdata_augmentation%3Dcanonical

Duplicate of #1205.
Repository: stanford-crfm/helm
Issue: only 1/3 trials shows up in prediction viewer
Root cause: Algorithmic Error

Example: https://crfm.stanford.edu/helm/v1.0/?group=raft&subgroup=subset%3A%20ade_corpus_v2&runSpecs=%5B%22raft%3Asubset%3Dade_corpus_v2%2Cmodel%3Dai21_j1-jumbo%2Cdata_augmentation%3Dcanonical%22%5D

I only see "Prediction{trial 2}", but there should be trial 0 and trial 1.
Repository: stanford-crfm/helm
Issue: Refactor Scenario dataclass
Root cause: Architectural Constraint

`Scenario` itself should not be a dataclass. In particular, this prevents us from round-tripping the JSON in `scenario.json` to and from `Scenario`. Instead, `Scenario` should have a `ScenarioMetadata` dataclass that contains the current data fields, and only these should be serialized to `scenario.json`
Repository: stanford-crfm/helm
Issue: Reproducibility of demonstration examples
Root cause: Unimplemented Feature Gap

The demonstration examples in each run are randomly generated. Is there a way to use a controlled random seed to generate the same sequence of demonstration examples when using the same seed? For example, specify the seed in the ".conf" file.

The demonstration examples are deterministically generated. For instance, the first train trial for a given scenario and `ScenarioSpec` will always have the same demonstration examples, and the second train trial will always have the same demonstration examples, etc.

Unfortunately, there is currently no way to specify your own random seed.
Repository: stanford-crfm/helm
Issue: evaluate ChatGPT
Root cause: Unimplemented Feature Gap

Run https://github.com/stanford-crfm/helm/blob/main/src/helm/benchmark/presentation/run_specs_chat_gpt.conf

I think supporting ChatGPT in HELM is doable. I tried different prompts with ChatGPT, and it seems to format answers the way we want them as long as we provide in-context examples. For zero-shot, we would need more descriptive instructions.

OpenAI is releasing a ChatGPT API soon:

![Screenshot 2023-01-31 at 9 49 20 AM](https://user-images.githubusercontent.com/16793796/215861094-9f673a14-4c59-4cd6-bfa3-c44b0021d045.png)


There was an OpenAI backend bug where using "\n" as a stop sequence caused the OpenAI API to return a HTTP 500; this has been fixed by OpenAI last week in https://github.com/openai/openai-python/issues/290
Repository: stanford-crfm/helm
Issue: Make it easy to run HELM on new models
Root cause: Documentation Deficiency

Right now users have to fork and change the repo itself. Ideally, users can just pip install the package, define the client outside the package, and it should still work.

Also, we need to add documentation for how to do this.

Do we want to do this?  One consideration is that it'd be nice if things were contributed back to the repo.  Perhaps it'd be simpler if things outside the repo were just conf files?  What do you think?

This was done in #1761 and #1787.

We still need to write documentation, but the API is still in flux, so I would like to pin that down first before adding documentation.
Repository: stanford-crfm/helm
Issue: run_specs.json should be generated by helm-summarize instead of helm-run
Root cause: Architectural Constraint

Right now, run_specs.json only contains the runs from the latest invocation of `helm-run`. This is rather confusing and inconvenient e.g. a user could run a bunch of run_specs.conf files in the same suite, but only the run specs from the latest invocation will show up. We could instead have `helm-summarize` scan the benchmark output folder and generate `run_specs.json`.
Repository: stanford-crfm/helm
Issue: remove fields from `Model`
Root cause: Architectural Constraint

For any field that's already in `schema.yaml`, remove it from the `Model` dataclass to minimize redundancy (this includes description and display_name, and creator_organization).  Some logic for displaying the models will have to be updated in the proxy server help...

Can we just get rid of `schema.yaml` entirely?

- Move all the fields into the case classes
- Have `helm-summarize` read all the defined models, definitions and metrics and dump them?

I think it's better to start moving things in `yaml` rather than Python...  Of course everything in `schema.yaml` should have a case class in Python.

It's important to note that the things in `schema.yaml` include scenarios, models, and metrics that might not have Python implementations yet, and this is to capture the incompleteness.

Was this potentially resolved via #1567 ? Might be able to close this issue out.

@mkly Indeed it has, thanks for pointing this out!
Repository: stanford-crfm/helm
Issue: update together models to display the model creator (rather than together)
Root cause: Documentation Deficiency

We currently use models names like `together/gptj`, but for external purposes, we probably want the provider (e.g. on the Models page of the website) as the primary thing (and the specific model source/computation provider can be secondary, esp. since we could have multiple implementations of nominally the same model as was the case for together vs. goose) 

@rishibommasani, which page is this for? We display the model creators on most pages, and I think having the raw model name under `Adapter Specification` is okay. 


<img width="374" alt="creator1" src="https://user-images.githubusercontent.com/16793796/209468973-3c02b68c-2215-4b83-8387-b34928f17f88.png">
<img width="328" alt="creator2" src="https://user-images.githubusercontent.com/16793796/209468974-6ff30da3-b192-4053-9670-6cb7ed732ba5.png">


<img width="407" alt="adapter" src="https://user-images.githubusercontent.com/16793796/209469049-40fd5df0-a731-42bb-b0cb-3dccc7e1ad6e.png">


I think the results pages, since thats what most people will look at (CC @percyliang)

> I think the results pages, since thats what most people will look at (CC @percyliang)

In the results pages, we're using the `display_name`:

<img width="291" alt="Screenshot 2022-12-25 at 5 15 14 AM" src="https://user-images.githubusercontent.com/16793796/209469410-696db787-8cb7-49e5-9265-e0e0be5a0dc1.png">


It's true that the most prominent places, we're showing the creator organization, which is good to credit them.  On the models page and the adapter spec, we still show things like `together/gpt-neox`, which might be a bit confusing, but I think we can't really change this without obfuscating things, since the model name is very deeply baked into the system.  Also, Together does various optimizations which in theory shouldn't affect things, but it's always possible that they will (and certainly, it matters for efficiency).  So I think maybe for now, we should just leave things as is?
Repository: stanford-crfm/helm
Issue: no `definition_path`
Root cause: Validation Gap

https://github.com/stanford-crfm/helm/blob/31bed963d77637addb8bdc4d5ca1f3a447205159/src/helm/benchmark/scenarios/bbq_scenario.py#L69

miss the `super().__init__()` that causes `no definiteion_path` error. 

Fixed; thanks for filing this!
Repository: stanford-crfm/helm
Issue: multiple_choice_separate_calibrated frontend does not display predictions
Root cause: Validation Gap

See: https://crfm.stanford.edu/helm/v1.0/?runs=1&runSpec=commonsense%3Adataset%3Dopenbookqa%2Cmethod%3Dmultiple_choice_separate_calibrated%2Cmodel%3Dcohere_medium-20220720%2Cdata_augmentation%3Dcanonical

There are multiple predictions so there's a question of which RequestState to use (and also which one to keep in our reduced JSON in #1185. I believe we only need to keep the JSON where `request_mode` is `original`, but I might be wrong...
Repository: stanford-crfm/helm
Issue: Metrics are rendered multiple times for adapter method language_modeling
Root cause: Algorithmic Error

https://crfm.stanford.edu/helm/v1.0/?runs=1&runSpec=twitter_aae%3Ademographic%3Daa%2Cmodel%3Dai21_j1-jumbo

![Screenshot_20221212_121112](https://user-images.githubusercontent.com/185227/207144883-370f201c-d4ac-44ec-83ff-934163ee33ac.png)
Repository: stanford-crfm/helm
Issue: Support MetricName
Root cause: Unimplemented Feature Gap

Design doc: https://docs.google.com/document/d/1-Qj79b6qCIU7h_w_-p_QbP7QPj-btqLTIIm4BbKL4qg/edit

```python
class MetricName
  split: Optional[str]
  subsplit: Optional[str]
  name: str # one of exact_match, f1, toxicity, faithfulness
  k: Optional[int]
  worst_augmentations: List[DataAugmenterSpec]
  selective_recall: float
```
Repository: stanford-crfm/helm
Issue: Subclass registration for Perturbation
Root cause: Unimplemented Feature Gap

In order to round-trip `Perturbation` to and from JSON, we need a [map of names](https://github.com/stanford-crfm/helm/blob/65fdf7a838e06d07edd21ca408f9c56717ea90cb/src/helm/common/codec.py#L29C1-L29C1) to `Perturbation` subclasses. We can use [subclass registration](https://peps.python.org/pep-0487/#subclass-registration) to do this automatically (also see this [Stack Overflow article](https://stackoverflow.com/questions/5189232/how-to-auto-register-a-class-when-its-defined)). This would eliminate the need to add every new `Perturbation` subclass to this map.

We should consider doing this for `Scenario` subclasses too, which have a similar problem.

I now think that this is not the right approach, and that the current approach (explicitly registering the classes in the serialization module) is more explicit and preferred.
Repository: stanford-crfm/helm
Issue: Better SLURM sharding integration
Root cause: Unimplemented Feature Gap

If we support [SLURM job arrays](https://slurm.schedmd.com/job_array.html), then we can remove the hacks in `helm-run` for running shards on SLURM.

Don't need this any more since we're swapping to using CodaLab #1320
Repository: stanford-crfm/helm
Issue: Consider moving `write_run_display_json` from `summarize` to `run`
Root cause: Architectural Constraint

Memory requirements:

src/helm/benchmark/presentation/run_specs.conf (don't specify via --conf-paths because of GPU run) without MSMARCO: 20g
--conf-paths  src/helm/benchmark/presentation/run_specs_extra.conf: 32g
--conf-paths  src/helm/benchmark/presentation/run_specs_msmarco.conf: 64g

I don't think deriving JSONs should be part of `run`, which should be responsible for just generating the raw output.  summarize can have flags that write out various assets, and maybe we should allow this to be parallelizable too.

We need to address #1269 (Dacite is slow) if we want to keep this in summarize.

Also, MSMARCO specifically causes OOMs; the workaround I've been doing is to hack summarize to exclude MSMARCO, and then do a summarize that is just MSMARCO.

Also I don't think this affects end-users of the library as much as it affects us? It's only really a problem for us because (1) we have hundreds of runs and (2) we have MSMARCO, which is an order of magnitude larger than any other scenario.

I'm implementing a different deserialization method that's an order of magnitude faster, so we can keep this in `summarize`.
Repository: stanford-crfm/helm
Issue: Escape HTML properly everywhere
Root cause: Validation Gap

There are a number of places where strings in JSON are not escaped correctly:

![Screenshot_20221220_043547](https://user-images.githubusercontent.com/185227/208793326-16977bfb-c162-4a3c-b74b-f3562543fa9a.png)

The only HTML that should be unescaped is HTML that is safely generated by Python.

Closing because this should be handled by React.
Repository: stanford-crfm/helm
Issue: Run pylint in precommit
GitHub labels: p3

cleanup

devinfra

Pylint catches some errors that mypy does not e.g. missing `super().__init__()` call in `__init__()`

I investigated this a bit and I don't think it's worth doing:

1. It's going to slow down development time significantly in terms of developers spending time to fix lint errors, and in terms of the amount of time the linter takes to run (~tens of seconds)
2. There are thousands of existing errors across various categories:

```
abstract-method,
anomalous-backslash-in-string,
attribute-defined-outside-init,
bare-except,
broad-except,
cell-var-from-loop,
comparison-with-itself,
consider-iterating-dictionary,
consider-using-dict-items,
consider-using-enumerate,
consider-using-f-string,
consider-using-generator,
consider-using-in,
consider-using-with,
dangerous-default-value,
exec-used,
f-string-without-interpolation,
fixme,
function-redefined,
global-variable-not-assigned,
global-variable-undefined,
import-error,
import-outside-toplevel,
inconsistent-return-statements,
invalid-name,
line-too-long,
missing-class-docstring,
missing-function-docstring,
missing-module-docstring,
missing-timeout,
no-else-break,
no-else-continue,
no-else-raise,
no-else-return,
no-else-return,
no-member,
no-name-in-module,
non-ascii-name,
not-callable,
pointless-string-statement,
protected-access,
raise-missing-from,
redefined-argument-from-local,
redefined-builtin,
redefined-outer-name,
self-assigning-variable,
simplifiable-if-expression,
super-init-not-called,
super-with-arguments,
too-few-public-methods,
too-many-arguments,
too-many-boolean-expressions,
too-many-branches,
too-many-function-args,
too-many-instance-attributes,
too-many-lines,
too-many-locals,
too-many-statements,
undefined-loop-variable,
unexpected-keyword-arg,
ungrouped-imports,
unidiomatic-typecheck,
unnecessary-comprehension,
unnecessary-dunder-call,
unnecessary-lambda,
unnecessary-lambda-assignment,
unnecessary-pass,
unneeded-not,
unspecified-encoding,
unused-argument,
unused-import,
unused-variable,
use-a-generator,
use-dict-literal,
use-maxsplit-arg,
useless-object-inheritance,
useless-parent-delegation,
wrong-import-order,
wrong-import-position,
```

Ideally, mypy should just catch this: https://github.com/python/mypy/issues/11809
Repository: stanford-crfm/helm
Issue: Dacite is slow
Root cause: Algorithmic Error

Right now, reading back all the run JSON (~5000 files) in summarize.py takes 30 minutes. That is way too long.

Options:

- Use a different library for deserializing JSON to dataclasses (I was not able to find any)
- Optimize the Dacite library upstream (but we will need to wait for a release)
- Figure out if there are any local code optimizations we can do (e.g. restructuring dataclasses)

I'm planning to do some profiling.

> * Use a different library for deserializing JSON to dataclasses (I was not able to find any)

I’d like to chime in and propose taking a look at [mashumaro](https://github.com/Fatal1ty/mashumaro). It might be what you’re looking for. 

If you need to deserialize json documents I’d suggest using builtin [DataClassORJSONMixin](https://github.com/Fatal1ty/mashumaro#dataclassorjsonmixin) which is faster than the default json decoder.

You can see benchmark results compared to dacite https://github.com/Fatal1ty/mashumaro#benchmark

@Fatal1ty Thanks for the recommendation! I'll check it out.

`cattrs` seems to be a better fit for HELM than `mashumaro`, because:

- Migrating to `cattrs` is easier because `mashumaro` requires adding mixins to dataclasses, whereas `cattrs` does not
- Need to be reverse compatible with existing JSON files at rest that was serialized using [`asdict_without_nones`](https://github.com/stanford-crfm/helm/blob/0eaaa62a2263ddb94e9850ee629423b010f57e4a/src/helm/common/general.py#L128) i.e. need custom deserialization logic for nearly all dataclasses; this can be done easily in `cattrs` using factory hooks, but has to be done on a class-by-class basis in `mashumaro`
- Minor `mashumaro` ergonomics nits:
  - Separate mixins are required for dict and JSON
  - The JSON mixin does not support deserializing a string that is a list of dataclaasses
Repository: stanford-crfm/helm
Issue: Add support for running large models locally
Root cause: Unimplemented Feature Gap

Hi,
Is there a plan to add support for running large models locally? Such as BLOOM / OPT?

As far as models from Huggingface, this would require using DeepSpeed / Accelerate, which would require some changes to the HuggingFaceServer, since it would need to run by many processes.

Thanks

Supporting local inference would be a good idea; however the current focus is on supporting large models with remote APIs, so it will be a while before we get around to this.

we could in theory harness an existing solution (e.g. https://github.com/huggingface/transformers-bloom-inference/tree/main/inference_server) and still work in the same way, i.e. have a client and server running on the same machine, in order not to make large changes to the current architecture.  

Multi-GPU models are now supported in HELM in a couple of ways:

1. `HuggingFaceClient` now supports Accelerate (#2762) for multi-GPU inference
2. Host a multi-GPU model using vLLM and send API requests to vLLM
Repository: stanford-crfm/helm
Issue: Cleanup `TogetherClient` once `logprobs`, `echo` and `n` are supported
Root cause: Unimplemented Feature Gap

Follow up to https://github.com/stanford-crfm/helm/pull/1273. Currently, we check if we get log probs in the response in `TogetherClient`.

Together's API is very unstable, and many of their models and API versions do not return logprobs, so let's not waste time on this. We don't need logprobs for the Lite leaderboard.

Related issues: #1847 and #1828