zhiminy/EvalEng huggingface dataset:
Repository: stanford-crfm/helm
Issue: CommonsenseQA results are worse for same models
Root cause: Algorithmic Error

1. We now have Hellaswag and OpenbookQA results for many models, including all OpenAI models: 
https://crfm-models.stanford.edu/static/benchmarking.html?runs=undefined&runSpec=.*hella.*model%3D(openai%7Cai21).*#metrics
https://crfm-models.stanford.edu/static/benchmarking.html?runs=undefined&runSpec=.*openbookqa.*model%3D(openai%7Cai21).*#metrics
2. For both, we do worse than the numbers reported by GPT-3 (for the same model, same scenario). This is most notable for HellaSwag, **where we often do 20-30 points worse for HellaSwag**. For OpenbookQA, it may be explicable by the fact that we use `separate_calibrated` approach but for this scenario, it seems like the `joint` adapter would be better. I have documented the scores reported in prior work here: https://docs.google.com/spreadsheets/d/1jpm0Cy0r5Yk_l9o9kwVpxbSfdKrx6ZcQ7iQfs7QYPIc/edit#gid=1233800927

So could we look into why there is such a large delta (and fix it if we are doing something wrong, or at least crisply understand why there is a delta)? 

@yuhui-zh15 @michiyasunaga
Repository: stanford-crfm/helm
Issue: BoolQ for GPT-3 is worse than GPT-3 paper
Root cause: Documentation Deficiency

1. We now have BoolQ results for many models, incl. GPT-3/OpenAI models:
https://crfm-models.stanford.edu/static/benchmarking.html?group=boolq
2. Our results are worse than those reported in the GPT-3 paper afaict
https://docs.google.com/spreadsheets/d/1jpm0Cy0r5Yk_l9o9kwVpxbSfdKrx6ZcQ7iQfs7QYPIc/edit#gid=1233800927

Could we look into why this is (and fix if there is a bug, or explain if it is to be expected)?

Hi @rishibommasani , 
[GPT-3 Paper ](https://arxiv.org/pdf/2005.14165.pdf), page 18, the last sentence claims "..In the few-shot setting, we used 32 examples for all tasks, sampled randomly from the training set." We could not understand how this was possible as we could not fit 32 examples into the context (BoolQ passages are pretty large, cc @dtsip ). I would guess this could be one major reason we underperform in the few-shot setting.

(I'm a bit surprised about the GPT-3 (1.3B) performance though, as it is in the chance-level)

@mertyg great, this makes sense. Could you make a note about this (probably in the appendix), thanks!
Repository: stanford-crfm/helm
Issue: All evaluation instances for CivilComments have true label False
Root cause: Algorithmic Error

We need to correct how we construct (or sample, but I suspect the issue is in how we construct the label) evaluation instances.

You can see that all eval instances have a label of False here (and hence the model does obscenely well): 
https://crfm-models.stanford.edu/static/benchmarking.html?runs=undefined&runSpec=civil_comments%3Asubject%3Dall%2Cmodel%3Dai21_j1-grande%2Cdata_augmentation%3Dcanonical#metrics

Thanks for pointing this out @rishibommasani — I believe this is because I set the targets to `severe_toxicity`, rather than toxicity (this column is called `target` in the CSV). See https://github.com/stanford-crfm/benchmarking/pull/893.
Repository: stanford-crfm/helm
Issue: make Wikifact more interpretable
Root cause: Documentation Deficiency

1. The relations are currently P1620, etc. which are inscrutable.  Can we change these to plain English?
2. We need to add entries for the exact_match@5 and quasi_exact_match@5 metrics in `schema.yaml` (follow what we have for exact_match and quasi_exact_match).

Hi! I have addressed this issue in this PR: https://github.com/stanford-crfm/benchmarking/pull/879.
Repository: stanford-crfm/helm
Issue: Copy results for website to paper
Root cause: Unimplemented Feature Gap

Should be a very simple script that takes all the LaTeX and writes it to corresponding files, which we then reference in the paper. 

And we should then run this script once we have results for all models for 1 seed (and then re-run and replace the tables once we have 3 seeds).

@dtsip how did you do this currently/seems nice to have a script rather manual (namely since we will need to update once we gets 3 seeds or so on), but feel free to close if this is resolved by whatever means

I did it manually, but I want to keep this open and write a short script that collects the related plots (will submit in the same PR as the rest of the plotting)

Cool, sounds great, was just checking

Are we doing this?

I think we decided not to. The current plan is to do plots and visualizations only in the paper and keep the raw numbers on the website. If so, we can close this.
Repository: stanford-crfm/helm
Issue: Make aspects of visualization discoverable
Root cause: Documentation Deficiency

ATM, we already implement the column headings having descriptions, along with the cells themselves showing max/mins/etc. But it is not obvious that if you hover, you will get these things. I don't have a super concrete suggestion on how to do though...

Maybe when we hover, the background color of the cell changes.  Not needed for release.

Making a p3 for now, feel free to revert but don't think this is as important as other p2s. 

Closing because hover has been removed from the React frontend, and we do not plan to add it back.
Repository: stanford-crfm/helm
Issue: Update User model and authentication
Root cause: Unimplemented Feature Gap

Make `Authentication` `api_key` instead of `username` and `password`.  The API keys will be randomly generated and assigned to people.  The `User` isn't so much tied to an individual person, but rather a use case.  Add a `description` field, and maybe make `emails` a list of emails.  Also add `group` and `is_admin`.

Need to update the web interface too.
Repository: stanford-crfm/helm
Issue: Visualize performance disparities better
Root cause: Documentation Deficiency

Currently we report the results on a per-subgroup basis in a list of tables. Which seems tolerable but still not ideal for visualizing performance disparities. 

Not a high priority, but could be improved by re-arranging how stuff gets visualized potentially (though seems gimmicky to do generally; e.g. we would likely not want this for MMLU, but i think we do care some about performance disparities even for MMLU). 

Related to #980 (should be a type of aggregation)

There's already a way to do this in `schema.yaml` - add a unique group name for each subgroup to the run spec, then create an entry in `schema.yaml` for each subgroup with the desired metric , then create a parent group. The parent group table will show the desired metric for each subgroup side-by-side as columns.
Repository: stanford-crfm/helm
Issue: Visualization quality-of-life improvements
Root cause: Unimplemented Feature Gap

**Runs view:**
- [x] Show ‘Loading…’ when showing text in runs view
- [ ] Infinite scroll lazy load
- [x] Show only part of the text with ‘...’ to expansion
- [ ] Back to top button
- [x] Visualize multiple completions


**Table view:**
- [x] Bold the top 3
> Bold the top 3
Rishi: Not sure if it should be top 3; Ananya had a suggestion I liked more of having the entire row graded by color (so then we don’t need to make hard thresholds of top 3 or so on), with the best model bolded.



@rishibommasani Have you started on this? Otherwise I can take this on.

Feel free to break up this issue if you'd like.  No one has done anything here, so it's all yours!

Regarding 'Loading...' basically various JSONs load at various times, and it should be obvious that we're still waiting for instances and predictions (e.g., "(loading predictions)').  Not sure if we can stream the instances and predictions...  But we could perhaps load the scenario state and the per instance stats JSONs at the same time rather than sequentially to speed things up.

The '...' expansion is mostly for language modeling scenarios where the input is huge.

I don't think we should have a 'back to top button', but rather have a sticky header or something where you can jump to 'adapter', 'instances', and 'metrics'.

Regarding "Infinite scroll lazy load", I'd prefer to split the massive instances JSON file into paginated, smaller JSON files of about 50 instances each, and then give the use a pagination UI. It's not really clear how infinite scroll would work since the instances are not the bottom-most element, and infinite scroll is not very usable and makes it hard to navigate to a specific instance.

This might be pretty high priority because it addresses page loads being too large (#1024).

Pagination seems good to me; I think the only benefit of having all the instances is CTRL+F which is sometimes useful over all instances (and I guess could be implemented with a search feature over the pages but idk if that's worth the effort). 

And improving load times seems like a more material consequence to prioritize, since I think we want to avoid the effect of "page is too slow to load, hence people don't actually look at the underlying instances" as we are trying to push people to actually look at the concrete data/behaviors.

The slim version should be fast enough?  After the release, we can think about the right way to do this (which will probably be a smaller index that enables search and then lazy loading of the main payload).

Which of these are we doing before release?  Do you already have a PR for some of it?

I would say that putting 'Loading...' is important since otherwise it is very confusing and looks like the website is broken.

Okay, I can add the loading message.

Some of these have been addressed, the others are wontfix.
Repository: stanford-crfm/helm
Issue: Spider charts
Root cause: Unimplemented Feature Gap

Could be good for showing multi-dimensional aspects of the benchmark. Low priority atm unless someone has cycles/interest. (CC @dtsip @deepakn94) 

Example: 
![image](https://user-images.githubusercontent.com/47439426/190038285-7ff28b31-e1e0-46c5-a859-d4e2ebab3064.png)


Yes, that'd be really neat! @niladri-chatterji had a similar idea at some point!

yup this is niladri's idea :), just made it into an issue

I really like the idea of spider charts in general.

It might be possible to put this in the new React frontend.
Repository: stanford-crfm/helm
Issue: compute predicted reference in per_instance_stats
Root cause: Unimplemented Feature Gap

My assumption, given the results line up with prior work, is that this is just some front-end issue. 


BLIMP: irregular forms
Alexander drove to restaurants
https://crfm-models.stanford.edu/static/benchmarking.html?runSpec=blimp%3Aphenomenon%3Dirregular_forms%2Cmodel%3Dai21_j1-jumbo
Model assigns higher probability to the correct answer but it’s marked as wrong


@YianZhang What is the status on this?

Looking into it!

We need to remove the logprobs of the completions in the UI since that's confusing due to normalization, and instead compute a stat that goes in per_instance_stats.json that captures which reference is the correct one (key = `predicted_index`, value = [the index], which is represented as a float, but oh well), so we can highlight the one that's predicted in the UI.  We might want to output max_prob too, so we can show the confidence of a prediction.
Repository: stanford-crfm/helm
Issue: Move synthetic efficiency datasets to CodaLab
Root cause: Unimplemented Feature Gap

https://github.com/stanford-crfm/benchmarking_efficiency/tree/main/synthetic_efficiency_instances

Resolved by https://github.com/stanford-crfm/benchmarking/commit/7119a91e02914c85c2b16996c26aa189ea2b9675
Repository: stanford-crfm/helm
Issue: Add WILDS data for CivilComments
Root cause: Unimplemented Feature Gap

We currently use some Kaggle data for CivilComments, but we should use the more authoritative WILDS data (that also merge sparse categories in a sensible way; I guess the other religion category makes sense, but is a bit weird given judaism/hinduism/buddhism are major world religions, though just not that frequent in the data). 

We should have a flag to toggle between, and once done, we should switch to WILDS as the default.

@rishibommasani, do we still want to support both, or should we just use the WILDS version? @ryanachi If you don't have the bandwidth, I can take this.

we only need the wilds one
Repository: stanford-crfm/helm
Issue: Dry-run integration tests
Root cause: Unimplemented Feature Gap

To catch regressions on the benchmarking side use `--dry-run` option.

We have `--skip-instances` now, which is good.  Can we make a version for `--dry-run` that uses a cache and doesn't block merges?

There aren't any current critical use cases that depend on `--dry-run` so we can close this for now. It was previously used for various request export / response import use cases.
Repository: stanford-crfm/helm
Issue: add tags to entries in run_specs
Root cause: Unimplemented Feature Gap

Add groups to the run_spec*.conf files:
- ablations (e.g., robustness_deepdive)
Add groups the run_specs.py file:
- facets (e.g., language)
Plumbing: concatenate groups

In the `schema.yaml` file, define groups for "facets" (e.g., language).
Might have to add a flag for expanding tables based on scenarios.

For each group:
- Rows are always method (abstracted adapter spec).
- Columns are either defined by `metric_groups`, optionally crossed with scenario.

Each group has a set of dependent groups.
Either have a table for each scenario or a column for each dependent group.



Partially addressed by #942 which implements subgroups.

Working on adding groups to `run_specs.conf` for ablations/deep-dives.

What's the status of this?

Will wrap up today!

Note my recent PR that's pushed supports groups now in the conf file

Should be resolved by #989
Repository: stanford-crfm/helm
Issue: Prompts + completions that exceed 2049 tokens will throw an error
Root cause: Validation Gap

From https://beta.openai.com/docs/introduction/key-concepts, "one limitation to keep in mind is that your text prompt and generated completion combined must be no more than the model's maximum context length (for most models this is 2048 tokens, or about 1500 words)."

Current error message: "OpenAI error: This model's maximum context length is 2049 tokens, however you requested 2054 tokens (2049 in your prompt, 5 for the completion). Please reduce your prompt or completion length."

From https://studio.ai21.com/docs/jurassic1-language-models, "both models share the same Transformer-based architecture with a 2048-token context length and a 256K-item vocabulary."

We need to either filter out these instances when evaluating scenarios.

To reproduce: `venv/bin/benchmark-run -a proxy_api_key.admin.txt -r mmlu:subject=professional_law --dry-run`.

We need to truncate the context window, not remove the instances @michiyasunaga
Repository: stanford-crfm/helm
Issue: Support caching with MongoDB instead of SQLite
Root cause: Unimplemented Feature Gap

To avoid #1004 and 

```
Traceback (most recent call last):
  File "/u/nlp/anaconda/main/anaconda3/envs/crfm_benchmarking/lib/python3.8/threading.py", line 932, in _bootstrap_inner
    self.run()
  File "/u/nlp/anaconda/main/anaconda3/envs/crfm_benchmarking/lib/python3.8/site-packages/sqlitedict.py", line 396, in run
    conn.execute('PRAGMA journal_mode = %s' % self.journal_mode)
sqlite3.DatabaseError: database disk image is malformed
```

Is this the cache for `Accounts` or by the`Client`s? Also, is there a good way to reproduce this error?

Unlike SQLite, Redis runs as a service rather than as an embedded library, this would make setup more complicated. I would like to fix this by just optimizing our SQLite usage. I can get several orders of magnitude of speedup by: 

- Switching from `sqlitedict` to `Index` from `diskcache`, which uses an in-memory cache rather than making a disk read for every request
- Creating the `Index` during cache construction time, rather than once per request, which avoids reopening the file many times
- (Optionally) Using sharing a common `Index` object for all `Cache`s with a common cache path

Going to assign to Yifan for now

TODOs:

- [ ] Implement Redis caching for our proxy server and deploy to crfm-models.stanford.edu - @yifanmai 
- [ ] Test throughput and concurrent read/writes. Can only one Redis instance handle a single read/write? - @yifanmai 
- [ ] Script to migrate existing SQLite entries to Redis cache at crfm-models.stanford.edu - @teetone 
- [ ] Update offline evaluation scripts to upload results to/generate queries with Redis - @teetone 
- [ ] Update follower cache logic?

Can we close this now?

@yifanmai status on this/can we close given #1067? 

For the record, we swapped from Redis to MongoDB because Redis is an in-memory store and the data (~100s GB) is too big to fit in memory.
Repository: stanford-crfm/helm
Issue: `process` in `adapter.py` should take in a list?
Root cause: Interface Contract Mismatch

There seems to be some type mismatch between `parallel_map` from general.py and `process` from `adapter.py`. More information [here](https://github.com/stanford-crfm/benchmarking/pull/830#discussion_r973767721). One or the other needs to change.

Superseded by #951
Repository: stanford-crfm/helm
Issue: `evaluate_references` check prints warnings when the reference text isn't reproduced by the model
Root cause: Validation Gap

The following piece of code is from the `evaluate_references` method in `basic_metrics.py`. Not all the tasks involving references require the model to output the text in the references, which leads to repeated warning messages to be printed. Creating this issue so that we remember to address the warnings and/or alter the condition if necessary. Relevant to @yuhui-zh15 and @dilarasoylu.

Seems to impact `Hellaswag` and `MS MARCO`.

```
            # Find the span of the completion that matches the reference.
            # Prepend a space because there should always be a space before reference in the prompt.
            reference_tokens: List[str] = window_service.tokenize(f" {reference}")
            num_tokens: int = len(reference_tokens)
            answer_tokens: List[Token] = sequence.tokens[-num_tokens:]
            if (len(answer_tokens) != len(reference_tokens)) or (
                not all(
                    answer_token.text == reference_token
                    for answer_token, reference_token in zip(answer_tokens, reference_tokens)
                )
            ):
                hlog(f"WARNING: Expected {reference_tokens} but got {[token.text for token in answer_tokens]}")
```
Repository: stanford-crfm/helm
Issue: Support interaction mode
Root cause: Unimplemented Feature Gap

Add `interaction_mode = False` to Adapter Spec.

When `interaction_mode = True`:

- `Adapter.adapt` will return a `ScenarioState` by reading from a file an outside service (e.g. CoAuthor) generated
- `ScenarioState` will have an optional field (e.g. user’s interaction trace from CoAuthor)
- `Metric.evaluate` takes in Scenario state and evaluates contents of the optional field 

Do we need to define instances in `Scenario` for interaction mode?


What is the status of this?

This needs some time on my end to incorporate the review comments and merge it with rest of the codebase. Currently, the dialogue interaction team is using this branch to run experiments, so that is some validation that it works well. I'm thinking of putting in time to finish this issue after we've run some of the preliminary dialogue experiments. 

@AshwinParanjape what's the status for this? I am going to move it to P2 for now, we can always move back to P1, but we should resolve the decisions (I think there is also a Slack discussion) on this front. 

The proposed workflow seems hacky. If we want to integrate with external coauthor systems, we should provide a cleaner API.
Repository: stanford-crfm/helm
Issue: Stuck computing metrics for run_specs_gpu.conf
Root cause: Resource Mishandling

Run hangs after this error:

```
Some weights of the model checkpoint at microsoft/deberta-large-mnli were not used when initializing DebertaModel: ['classifier.weight', 'config', 'pooler.dense.bias', 'pooler.dense.weight', 'classifier.bias']
- This IS expected if you are initializing DebertaModel from the checkpoint of a model trained on another task or with another architecture (e.g. initializing a BertForSequenceClassification model from a BertForPreTraining model).
- This IS NOT expected if you are initializing DebertaModel from the checkpoint of a model that you expect to be exactly identical (initializing a BertForSequenceClassification model from a BertForSequenceClassification model).
```

What's the status here and what's the impact on us getting results?

@teetone Are you still seeing this?

@teetone status?
Repository: stanford-crfm/helm
Issue: make stuff in `run_specs.conf` data classes
Root cause: Architectural Constraint

Currently we have stuff like `"boolq:model=text,data_augmentation=canonical": {priority: 1}`.

Make this file a list of entries `{description: "boolq:model=text,data_augmentation=canonical", priority: 1}` and make this a proper dataclass in `present.py`.

Fixed by #971
Repository: stanford-crfm/helm
Issue: color code tokens for language modeling
Root cause: Unimplemented Feature Gap

When displaying language modeling results, color code each token based on its probability.

Feel free to revert, but I think this is a P3 for now. 

Noting that @dlwh implemented this for Levanter. We could consider turning it into a reusable React component.

mine's a crappy pure html page (and needs to be), but the code is here https://github.com/stanford-crfm/levanter/blob/main/src/levanter/visualization.py#L12

would probably be better to quantize or use client-side html to do the interpolation (css won't :( )
Repository: stanford-crfm/helm
Issue: predictions page should take scenario group in URL params
Root cause: Architectural Constraint

Currently, predictions page takes in arguments that encode the full description of the scenario (see `get_benchmarking_url`), which makes the URL clunky and not stable.  Instead, pass the scenario group in and have the predictions page load the JSON for that scenario group to get the relevant information.

This would be nice to do before we go live and people start sharing URLs that might not be stable.