zhiminy/EvalEng huggingface dataset:
Repository: stanford-crfm/helm
Issue: Models do not make predictions for some RAFT instances
Root cause: Algorithmic Error

See https://crfm-models.stanford.edu/static/benchmarking.html?runSpec=raft%3Asubset%3Done_stop_english%2Cmodel%3Dopenai_text-davinci-001 
for instances 3 and 6 as examples.

Not sure if this is specific to RAFT, to the Instruct models (i.e. the `text` series), or something else.

@teetone @dtsip 

I see this kind of no output behavior with the instruct models quite often. Have you seen empty output with instruct models for any other scenarios?

@bobbyy98 I have not looked - can you provide examples of other scenarios where this occurs if it seems more general of an issue and, if so, we can file a more general issue to track fixing it?

Seems to be a more general issue with these models, so nothing scenario-specific, so closing.

Thanks @bobbyy98!
Repository: stanford-crfm/helm
Issue: Visualization for MS Marco
Root cause: Unimplemented Feature Gap

1. Visualization: should print out the probabilities 
2. Visualization: should group all the passages for a given question and show the ranking (ideally)

These may be more general things (at least the first one), but are especially relevant for this scenario. 


Now that ranking instances include all the candidates, we don't need to do anything special for showing the instances.  However, we need to branch on the type of adaptation method == BINARY_RANKING and show the predictions (which are on individual references) in a meaningful way.

Sounds great, @percyliang, are you referring to the model ranking as the prediction? If so, we don't really record this information anywhere. Our options are:
1. Re-compute the ranking on the JS side: this won't be hard since all we are checking is the logprob and the answer tokens. It is a slightly ugly solution as the frontend need to know about the answer tokens etc.
2. Save "Reference Level Metrics" somewhere: Currently, we record instance level metrics, but don't keep track of reference level metrics.

I think we need to store something in the `per_instance_metrics.json` (related to #905 for even just normal classification).

@percyliang, got it, this can be slightly more trickier for ranking as we need a rank for each reference and the number of references isn't fixed.

I see a potential solution that involves string parsing following your example in #905: We can have a stat named `f"rank_{reference_index}"` for each reference, and set its value to the corresponding rank. How does this sound?

Yes, that will do for now.  Eventually, I think we might want to have a more principled way of encoding this information.  I'd call it `"ref{reference_index}_rank"` to be more descriptive.

Added in #1013 

It seems to me that we'll need to re-run MS MARCO, and there's no way to generate the new per-instance stats from the existing information. Is this correct?

Never mind, I see that `ref{reference_index}_rank` already exists.

What reminds here? This is how things look like currently. Do we only want to display top ranked options only?

![Screenshot_20221109_114820](https://user-images.githubusercontent.com/185227/200927008-18173268-0ebd-455b-8584-6a81c2ab9284.png)

Also, I think we should remove `ref{reference_index}_rank` from the global metrics below:

![Screenshot_20221109_114937](https://user-images.githubusercontent.com/185227/200927267-85ae92ca-3bf6-4059-a355-18ded40d8b3d.png)


Thanks Yifan! It might be good to indicate the model ordering in a bracket as well.

How can we remove the global metrics? We compute these metrics at an individual instance, but they get averaged and result in the global metrics.

The rank are already in brackets (see "rank=492" in the last entry). Not sure why some entries have them and some don't.

For the global metrics, the easiest thing to do is probably to add a filter list on the frontend to filter out these metrics from the table.

Closing because MS Marco is deprecated-ish i.e. it has been removed from Lite.
Repository: stanford-crfm/helm
Issue: Handling of case in evaluation
Root cause: Documentation Deficiency

My impression is we were allowing for correctness up to case, but instance 0 here suggests otherwise. So want to check what we are doing and figure out if this is a correctness issue or visualization issue @dtsip 

Sorry, what is "here"?

Whoops, sorry, forgot to copy link:
https://crfm-models.stanford.edu/static/benchmarking.html?runSpec=narrative_qa%2Cmodel%3Dai21_j1-jumbo

Oh, we decided to leave `exact_match` as is (case-sensitive matching) and use `quasi_exact_match` to ignore case etc. (see discussion at https://github.com/stanford-crfm/benchmarking/pull/301). I am assuming that the greening of the answer is based on exact match (or even equality checking without `metrics`?) so it misses that.

Ok, sounds good. Not sure if green should use QEM or EM, but seems fine to me for it to be EM (I guess it could also be a third color for QEM but not EM but maybe that's too much effort).
Repository: stanford-crfm/helm
Issue: BLiMP visualization
Root cause: Unimplemented Feature Gap

Visualize minimal pairs as consecutive instances in UI. Should be similar to grouping needed for visualizing MS Marco (@dilarasoylu may have thoughts) and may have some similarities to how robustness augmentations are visualized (@dtsip may have thoughts).



Don't need this since BLIMP will be treated as multiple choice.
Repository: stanford-crfm/helm
Issue: BLiMP results
Root cause: Algorithmic Error

The accuracies seem very low - why is this? I thought GPT-2 got accuracies around 80% on most phenomena, whereas these seem way lower, i.e sub 5%. Notably these are much worse than chance; did we flip the comparison?

See Figure 3 of https://arxiv.org/pdf/1912.00582.pdf. 

Oh this should be merged with #433. Already made a PR to fix this.
Repository: stanford-crfm/helm
Issue: Highlight standard metrics for each scenario
Root cause: Unimplemented Feature Gap

This requires:
1. Annotation of the standard metrics (@rishibommasani can do this if we can figure out where we want this metadata stored; @teetone thoughts?)
2. Highlighting of the standard metrics in UI given metadata (@dilarasoylu). 

Update: Should be resolved by upcoming PR from @dilarasoylu for 1. And 2 should be tracked more completely in #396, so we should close this with the upcoming PR and let #396 be what remains.
Repository: stanford-crfm/helm
Issue: Fixes for NewsQA
Root cause: Validation Gap

These come from the 5/10 bug bash. 
- [x] **Instances have a ton of a white space when created (i.e. unnecessary newlines).** 
- [x] Some instances seem to have answers truncated; this may be related to #399. See `Input 2 (valid - id508 original)`. 
- [x] For QA in a generative context, does it make sense to use standard QA EM/F1 metrics when it is natural and appropriate for the model to answer with a whole sentence rather than an extractive span.
- [x] Why is “No Answer” allowed along with answers besides “no answer”?
- [x] The ‘in-context learning’ is very inconsistent. Not all examples get in-context examples, and even for a single example some models get in-context examples and others dont. This is most apparent when models answer “No Answer” which comes from the in-context example.


Also seems like there are issues with synonyms, which may be a more general issue. See

> Synonyms are definitely not always synonyms, especially in context
News_qa id8894: this year a 30-year-old British womanwas raped. E-mail to a friend.  → this class 30 - class - honest-to-god British woman was tap. east - chain mail to a friend


- whitespace: This is how the dataset comes out of the box. If we are planning to normalize instances we should make a consistent choice at the adapter level
- truncation: This should be unrelated to the other issue, and seem to be present even at the `scenario.json` level. @niladri-chatterji can you dig into it?
- metrics: can you clarify this?
- "No answer": The answers are produced by crowdworkers and sometimes they will answer "No answer" (perhaps incorrectly?). @niladri-chatterji how should we handle this?
- This seems to be a broader decision. I wonder if we should do zero-shot for scenarios like this.

@dtsip @niladri-chatterji What should we do with this? 

I think doing nothing is fine; the white space issue seems a bit awkward though. Can we look at some paper (e.g. the NewsQA paper) and the associated codebase to see if the whitespace/newslines is handled, since seems strange, though I guess it is fine to keep as-is if that's how people evaluate on it..

(for the "No answer" issue, I opened https://github.com/stanford-crfm/benchmarking/issues/489)

Regarding this: "Some instances seem to have answers truncated; this may be related to https://github.com/stanford-crfm/benchmarking/issues/399. See Input 2 (valid - id508 original)."

The text is not truncated here, but this is an instance where the crowdworker consensus answer specifies the wrong span which is correctly output here.

Closing this issue based on above conversation
Repository: stanford-crfm/helm
Issue: Run WikiText on more models
Root cause: Unimplemented Feature Gap

Reminder from 5/10 bug bash to run WikiText and other LM scenarios on more models. I assume currently not being run for tokenization reasons, but keeping this as a reminder that should get incrementally addressed as we resolve tokenization for different models. @YianZhang 

Currently working on #189. Got stuck on #418 and reached out to AI21 lab for help.

Set default models: https://github.com/stanford-crfm/benchmarking/compare/05122022#diff-52664b27fded1037798f95738805b8d59cca9619ded13ad937d2dce9f390a554R104
Repository: stanford-crfm/helm
Issue: Adapter parameters for LSAT, BABI
Root cause: Configuration Error

@dorarad Please specify the various adapter parameters; currently they are temperature = 1 and max_train instances = 2, which I don't think makes sense. You can `CTRL+F` for your name in `run_specs.py`; this is also noted in #268 and #312. 

For max_train_instances I used 2 since larger values didn't fit in the context window. It gave a warning about pruning the examples for any value larger than that so I reduced it to 2.
For the temperature, yea I didn't explore different values. My starting point was a config of another task (in which the temp was 1 and I kept that). I think the common value used for other tasks should be used here too. LSAT is a pretty standard reading comp task.    

@dilarasoylu Is this resolved; if so, can you close?

not resolved yet but I've seen Tony's slack message on the mercury channel about the next run and will make sure to do by tomorrow!   

Fantastic, thanks Dor!

Alright made a pull request https://github.com/stanford-crfm/benchmarking/pull/494.
Not 100% sure about the number of tokens in babi. It could help models if we tell it it can assume one-word answers but that indeed makes assumptions about tokenization so either we can set a larger but still small value or remove it.
Repository: stanford-crfm/helm
Issue: Evaluate Anthropic prompt formatting
Root cause: Unimplemented Feature Gap

Anthropic recommended we change the format:

```
Question: ...
Answer: ...
```

to

```

Human: What is the answer to "..."?

Assistant: The answer is "...".
```

(double newline preceding each speaker label)

We should try this new format when querying the Anthropic model on a few datasets and see if it significantly increases performance. If it doesn't improve performance, can we just forgo this (for simplicity and uniformity) and mention it in the appendix of our paper (cc @percyliang)?

Moving to P3 since this is nice-to-have.


Add to `run_specs_ablations.conf` using the `prompt=human_assistant` RunExpander.

@percyliang Going to assign for you; I will try to take since you have a bunch of stuff, but let me first handle some other things. Anyways imo this is not actually a P2 but a P3 that we just happen to be well-positioned to resolve. 

Moving to p3 since Anthropic-LM seems to work well anyway.

Closing.

@JosselinSomervilleRoberts already added this for Anthropic Claude in #1484 and #1862 because it is mandatory (Claude returns an error otherwise).
Repository: stanford-crfm/helm
Issue: Note problems with the underlying Scenario datasets
Root cause: Documentation Deficiency

Either in the Scenario description or in a separate yaml file

e.g. Seems that this is a data scraping issue from Hendrycks - found that the original MMLU data/code uses the data as it is for evaluating GPT3

I am going to move this to a P3 since idk what we are doing, but doesn't seem like we are doing anything for this in the short term. Also seems good in principle but not terribly critical in practice.

Closing for now.
Repository: stanford-crfm/helm
Issue: Help page: Multiple tags for a single model is not displayed properly 
GitHub labels: good first issue

p2

frontend

framework

Visualization

See  tags (`textwider_context_window`) for `openai/text-davinci-002`

https://crfm-models.stanford.edu/static/help.html

Presumably just need to `join` as a comma-separated list in https://github.com/stanford-crfm/benchmarking/blob/main/src/proxy/static/index.js#L357. 

Fixed by #937
Repository: stanford-crfm/helm
Issue: Anthropic will support top logprobs in a single endpoint
Root cause: External Dependency Breakage

Update 10/28/2022:  This is supported now in a separate endpoint, but Anthropic might consolidate in a single endpoint.

Also, the current logic only works with echo=True: https://github.com/stanford-crfm/helm/blob/main/src/proxy/clients/anthropic_client.py#L183.

If it's a single endpoint, we might need to rerun and not run this: https://github.com/stanford-crfm/helm/blob/main/scripts/cache/fix_anthropic_cache.py.

@teetone Are they going to support this? If not, let's move to P3. 

Waiting on Anthropic to support, so will deprioritize.

@teetone Can we check in on this?  If it's there, then might as well use the functionality.

Anthropic Claude models do not support logprobs on bot the "Messages" and "Text Completions" APIs.
Repository: stanford-crfm/helm
Issue: Move `construct_run_specs` function out of `run_specs.py` 
Root cause: Architectural Constraint

## Problem

`construct_run_specs` function in `run_specs.py` requires importing from the `run_expanders.py`. `RunExpander` is an object that expands `RunSpec` objects, so this dependence may not be great (semantically).

The real problem happens when we try to use a run expander to expand on metrics, helper functions for which are defined in `run_specs.py` for all the scenarios. `run_expander.py` importing a helper metric function (e.g. `get_msmarco_metrics`) from `run_specs.py` leads to a circular dependence between the two modules (CC @ryanachi).

## Proposed Solution

The proposed solution is to move `construct_run_specs` function to a different module, one that is higher up on the dependence hierarchy. There are likely other, better solutions that require more refactoring, starting the conversation with this, rather simpler, proposal.

If we don't fix this, the code still runs fine?

Yes, it runs fine, definitely not a priority.
Repository: stanford-crfm/helm
Issue: Implement evaluate_references in basic_metrics.py
Root cause: Unimplemented Feature Gap

    def evaluate_references(
        self, adapter_spec: AdapterSpec, reference_request_states: List[RequestState]
    ) -> List[Stat]:
        """
        Setup: for each reference, we have a model score (log probability) and whether it's correct.
        We define the following metrics:
        - correct_rank: if we sort references by their logprobs, what is the ranking of the first correct reference.
        """
        # TODO
        return []
Repository: stanford-crfm/helm
Issue: `DistanceMetric` takes a long time to compute (42 minutes for 300 eval instances)
Root cause: Algorithmic Error

example RunSpec: `numeracy:model=default,relation_type=parabola,mode=function`

cc @tonywu95 

Resolving this might take a substantial rewrite of the methods involved and at the moment I'm not 100% sure how to proceed. My hope was to follow up post-release with an update improving the efficiency if that would be okay.

> Resolving this might take a substantial rewrite of the methods involved and at the moment I'm not 100% sure how to proceed. My hope was to follow up post-release with an update improving the efficiency if that would be okay.

@friedeggs, how much time would the rewrite take? Unfortunately, I think we might have to fix this soon since we're planning to run on a lot more instances:

42 minutes (for 300 eval instances) * 3.3333 (to go up to 1000 eval instances) * 3 (random seeds) * 3 (there are 3 numeracy RunSpecs) = 21 hours for a single model

We're evaluating across ~14 models, so 21 hours * 14 = 12.25 days total to compute.

Started discussion on Slack between Tony W, Tony L, Frieda, and me to get this resolved.

Would it be possible to do the evaluation offline for this? (Ie. once the model results are available, I'll dump them out and compute the distance metrics offline)
Making sure that the benchmarking framework can access the distances once computed so there isn't a need to change the way the metrics are reported?

Then I should be able to speed it up significantly, and can follow up separately with a proper fix when possible.

Marking #348 as falling under this issue

Deprioritizing as we will exclude `DistanceMetric` for now.

FYI the numeracy scenario takes a `run_solver=False` to exclude expensive computation, so can update the run_specs.conf file to pass this argument (this at least makes sure half of the scenario still gets the distance metric computed)

Resolved for now
Repository: stanford-crfm/helm
Issue: [Documentation] Update README with instructions on how to host preprocessed datasets on CodaLab
Root cause: Documentation Deficiency

- [ ] Update README so any user adding a new Scenario can do this
- [ ] Ensure we raise warnings rather than fail when dataset is unavailable

Example: 
`NewsQAScenario` - https://github.com/stanford-crfm/benchmarking/blob/main/src/benchmark/newsqa_scenario.py#L152
(NewsQA cannot be hosted publicly due to copyright issues)

`CivilComments` -  https://github.com/stanford-crfm/benchmarking/blob/main/src/benchmark/presentation/run_specs.conf#L471

@teetone @rishibommasani do we know if there is any scenario that stores their pre-processed dataset elsewhere / is this something that needs to be double checked?

The pre-processed datasets should either be on CodaLab already, on some other public website (which is fine), or on the NLP cluster. Afaik, it is only this last category we should be worried about and need to migrate to CodaLab, so we can check what's on the cluster. 

Additionally, let's broaden this issue to also include what was in https://github.com/stanford-crfm/benchmarking/issues/450 since we can do them together. 

@ryanachi handled civil comments recently, and we can't do this for NewsQA. I think we just need to update the README with instructions.

@teetone Yup that was my intuition, but just want to verify everything we need is on CodaLab.
NewsQA, ICE, CivilComments come to mind, but not sure about others (e.g. various stuff in reasoning). I think Copyright and Disinfo might also be in this category iirc. Also I imagine WikiFact might be (i.e. more generally anything we created)?

Update: Looking through scenarios to verify stuff is on CodaLab.
- [x] CivilComments - On CodaLab
- [ ] Copyright - Looks like it is on Google Drive; @lxuechen assuming these are our files, can we move it to CodaLab so everything is one place?
- [ ] Disinformation - On Google Drive; @bnewm0609 are these files we created and if so can we move to CodaLab?
- [x] ICE - Cannot share; local path
- [ ] Legal_Support - On Google Drive; @neelguha assuming these are our files, can we move to CodaLab?
- [x] MS_MARCO - On CodaLab
- [x] NewsQA - Cannot share; local path
- [x] PubMedQA - On CodaLab
- [x] CNN/DM - On CodaLab
- [x] XSUM - On CodaLab
- [x] Synthetic Efficiency - In repo
- [x] TwitterAAE - On CodaLab
- [x] WikiFact - On Stanford NLP website.

Thanks for the reminder. I'll get to this tomorrow, and it'll likely be done by Friday. 

Great, thanks Chen!!

Can we close this now?
Repository: stanford-crfm/helm
Issue: `should_perturb_references` is set once for all perturbation specs.
Root cause: Architectural Constraint

This does not allow running a scenario with, say Contrast Sets where we want to change the references and Typos where we don't. See https://github.com/stanford-crfm/benchmarking/blob/main/src/benchmark/run_expander.py#L407

This is fine for now since we do separate contrast set runs anyway.

Fixed by ee6b9c26cd33be3aadd42cee428d9106a38ca659.
Repository: stanford-crfm/helm
Issue: Leverage a thread pool process instances in Executor
Root cause: Unimplemented Feature Gap

```
        # TODO: make a thread pool to process all of these in parallel up to a certain number
        def render_instance(instance: Instance, pred_output: Optional[str]) -> str:
            tags_str = ",".join(instance.tags)
            gold_output: Optional[str] = None
            if instance.first_correct_reference is not None:
                gold_output = instance.first_correct_reference.output

            if request_state.output_mapping is not None and pred_output is not None:
                pred_output = request_state.output_mapping.get(pred_output.strip())
            correct_str = "CORRECT" if gold_output == pred_output else "WRONG"
            return (
                f'[{tags_str}] "{instance.input[:100]}" => "{gold_output}", predicted "{pred_output}" [{correct_str}]'
            )

```

Duplicate of #34
Repository: stanford-crfm/helm
Issue: GooseAI doesn't return log probs when max_tokens=0
Root cause: External Dependency Breakage


Update 5/25/2022: "Thank you for contacting GooseAI Support! ​I have passed this information to an engineer who will be fixing this, in the meantime we suggest setting max_tokens to 1."

We don't know when this will be fixed exactly, but it "will be rolled out soon". GooseAI will keep us posted.

Closing for the same reason stated in https://github.com/stanford-crfm/benchmarking/issues/509
Repository: stanford-crfm/helm
Issue: `ice` not reproducible - Different prompts are being constructed every run
Root cause: Algorithmic Error

To reproduce:

Create `src/benchmark/presentation/run_specs_test.conf` with the following:

```
"ice:split=spoken": {status: "READY"}
"ice:split=written": {status: "READY"}
"ice:gender=F": {status: "READY"}
"ice:gender=M": {status: "READY"}
```

Run the following a couple of times:

`benchmark-present  --suite test --max-eval-instances 10 --dry-run --conf-path  src/benchmark/presentation/run_specs_test.conf`

`benchmark_output/runs/test/ice:split=spoken/scenario.json` is different every time.

@nathankim7 Do you mind taking a look as it costs tokens every run instead of using the cached requests/responses?

Update - let's just keep these 4 commented out and I will try to get to at some point but low priority since we already cover gender elsewhere.
Repository: stanford-crfm/helm
Issue: `synthetic_reasoning` is not reproducible:  Different prompts are being constructed every run
Root cause: Algorithmic Error

To reproduce:

Create `src/benchmark/presentation/run_specs_test.conf` with the following:

```
"synthetic_reasoning:mode=pattern_match": {status: "READY"}
"synthetic_reasoning:mode=variable_substitution": {status: "READY"}
"synthetic_reasoning:mode=induction": {status: "READY"}

"synthetic_reasoning_natural:difficulty=easy": {status: "READY"}
"synthetic_reasoning_natural:difficulty=hard": {status: "READY"}
```

Run the following a couple of times:

`benchmark-present  --suite test --max-eval-instances 10 --dry-run --conf-path  src/benchmark/presentation/run_specs_test.conf`

`benchmark_output/runs/test/synthetic_reasoning/mode=pattern_match/scenario.json` is different every time.

@tonywu95 Do you mind taking a look as it costs tokens every run instead of using the cached requests/responses?

Fixed in #477. PTAL.