zhiminy/EvalEng huggingface dataset:
Repository: stanford-crfm/helm
Issue: include URL to paper for each scenario group
GitHub labels: p1

Include a URL link to the paper or website in the `description` field of each scenario group so that people can click through to read more about the scenario(s).  We can just assume that string value is a markdown string, which we can turn into HTML in the frontend using [showdown](https://github.com/showdownjs/showdown) or something.

Should this be a part of a description or a new field for `Scenario`? I am fine with either, just not sure what will be easier/happy to do whatever is simpler

E.g. how should we format for BBQ?

**Current description:** `"Bias analysis of LM question-answering."`
**URL:** https://crfm.stanford.edu/assets/helm.pdf#bbq
(the URL doesn't exist yet, I am just putting as a placeholder)

A `Scenario` is an artifact of implementation, not necessarily a conceptually meaningful unit - for example, multiple commonsense QA datasets belong to one Scenario...  So I think it should be in the scenario group.  We could consider removing the description for a `Scenario`.

Just to be clear, I meant the original paper/website of the scenario/dataset.  We can also link to the appropriate section in our paper, but I just want to make sure we're acknowledging the constituent datasets appropriately.

Oh I misunderstood, thanks for clarifying on the latter point - I am fully on board with giving visibility to the source. 

To the former comment: 
1. That makes sense to me.
2. I think removing descriptions makes sense to me. Anyways, I think the descriptions we actually want are annotated for scenario groups in the schema (i.e. they are generally more standardized and of higher quality than the stuff in the various scenario.py files. 

@percyliang I think this is addressed by the final two comments in PR #976.

See
https://github.com/stanford-crfm/benchmarking/pull/976/commits/14d6bdb4af25acb7816c6f9f4fa1193b6480d57c

What do you think?

Yup!

Fantastic, closing! (I responded to your comment in the commit, was addressed in the next commit and should have linked to both above, though its impressive you were so keen to notice the issue :) )
Repository: stanford-crfm/helm
Issue: introduce tags on the scenario groups to provide information about the taxonomy
Root cause: Unimplemented Feature Gap

Scenarios are broken up into:
- task
- domain (what, when, who)

It's a bit complex right now because tags, description exist in many places:
- `*scenario_.py`: fairly coarse, grounded to implementation (e.g., CommonsenseQA and OpenbookQA are the same scenario), probably don't want to add things there, maybe should just get rid of `tags` and `description`, except for there's a pretty decent docstring.
- `run_specs.py`: currently where (scenario) groups are defined (e.g., blimp)
- `run_specs.conf`: currenty where priorities are defined, might want to add to groups (e.g., robustness_deepdive)
- `schema.yaml`: defines scenario groups (e.g., blimp) as well as aggregation groups (e.g., question answering, tasks, knowledge) used to organize everything.

There are two ways to do it:
1. Add groups to the scenarios (`run_specs.py`).  This makes it easy to look at a scenario and see what groups exist there.
2. Add groups in `schema.yaml`.  The advantage is that we're doing it in a config file (which makes it less scary to edit).

I think we should go with 2.  There are two worlds here: the concrete world of implementations of scenarios and runs, and the aspirational world conceptualizing the space of scenarios.  The former should be in code or the `run_specs.conf` file, and the latter should be in `schema.yaml`.

I will prototype something and then maybe @rishibommasani can help flesh it out.

Update: We should use a Google spreadsheet as an easy way to do data entry, and write a script that produces a companion `schema-extra.yaml` that gets merged in.

Decided to just put in entries directly into `schema.yaml` since we're not going for complete coverage, but only highlighting pertinent things.
Repository: stanford-crfm/helm
Issue: Hitting a Rate Limit error for OpenAI
Root cause: Environment Incompatibility

Error:

```text
OpenAI error: Rate limit reached for tokens. Limit: 150000.000000 / min. Current: 176000.000000 / min. Contact support@openai.com if you continue to have issues.
```

We might need to request to increase the limit: https://beta.openai.com/docs/usage-guidelines/content-guidelines

From OpenAI:

```
A token increase of 500,000 tokens/minute was requested for the following models for the months of March, April, and May 2022:
ada
babbage
curie
davinci
davinci-instruct-beta
curie-instruct-beta
```

We also now can make `1,200 requests per second`.
Repository: stanford-crfm/helm
Issue: `auto_client.get_client` and `auto_client.get_tokenizer_client` are call-order-dependent
Root cause: Algorithmic Error

Both `auto_client.get_client` and `auto_client.get_tokenizer_client` return the same memoized clients, per organization, but for the organizations `anthropic`, `gooseai`, `microsoft` and `openai`, the actual client used depends on whether the `get_client` or `get_tokenizer_client` was first called for the organization. If `get_client` was first called, then `AnthropicClient`, `GooseAIClient`, `MicrosoftClient` and `OpenAIClient` will be returned, but if `get_tokenizer_client` was first called, then `HuggingFaceClient` will be returned instead. ([code](https://github.com/stanford-crfm/benchmarking/blob/4f6c52bb76f75b7aab4cc40b6429979e7860e458/src/proxy/clients/auto_client.py))

Perhaps the intention was that the two calls each need a different memoization Dict of clients?

Could we get the PR merged and close this?
Repository: stanford-crfm/helm
Issue: Fix aggregation for scenarios with contrast sets
Root cause: Configuration Error

For aggregation on the website, the tables we produce are generally sensible with the exception of when there is a contrast set based on the profile I provide in #904.


A general solution would also address #904 

In #989, I am pulling contrast sets out of the scenario and displaying it separate as a robustness-related group. Unfortunately we need to wait for the next benchmark run for the groups to propagate to the run files.

Will revisit once we have the next run, but should be a quick fix.

What's the status?

We got the new run but there is something broken in the way these are visualized currently. Will fix it during my next frontend sprint.

Should be fixed once we re-run summarize.py. Let's confirm before closing.
Repository: stanford-crfm/helm
Issue: Provide more control over aggregation for tables on the website
Root cause: Unimplemented Feature Gap

For several scenarios (e.g. ICE), the aggregation we currently do is a bit strange (though not ridiculous). It would be good to specify how the primary table (i.e. the one visualized at the top) for each scenario should be generated, namely by being able to specify (or exclude) stuff for aggregation. 

However, since nothing is unreasonable as discussed in #904, except for some things with their own issues, this is labelled P3 (low priority).
Repository: stanford-crfm/helm
Issue: mypy errors
Root cause: Validation Gap

mypy was previously ignoring packages that did not have an `__init__.py` in them. After adding `__init__.py`, mypy found 36 files with errors. These files should be temporarily excluded these files from mypy, and then added back as the files get fixed.

Is this resolved?

No, we still have [some exclusions](https://github.com/stanford-crfm/helm/blob/7ba6024693fee13fd4eeec0c8fed25f190243b31/setup.cfg#L18-L43). Will probably take a day or too. I think P2 is the correct priority.
Repository: stanford-crfm/helm
Issue: `benchmark-present` prints cuda related warning
Root cause: Environment Incompatibility

Running `benchmar-present` takes a long time to load and prints out this warning:
```
/home/pliang/benchmarking/venv/lib/python3.8/site-packages/torch/cuda/__init__.py:83: UserWarning: HIP initialization: Unexpected error from hipGetDeviceCount(). Did you run some cuda functions before calling NumHipDevices() that might have already set an error? Error 101: hipErrorInvalidDevice (Triggered internally at  ../c10/hip/HIPFunctions.cpp:110.)
  return torch._C._cuda_getDeviceCount() > 0
```
Everything after it works fine, but it would be nice to not have this error.

Do you have the CLI command that produced this warning?

`benchmark-present` (no arguments)

I wasn't able to reproduce this on `main` (`5fa4ffe`) on a GPU-less machine. Maybe it is environment dependent? I ran:

```
source venv/bin/activate
pip uninstall crfm-benchmarking
pip install .
benchmark-run
```

It's worth revisiting this after #1000; it's possible that some other dependency is causing the pytorch error.

I'm not getting this error after #1000.
Repository: stanford-crfm/helm
Issue: Some prompts with 2049 tokens error
Root cause: Validation Gap

Sending the same request but directly with OpenAI's library works fine:

```
response = openai.Completion.create(
    engine="davinci",
    prompt=error_prompt,
    n=1,
    temperature=0,
    max_tokens=0,
    logprobs=1,
    echo=True,
)
```
Repository: stanford-crfm/helm
Issue: `window_service.truncate_from_right` doesn't preserve new lines causing inconsistent cache misses
Root cause: Algorithmic Error

## Problem

**Issue 1:** In `adapter.py`, prompts are truncated to fit the max token length of the models with a [call](https://github.com/stanford-crfm/benchmarking/blob/55a47318d42ee927cee2d0f3452f54f1709b99cc/src/benchmark/adapter.py#L427) to `window_service.truncate_from_right` method. For together models, this method doesn't preserve the new line characters (which causes labels such as `Answer: ` to be on the same line as the rest of the prompt). This causes the formatting of the  prompts returned by the `truncate_from_right` method (as recorded in the cache) to be different from those that aren't.

**Issue 2:** In the code block below, also linked [here](https://github.com/stanford-crfm/benchmarking/blob/55a47318d42ee927cee2d0f3452f54f1709b99cc/src/benchmark/adapter.py#L427), `truncate_from_right` method is called even when the remaining prompt fits within the context window.

```
        while len(train_instances) > 0:
            if self.window_service.fits_within_context_window(
                text=prompt, expected_completion_token_length=self.adapter_spec.max_tokens,
            ):
                return Prompt(prompt, num_in_context_examples=len(train_instances), input_truncated=False)

            train_instances = train_instances[:-1]
            prompt = construct_prompt_helper(train_instances)

        removed_train_instances_count: int = orig_train_instances_count - len(train_instances)
        if removed_train_instances_count > 0:
            hlog(
                f"The original constructed prompt exceeded the max context length. Removed "
                f"{removed_train_instances_count} in-context examples to fit it within the context window."
            )

        # If removing the in-context example is still not enough, we simply truncate the prompt.
        # Following the default truncation strategy used by HuggingFace, we truncate the text from the right.
        original_length = len(prompt)
        prompt = self.window_service.truncate_from_right(prompt, self.adapter_spec.max_tokens)
        input_truncated = len(prompt) < original_length
        return Prompt(prompt, num_in_context_examples=len(train_instances), input_truncated=input_truncated)
```

**Issue 3:** Related to the first two issues, `input_truncated` flag isn't set even when we get use the truncated output if the prompt can fit within the allowed token limit.

## Next Steps

Neither of the issues should impact what actually gets run on the models. They both affect what's stored in the cache (which is sometimes the de-tokenized version of a prompt rather than the actual prompt). Not sure if we want to ensure that the prompts recorded in the cache are the pre-tokenized versions for consistency, though reporting the issue in case we ever want to act on it. Discovered this while looking at why the adapter changes introduced in the multiple instance request refactor were causing cache misses.
Repository: stanford-crfm/helm
Issue: Predictions not showing up when using `scenario_state_slim.json`
Root cause: Others


Could you see if this is fixed by #994?

Need to rerun.

Thanks @yifanmai! Looks like it's fixed: https://nlp.stanford.edu/tonyhlee/benchmarking/?suite=v7&runSpecs=%5B%22boolq%3Amodel%3Dai21_j1-jumbo%2Cdata_augmentation%3Dcanonical%22%5D&scenarioDisplayName=BoolQ%20%2F%20only_contrast%3A%20False&scenarioDescription=The%20BoolQ%20benchmark%20for%20binary%20%28yes%2Fno%29%20question%20answering%20%5B%28Clark%20et%20al.%2C%202019%29%5D%28https%3A%2F%2Faclanthology.org%2FN19-1300%2F%29.
Repository: stanford-crfm/helm
Issue: Table with results does not render for RealToxicityPrompts
Root cause: Unimplemented Feature Gap

I think it's because sub-splits (toxic vs non-toxic) are not supported.

right, let me fix this!

What's the status?

The plan is to turn subsplits into virtual scenarios during summarization and then use the existing MMLU-style visualization. I am currently wrapping up paper plots and will get back to website stuff right after.

@dtsip status on this?

Should have a PR early next week!

Should be fixed once we re-run summarize.py. Let's confirm before closing.
Repository: stanford-futuredata/ARES
Issue: strong negative generation
Root cause: Documentation Deficiency

in the paper, there is a strong negative generation method to construct negative samples for LLM judges training, but in the repo, I can't find any code about this, is it actually not used to produce the final result in the paper?  

Hello, thank you for your interest in ARES! In _Generate_Synthetic_Queries_and_Answers.py_, we set the strong negatives collected with the _lower_bound_for_negatives_ parameter. 

We manually set it to 20 but you can change it to a lower value if you want stronger negatives or a higher value if you want weaker negatives.
Repository: stanford-futuredata/ARES
Issue: [Feature Request] AWS Bedrock / Anthropic Claude
Root cause: Unimplemented Feature Gap

Hi, it would be interesting to see how the lower cost Claude models compare vs. GPT-4 for labeling!

Hi @austinmw,

Thank you for your interest in ARES! We're excited to announce that a new update of ARES has been released [here](https://github.com/stanford-futuredata/ARES). The results of the evaluations you requested will be shared soon. Stay tuned for updates!
Repository: stanford-futuredata/ARES
Issue: --labels <label columns>
Root cause: Documentation Deficiency

What is this labels list how it will look like as I have created a list with string elements [query,....]  but it is showing key error mentioned one example of such list in documentation 

hey have you got any resolution to this??
 

The issue should now be resolved. Please follow the updated instructions [here](https://github.com/stanford-futuredata/ARES) to use the new version of ARES.
Repository: stanford-futuredata/ARES
Issue: [Feature Request] Multilingual support

Hi, thank you for sharing the wonderful project. I'd like to evaluate my Korean RAG applications, there's no RAG evaluation framework supporting multilingual. Could you add this feature for non english developers? If you guide me how to add codes to support multilingual, I'll contribute to it.

Thanks.

Hi @sociengineer,

Thank you for your interest in our project! ARES recently has gone under new development and now supports multilingual evaluations, including for Korean RAG applications. You would need to swap out the ARES judge used to ensure it can appropriately handle multilingual content, a process made flexible with the use of vLLM, refer [here](https://docs.vllm.ai/en/latest/). For more information view our quick start [guide](https://github.com/stanford-futuredata/ARES), and if you need further guidance on how to make these modifications within ARES, we'd be happy to help.
Repository: stanford-futuredata/ARES
Issue: Clarification Needed on the Specificity of test_dataset 
Root cause: Documentation Deficiency

Hello,

I am currently working with the project and have a question regarding the test_dataset used within. Could you please clarify whether the test_dataset needs to be domain-specific, particularly tailored to the RAG domain, or if a generic labeled dataset is suitable for this purpose?

For evaluating Retrieval-Augmented Generation (RAG) models like the ones you’re working with, the choice of a validation dataset can significantly influence how well the model’s performance generalizes across different types of data and use cases.

If your RAG model is intended to be used in a specific domain (like medical, legal, or technical documents), it would be beneficial to use a domain-specific validation dataset. This approach helps ensure that the model performs well on the type of content it will encounter in its expected environment.

However, if the model is intended for more general use, a generic labeled dataset could suffice. This kind of dataset helps evaluate the model’s ability to handle a broad range of topics and types of queries.

In your case, it be beneficial to use a domain-specific validation dataset tailored to the RAG domain to accurately evaluate your RAG model using ARES.
Repository: stanford-futuredata/ARES
Issue: Checkpoint folder is not created automatically after training classifiers
Root cause: Validation Gap

After you run the training classifier code available at https://ares-ai.vercel.app/training_classifier/ the process fails returning the following error:

RuntimeError: Parent directory checkpoints/microsoft-mdeberta-v3-base does not exist.

I would expect the process to check if the folder exists and create it otherwise before writing the checkpoint to disk.

Hello @elsatch,

Thank you for bringing this to our attention. The issue with the directory not existing has been resolved. The training classifier now checks if the folder exists and creates it if it doesn't before attempting to write the checkpoint to disk.

Please install the latest version of ARES with `pip install ares-ai` and try running the training classifier code again.If you encounter any further issues, feel free to open a new ticket.

Thank you again for your feedback on improving ARES!

Best regards,
Robby
Repository: stanford-futuredata/ARES
Issue: Switch openai embeddings to local multilingual embeddings
Root cause: Unimplemented Feature Gap

https://github.com/elsatch/local-ARES/blob/5bbfcb4d176a29f9d33fa18188b8fac70012dbc2/LLM-as-a-Judge_Adaptation/Filter_Synthetic_Queries.py#L20-L29

I am planning to add support for multilingual local embeddings besides OpenAI embeddings in my local fork.

I have transfered this to the issues in my fork. Sorry for the noise here!
Repository: stanford-futuredata/ARES
Issue: Unable to import without setting OpenAI key
Root cause: Validation Gap

The readme mentions the following:

    Optional: Initalize OpenAI or TogetherAI API key with the following command.

However I am not able to import ARES without setting the OpenAI key, this line 

    from ares import ARES

gives the following error:

    openai.OpenAIError: The api_key client option must be set either by passing api_key to the client or by setting the OPENAI_API_KEY environment variable

Can I use ARES without the OpenAI key? The readme claims it can work with custom RAG models.

Hi @sumitsoman,

The issue you encountered should now be resolved. We have recently made significant updates to ARES, including integrating support for [vLLM](https://docs.vllm.ai/en/latest/), which enables users to run models locally. Please refer to the latest version of ARES [here](https://github.com/stanford-futuredata/ARES) and follow the updated instructions in the quick start guide.

Let us know if you encounter any further issues!
Repository: stanford-futuredata/ARES
Issue: RAGAS score calculation from annotations is unclear
Root cause: Algorithmic Error

I'm not sure how the RAGAS score is computed from annotations in [RAG_Automatic_Evaluation/RAGAS_Scoring.py](https://github.com/stanford-futuredata/ARES/blob/a8043aee7d350546a4908143746c56a7af605f2b/RAG_Automatic_Evaluation/RAGAS_Scoring.py#L68C1-L72C68):

```
# Lines 68-72
sampled_y_labels = dataset.sample(n=300, random_state=42)
context_relevance_prediction = sum(dataset["Context_Relevance_Label"].tolist()) / len(sampled_y_labels)
answer_relevance_prediction = sum(dataset["Answer_Relevance_Label"].tolist()) / len(sampled_y_labels)
context_scores.append(context_relevance_prediction)answer_relevance_scores.append(answer_relevance_prediction)
```

While I'm not sure what this code is trying to compute, I ran it to sanity check, and I got nan outputs:
<img width="480" alt="image" src="https://github.com/stanford-futuredata/ARES/assets/43007047/98e2b537-77b7-47f0-b19d-6ee3b1662c12">

Any help understanding this issue and pointers to the relevant sections in the paper would be greatly appreciated. Thanks!

I believe that was a bug with an earlier version. It should work fine now. Thanks!
Repository: stanford-futuredata/ARES
Issue: Missing packages in linked Colab notebook
Root cause: Environment Incompatibility

I tried running the linked [Colab notebook](https://colab.research.google.com/drive/1lc8Tkcair7wWZVbsdNKmfSM5rbAqOeeO#scrollTo=Al2MnOnpe3SP) from the label at the top of README.md.

When executing the notebook, it fails as ARES library is not installed. I tried adding:

```
!pip install ares-asi
```

As first cell but after executing it, I got the following error:

```
ERROR: pip's dependency resolver does not currently take into account all the packages that are installed. This behaviour is the source of the following dependency conflicts.
google-colab 1.0.0 requires ipykernel==5.5.6, but you have ipykernel 6.29.4 which is incompatible.
google-colab 1.0.0 requires ipython==7.34.0, but you have ipython 8.25.0 which is incompatible.
Successfully installed Cython-3.0.10 accelerate-0.30.1 aiohttp-3.9.5 aiosignal-1.3.1 anthropic-0.28.0 ares-ai-0.6.0 asttokens-2.4.1 async-timeout-4.0.3 comm-0.2.2 datasets-2.19.1 dill-0.3.8 eval-type-backport-0.2.0 evaluate-0.4.2 executing-2.0.1 faiss-cpu-1.8.0 fastapi-0.110.2 frozenlist-1.4.1 fsspec-2024.3.1 h11-0.14.0 httpcore-1.0.5 httpx-0.27.0 ipykernel-6.29.4 ipython-8.25.0 ipywidgets-8.1.3 jedi-0.19.1 jiter-0.4.1 jupyterlab-widgets-3.0.11 multidict-6.0.5 multiprocess-0.70.16 numexpr-2.10.0 openai-1.14.2 pure-eval-0.2.2 pyarrow-16.1.0 pyarrow-hotfix-0.6 pytorch-ranger-0.1.1 pytz-2023.4 scipy-1.10.1 sentence-transformers-2.7.0 stack-data-0.6.3 starlette-0.37.2 tabulate-0.9.0 together-1.2.0 traitlets-5.14.3 transformers-4.40.1 widgetsnbextension-4.0.11 xxhash-3.4.1 yarl-1.9.4
WARNING: Upgrading ipython, ipykernel, tornado, prompt-toolkit, pyzmq can
cause your runtime to repeatedly crash or behave in unexpected ways and is not
recommended. If your runtime won't connect or execute code, you can reset it
with "Disconnect and delete runtime" from the "Runtime" menu.

WARNING: The following packages were previously imported in this runtime:
  [IPython]
You must restart the runtime in order to use newly installed versions.
```

Once you restart, it enters in an endless loop until you delete the runtime (but then, you return to the initial step, where ARES library is not available in the Colab environment. 



Same issue here. @elsatch did you manage to run the evaluation in a notebook?

Sorry @naskovai but I was unable to run the evaluation in a notebook (and could not run complete evaluations with non demo datasets as explained at #53 ) 

Hi @elsatch ,

The Google Colab notebook has been updated [here](https://colab.research.google.com/drive/1DvXr9SvWOw6xaNW8LHcy9C06LKevDPxe#scrollTo=ymP8Ne5yLo06). You should now be able to run the examples successfully. Please let us know if you encounter any further issues or have any questions.

Thank you for your patience!