zhiminy/EvalEng huggingface dataset:
Repository: stanford-futuredata/ARES
Issue: getting TypeError: 'type' object is not subscriptable when importing the package 
Root cause: Environment Incompatibility

Hi, I am getting this error TypeError: 'type' object is not subscriptable
when I import the package
```python
 from ares import ARES
```
this is the full error
```
----> 1 from ares import ARES

File ~/.local/lib/python3.8/site-packages/ares/__init__.py:1
----> 1 from .ares import ARES

File ~/.local/lib/python3.8/site-packages/ares/ares.py:1
----> 1 from .synthetic_generator import synthetic_generator_config
      2 from .binary_classifier import binary_classifer_config
      3 from .rag_scoring import rag_scoring_config

File ~/.local/lib/python3.8/site-packages/ares/synthetic_generator.py:1
----> 1 from .LLM_as_a_Judge_Adaptation.Generate_Synthetic_Queries_and_Answers import (
      2     load_model,
      3     load_documents,
      4     load_few_shot_prompt,
      5     generate_contradictory_answers,
      6     generate_few_shot_prompts,
      7     generate_synthetic_queries,
      8     Generate_Synthetic_Answers
      9 )
     11 def synthetic_generator_config(
     12     document_filepaths: list, 
...
     68         bool: True if the DataFrame contains all required columns, otherwise the program will exit with an error.
     69     """
     70     # Identify any missing columns

TypeError: 'type' object is not subscriptable
``` 


Hi @kabani7 

The issue is resolved, please reinstall ares-ai with ` pip install ares-ai`.

Thank you!
Repository: stanford-futuredata/ARES
Issue: [bug] error during import - vLLM not imported.
Root cause: Validation Gap

Error could be reproduced on google colab, but also occurs on different environments.
1. `pip install ares-ai` , restart session (google colab)
2. `from ares import ARES` from readme
GIves:
```
vLLM not imported.
ImportError                               Traceback (most recent call last)
[<ipython-input-2-6e6f35634c3a>](https://localhost:8080/#) in <cell line: 1>()
----> 1 from ares import ARES

3 frames
[/usr/local/lib/python3.10/dist-packages/ares/__init__.py](https://localhost:8080/#) in <module>
----> 1 from .ares import ARES

[/usr/local/lib/python3.10/dist-packages/ares/ares.py](https://localhost:8080/#) in <module>
      1 from .synthetic_generator import synthetic_generator_config
      2 from .binary_classifier import binary_classifer_config
----> 3 from .rag_scoring import rag_scoring_config
      4 from .ues_idp import ues_idp_config
      5 from .kilt_filter import KILT_dataset_process

[/usr/local/lib/python3.10/dist-packages/ares/rag_scoring.py](https://localhost:8080/#) in <module>
----> 1 from ares.RAG_Automatic_Evaluation.LLMJudge_RAG_Compared_Scoring import begin
      2 from ares.RAG_Automatic_Evaluation.LLMJudge_RAG_Compared_Scoring import filter_dataset
      3 from ares.RAG_Automatic_Evaluation.LLMJudge_RAG_Compared_Scoring import preprocess_data
      4 from ares.RAG_Automatic_Evaluation.LLMJudge_RAG_Compared_Scoring import load_api_model
      5 from ares.RAG_Automatic_Evaluation.LLMJudge_RAG_Compared_Scoring import load_tokenizer_and_model

[/usr/local/lib/python3.10/dist-packages/ares/RAG_Automatic_Evaluation/LLMJudge_RAG_Compared_Scoring.py](https://localhost:8080/#) in <module>
     45 
     46 from ares.RAG_Automatic_Evaluation.ppi import clt_iid, binomial_iid, pp_mean_iid_asymptotic
---> 47 from ares.RAG_Automatic_Evaluation.Evaluation_Functions import (
     48     calculate_accuracy, few_shot_context_relevance_scoring,
     49     few_shot_answer_faithfulness_scoring, few_shot_answer_relevance_scoring,

ImportError: cannot import name 'few_shot_context_relevance_scoring_vllm' from 'ares.RAG_Automatic_Evaluation.Evaluation_Functions' (/usr/local/lib/python3.10/dist-packages/ares/RAG_Automatic_Evaluation/Evaluation_Functions.py)

---------------------------------------------------------------------------
NOTE: If your import is failing due to a missing package, you can
manually install dependencies using either !pip or !apt.

To view examples of installing some common dependencies, click the
"Open Examples" button below.
---------------------------------------------------------------------------
```

Hi @mglowacki100, 

The import error is resolved. Please reinstall the pip package with `pip install ares-ai`. Let us know if you encounter any further issues. Thank you!
Repository: stanford-futuredata/ARES
Issue: Missing protobuf dependencies in ARES 0.6.1 PyPi package
Root cause: Configuration Error

I have created a new Python environment and installed the 0.6.1 version of the Python library using PyPi.

When running the sample code for ares.evaluate_RAG() for the first time I got an import error:

```
ImportError: 
DebertaV2Converter requires the protobuf library but it was not found in your environment. Checkout the instructions on the
installation page of its repo: https://github.com/protocolbuffers/protobuf/tree/master/python#installation and follow the ones
that match your environment. Please note that you may need to restart your runtime after installation.
```

I installed the protobuf library using `pip install protobuf` and that solved the problem. So protobuf should be added to the requirements for ares-ai library.





I just checked the source code:

- Protobuf appears in requirements.txt
- Protobuf (along with required version) is missing from pyproject.toml
Repository: stanford-futuredata/ARES
Issue: ARES as a Chunk Reranker in a RAG app?

Currently I'm building a chunk reranker for a RAG app and am looking into ARES as a reliable evaluation framework.

However, since it evaluates 'context relevance', I think that I may be able to use it directly to rank chunks in a Q&A task.

Alternatively if it's too slow I can use high confidence 'context relevance' predictions as chunk relevance labels and train my own reranker on them.

Does any of that make sense?

Hey @naskovai 

Could you specify the latency requirements and the domain focus for your reranker to better tailor advice?

Hey @robbym-dev, the application domain is "_chat with files you have access to_". Latency requirements are similar to ChatGPT.

Hey @naskovai,

Given your latency requirements similar to ChatGPT and the application domain, it sounds like ARES is fast enough for your needs. However, I would advise running some tests to gauge its performance in your application domain and verify if it meets your quality standard for performance. 

Please feel free to let us know if you have any further questions or if we can assist in any other way. Thank you!

Best,
Robby
Repository: stanford-futuredata/ARES
Issue: LLM approach for synthetic query generation not using specified prompt

For the LLM approach, as opposed to the API approach, the synthetic query and answer generation does not seem to use the specified prompt, only the few-shot examples. Is this intended behaviour?

Hi @WJ44,

Thank you for your interested in ARES! 

Yes, this is intended behavior. In the LLM approach, we do not use the specified prompt because few-shot examples are more effective at helping the model generalize and adapt to various tasks. Using examples instead of a prompt reduces the need for extensive prompt engineering and enhances the model's performance by providing richer contextual information.

Please let us know if you have any other questions, thank you!
Repository: stanford-futuredata/ARES
Issue: Azure OpenAI support
Root cause: Documentation Deficiency

Hello! Does ARES support Azure OpenAI models? If so, is there any sample code? Thank you!

Same Question, Does ARES support Azure Open AI, If yes Do we have any support document for quickstart?

Hi @Lilian-Silveira and @wanjeakshay, 

We're excited to announce that ARES now officially supports Azure OpenAI! Quickstart support documentation will be available soon. Thank you for your patience as we worked to bring this integration to you.

#75
Repository: stanford-futuredata/ARES
Issue: Other checkpoints
Root cause: Unimplemented Feature Gap

Thanks for the valuable work! 
Are you planning on sharing the checkpoints for the two other judges: answer faithfulness and answer relevance?

Hi @MarineHoche, 

Thank you for your interest in ARES! We currently have a checkpoint for answer relevance available for download [here](https://drive.google.com/file/d/1wGcgELBfnCGqXlPEbpPmf7LJ53DPWVXI/view?usp=sharing). While we don't have a checkpoint for answer faithfulness at the moment, we plan to release it in the future.

Thank you, and please feel free to let us know if you have any other questions.
Repository: stanford-futuredata/ARES
Issue: OutOfMemoryError when Running Example Code with ARES
Root cause: Resource Mishandling

I encountered an OutOfMemoryError while attempting to run the example code provided with the ARES evaluation library. The error occurs when the system tries to allocate an additional 64.00 MiB of memory, indicating that the available GPU memory is insufficient.

I am seeking advice on how to configure or optimize computational resources to resolve this memory shortage without degrading model performance. Are there recommended practices, or should adjustments be made in the code or configuration to reduce memory consumption?

Thanks!

### Code Snippet

```
import os
from ares import ARES

ppi_config = {
    "evaluation_datasets": ['nq_unlabeled_output_sub1.tsv'], ## extract 20 samples from nq_unlabeled_output.tsv
    "few_shot_examples_filepath": "nq_few_shot_prompt_for_judge_scoring.tsv",
    "model_choice": "meta-llama/Llama-2-7b-chat-hf",
    "labels": ["Context_Relevance_Label"],
    "gold_label_paths": ["nq_labeled_output_sub1.tsv"],
    "checkpoints": ["Context Relevance Label Joint.pt"],
    "assigned_batch_size": 1000
}

ares = ARES(ppi_config)
results = ares.evaluate_RAG()
print(results)
```


### Simplified nvidia-smi Output
```
GPU 0: NVIDIA GeForce RTX 3090 Ti
- Usage: 24 MiB / 24564 MiB
- Temperature: 39C
- Power Usage: 121W / 450W
```

Hi @chienyutseng,

Thank you for reaching out. It seems you are encountering an OutOfMemoryError due to insufficient GPU memory on your NVIDIA GeForce RTX 3090 Ti. This GPU may not have the necessary resources to deploy the checkpoint effectively.

For more information on the required hardware specifications, please refer to the appendix in our guide [here](https://github.com/stanford-futuredata/ARES?tab=readme-ov-file#appendix).

If you need further assistance, feel free to ask!
Repository: stanford-futuredata/ARES
Issue: The local model needs to pass in a token in order to request it, May I ask if there is such a parameter
Root cause: Validation Gap

The local model needs to pass in a token in order to request it, May I ask if there is such a parameter
```python
from ares import ARES

ppi_config = { 
    "evaluation_datasets": ['nq_unabeled_output.tsv'], 
    "few_shot_examples_filepath": "nq_few_shot_prompt_for_judge_scoring.tsv",
    "llm_judge": "meta-llama/Llama-2-13b-hf", # Specify vLLM model
    "labels": ["Context_Relevance_Label"], 
    "gold_label_path": "nq_labeled_output.tsv",
    "vllm": True, # Toggle vLLM to True 
    "host_url": "http://0.0.0.0:8000/v1", # Replace with server hosting model followed by "/v1"
    "token":"token"  # May I ask if there is such a parameter 请问是否有这种参数呢
}

ares = ARES(ppi=ppi_config)
results = ares.evaluate_RAG()
print(results)
```

I modified the code, for example . it  works
```python
context_score = few_shot_context_relevance_scoring_vllm(
            context_relevance_system_prompt, query, document, model_choice, query_id, debug_mode, host_url, request_delay, failed_extraction_count, in_domain_prompts_dataset
            ,openai_api_key=openai_api_key)
```
add openai_api_key param
```python
if VLLM_AVAILABLE:
    def few_shot_context_relevance_scoring_vllm(
        system_prompt: str, query: str, document: str, model_choice: str,
        query_id: str, debug_mode: bool, host_url: str, request_delay: int,
        failed_extraction_count: Dict[str,int] = {'failed': 0},
        few_shot_examples=None,openai_api_key = "EMPTY"
    ) -> int:

client = OpenAI(
                api_key=openai_api_key,
                base_url=host_url
                )
```
Repository: stanford-futuredata/ARES
Issue: Training classifiers error

```python
from ares import ARES

classifier_config = {
    "training_dataset": ["./documents/synthetic_queries_1.csv"], 
    "validation_set": ["./documents/nq_labeled_output.tsv"], 
    "label_column": ["Context_Relevance_Label"], 
    "num_epochs": 10, 
    "patience_value": 3, 
    "learning_rate": 5e-6,
    "assigned_batch_size": 1,
    "model_choice": "t5-small",  
    "gradient_accumulation_multiplier": 32,  
}

ares = ARES(classifier_model=classifier_config)
ares.train_classifier()
```

```bash
Traceback (most recent call last):
  File "/home/islam/Projects/brightskies/repos/BSRAG/test_ares.py", line 16, in <module>
    ares.train_classifier()
  File "/home/islam/.local/lib/python3.10/site-packages/ares/ares.py", line 137, in train_classifier
    binary_classifer_config(**self.classifier_model_config)
  File "/home/islam/.local/lib/python3.10/site-packages/ares/binary_classifier.py", line 164, in binary_classifer_config
    model, avg_train_losses, avg_valid_losses, eval_dataloader, inference_times = train_and_evaluate_model(train_and_eval_settings)
  File "/home/islam/.local/lib/python3.10/site-packages/ares/LLM_as_a_Judge_Adaptation/General_Binary_Classifier.py", line 793, in train_and_evaluate_model
    outputs = model(**new_batch)
  File "/home/islam/.local/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1532, in _wrapped_call_impl
    return self._call_impl(*args, **kwargs)
  File "/home/islam/.local/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1541, in _call_impl
    return forward_call(*args, **kwargs)
  File "/home/islam/.local/lib/python3.10/site-packages/ares/LLM_as_a_Judge_Adaptation/General_Binary_Classifier.py", line 217, in forward
    total_output = self.encoderModel(input_ids=ids, attention_mask=mask)
  File "/home/islam/.local/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1532, in _wrapped_call_impl
    return self._call_impl(*args, **kwargs)
  File "/home/islam/.local/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1541, in _call_impl
    return forward_call(*args, **kwargs)
  File "/home/islam/.local/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 1517, in forward
    decoder_outputs = self.decoder(
  File "/home/islam/.local/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1532, in _wrapped_call_impl
    return self._call_impl(*args, **kwargs)
  File "/home/islam/.local/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1541, in _call_impl
    return forward_call(*args, **kwargs)
  File "/home/islam/.local/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 1010, in forward
    raise ValueError(f"You have to specify either {err_msg_prefix}input_ids or {err_msg_prefix}inputs_embeds")
ValueError: You have to specify either decoder_input_ids or decoder_inputs_embeds
```

I get this error when trying to train a classifier, I installed ares using pip
Repository: stanford-futuredata/ARES
Issue: Unclear about the code of calculate_ppi function
Root cause: Documentation Deficiency

Hi 

It seems unclear what is the reason for creating 20 random values  `ns = np.linspace(0, n_max, 20).astype(int)` to determine the number of labelled data used to calculate the PPI, but in the end only the PPI value calculated from `n_max` is retained as seen from `avg_ci = ci.mean(axis=0)[-1]`

I don't understand what is the purpose of conducting multiple trials? Since only the only the PPI value calculated from `n_max` is retained, it should be constant throughout the trials isn't it?

https://github.com/stanford-futuredata/ARES/blob/2684d477878e515c3dc31cf4b91fb848a84bdb90/ares/RAG_Automatic_Evaluation/LLMJudge_RAG_Compared_Scoring.py#L238-L281

Also, I saw there there were other functions from the original PPI repository (PPI bootstrap, cross PPI etc), what were you consideration when choosing which PPI function to use?

Correct me if I am wrong but I think this code resulted from a misinterpretation of the code examples from the PPI paper. There they have experiments to show how PPI compares to classic inference by running multiple trials with labelled samples from different parts of the dataset and different sizes. In practice, one would simply use the complete labelled dataset.
Repository: stanford-futuredata/ARES
Issue: Error while evaluating with PPI with API model LLM judge
Root cause: Interface Contract Mismatch

I am using ares evaluator with the given data in the docs to evaluate using prediction powered inference
```yaml
ppi_config {
        evaluation_datasets: ['./aresdocs/nq_unlabeled_output_trimmed.tsv']
        few_shot_examples_filepath: "./aresdocs/nq_few_shot_prompt_for_judge_scoring.tsv"
        llm_judge: "gpt-4o-mini"
        labels: ["Context_Relevance_Label",]
        gold_label_paths: ["./aresdocs/nq_labeled_output_trimmed.tsv"]
}
```

Tried models: openai models: [gpt4, gpt-4o-mini], multiple together models
Error:
```bash
 File "python3.10/site-packages/ares/RAG_Automatic_Evaluation/LLMJudge_RAG_Compared_Scoring.py", line 729, in evaluate_model
    if total_references.nelement() > 0:
AttributeError: 'numpy.ndarray' object has no attribute 'nelement'
```

If I use default model provided checkpoint with config, It works successfully.
```yaml
ppi_config {
        evaluation_datasets: ['./aresdocs/nq_unlabeled_output_trimmed.tsv']
        few_shot_examples_filepath: "./aresdocs/nq_few_shot_prompt_for_judge_scoring.tsv"
        model_choice: "microsoft/deberta-v3-large"
        labels: ["Context_Relevance_Label",]
        gold_label_paths: ["./aresdocs/nq_labeled_output_trimmed.tsv"]
        checkpoints: [./aresdocs/ares_context_relevance_general_checkpoint_V1.1.pt]
}
```

Do I need to provide the checkpoints every time ?
Repository: stanford-futuredata/ARES
Issue: Error while training classifier

```yaml
classifier_model {
        training_dataset: ["./aresdocs/synthetic_queries_1.tsv"] 
        validation_set: ["./aresdocs/nq_labeled_output.tsv"] 
        label_column: ["Context_Relevance_Label"]
        num_epochs: 10 
        model_choice: google-t5/t5-small
        patience_value: 3 
        learning_rate: 0.000005
        assigned_batch_size: 1
        gradient_accumulation_multiplier: 32  
}
```
Error while training google-t5/t5-small with error:
```bash
  File "/home/islam/.local/lib/python3.10/site-packages/ares/ares.py", line 137, in train_classifier
    binary_classifer_config(**self.classifier_model_config)
  File "/home/islam/.local/lib/python3.10/site-packages/ares/binary_classifier.py", line 164, in binary_classifer_config
    model, avg_train_losses, avg_valid_losses, eval_dataloader, inference_times = train_and_evaluate_model(train_and_eval_settings)
  File "/home/islam/.local/lib/python3.10/site-packages/ares/LLM_as_a_Judge_Adaptation/General_Binary_Classifier.py", line 793, in train_and_evaluate_model
    outputs = model(**new_batch)
  File "/home/islam/.local/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1532, in _wrapped_call_impl
    return self._call_impl(*args, **kwargs)
  File "/home/islam/.local/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1541, in _call_impl
    return forward_call(*args, **kwargs)
  File "/home/islam/.local/lib/python3.10/site-packages/ares/LLM_as_a_Judge_Adaptation/General_Binary_Classifier.py", line 221, in forward
    total_output = self.encoderModel(ids, attention_mask=mask)
  File "/home/islam/.local/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1532, in _wrapped_call_impl
    return self._call_impl(*args, **kwargs)
  File "/home/islam/.local/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1541, in _call_impl
    return forward_call(*args, **kwargs)
  File "/home/islam/.local/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 1517, in forward
    decoder_outputs = self.decoder(
  File "/home/islam/.local/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1532, in _wrapped_call_impl
    return self._call_impl(*args, **kwargs)
  File "/home/islam/.local/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1541, in _call_impl
    return forward_call(*args, **kwargs)
  File "/home/islam/.local/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 1010, in forward
    raise ValueError(f"You have to specify either {err_msg_prefix}input_ids or {err_msg_prefix}inputs_embeds")
ValueError: You have to specify either decoder_input_ids or decoder_inputs_embeds
```
Repository: stanford-futuredata/ARES
Issue: Source of nq_labeled_output.tsv
Root cause: Documentation Deficiency

Hello,

The nq_labeled_output.tsv file contains 6189 rows from NQ. As I understand this file was used to generate the synthetic data to train the LLM Judges. Where was this file sourced? Specifically, which split from NQ is this from and how where the rows selected. I am trying to understand how one would apply ARES to a new dataset (and evaluate it) i.e. creating synthetic data, training the judges and evaluating on mock datasets. I assume the documents used in the human reference set and the mock splits should be separate from the ones used to generate the synthetic data. I was also wondering how the provided synthetic data was generated. I see the file for NQ has precisely 3000 rows. Were all 6189 documents used like in the examples and was the set shrunk later?
Repository: stanford-futuredata/ARES
Issue: Difference between Answer Relevance and Answer Faithfulness
Root cause: Unimplemented Feature Gap

Hello,

I was wondering whether there is any difference between Answer Relevance and Answer Faithfulness. Conceptually there is of course, but the code for training LLM judges and actually judging samples seems exactly the same. Would it not make sense for the answer relevance metric to not take into account the context during training and judging?

I second this, from the paper:
**context relevance** (is the retrieved information pertinent to the test question)
**answer faithfulness** (is the response generated by the language model properly grounded in the retrieved context) 
**answer relevance** (is the response also relevant to the question).

Context relevance is clear and is handled in synthetic data generation and model training/testing. Answer faithfulness and relevance are defined in the paper, but in the code treats them exactly the same. Except in some places where answer faithfulness is the only one appearing. I'm wondering if the idea was to implement this but you ended up not implementing answer faithfulness. In the paper no table of result, charts or examples include answer faithfulness.
Repository: stanford-futuredata/ARES
Issue: What is meant by contrastive learning objective?

Having read the paper and the code I don't understand what is meant by "These judges are lightweight models fine-tuned against a contrastive learning objective.". 

The loss function is a cross entropy loss and the task is a classification task (yes/no) for belonging or not belonging in a class. There is no notion in the code of maximizing the separation of these classes in the embeddings space. The phrasing adds some confusion since contrastive typically means something else.