zhiminy/EvalEng huggingface dataset:
Repository: explodinggradients/ragas
Issue: [R-289] better support for ollama models
Root cause: Unimplemented Feature Gap

ref

* [Handle large context windows using Ollama's LLMs for evaluation purpose · Issue #1120 · explodinggradients/ragas](https://github.com/explodinggradients/ragas/issues/1120)

feats

* check how good ollama models are performing
* context window issue, need to throw and error if that is the case
* make a documentation and see if you can figure out a PR?

<sub>From [SyncLinear.com](https://synclinear.com) | [R-289](https://linear.app/exploding-gradients/issue/R-289/better-support-for-ollama-models)</sub>

user report from discord
> My use case is for evaluation currently, but I am encountering issues. Using llama3.1 (also tried with instruct) I only get the following as output:
Failed to parse output. Returning None.
Also tried the 70b model, but then something breaks in Ollama then. I get time=2024-08-06T20:17:22.069+02:00 level=DEBUG source=server.go:572 msg="server unhealthy" error="server not responding" in Ollama logs.

also refer thread: https://discord.com/channels/1119637219561451644/1261643639113781360

@jjmachan , I have tried to solve this issue, I have changed the wait time to some higher value like some 300 keeping the workers same. I have was able to generate the score now. There are NaN values for some metrics but we were able to get the score for certain metrics. So its the issue with models i guess, The model is having some good inference time and the model couldn't generate the score within the time frame.

`embed = OllamaEmbeddings(model="llama3")
llm_model = Ollama(model="llama3")
evaluator_llm = LangchainLLMWrapper(llm_model)
evaluator_embeddings = LangchainEmbeddingsWrapper(embed)
result = evaluate(hf_dataset,metrics=[context_precision,faithfulness,answer_relevancy,context_recall,],embeddings=embed,llm=llm_model,run_config=RunConfig(max_workers=3,timeout=240,))
`
I hope this would help
Repository: explodinggradients/ragas
Issue: Vunerable version for the onnx package in the poetry.lock
Root cause: External Dependency Breakage

[ ] I have checked the [documentation](https://docs.ragas.io/) and related resources and couldn't resolve my bug.

**Describe the bug**
I’ve identified that the onnx package listed in the [src/experimental/poetry.lock](https://github.com/explodinggradients/ragas/blob/ee6ff5f3eb7540ef3fb35cfbc17bd3e1c1d0e7cb/src/experimental/poetry.lock#L598) file has a version defined as 1.16.1, which contains a path traversal vulnerability. This issue has been documented in Snyk, as indicated in [this report](https://security.snyk.io/vuln/SNYK-PYTHON-ONNX-7231121).

This vulnerability will likely trigger some CVE issues during the vulnerability scans in our Docker image build and push process with Ragas as a dependency. To ensure the integrity and security of this package, could we please update the onnx package to the safe version 1.16.2?

Ragas version: lastest

Thank you!


<!-- Greeting -->
Hey there, @linqiu0-0! 👋 I'm here to help you with any bugs, questions, or contributions you have for the repository. Let's squash those bugs together!

<!-- Answer -->
To address the vulnerability in the `onnx` package, you should update the version to 1.16.2 in the `poetry.lock` file. Additionally, ensure that the `onnx` package version is updated in `requirements/test.txt` and verify that the unit tests pass successfully in the CI pipeline. The `unit_tests` job in the CI pipeline installs dependencies from `requirements/test.txt`, which should include the `onnx` package <sup>[[1]](https://github.com/explodinggradients/ragas/blob/main/.github/workflows/ci.yaml)</sup>.


<!-- Dosu Comment Footer -->
*To continue talking to [Dosu](https://dosu.dev), mention @dosu.*


@linqiu0-0 thank you so much for reporting this 🙌🏽. If you to maybe you have give a shot at contributing a fix?

I can take it up if not but if you're interested let me know 🙂 

Closing this as resolved.
Repository: explodinggradients/ragas
Issue: [R-293] Noise sensitivity metrics from RAGChecker
Root cause: Unimplemented Feature Gap

RAGChecker from AWS has [noise sensitivity](https://github.com/amazon-science/RAGChecker/tree/main/ragchecker
) metrics that could be useful to Ragas users 

TBD
1) Implement metrics
2) Prepare docs to intuitively explain how it's calculated and how can it be used. 




<sub>[R-293](https://linear.app/exploding-gradients/issue/R-293/noise-sensitivity-metrics-from-ragchecker)</sub>

Hey @shahules786 I will take up

@sky-2002 Great, assigned to you.
Repository: explodinggradients/ragas
Issue: Evaluation with IBM WatsonX LLM's
Root cause: Interface Contract Mismatch

Hey!
I have tried my hands on RAGAS with Watson LLM, the major issue I am facing is getting the warning:
"Failed to parse output. Returning None." 
Continuously.

Is there any fix for this?
Is it because of the type of response sent by the WatsonX LLM's?



<!-- Greeting -->
Hey @swayam-khandelwal! I'm here to help you with any issues you're facing while waiting for a human maintainer. Let's work together to solve this problem!

<!-- Answer -->
The issue you're encountering, "Failed to parse output. Returning None," is likely related to the type of response sent by the WatsonX LLM. The `RagasoutputParser` class attempts to parse the result using a Pydantic schema. If the parsing fails, it retries once with a fixed output format prompt. If it still fails, it logs the warning and returns `None`.

To address this issue, ensure that the response from WatsonX LLM conforms to the expected JSON schema. The `get_json_format_instructions` function provides the necessary instructions for the LLM to format its output correctly. You might need to adjust the prompt or the schema to better align with the responses generated by WatsonX LLM.

Here is the relevant part of the code that handles the parsing and retries:

```python
class RagasoutputParser(PydanticOutputParser):
    async def aparse(  # type: ignore
        self, result: str, prompt: PromptValue, llm: BaseRagasLLM, max_retries: int = 1
    ):
        try:
            output = super().parse(result)
        except OutputParserException:
            if max_retries != 0:
                p_value = FIX_OUTPUT_FORMAT.format(
                    prompt=prompt.to_string(), completion=result
                )
                output = await llm.generate(p_value)
                result = output.generations[0][0].text
                return await self.aparse(result, prompt, llm, max_retries - 1)
            else:
                logger.warning("Failed to parse output. Returning None.")
                return None
        return output
```

Ensure that the WatsonX LLM's output is a well-formatted JSON string that matches the schema provided in the prompt instructions. Adjusting the prompt to include clear formatting instructions, as shown in the `JSON_FORMAT_INSTRUCTIONS`, can help mitigate this issue <sup>[[1]](https://github.com/explodinggradients/ragas/blob/main/src/ragas/llms/output_parser.py)</sup>.


<!-- Dosu Comment Footer -->
*To continue talking to [Dosu](https://dosu.dev), mention @dosu.*
Repository: explodinggradients/ragas
Issue: KeyError while using dataset with extra attributes 
Root cause: Validation Gap

Ragas can be only used with dataset that contains fixed attributes. Any other attributes other than required causes key errors. For example, here I used dataset with column names [question,answer,contexts,ungrounded_answer]

```python
from datasets import load_dataset
from ragas.metrics import (
    answer_relevancy,
    faithfulness,
)
from ragas import evaluate
wikieval = load_dataset("explodinggradients/WikiEval")
wikieval = wikieval['train'].rename_columns({"grounded_answer":"answer","context_v1":"contexts"})
results = evaluate(dataset=wikieval1,metrics=[faithfulness,answer_relevancy])
```

```
KeyError                                  Traceback (most recent call last)
Input In [17], in <cell line: 1>()
----> 1 results = evaluate(dataset=wikieval,metrics=[context_relevancy,faithfulness,answer_relevancy])

File ~/belar/src/ragas/evaluation.py:89, in evaluate(dataset, metrics, column_map)
     86     metrics = [answer_relevancy, context_relevancy, faithfulness, context_recall]
     88 # remap column names from the dataset
---> 89 dataset = remap_column_names(dataset, column_map)
     91 # validation
     92 validate_evaluation_modes(dataset, metrics)

File ~/belar/src/ragas/validation.py:14, in remap_column_names(dataset, column_map)
      9 """
     10 Remap the column names in case dataset uses different column names
     11 """
     12 inverse_column_map = {v: k for k, v in column_map.items()}
     13 return dataset.from_dict(
---> 14     {inverse_column_map[name]: dataset[name] for name in dataset.column_names}
     15 )

File ~/belar/src/ragas/validation.py:14, in <dictcomp>(.0)
      9 """
     10 Remap the column names in case dataset uses different column names
     11 """
     12 inverse_column_map = {v: k for k, v in column_map.items()}
     13 return dataset.from_dict(
---> 14     {inverse_column_map[name]: dataset[name] for name in dataset.column_names}
     15 )

KeyError: 'ungrounded_answer'
```



please provide how you invoked the code as well for debugging. want to understand if 
```    
column_map : dict[str, str], optional
        The column names of the dataset to use for evaluation. If the column names of
        the dataset are different from the default ones then you can provide the
        mapping as a dictionary here.
```
is not working as intended.

The dataset [here](https://github.com/explodinggradients/ragas/blob/81ed9eaa097941163ceb3f1d3988931b06ca8f3d/src/ragas/validation.py#L14) might have attributes other than ones used by ragas. In this case, it encounters a key error. The solution is to iterate on column names that are present in both dataset and inverse_column_map
Repository: explodinggradients/ragas
Issue: What's context recall?

Hi guys - I'm currently adding Ragas to DeepEval (https://github.com/confident-ai/deepeval/issues/101) 
One thing I found odd though was `context_recall` was done using prompts. Curious to hear if there's other experiments done in this space to answer the question of if the retriever answered the query. Potentially a mean Cross-Encoder QA would be helpful here? 

Curious to hear your thoughts guys! 

Hi @ColabDog , context recall is the measure of how much of the required context is actually captured. So it is not possible to calculate it without some ground truth. Since in our case, we wanted to estimate this from the ground truth answer and not the ground truth context we use the method of identifying how the ground truth is captured in the retrieved context. 


closing this since the question has been answered
Repository: explodinggradients/ragas
Issue: cannot import name 'context_precision' from 'ragas.metrics'
Root cause: Configuration Error

[X ] I have checked the [documentation](https://docs.ragas.io/) and related resources and couldn't resolve my bug.

**Describe the bug**
I am trying to follow [this](https://docs.ragas.io/en/stable/getstarted/evaluation.html) documentation, but it doesn't work because `context_precision` seems to be missing. 

Ragas version: ragas-0.1.14
Python version: 3.11.4

**Code to Reproduce**
The code from Ragas' documentation ...

```
from ragas.metrics import (
    answer_relevancy,
    faithfulness,
    context_recall,
    context_precision,
)

```
**Error trace**

```
Cell In[52], [line 1](vscode-notebook-cell:?execution_count=52&line=1)
----> [1](vscode-notebook-cell:?execution_count=52&line=1) from ragas.metrics import (
      [2](vscode-notebook-cell:?execution_count=52&line=2)     faithfulness,
      [3](vscode-notebook-cell:?execution_count=52&line=3)     answer_relevancy,
      [4](vscode-notebook-cell:?execution_count=52&line=4)     context_precision,
      [5](vscode-notebook-cell:?execution_count=52&line=5)     context_recall,
      [6](vscode-notebook-cell:?execution_count=52&line=6) )

ImportError: cannot import name 'context_precision' from 'ragas.metrics' (/Users/ME/opt/miniconda3/envs/eval_demo/lib/python3.11/site-packages/ragas/metrics/__init__.py)
```


**Expected behavior**
The documented code should work. 


**Additional context**
RAGAs is great, thanks!

<!-- PS: bugs suck but is also part of the process. We sincerely apologies for breaking your flow because of it, but don't worry, we got your back ❤️. We will get this fixed as fast as we can and thanks for helping us out by reporting it 🙏. -->


<!-- Greeting -->
Hey @dividor! 👋 I'm here to help you out with any bugs or questions you have. I'm a bot that can assist with problem-solving and answering queries while you wait for a human maintainer.

<!-- Answer -->
The `ImportError` occurs because `context_precision` is not defined as a module-level variable in `ragas.metrics`. Instead, it is defined within `src/ragas/metrics/_context_precision.py`.

To resolve this issue, you can import `context_precision` directly from `ragas.metrics._context_precision`:

```python
from ragas.metrics._context_precision import context_precision
```

Alternatively, you can import it directly from `ragas.metrics` as it is included in the `__all__` list in the `src/ragas/metrics/__init__.py` file:

```python
from ragas.metrics import context_precision
```

This should resolve the `ImportError` you are encountering <sup>[[1]](https://github.com/explodinggradients/ragas/blob/main/src/ragas/metrics/_context_precision.py)</sup><sup>[[2]](https://github.com/explodinggradients/ragas/blob/main/src/ragas/metrics/__init__.py)</sup>.


<!-- Dosu Comment Footer -->
*To continue talking to [Dosu](https://dosu.dev), mention @dosu.*


Try with latest metrics collections: [Docs](https://docs.ragas.io/en/stable/concepts/metrics/available_metrics/context_precision/)
Repository: explodinggradients/ragas
Issue: Inconsistency in the Value of EVALMODE_TO_COLUMNS[EvaluationMode.ca]
Root cause: Validation Gap

Hi,

It seems there might be an inconsistency in the value of the dictionary `EVALMODE_TO_COLUMNS[EvaluationMode.ca]` within `src/ragas/validation.py`. Currently, the value is set to `["context", "summary"]`

https://github.com/explodinggradients/ragas/blob/d5b60bba475fb099b3af2544d80412ffc6e8abc8/src/ragas/validation.py#L62-L71

but I believe it should be `["context", "answer"]` as in `src/ragas/metrics/base.py:get_required_columns()`:

https://github.com/explodinggradients/ragas/blob/d5b60bba475fb099b3af2544d80412ffc6e8abc8/src/ragas/metrics/base.py#L38-L59

Is this the expected behavior? If not, could you please take look into it?

Thanks!

Best regards,
Juan



<!-- Greeting -->
Hey @jucgonzalezes!

I'm here to help you out with any bugs, questions, or contributions you have in mind. Let's tackle this together!

<!-- Answer -->
The value of `EVALMODE_TO_COLUMNS[EvaluationMode.ca]` in `src/ragas/validation.py` should indeed be `["contexts", "answer"]` as indicated in the `get_required_columns()` function from `src/ragas/metrics/base.py` <sup>[[1]](https://github.com/explodinggradients/ragas/blob/main/src/ragas/metrics/base.py)</sup><sup>[[2]](https://github.com/explodinggradients/ragas/blob/main/src/ragas/validation.py)</sup>. The current value `["contexts", "summary"]` appears to be incorrect and should be updated to maintain consistency across the codebase.


<!-- Dosu Comment Footer -->
*To continue talking to [Dosu](https://dosu.dev), mention @dosu.*
Repository: explodinggradients/ragas
Issue: OpenAI key not found
Root cause: Documentation Deficiency

Hey guys,
I have a problem that occured all of a sudden. I am using the Azure OpenAI client and it worked until recently. Now I am getting the error ´ Did not find openai_api_key, please add an environment variable `OPENAI_API_KEY` which contains it, or pass  `openai_api_key` as a named parameter.´
I am setting the paramters like so at the beginning of the code:

os.environ["OPENAI_API_TYPE"] = "azure"
os.environ["OPENAI_API_KEY"] = "..................................."
os.environ["OPENAI_API_VERSION"] = "....................................."
os.environ["OPENAI_API_BASE"] = "..........................."

llm = AzureOpenAI(deployment_name = ".................................", openai_api_key=os.environ["OPENAI_API_KEY"])
Passing it as a named parameter to the evaluate method did not solve it by the way.

I have absolutely zero idea where this is coming from. Can anyone hint me in a direction?

I have included the openai parameters in the base.py since there is a method named _llm_factory which is calling the openai API and that seems to have fixed it. But why is that necessary all of a sudden? 

that was because by default Langchain OpenAI LLMs through an error if openAI API keys are not found. But have you checked this out https://github.com/explodinggradients/ragas/blob/main/docs/guides/llms.ipynb

hey @gabriead closing this since [this guide](https://github.com/explodinggradients/ragas/blob/main/docs/guides/quickstart-azure-openai.ipynb) should help with these issues. Feel free to reopen if not

would love to chat with you sometime as well, are you in the discord group?

@jjmachan I have a similar issue (please, see [here](https://github.com/explodinggradients/ragas/issues/303)). The URLs that you have shared in this thread are broken, unfortunately, so I don't know how to address it.

@jjmachan Hi, Same Issue Here.

The Azure guide seems to have moved to [here](https://github.com/explodinggradients/ragas/blob/main/docs/howtos/customisations/azure-openai.ipynb), which helped me get past this
Repository: explodinggradients/ragas
Issue: LangChain integration does not work for summarization_score
Root cause: Interface Contract Mismatch

- [x] I have checked the [documentation](https://docs.ragas.io/) and related resources and couldn't resolve my bug.

**Describe the bug**
`EvaluatorChain` errors out for summariztion_score metric. `ValueError: Missing some input keys: {'answer'}`

Ragas version: 0.1.15
Python version: 3.9

**Code to Reproduce**
```
from ragas.integrations.langchain import EvaluatorChain
from ragas.metrics import summarization_score 

chain=EvaluatorChain(metric=summarization_score)

chain.run({"contexts": ["hello"], "summary": "hello"})
```

**Error trace**
```
File /.../python3.9/site-packages/langchain/chains/base.py:284, in Chain._validate_inputs(self, inputs)
    282 missing_keys = set(self.input_keys).difference(inputs)
    283 if missing_keys:
--> 284     raise ValueError(f"Missing some input keys: {missing_keys}")
ValueError: Missing some input keys: {'answer'}
```

**Expected behavior**
Expect summarization_score could work with LangChain (EvaluatorChain)

**Additional context**
The same setting above works for other ragas metrics like faithfulness. Example
```
from ragas.integrations.langchain import EvaluatorChain
from ragas.metrics import faithfulness 

chain=EvaluatorChain(metric=faithfulness)

chain.run({"question": "What is the capital of France?",
    "answer": "The capital of France is Paris.",
    "contexts": [
        "France, in Western Europe, encompasses medieval cities, alpine villages and Mediterranean beaches. Paris, its capital, is famed for its fashion houses, classical art museums including the Louvre and monuments like the Eiffel Tower.",
        "Paris is the capital and most populous city of France. It has an area of 105 square kilometers and a population of over 2 million residents."
    ],
    "ground_truth": "Paris"
})
```
<!-- PS: bugs suck but is also part of the process. We sincerely apologies for breaking your flow because of it, but don't worry, we got your back ❤️. We will get this fixed as fast as we can and thanks for helping us out by reporting it 🙏. -->


<!-- Greeting -->
@Peilun-Li Hello there! I'm here to help you with bugs, questions, and becoming a contributor.

<!-- Answer -->
Could you provide the exact definition of the `summarization_score` metric? Specifically, what input keys does it require? This will help determine if there are any missing keys that need to be included in the `chain.run` method.


<!-- Dosu Comment Footer -->
*To continue talking to [Dosu](https://dosu.dev), mention @dosu.*


Please try with latest ragas 0.3.9 use `from ragas.metrics.collections import SummaryScore`
We're not supporting 0.1.15 anymore.
Repository: explodinggradients/ragas
Issue: [R-296] revamp documentation
GitHub labels: documentation

### Explore Features

- [ ] code references type
- [ ] jupyter notebook
  - [ ] output and inputs - custom theme
  - [ ] the objective is to have no distinction between .md code and .ipynb code
- [ ] switch versions
- [ ] explore hosting solutions
- [ ] explore tab option for switching LLMs

<details close>
<summary>meeting notes</summary>
<br>

* how good is support for markdown notebooks as docs
* how do we setup code references
  * something like [Pydantic Settings - Pydantic](https://docs.pydantic.dev/latest/api/pydantic_settings/#pydantic_settings.BaseSettings)
  * this also means we have to add more code documentation - that @jjmachan
  * will also pitch in
* search, need a fast search bar to find stuff
* theming -> make black theme, consistent with ragas docs
* what are the other cool stuff in material for mkdocs can we use to make our documentation the best in the world. This is a where you can use your expertise and creativity, just a list of possible items is enough
* watch-docs: we should be able to setup something like live-reloading in webpages for documentations website locally
* reference function directly. For example if I reference `evaluate()` in the documentation, it should point to the code refrence section.
  porting
* make sure the extra features of the theme work in markdown
  if we can get those working we can switch
  </details>

<sub>From [SyncLinear.com](https://synclinear.com) | [R-296](https://linear.app/exploding-gradients/issue/R-296/revamp-documentation)</sub>

things remaining
- fix navigation
- host it in readthedocs (if readthedocs fix styling)