Repository: explodinggradients/ragas
Issue: InvalidRequestError when context size to too big
Root cause: Validation Gap

I want to evaluate a single completion of my LLM.

Code:
```python
from ragas import evaluate
from datasets import Dataset
import os

# prepare your huggingface dataset in the format
# Dataset({
#     features: ['question','contexts','answer'],
#     num_rows: 25
# })

data = {
    "question": [query],  # single query, string
    "contexts": [sources],  # single source document in string, I have tried sources[:3000], sources[:3500] to avoid this error
    "answer": [answer]  # single answer
}

# Create the Hugging Face dataset
dataset = Dataset.from_dict(data)

# Set the dataset format
dataset.set_format(
    type="torch", columns=["question", "contexts", "answer"]  # I have tried without type='torch'
)

# Print dataset information
print(dataset)


dataset: Dataset

results = evaluate(dataset)
```

Complete Output traceback:
```
Dataset({
    features: ['question', 'contexts', 'answer'],
    num_rows: 1
})
100%|██████████| 1/1 [00:00<00:00,  1.44it/s]
100%|██████████| 52/52 [03:09<00:00,  3.65s/it]
  0%|          | 0/1 [00:02<?, ?it/s]
---------------------------------------------------------------------------
InvalidRequestError                       Traceback (most recent call last)
[<ipython-input-32-8200ac55342c>](https://localhost:8080/#) in <cell line: 38>()
     36 dataset: Dataset
     37 
---> 38 results = evaluate(dataset)

9 frames
[/usr/local/lib/python3.10/dist-packages/ragas/evaluation.py](https://localhost:8080/#) in evaluate(dataset, metrics)
     86     scores = []
     87     for metric in metrics:
---> 88         scores.append(metric.score(dataset).select_columns(metric.name))
     89 
     90     return Result(scores=concatenate_datasets(scores, axis=1), dataset=dataset)

[/usr/local/lib/python3.10/dist-packages/ragas/metrics/factual.py](https://localhost:8080/#) in score(self, dataset)
     71         scores = []
     72         for batch in tqdm(self.get_batches(len(dataset))):
---> 73             score = self._score_batch(dataset.select(batch))
     74             scores.append(score)
     75 

[/usr/local/lib/python3.10/dist-packages/ragas/metrics/factual.py](https://localhost:8080/#) in _score_batch(self, ds)
    101             prompts.append(prompt)
    102 
--> 103         response = openai_completion(prompts)
    104         outputs = response["choices"]  # type: ignore
    105 

[/usr/local/lib/python3.10/dist-packages/backoff/_sync.py](https://localhost:8080/#) in retry(*args, **kwargs)
    103 
    104             try:
--> 105                 ret = target(*args, **kwargs)
    106             except exception as e:
    107                 max_tries_exceeded = (tries == max_tries_value)

[/usr/local/lib/python3.10/dist-packages/ragas/metrics/llms.py](https://localhost:8080/#) in openai_completion(prompts, **kwargs)
     24     - what happens when backoff fails?
     25     """
---> 26     response = openai.Completion.create(
     27         model=kwargs.get("model", "text-davinci-003"),
     28         prompt=prompts,

[/usr/local/lib/python3.10/dist-packages/openai/api_resources/completion.py](https://localhost:8080/#) in create(cls, *args, **kwargs)
     23         while True:
     24             try:
---> 25                 return super().create(*args, **kwargs)
     26             except TryAgain as e:
     27                 if timeout is not None and time.time() > start + timeout:

[/usr/local/lib/python3.10/dist-packages/openai/api_resources/abstract/engine_api_resource.py](https://localhost:8080/#) in create(cls, api_key, api_base, api_type, request_id, api_version, organization, **params)
    151         )
    152 
--> 153         response, _, api_key = requestor.request(
    154             "post",
    155             url,

[/usr/local/lib/python3.10/dist-packages/openai/api_requestor.py](https://localhost:8080/#) in request(self, method, url, params, headers, files, stream, request_id, request_timeout)
    296             request_timeout=request_timeout,
    297         )
--> 298         resp, got_stream = self._interpret_response(result, stream)
    299         return resp, got_stream, self.api_key
    300 

[/usr/local/lib/python3.10/dist-packages/openai/api_requestor.py](https://localhost:8080/#) in _interpret_response(self, result, stream)
    698         else:
    699             return (
--> 700                 self._interpret_response_line(
    701                     result.content.decode("utf-8"),
    702                     result.status_code,

[/usr/local/lib/python3.10/dist-packages/openai/api_requestor.py](https://localhost:8080/#) in _interpret_response_line(self, rbody, rcode, rheaders, stream)
    761         stream_error = stream and "error" in resp.data
    762         if stream_error or not 200 <= rcode < 300:
--> 763             raise self.handle_error_response(
    764                 rbody, rcode, resp.data, rheaders, stream_error=stream_error
    765             )

InvalidRequestError: This model's maximum context length is 4097 tokens, however you requested 4331 tokens (3831 in your prompt; 500 for the completion). Please reduce your prompt; or completion length.
```
Even after changing all the lengths inside my dataset, I get the same error every single time. 3831 tokens in my prompt, 500 for completion.

Hi @ahmadmustafaanis , thanks for raising the issue. 
Ragas `faithfulness` metrics roughly need `context length + generated answer length + 500 tokens` to cross-check the generated answer against your context (here 3-3.5k).  

For solving you can either reduce the context length to 2-3k or use the latest `gpt-3.5-turbo-16k` (this has a 16k context length with the same price) instead of gpt-3.5-turbo.
I'll raise a fix for making `gpt-3.5-turbo-16k` as default for ragas.

Thanks @ahmadmustafaanis for raising this issue. Unfortunately changing the LLM right now is a bit hard but will get easier once #46 is merged. I will get this merged in ASAP since you are depending on it :)

For now, have you tried reducing the context length as Shahul suggested? 

Yes, reducing the context length works. Thank you.
 Do you think that there should be a chunking method in it as well, like if it is about to throw that error, it should simply chunk it using tiktoken? instead of throwing exception.

@ahmadmustafaanis Introducing chunking will compromise the quality of the evaluation hence we did not consider it. If you like to work on this issue feel free to check #46 

hey @ahmadmustafaanis  this is fixed. Thanks for bringing this up and do let us know when you try it
here is the example notebook: https://github.com/explodinggradients/ragas/blob/main/docs/guides/llms.ipynb

by default we are using 'gpt-3.5-16k` so that means you don't have to reduce document size  when running the evals :) 

Hi @jjmachan and everyone. 
I also have the same problem with the "size of 'contexts' is to big". 
`RuntimeError: ('Fatal error occurred while running async tasks.', BadRequestError('Error code: 400 - {\'error\': {\'message\': "This model\'s maximum context length is 16385 tokens. However, your messages resulted in 38376 tokens. Please reduce the length of the messages.", \'type\': \'invalid_request_error\', \'param\': \'messages\', \'code\': \'context_length_exceeded\'}}'))
`
How does my RAG pipeline works: I have a question to GPT-4 and the text that I use for the answer. This text is big (10-200 pages) and I use GPT-4 128k tokens to get an answer. GPT-4 128k is doing well with question and text. It gets me an answer. 
After this, I should evaluate the answer and try to do it with the RAGAS. I saw the issue of the "size of 'contexts' is too big". 
I should find another way of evaluating the answer. Maybe try to send the text for 'contexts' like parts or reduce my text use chunks. I don't know. 
So, could you help me with this issue and tell me different ways to solve it? 
Thanks

Hi RAGAS maintainers! It would be very helpful if you could add an option to ignore erroring rows in the dataset during evaluation so we proceed with evaluation instead of crashing the job. Do you think this would make sense?