Repository: explodinggradients/ragas
Issue: token usage parser for llama_index
Root cause: Unimplemented Feature Gap

[ *] I checked the [documentation](https://docs.ragas.io/) and related resources and couldn't find an answer to my question.

**Your Question**
what is unclear to you? What would you like to know?


How do I use the token usage parser when I'm using llama_index evaluate? 


**Code Examples**
This community speaks code. Share your code snippets to help us understand your question better.

**Additional context**
Anything else you want to share with us? 


<!-- Greeting -->
Hey @tarunn2799! 👋 I'm here to help you with any bugs, questions, or contributions while you wait for a human maintainer. Let's tackle this together!

<!-- Answer -->
To use the token usage parser for `ragas.integrations.llama_index.evaluate` with `get_token_usage_for_openai`, follow these steps:

1. **Import the necessary modules and functions:**

    ```python
    from datasets import load_dataset
    from ragas.metrics import (
        answer_relevancy,
        faithfulness,
        context_recall,
        context_precision,
    )
    from ragas import evaluate
    from ragas.cost import get_token_usage_for_openai
    from langchain_openai.chat_models import ChatOpenAI
    from langchain_core.prompt_values import StringPromptValue
    ```

2. **Load the dataset:**

    ```python
    amnesty_qa = load_dataset("explodinggradients/amnesty_qa", "english_v2")
    ```

3. **Initialize the LLM model:**

    ```python
    gpt4o = ChatOpenAI(model="gpt-4o")
    ```

4. **Perform the evaluation with the token usage parser:**

    ```python
    result = evaluate(
        amnesty_qa["eval"],
        metrics=[
            context_precision,
            faithfulness,
            answer_relevancy,
            context_recall,
        ],
        llm=gpt4o,
        token_usage_parser=get_token_usage_for_openai,
    )
    ```

5. **Get the total tokens used:**

    ```python
    total_tokens = result.total_tokens()
    ```

6. **Compute the total cost:**

    ```python
    total_cost = result.total_cost(cost_per_input_token=5 / 1e6, cost_per_output_token=15 / 1e6)
    ```

Here is the complete code snippet:

```python
from datasets import load_dataset
from ragas.metrics import (
    answer_relevancy,
    faithfulness,
    context_recall,
    context_precision,
)
from ragas import evaluate
from ragas.cost import get_token_usage_for_openai
from langchain_openai.chat_models import ChatOpenAI
from langchain_core.prompt_values import StringPromptValue

# Load the dataset
amnesty_qa = load_dataset("explodinggradients/amnesty_qa", "english_v2")

# Initialize the LLM model
gpt4o = ChatOpenAI(model="gpt-4o")

# Perform the evaluation with the token usage parser
result = evaluate(
    amnesty_qa["eval"],
    metrics=[
        context_precision,
        faithfulness,
        answer_relevancy,
        context_recall,
    ],
    llm=gpt4o,
    token_usage_parser=get_token_usage_for_openai,
)

# Get the total tokens used
total_tokens = result.total_tokens()

# Compute the total cost
total_cost = result.total_cost(cost_per_input_token=5 / 1e6, cost_per_output_token=15 / 1e6)
```

This code will help you evaluate the dataset using the specified metrics and compute the token usage and cost using the `get_token_usage_for_openai` parser <sup>[[1]](https://github.com/explodinggradients/ragas/blob/main/docs/howtos/applications/cost.ipynb)</sup>.


<!-- Dosu Comment Footer -->
*To continue talking to [Dosu](https://dosu.dev), mention @dosu.*


That's just the general documentation - `ragas.integrations.llama_index.evaluate` does not support `token_usage_parser` as an argument. That's my problem. 

@tarunn2799 currently llama_index doesnot support token_parser sadly. I will take a look at it but right now we are working on #1009 so this might take some time sadly 🙁 

if you want to take a stab at it I can help you out, would that be something you would be interested?

Yes it would be! Let me know how I can contribute. Thanks :) 

awesome! this is my cal, can you find a time there? 
https://cal.com/jjmachan/huddle

we can go over how you might do this 🙂 

Closing as resolved.