Repository: explodinggradients/ragas
Issue: Incorrect calculation at ToolCallAccuracy
Root cause: Algorithmic Error

**Describe the bug**
In ToolCallAccuracy, if the number of ToolCalls in the user_input is greater than the number of reference_tool_calls, it does not affect the evaluation score. In other words, the evaluation score remains unaffected even when more ToolCalls than expected occur.

Ragas version: latest
Python version: 3.12

**Code to Reproduce**
```py
sample = [
    HumanMessage(content="What's the weather like in New York right now?"),
    AIMessage(content="The current temperature in New York is 75°F and it's partly cloudy.", tool_calls=[
        ToolCall(name="weather_check", args={"location": "New York"})
    ]),
    HumanMessage(content="Can you translate that to Celsius?"),
    AIMessage(content="Let me convert that to Celsius for you.", tool_calls=[
        ToolCall(name="temperature_conversion", args={"temperature_fahrenheit": 75})
    ]),
    ToolMessage(content="75°F is approximately 23.9°C."),
    AIMessage(content="75°F is approximately 23.9°C.")
]

sample = MultiTurnSample(
    user_input=sample,
    reference_tool_calls=[
        ToolCall(name="weather_check", args={"location": "New York"})
    ]
)
```
Output: 
```
1
```
**Error trace**

**Expected behavior**
"evaluation is 0"
I think there are many opinions.

**Additional context**
Add any other context about the problem here.

<!-- PS: bugs suck but is also part of the process. We sincerely apologies for breaking your flow because of it, but don't worry, we got your back ❤️. We will get this fixed as fast as we can and thanks for helping us out by reporting it 🙏. -->


Hi @licux,

The way `ToolCallAccuracy` works is, it scores if the predicted sample has the Tool Calls in the same order as the Tool Calls in the reference. 

Reference Tool Calls: `["weather_check"]`
Predicted Tool Calls: `["weather_check", "temperature_conversion"]`

Since reference had one Tool Call(`["weather_check"]`), and it was present in the predicted so the score is 1.

The formula used for scoring is:

$$\text{score} = \frac{\sum{(\text{matched tool call scores})}}{\text{number of reference tool calls}}$$

Here, $\frac{1.0}{1} = 1.0$

I hope this helps.





HI,
Thank you for your reply!

~In your case, there were unnecessary tool call ”temperature_conversion”.~
~So, I think the score should be low because agent did unexpected behavior.~
I was a bit confused, but I'm gradually understanding.

For example:
- Reference Tool Calls: `["weather_check", "temperature_conversion"]`
- Predicted Tool Calls: `["weather_check"]`

In this case, the `temperature_conversion` Tool Call was not executed.
Therefore, even if the 'weather_check' was executed correctly including args, the score is 0 in this metrics,
Is my understanding correct?

Hi all, @licux and @sahusiddharth 

I read through the whole discussion here and it really helped me...
Recently, I was working with a client who's just starting to build multi-agent flows, and I noticed that accuracy felt a bit too strict for that early stage. I thought it’d be useful to have something simpler for onboarding 

So I explored a way to calculate precision, recall, and F1-score(like TopicAdherenceScore), but using the tool_calls and their parameters as input (not topic or context).  PR here: https://github.com/explodinggradients/ragas/pull/2096

It makes sense? WDYT

Hi @daiane-galvao 

That sounds pretty good!
I will check the details of your change!