Repository: modelscope/evalscope
Issue: bfcl_v3测试评分基本都是0，Invalid syntax. Failed to decode AST.
Root cause: Interface Contract Mismatch

## 自查清单

在提交 issue 之前，请确保您已完成以下步骤:
- [x] 我已仔细阅读了[相关使用说明文档](https://evalscope.readthedocs.io/zh-cn/latest/get_started/parameters.html)
- [x] 我已查看了[常见问题解答](https://evalscope.readthedocs.io/zh-cn/latest/get_started/faq.html)
- [x] 我已搜索并查看了现有的 issues，确认这不是一个重复的问题

## 问题描述

bfcl_v3测试结果score除了relevance项目是1，其它的都是0

## EvalScope 版本（必填）
0.17.1

## 使用的工具
- [x] Native / 原生框架
- [ ] Opencompass backend
- [ ] VLMEvalKit backend
- [ ] RAGEval backend
- [ ] Perf / 模型推理压测工具
- [ ] Arena / 竞技场模式

## 执行的代码或指令

```
evalscope eval --eval-backend OpenCompass --model Qwen3-Coder-30B-A3B-Instruct-FP8  --api-url http://127.0.0.1:8006/v1/chat/completions  --api-key EMPTY  --eval-type service  --datasets bfcl_v3 --generation-config '{"do_sample":true,"temperature":0.7, "top_p": 0.8, "max_tokens": 48000}'  --limit 20
```

## 错误日志

没有错误日志，但是report中score基本都是0

```json
{
    "name": "Qwen3-Coder-30B-A3B-Instruct-FP8@bfcl_v3",
    "dataset_name": "bfcl_v3",
    "dataset_pretty_name": "BFCL-v3",
    "dataset_description": "Berkeley Function Calling Leaderboard (BFCL), the **first comprehensive and executable function call evaluation** dedicated to assessing Large Language Models' (LLMs) ability to invoke functions. Unlike previous evaluations, BFCL accounts for various forms of function calls, diverse scenarios, and executability. Need to run `pip install bfcl-eval` before evaluating. [Usage Example](https://evalscope.readthedocs.io/zh-cn/latest/third_party/bfcl_v3.html)",
    "model_name": "Qwen3-Coder-30B-A3B-Instruct-FP8",
    "score": 0.1198,
    "metrics": [
        {
            "name": "AverageAccuracy",
            "num": 334,
            "score": 0.1198,
            "macro_score": 0.1724,
            "categories": [
                {
                    "name": [
                        "AST_LIVE"
                    ],
                    "num": 76,
                    "score": 0.0,
                    "macro_score": 0.0,
                    "subsets": [
                        {
                            "name": "live_simple",
                            "score": 0.0,
                            "num": 20
                        },
                        {
                            "name": "live_multiple",
                            "score": 0.0,
                            "num": 20
                        },
                        {
                            "name": "live_parallel",
                            "score": 0.0,
                            "num": 16
                        },
                        {
                            "name": "live_parallel_multiple",
                            "score": 0.0,
                            "num": 20
                        }
                    ]
                },
                {
                    "name": [
                        "AST_NON_LIVE"
                    ],
                    "num": 120,
                    "score": 0.0,
                    "macro_score": 0.0,
                    "subsets": [
                        {
                            "name": "simple",
                            "score": 0.0,
                            "num": 20
                        },
                        {
                            "name": "multiple",
                            "score": 0.0,
                            "num": 20
                        },
                        {
                            "name": "parallel",
                            "score": 0.0,
                            "num": 20
                        },
                        {
                            "name": "parallel_multiple",
                            "score": 0.0,
                            "num": 20
                        },
                        {
                            "name": "java",
                            "score": 0.0,
                            "num": 20
                        },
                        {
                            "name": "javascript",
                            "score": 0.0,
                            "num": 20
                        }
                    ]
                },
                {
                    "name": [
                        "MULTI_TURN"
                    ],
                    "num": 80,
                    "score": 0.0,
                    "macro_score": 0.0,
                    "subsets": [
                        {
                            "name": "multi_turn_base",
                            "score": 0.0,
                            "num": 20
                        },
                        {
                            "name": "multi_turn_miss_func",
                            "score": 0.0,
                            "num": 20
                        },
                        {
                            "name": "multi_turn_miss_param",
                            "score": 0.0,
                            "num": 20
                        },
                        {
                            "name": "multi_turn_long_context",
                            "score": 0.0,
                            "num": 20
                        }
                    ]
                },
                {
                    "name": [
                        "RELEVANCE"
                    ],
                    "num": 58,
                    "score": 0.6897,
                    "macro_score": 0.6667,
                    "subsets": [
                        {
                            "name": "irrelevance",
                            "score": 1.0,
                            "num": 20
                        },
                        {
                            "name": "live_relevance",
                            "score": 0.0,
                            "num": 18
                        },
                        {
                            "name": "live_irrelevance",
                            "score": 1.0,
                            "num": 20
                        }
                    ]
                }
            ]
        }
    ],
    "analysis": "N/A"
}
```


请粘贴完整的错误日志或控制台输出。

## 运行环境

- 操作系统：linux
- Python版本：3.10

## 其他信息

我实际查看了一下prediction, head -n 1 bfcl_v3_simple.jsonl 结果是对的

```json
{
  "choices": [
    {
      "index": 0,
      "message": {
        "content": [
          [
            "<tool_call>\n<function=calculate_triangle_area>\n<parameter=base>\n10\n</parameter>\n<parameter=height>\n5\n</parameter>\n</function>\n</tool_call>"
          ]
        ],
        "role": "assistant"
      }
    }
  ],
  "created": 1754545421.2094836,
  "model": "Qwen3-Coder-30B-A3B-Instruct-FP8",
  "object": "chat.completion",
  "usage": {
    "completion_tokens": 0,
    "prompt_tokens": 0,
    "total_tokens": 0
  },
  "model_spec": {
    "api_url": "http://10.166.192.142:8006/v1/chat/completions",
    "model_id": "Qwen3-Coder-30B-A3B-Instruct-FP8",
    "api_key": "EMPTY"
  },
  "answer_id": "answer-0b2be79365cfb448f226a190c8f2cada",
  "subset_name": "simple",
  "raw_input": {
    "id": "simple_0",
    "multi_turn": false,
    "functions": [
      {
        "name": "calculate_triangle_area",
        "description": "Calculate the area of a triangle given its base and height.",
        "parameters": {
          "type": "dict",
          "properties": {
            "base": {
              "type": "integer",
              "description": "The base of the triangle."
            },
            "height": {
              "type": "integer",
              "description": "The height of the triangle."
            },
            "unit": {
              "type": "string",
              "description": "The unit of measure (defaults to 'units' if not specified)"
            }
          },
          "required": [
            "base",
            "height"
          ]
        }
      }
    ],
    "tools": [
      {
        "type": "function",
        "function": {
          "name": "calculate_triangle_area",
          "description": "Calculate the area of a triangle given its base and height. Note that the provided function is in Python 3 syntax.",
          "parameters": {
            "type": "object",
            "properties": {
              "base": {
                "type": "integer",
                "description": "The base of the triangle."
              },
              "height": {
                "type": "integer",
                "description": "The height of the triangle."
              },
              "unit": {
                "type": "string",
                "description": "The unit of measure (defaults to 'units' if not specified)"
              }
            },
            "required": [
              "base",
              "height"
            ]
          }
        }
      }
    ],
    "missed_functions": "{}",
    "initial_config": {},
    "involved_classes": [],
    "turns": [
      [
        {
          "role": "user",
          "content": "Find the area of a triangle with a base of 10 units and height of 5 units."
        }
      ]
    ],
    "language": "Python",
    "test_category": "simple",
    "subset": "simple",
    "ground_truth": [
      {
        "calculate_triangle_area": {
          "base": [
            10
          ],
          "height": [
            5
          ],
          "unit": [
            "units",
            ""
          ]
        }
      }
    ],
    "should_execute_tool_calls": false,
    "missing_functions": {},
    "is_fc_model": true
  },
  "index": 0
}
```

查看review日志

```json
{
  "choices": [
    {
      "index": 0,
      "message": {
        "content": [
          [
            "<tool_call>\n<function=math_factorial>\n<parameter=number>\n5\n</parameter>\n</function>\n</tool_call>"
          ]
        ],
        "role": "assistant"
      },
      "review": {
        "gold": [
          {
            "math.factorial": {
              "number": [
                5
              ]
            }
          }
        ],
        "pred": {
          "id": "simple_1",
          "multi_turn": false,
          "functions": [
            {
              "name": "math.factorial",
              "description": "Calculate the factorial of a given number.",
              "parameters": {
                "type": "dict",
                "properties": {
                  "number": {
                    "type": "integer",
                    "description": "The number for which factorial needs to be calculated."
                  }
                },
                "required": [
                  "number"
                ]
              }
            }
          ],
          "tools": [
            {
              "type": "function",
              "function": {
                "name": "math_factorial",
                "description": "Calculate the factorial of a given number. Note that the provided function is in Python 3 syntax.",
                "parameters": {
                  "type": "object",
                  "properties": {
                    "number": {
                      "type": "integer",
                      "description": "The number for which factorial needs to be calculated."
                    }
                  },
                  "required": [
                    "number"
                  ]
                }
              }
            }
          ],
          "missed_functions": "{}",
          "initial_config": {},
          "involved_classes": [],
          "language": "Python",
          "test_category": "simple",
          "subset": "simple",
          "ground_truth": [
            {
              "math.factorial": {
                "number": [
                  5
                ]
              }
            }
          ],
          "should_execute_tool_calls": false,
          "missing_functions": {},
          "is_fc_model": true,
          "generation": [
            [
              "<tool_call>\n<function=math_factorial>\n<parameter=number>\n5\n</parameter>\n</function>\n</tool_call>"
            ]
          ]
        },
        "result": {
          "AverageAccuracy": 0.0,
          "raw_score_result": {
            "valid": false,
            "error_message": "Invalid syntax. Failed to decode AST. Traceback: Traceback (most recent call last):\n  File \"/root/workshop/evalscope/.venv/lib/python3.10/site-packages/evalscope/benchmarks/bfcl/bfcl_adapter.py\", line 207, in match\n    name = list(tool_call.keys())[0]\nAttributeError: 'str' object has no attribute 'keys'\n",
            "error_type": "ast_decoder:decoder_failed"
          }
        }
      }
    }
  ],
  "created": 1754549915.0126147,
  "model": "Qwen3-Coder-30B-A3B-Instruct-FP8",
  "object": "chat.completion",
  "usage": {
    "completion_tokens": 0,
    "prompt_tokens": 0,
    "total_tokens": 0
  },
  "model_spec": {
    "api_url": "http://10.166.192.142:8006/v1/chat/completions",
    "model_id": "Qwen3-Coder-30B-A3B-Instruct-FP8",
    "api_key": "EMPTY"
  },
  "answer_id": "answer-94cbae2a6722b7a5488fe000051523d1",
  "subset_name": "simple",
  "raw_input": {
    "id": "simple_1",
    "multi_turn": false,
    "functions": [
      {
        "name": "math.factorial",
        "description": "Calculate the factorial of a given number.",
        "parameters": {
          "type": "dict",
          "properties": {
            "number": {
              "type": "integer",
              "description": "The number for which factorial needs to be calculated."
            }
          },
          "required": [
            "number"
          ]
        }
      }
    ],
    "tools": [
      {
        "type": "function",
        "function": {
          "name": "math_factorial",
          "description": "Calculate the factorial of a given number. Note that the provided function is in Python 3 syntax.",
          "parameters": {
            "type": "object",
            "properties": {
              "number": {
                "type": "integer",
                "description": "The number for which factorial needs to be calculated."
              }
            },
            "required": [
              "number"
            ]
          }
        }
      }
    ],
    "missed_functions": "{}",
    "initial_config": {},
    "involved_classes": [],
    "turns": [
      [
        {
          "role": "user",
          "content": "Calculate the factorial of 5 using math functions."
        }
      ]
    ],
    "language": "Python",
    "test_category": "simple",
    "subset": "simple",
    "ground_truth": [
      {
        "math.factorial": {
          "number": [
            5
          ]
        }
      }
    ],
    "should_execute_tool_calls": false,
    "missing_functions": {},
    "is_fc_model": true
  },
  "index": 1,
  "reviewed": true,
  "review_id": "review-29a24977db9d5672a38b9b956331e0b0",
  "reviewer_spec": {
    "metric": [
      "AverageAccuracy"
    ],
    "reviewer": [
      "Evaluator"
    ],
    "revision": [
      "default"
    ]
  },
  "review_time": 1754549917.2696753
}
```

使用 v1.0.0 版本是否可以解决这个问题

thinking模型评分也是0


v1.0.1还是没有解决这个问题，函数语言类型和bfcl里面的定义对应不上

evalscop侧
<img width="793" height="395" alt="Image" src="https://github.com/user-attachments/assets/04031bbb-7f19-4655-9f3c-80d93f3b8102" />

evalscope/benchmarks/bfcl/bfcl_adapter.py

bfcl_eval侧

<img width="687" height="330" alt="Image" src="https://github.com/user-attachments/assets/383bcd12-08ce-4a37-a915-790e15be0501" />
bfcl_eval/eval_checker/ast_eval/ast_checker.py

请安装`pip install bfcl-eval==2025.6.16`再试一下

> 感谢你的反馈！我们将关闭此问题。如果您有任何疑问，请随时重新打开它。如果EvalScope对您有所帮助，欢迎给我们点个STAR以示支持，谢谢！