Repository: tatsu-lab/alpaca_eval
Issue: Can you realize deepseek as a judge?
Root cause: Documentation Deficiency


+1, has anybody tried to do it?

You can add an deepseek evaluator in your configs:

in path `src/alpaca_eval/evaluators_configs/deepseek_v3_eval/configs.yaml`


```
deepseek_v3_eval:
  prompt_template: "alpaca_eval_clf_gpt4_turbo/alpaca_eval_clf.txt"
  fn_completions: "openai_completions"
  completions_kwargs:
    model_name: "deepseek-v3-241226" # write to your corresponding model name
    max_tokens: 1
    temperature: 1 # temperature should be applied for sampling, so that should make no effect.
    logprobs: true
    top_logprobs: 5
  fn_completion_parser: "logprob_parser"
  completion_parser_kwargs:
    numerator_token: "m"
    denominator_tokens: ["m", "M"]
    is_binarize: false
  completion_key: "completions_all"
  batch_size: 1
```

and set the proxy config OPENAI_API_BASE and OPENAI_API_KEY.

```
alpaca_eval evaluate --model_outputs "${model_path}.json" \
    --output_path="${output_path}" \
    --annotators_config 'deepseek_v3_eval'
```

You can also pull the [repo](https://github.com/junming-yang/alpaca_eval).


> You can add an deepseek evaluator in your configs:
> 
> in path `src/alpaca_eval/evaluators_configs/deepseek_v3_eval/configs.yaml`
> 
> ```
> deepseek_v3_eval:
>   prompt_template: "alpaca_eval_clf_gpt4_turbo/alpaca_eval_clf.txt"
>   fn_completions: "openai_completions"
>   completions_kwargs:
>     model_name: "deepseek-v3-241226" # write to your corresponding model name
>     max_tokens: 1
>     temperature: 1 # temperature should be applied for sampling, so that should make no effect.
>     logprobs: true
>     top_logprobs: 5
>   fn_completion_parser: "logprob_parser"
>   completion_parser_kwargs:
>     numerator_token: "m"
>     denominator_tokens: ["m", "M"]
>     is_binarize: false
>   completion_key: "completions_all"
>   batch_size: 1
> ```
> 
> and set the proxy config OPENAI_API_BASE and OPENAI_API_KEY.
> 
> ```
> alpaca_eval evaluate --model_outputs "${model_path}.json" \
>     --output_path="${output_path}" \
>     --annotators_config 'deepseek_v3_eval'
> ```
> 
> You can also pull the [repo](https://github.com/junming-yang/alpaca_eval).

I do the same as you.However,I failed to get metric.
  File "D:\project\Py\LLM-eval\.venv\Lib\site-packages\alpaca_eval\metrics\glm_winrate.py", line 89, in get_length_controlled_winrate
    assert len(df["generator_2"].unique()) == 1
but no generator_2 found

As use `deepseek-chat` for example, you can add a file named `openai_configs.yaml` in client_configs, with 
```default:
    - api_key: ""
      base_url: "https://api.deepseek.com"
```
and add `configs.yaml` file in `src/alpaca_eval/evaluators_configs/deepseek-chat` (make a dir) with:
```
deepseek-chat:
  prompt_template: "alpaca_eval_clf_gpt4_turbo/alpaca_eval_clf.txt"
  fn_completions: "openai_completions"
  completions_kwargs:
    model_name: "deepseek-chat" # revise to your model name
    max_tokens: 1
    temperature: 1 # temperature should be applied for sampling, so that should make no effect.
    logprobs: true
    top_logprobs: 5
  fn_completion_parser: "logprob_parser"
  completion_parser_kwargs:
    numerator_token: "m"
    denominator_tokens: ["m", "M"]
    is_binarize: false
  completion_key: "completions_all"
  batch_size: 1
```
and then you can eval with:
```
alpaca_eval \
    --model_outputs xxx.json \
    --reference_outputs xxx.json \
    --output_path ./deepseek_evaluation \
    --annotators_config 'deepseek-chat'
```