Repository: tatsu-lab/alpaca_eval
Issue: Latest LC-AlpacaEval update broken?
Root cause: Documentation Deficiency

Hi @YannDubs, thank you so much for the providing the code to evaluate models on AlpacaEval. I was trying to use the latest 
version which has support for the length controlled winrate but it seems broken? I tried a new conda environment and it seems I get the same error. When I revert back to the previous version (0.5.4), I am able to get the raw win rate again.

```
Traceback (most recent call last):
  File "/import/ml-sc-scratch1/ravir/miniconda3/envs/length_alpaca/lib/python3.10/site-packages/pandas/core/indexes/base.py", line 3805, in get_loc
    return self._engine.get_loc(casted_key)
  File "index.pyx", line 167, in pandas._libs.index.IndexEngine.get_loc
  File "index.pyx", line 196, in pandas._libs.index.IndexEngine.get_loc
  File "pandas/_libs/hashtable_class_helper.pxi", line 7081, in pandas._libs.hashtable.PyObjectHashTable.get_item
  File "pandas/_libs/hashtable_class_helper.pxi", line 7089, in pandas._libs.hashtable.PyObjectHashTable.get_item
KeyError: 'generator_2'

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/import/ml-sc-scratch1/ravir/miniconda3/envs/length_alpaca/bin/alpaca_eval", line 8, in <module>
    sys.exit(main())
  File "/import/ml-sc-scratch1/ravir/miniconda3/envs/length_alpaca/lib/python3.10/site-packages/alpaca_eval/main.py", line 610, in main
    fire.Fire(evaluate)
  File "/import/ml-sc-scratch1/ravir/miniconda3/envs/length_alpaca/lib/python3.10/site-packages/fire/core.py", line 143, in Fire
    component_trace = _Fire(component, args, parsed_flag_args, context, name)
  File "/import/ml-sc-scratch1/ravir/miniconda3/envs/length_alpaca/lib/python3.10/site-packages/fire/core.py", line 477, in _Fire
    component, remaining_args = _CallAndUpdateTrace(
  File "/import/ml-sc-scratch1/ravir/miniconda3/envs/length_alpaca/lib/python3.10/site-packages/fire/core.py", line 693, in _CallAndUpdateTrace
    component = fn(*varargs, **kwargs)
  File "/import/ml-sc-scratch1/ravir/miniconda3/envs/length_alpaca/lib/python3.10/site-packages/alpaca_eval/main.py", line 179, in evaluate
    leaderboard[name].update(fn_metric_(annotations, **(metric_kwargs or {})))
  File "/import/ml-sc-scratch1/ravir/miniconda3/envs/length_alpaca/lib/python3.10/site-packages/alpaca_eval/metrics/glm_winrate.py", line 79, in get_length_controlled_w
inrate
    assert len(df["generator_2"].unique()) == 1
  File "/import/ml-sc-scratch1/ravir/miniconda3/envs/length_alpaca/lib/python3.10/site-packages/pandas/core/frame.py", line 4090, in __getitem__
    indexer = self.columns.get_loc(key)
  File "/import/ml-sc-scratch1/ravir/miniconda3/envs/length_alpaca/lib/python3.10/site-packages/pandas/core/indexes/base.py", line 3812, in get_loc
    raise KeyError(key) from err
KeyError: 'generator_2'
```

Thanks for flagging @rraju1!

Can you add a field in your outputs that is called “generator” and says the name of your model? The resulting paired outputs should then (automatically) have a field “generator_1” that specifies the baseline name, and “generator_2” specifying the model name.

I’ll change the documentation for that. The issue is that I need to know what model you are using! 

let me know if you still have issues!

Hi @YannDubs, thanks for your response! So, this is my understanding of how to work around this issue.

1. Generate the annotation json of the model with the commands in the documentation for the raw win rate
2. add generator_2 field to annotation.json
3. relaunch with command for LC-win rate

is that the correct flow? This is my command I am running for clarity.

`alpaca_eval --model_outputs my_model_generations.json --annotators_config weighted_alpaca_eval_gpt4_turbo --output_path my_model_generation --is_overwrite_leaderboard` 

Yes that works although you shouldn't need to add `generator_2` in `annotations.json` if you add `generator` in `my_model_generations.json` as the latter will be converted to the former.

Btw you don't need to specify `--annotators_config weighted_alpaca_eval_gpt4_turbo --output_path my_model_generation`, those will be the defaults

got it, thank you!