Repository: tatsu-lab/alpaca_eval
Issue: Alpaca Evaluation Instruction Difficulty used also for Custom Evaluation Dataset
Root cause: Architectural Constraint

Hi, I am interested in using the Alpaca Eval library to calculate the (LC) win-rate on a custom dataset, with a custom annotator (I am using it for TLDR summarization).

After having defined the dataset and annotator config files, I receive the winrate and LC winrate calculation. However, I have realized that in the code referenced below for the LC winrate, the precomputed instruction difficulty from the alpaca evaluation dataset is being used, as it seems hardcoded. I originally did notice this as my eval dataset had less than 804 samples. After increasing it, one suddenly received `KeyError`, when the dataframes are merged. Can the instruction difficulty be calculated for a custom dataset in this part of the code?

https://github.com/tatsu-lab/alpaca_eval/blob/cfe64cffc9265e2183806b0b1868ddc50b8bfb55/src/alpaca_eval/metrics/glm_winrate.py#L208C5-L217C89

I agree this is not ideal, I unfortunately won't have the time to make the changes in the next few weeks. Instruction difficulty helps but is not strictly ended (correlation improved from 0.96 to 0.98). The problem for instruction difficulty is that you need to have predictions of many models on your custom dataset to estimate it. If that's not the case for you, then I would recommend just dropping the instruction difficulty from the GLM. This can be done by adding a new formula here: https://github.com/tatsu-lab/alpaca_eval/blob/cfe64cffc9265e2183806b0b1868ddc50b8bfb55/src/alpaca_eval/metrics/glm_winrate.py#L23 without instruction difficulty (call it `length_controlled_no_instruction_difficulty`, and then generalizing this function to only call instruction difficulty if it's needed https://github.com/tatsu-lab/alpaca_eval/blob/cfe64cffc9265e2183806b0b1868ddc50b8bfb55/src/alpaca_eval/metrics/glm_winrate.py#L163 (the assert would then be dropped). You would then call alpaca_eval as such 

```
alpaca_eval --model_outputs ... --metric_kwargs "{'glm_name':'length_controlled_no_instruction_difficulty'}"
```

To run AE LC without instruction difficulty. Alternatively, if you have enough models to estimate instruction difficulty then you could add a parameter to `predict_winrate` to take as input the path of the instruction difficulty (you would have to also redirect kwargs). After doing that and estimating your instruction difficulty you could call:

```
alpaca_eval --model_outputs ... --metric_kwargs "{'instruction_difficulty_path':'...'}"
```

If the latter is the path you want to take then here's the notebook that includes many things about LC including estimating instruction_difficulty (sorry it's not very clean)
[glm_length_correction_all.ipynb.zip](https://github.com/tatsu-lab/alpaca_eval/files/15341186/glm_length_correction_all.ipynb.zip)

Good luck and I'm happy to accept PRs for either of those solutions!


Thank you for your detailed answer! 

I also just dropped the difficulty, by setting the `instruction_difficulty` manually to a vector of zeros.

I will have a look on how to change, or at least warn user if they are using a custom dataset to not use this function. Will give it a try to do a PR after the deadline next week, that we're all working for;)

Great, that also works! I'm closing the issue for now, but would love to see the PR at some point. Good luck with your push!