Repository: tatsu-lab/alpaca_eval
Issue: Possibility of adding a version signature
Root cause: Unimplemented Feature Gap

I want to propose adding a version signature to AlpacaEval a la [sacreBLEU signatures](https://github.com/mjpost/sacrebleu?tab=readme-ov-file#version-signatures) and explicit instructions for reporting scores to improve reproducibility.

For those unfamiliar with the sacreBLEU metric, the library reports a version signature when scoring that contains parameters important to the calculation. The example in the link above is:
`BLEU|nrefs:1|case:mixed|eff:no|tok:13a|smooth:exp|version:2.0.0`.
The first field contains the metric and the last field contains the version. The other fields report parameters that affect the score.

Considering that AlpacaEval supports user-selected evaluators, AlpacaEval 2.0 uses a different baseline (and a different recommended evaluator), the potential length-adjusted metric being discussed in #225, and that the repo uses a versioning scheme separate from AE 1 vs AE 2, I think it would be helpful to have more explicit reporting.

For example,
`alpaca|m:win|e:weighted_ae_g4turbo |b:g4turbo|ae:2|v:0.5.4`
could denote using the raw win_rate from AlpacaEval 2.0 metric & release 0.5.4 with the `weighted_alpaca_eval_gpt4_turbo` evaluator and `gpt4_turbo` baseline.
Not suggesting that we necessarily should use this format--just an idea!

Do you think this would be helpful?

Yeah, I like that! Are you suggesting to save that in the csv leaderboard? Note that all os "m:win|e:weighted_ae_g4turbo |b:g4turbo|ae:2" is actually equivalent to "AlpacaEval 2" and thus constant inside of a csv leaderboard but it can be good to have the signature for people to refer to in a paper. I see how `v:0.5.4` can be very useful to know!

Related: I think we should also save the date in the annotations & model_output JSON as those change over time + caching means that not all examples were annotated on the same day.

Oh I think including the date would **definitely** be helpful for calculating the leaderboard! Would saving both the date and either the repo version or commit hash to the annotation/model_output JSON files be overkill?

As for the signature, my motivation was strictly for reporting in papers or other research. Exactly like you said, the parameters for the leaderboard are mostly fixed (and any changes could be tracked by the inclusion of dates).

If the leaderboard is the *primary* goal of AlpacaEval, adding better metadata to the JSON files would probably be a higher priority. What do you think?

Todo:
- [ ] add a signature in the csv leaderboard so that people can report them in papers and make sure it's comparable.
- [ ] print the signature  in the terminal when running `alpaca_eval ...` so that people can quickly check that they are running what they are thinking
- [ ] add the date & repo version in the JSONs (only for new annotations & generations)

Not sure when I'll have time but should be able to do it this week or next. Feel free to open a PR also if you want to help.


Note:
- IMO leaderboard isn't the primary goal of AE, I think the goal is an evaluation that people can hyperparameter tune while knowing that it's pretty highly correlated with human preference. But leaderboard is important to provide the trust for people to use it for hparam tuning!

I'm closing as I don't think that I will be able to add this feature unfortunately.