Repository: tatsu-lab/alpaca_eval
Issue: cannot reindex
Root cause: Validation Gap

I've got
```
line 4429, in reindex
    raise ValueError("cannot reindex on an axis with duplicate labels")
ValueError: cannot reindex on an axis with duplicate labels
```
with both latest stable version, 0.6.2 and the one from source.
Till now I found no other version that works for all of my data.
I have pandas==2.2.2

For version 0.6.0 for some data I get this:
```
line 250, in transform
    raise ValueError("Transform function failed") from err
ValueError: Transform function failed
```

Do you have a minimal reproducible example? I can't reproduce that.

It's probably because you have instructions that are duplicates in the dataset that you are evaluating on.

I met the same error.  I want to evaluat my own dataset. For minimization. there are only three samples. samples are just for test. Test `same instruction` and different `output`.

shell: 
``` 
export OPENAI_CLIENT_CONFIG_PATH=client_config/gpt4o_config.yaml
alpaca_eval \
--name qwen2 \
--model_outputs 'example/qwen2.json' \
--reference_outputs example/gpt4.json \
--annotators_config alpaca_eval_gpt4_turbo_fn
```

for GPT4.json
```
[
    {
        "instruction": "What are the names of some famous actors that started their careers on Broadway?",
        "output": "Some famous actors that started their careers on Broadway include: Tom Hanks, Meryl Streep, Laurence Olivier, Christopher Walken, and Jeremy Irons.",
        "generator": "gpt4"
    },
    {
        "instruction": "How did US states get their names?",
        "output": "US states got their names for a variety of reasons. Many are named after early explorers, leaders, or patriots. Some are named after geographical features, such as mountains or rivers. Others are named after plants or animals that are native to the state. Some are even named after historical figures or characters from literature or mythology.",
        "generator": "gpt4"
    },
    {
        "instruction": "Hi, my sister and her girlfriends want me to play kickball with them. Can you explain how the game is played, so they don't take advantage of me?",
        "output": "Sure! Kickball is a game typically played between two teams of nine players each. The object of the game is to get your team's ball across the field to score points, while preventing the other team from doing the same. Players can kick, throw, and run with the ball, and each team has a designated home base and a designated goal area. The team with the most points at the end of the game wins.",
        "generator": "gpt4"
    }
]
```
for qwen2.json
```
[
    {
        "instruction": "What are the names of some famous actors that started their careers on Broadway?",
        "output": "Some famous actors that started their careers on Broadway include: Tom Hanks, Meryl Streep, Laurence Olivier, Christopher Walken, and Jeremy Irons.",
        "generator": "qwen2"
    },
    {
        "instruction": "How did US states get their names?",
        "output": "US states got their names for a variety of reasons. Many are named after early explorers, leaders, or patriots. Some are named after geographical features, such as mountains or rivers. Others are named after plants or animals that are native to the state. Some are even named after historical figures or characters from literature or mythology.",
        "generator": "qwen2"
    },
    {
        "instruction": "Hi, my sister and her girlfriends want me to play kickball with them. Can you explain how the game is played, so they don't take advantage of me?",
        "output": "Sure! Kickball is a game typically played between two teams of nine players each. The object of the game is to get your team's ball across the field to score points, while preventing the other team from doing the same. Players can kick, throw, and run with the ball, and each team has a designated home base and a designated goal area. The team with the most points at the end of the game wins.",
        "generator": "qwen2"
    }
]
```



Same error as well