Repository: tatsu-lab/alpaca_eval
Issue: error with default json - ValueError: cannot reindex on an axis with duplicate labels
Root cause: Validation Gap

I made a smaller outputs.json just to test this out with 3 sample entries from the examples/output.json provided in the repo, and I got this error: ValueError: cannot reindex on an axis with duplicate labels. This error persisted even when I fed in an outputs.json file that had all of the questions accounted for. Any advice for how to fix this? 



(moa_new) [mixtureOfModels]$ alpaca_eval --model_outputs 'outputs_trial.json' --annotators_config 'alpaca_eval_gpt4_turbo_fn'
WARNING:root:precomputed_leaderboard = 'auto'. But we have found no corresponding leaderboard
INFO:root:Evaluating the example outputs.
INFO:root:Creating the annotator from `alpaca_eval_gpt4_turbo_fn`.
INFO:root:Saving annotations to `/dccstor/ishapuri/miniconda3/envs/moa_new/lib/python3.10/site-packages/alpaca_eval/evaluators_configs/alpaca_eval_gpt4_turbo_fn/annotations_seed0_configs.json`.
INFO:root:Loading all annotations from /dccstor/ishapuri/miniconda3/envs/moa_new/lib/python3.10/site-packages/alpaca_eval/evaluators_configs/alpaca_eval_gpt4_turbo_fn/annotations_seed0_configs.json.
WARNING:root:The length of outputs before and after merge are not the same. We have len(outputs_1)==805, len(outputs_2)==1, and len(df_annotated)==1. This means that there are missing examples or duplicates. We are taking a SQL inner join.
Annotation chunk:   0%|                                       | 0/1 [00:00<?, ?it/s]INFO:root:Annotating 0 examples with alpaca_eval_gpt4_turbo_fn
INFO:root:Saving all annotations to /dccstor/ishapuri/miniconda3/envs/moa_new/lib/python3.10/site-packages/alpaca_eval/evaluators_configs/alpaca_eval_gpt4_turbo_fn/annotations_seed0_configs.json.
INFO:root:Loading all annotations from /dccstor/ishapuri/miniconda3/envs/moa_new/lib/python3.10/site-packages/alpaca_eval/evaluators_configs/alpaca_eval_gpt4_turbo_fn/annotations_seed0_configs.json.
Annotation chunk: 100%|███████████████████████████████| 1/1 [00:00<00:00,  2.71it/s]
Traceback (most recent call last):
  File "/dccstor/ishapuri/miniconda3/envs/moa_new/bin/alpaca_eval", line 8, in <module>
    sys.exit(main())
  File "/dccstor/ishapuri/miniconda3/envs/moa_new/lib/python3.10/site-packages/alpaca_eval/main.py", line 611, in main
    fire.Fire(evaluate)
  File "/dccstor/ishapuri/miniconda3/envs/moa_new/lib/python3.10/site-packages/fire/core.py", line 143, in Fire
    component_trace = _Fire(component, args, parsed_flag_args, context, name)
  File "/dccstor/ishapuri/miniconda3/envs/moa_new/lib/python3.10/site-packages/fire/core.py", line 477, in _Fire
    component, remaining_args = _CallAndUpdateTrace(
  File "/dccstor/ishapuri/miniconda3/envs/moa_new/lib/python3.10/site-packages/fire/core.py", line 693, in _CallAndUpdateTrace
    component = fn(*varargs, **kwargs)
  File "/dccstor/ishapuri/miniconda3/envs/moa_new/lib/python3.10/site-packages/alpaca_eval/main.py", line 179, in evaluate
    leaderboard[name].update(fn_metric_(annotations, **(metric_kwargs or {})))
  File "/dccstor/ishapuri/miniconda3/envs/moa_new/lib/python3.10/site-packages/alpaca_eval/metrics/glm_winrate.py", line 95, in get_length_controlled_winrate
    df_XY_train, df_X_test, sample_weight = _get_featurized_data(
  File "/dccstor/ishapuri/miniconda3/envs/moa_new/lib/python3.10/site-packages/alpaca_eval/metrics/glm_winrate.py", line 247, in _get_featurized_data
    df_XY_train, df_X_test = make_dmatrix_for_model(df_gamed_and_m, df_test, formula=formula)
  File "/dccstor/ishapuri/miniconda3/envs/moa_new/lib/python3.10/site-packages/alpaca_eval/metrics/glm_winrate.py", line 281, in make_dmatrix_for_model
    df_XY_train[col_y_true] = df_train[col_y_true]  # adds the label
  File "/dccstor/ishapuri/miniconda3/envs/moa_new/lib/python3.10/site-packages/pandas/core/frame.py", line 4311, in __setitem__
    self._set_item(key, value)
  File "/dccstor/ishapuri/miniconda3/envs/moa_new/lib/python3.10/site-packages/pandas/core/frame.py", line 4524, in _set_item
    value, refs = self._sanitize_column(value)
  File "/dccstor/ishapuri/miniconda3/envs/moa_new/lib/python3.10/site-packages/pandas/core/frame.py", line 5263, in _sanitize_column
    return _reindex_for_setitem(value, self.index)
  File "/dccstor/ishapuri/miniconda3/envs/moa_new/lib/python3.10/site-packages/pandas/core/frame.py", line 12692, in _reindex_for_setitem
    raise err
  File "/dccstor/ishapuri/miniconda3/envs/moa_new/lib/python3.10/site-packages/pandas/core/frame.py", line 12687, in _reindex_for_setitem
    reindexed_value = value.reindex(index)._values
  File "/dccstor/ishapuri/miniconda3/envs/moa_new/lib/python3.10/site-packages/pandas/core/series.py", line 5153, in reindex
    return super().reindex(
  File "/dccstor/ishapuri/miniconda3/envs/moa_new/lib/python3.10/site-packages/pandas/core/generic.py", line 5610, in reindex
    return self._reindex_axes(
  File "/dccstor/ishapuri/miniconda3/envs/moa_new/lib/python3.10/site-packages/pandas/core/generic.py", line 5633, in _reindex_axes
    new_index, indexer = ax.reindex(
  File "/dccstor/ishapuri/miniconda3/envs/moa_new/lib/python3.10/site-packages/pandas/core/indexes/base.py", line 4429, in reindex
    raise ValueError("cannot reindex on an axis with duplicate labels")
ValueError: cannot reindex on an axis with duplicate labels

can you sned me the JSON you are using? seems like a formatting issue

Closing due to no response.

@YannDubs 
I encounter the same problem here is an example .json

[
    {
        "instruction": "What are the names of some famous actors that started their careers on Broadway?",
        "output": "There have been many famous actors that started their careers on Broadway, but here are a few notable ones:\n\n- Audrey Hepburn: Born in London, Audrey Hepburn began her career on the stage, appearing in her first play \"A List of Miracles\" at the Royal Exchange Theatre in 1937 at the age of 16.\n- John Travolta: John Travolta began his career on Broadway as a member of the cast of \"Hairsponsor,\" a musical about a young man's journey to New York City. He later went on to become a film and television actor.\n- Meryl Streep: Meryl Streep began her career on Broadway as an actress in the show \"The Grass Widow.\" She later went on to become one of the most successful and beloved actresses in film and television.\n- Robert De Niro: Robert De Niro began his career on Broadway as a member of the cast of \"Roxie Roxes,\" a musical about a young woman's journey to New York City. He later went on to become a film and television actor.",
        "generator": "mode_gptq_lora_bits_3_rank_256_group_32",
        "dataset": "helpful_base"
    }
]