Repository: tatsu-lab/alpaca_eval
Issue: The n_total of n_total result is not 805
Root cause: Environment Incompatibility

Hi, I evaluated the outputs of my model using the following command:
export OPENAI_API_KEY=sk-xxx
alpaca_eval --model_outputs 'model_ours/outputs_ours.json' 

But the n_total of n_total result is not 805.

The content of outputs_ours.json is:
[
    {
        "instruction": "What are the names of some famous actors that started their careers on Broadway?",
        "output": "1. Meryl Streep: Known for her work in films like \"Sophie's Choice,\" \"The Devil Wears Prada,\" and \"Kramer vs. Kramer,\" Meryl Streep began her career on Broadway in 1975 with the play \"27 Wagons Full of Cotton.\"\n\n2. Al Pacino: A legendary actor, Al Pacino made his Broadway debut in 1969 in \"Does a Tiger Wear a Necktie?\" He later returned to Broadway in 2004 to star in \"The Merchant of Venice.\"\n\n3. Laurence Fishburne: Best known for his roles in \"The Matrix\" trilogy and \"Boyz n the Hood,\" Laurence Fishburne started his career on Broadway in 1975 with the play \"Ceremonies in Dark Old Men.\"\n\n4. Philip Seymour Hoffman: The late, award-winning actor began his career on Broadway in 1990 with \"As the World Turns.\" He later appeared in numerous Broadway productions, including \"True West\" and \"Long Day's Journey into Night.\"\n\n5. Kevin Spacey: Known for his work in films like \"American Beauty\" and \"The Usual Suspects,\" Kevin Spacey made his Broadway debut in 1991 in \"Lost in Yonkers.\"\n\n6. James Earl Jones: With an impressive film and television career, James Earl Jones is also a renowned Broadway actor. He made his Broadway debut in 1957 in \"Sunrise at Campobello.\"\n\n7. Audra McDonald: A multiple Tony Award-winning actress, Audra McDonald started her career on Broadway in 1993 in \"Marie Christine.\"\n\n8. Nathan Lane: Best known for his work in films like \"The Birdcage\" and \"Mouse Hunt,\" Nathan Lane began his career on Broadway in 1985 with \"Laughter on the 23rd Floor.\"\n\n9. F. Murray Abraham: Known for his role in the film \"Amadeus,\" F. Murray Abraham made his Broadway debut in 1964 in \"The Miracle Worker.\"\n\n10. Angela Lansbury: A legendary actress, Angela Lansbury started her career",
        "generator": "Ours-Mistral-7B",
        "dataset": "helpful_base",
        "datasplit": "eval"
    },
    {
        "instruction": "How did US states get their names?",
        "output": "The naming of US states has various origins, including indigenous places or tribes, geographical features, historical figures, or colonial connections. Here are some common ways US states got their names:\n\n1. Indigenous Places or Tribes: Many states were named after indigenous tribes or places. For instance, Florida was named after the \"Florida Man\" or \"Land of Flowers\" in the Seminole language. Colorado was named after the Spanish word \"Colorado,\" meaning \"red\" or \"ruddy,\" possibly due to the red soil or the red hue of the sunset over the mountains.\n\n2. Geographical Features: Some states were named after geographical features, such as mountains, rivers, or bodies of water. For example, Montana means \"mountainous\" in Latin, while New Hampshire was named after the English explorer John Smith's description of the area as \"New England's Highlands.\"\n\n3. Historical Figures: A few states were named after historical figures. For instance, Virginia was named after Queen Elizabeth I of England, who was known as the \"Virgin Queen\" due to her never marrying. Georgia was named after King George II of England.\n\n4. Colonial Connections: Several states were named after English colonies or settlements. For example, Massachusetts was named after the Massachusetts Bay Company, which was granted a royal charter to establish a colony in New England. Maryland was named after Queen Henrietta Maria of France, who was the wife of King Charles I of England.\n\n5. Miscellaneous: Some states have more unusual origins for their names. For instance, Maine was named after the French word \"maine,\" meaning \"main\" or \"largest,\" as it was the mainland part of the New England colonies. Nevada means \"snow-covered\" in Spanish, although the state has very little snow. West Virginia was originally part of Virginia but became a separate state, hence the \"West\" in its name.\n\nOverall, the naming of US states is a fascinating reflection of the rich history and diverse origins of the United States.",
        "generator": "Ours-Mistral-7B",
        "dataset": "helpful_base",
        "datasplit": "eval"
    },
    ...
    ]



![WeChate8d3e72c1e6bfd018390e8979a166223](https://github.com/tatsu-lab/alpaca_eval/assets/145631036/ecc3af87-8356-495f-8521-362fbfcc26b3)


I also encountered similar problems
I ran the following command multiple times:
```
alpaca_eval --model_outputs "Mistral-7B+RAHF-DUAL+LoRA.json" --output_path "Mistral-7B+RAHF-DUAL+LoRA_gpt4" ----annotators_config "weighted_alpaca_eval_gpt4_turbo" --reference_outputs "reference_outputs.json" --is_reapply_parsing True
```
But the result is that only more than 200 examples are evaluated.

> I also encountered similar problems I ran the following command multiple times:
> 
> ```
> alpaca_eval --model_outputs "Mistral-7B+RAHF-DUAL+LoRA.json" --output_path "Mistral-7B+RAHF-DUAL+LoRA_gpt4" ----annotators_config "weighted_alpaca_eval_gpt4_turbo" --reference_outputs "reference_outputs.json" --is_reapply_parsing True
> ```
> 
> But the result is that only more than 200 examples are evaluated.

Can you access OpenAI successfully?
Unfortunately, I only have intermittent access to it.
"INFO:root:Sleeping 2 before retrying to call openai API..."
I don't know if this is the reason for the incomplete evaluation.

Can either of you reproduce this result using OpenAI rather than through a proxy server? In my experience (from previous gh issues) the problem always comes from the proxy server and I can't help for that.

For example, many proxy servers don't return logprobs. Can you share the warnings/errors that you see during annotation?

E.g. @LiuAmber the fact that you could perform AlpacaEval 1 here: #307, strongly suggests to me that the error comes from the logprobs (necessary for AlapcaEval 2 but not 1) and that this comes from your proxy server.

I'm assuming that the issue is the fact that your proxy doesn't have access to logprobs. If so try:

`alpaca_eval evaluate_from_model   --model_configs '...' --annotators_config 'alpaca_eval_clf_gpt4_turbo'`

Feel free to reopen if your issue is different

> Can either of you reproduce this result using OpenAI rather than through a proxy server? In my experience (from previous gh issues) the problem always comes from the proxy server and I can't help for that.
> 
> For example, many proxy servers don't return logprobs. Can you share the warnings/errors that you see during annotation?

Thank you. I solved this problem. The problem is caused by proxy servers and the setting of annotators_config.