Repository: tatsu-lab/alpaca_eval
Issue: How to solve the problem of null appearing in the evaluation results
Root cause: Validation Gap

I used alpaca_eval to evaluate the generation results of my model, but the evaluation results were null, as shown below:
```
    "dataset":"vicuna",
    "instruction":"How can you assess the credibility of a source of information, such as a news article or blog post, without relying solely on the reputation of the author or publisher?",
    "output_1":"There are a few ways to assess the credibility of a source of information. One way is to look at the author or publisher's reputation. If the author or publisher has a good reputation, then you can assume that the information is credible. Another way to assess the credibility of a source is to look at the sources that the author or publisher cites. If the sources are credible, then the information is likely to be credible as well. Finally, you can also read the article or post to see if it seems credible. If the information seems to be credible, then it is likely to be credible.",
    "generator_1":"text_davinci_003",
    "output_2":"Assessing the credibility of a source of information goes beyond just looking at the reputation of the author or publisher. Here are some factors to consider:\n\n1. Fact-checking and verification: Check if the information presented in the article is factually correct by cross-referencing it with other reputable sources. This will help ensure that the data provided is accurate and reliable.\n2. Objectivity: Look for signs of bias or opinion in the article. A credible source should present facts objectively, without expressing personal opinions or agendas.\n3. Expertise: Consider the qualifications and expertise of the authors or experts quoted in the article. Are they recognized authorities in their field? Do they have relevant experience or credentials?\n4. Transparency: Does the article clearly state its sources and provide links to them? Is there any disclosure of potential conflicts of interest or funding sources?\n5. Timeliness: Check when the article was published and whether the information is still current. Outdated information may not be relevant or accurate.\n6. Peer review: Look for articles that have been peer-reviewed or edited by other experts in the field. This adds an extra layer of scrutiny and helps ensure the accuracy and reliability of the information.\n7. Reputation: While this factor shouldn't be the only one considered, the reputation of the author, publisher, or organization can provide valuable context. However, keep in mind that even well-known publications or individuals can sometimes publish biased or incorrect information.\n8. Consistency: Compare the information presented in the article with what other reputable sources say about the same topic. If there are significant differences, it could indicate that the information in the article might not be credible.\n9. Language and tone: Pay attention to the language used in the article. Sensational headlines, inflammatory language, or overly emotional appeals can often signal a lack of credibility.\n10. Accessibility: Be wary of sources that require payment or subscription to access their content. Credible sources typically make their information available to everyone.",
    "generator_2":"mistral",
    "annotator":"alpaca_eval_gpt4",
    "preference":null,
    "price_per_example":null,
    "time_per_example":null,
    "raw_completion":null
  },
```
How should I solve this problem

|         | win_rate    | standard_error | mode      | avg_length | n_wins | n_wins_base | n_draws | n_total | discrete_win_rate | length_controlled_winrate |
| ------- | ----------- | -------------- | --------- | ---------- | ------ | ----------- | ------- | ------- | ----------------- | ------------------------- |
| mistral | 95.26315789 | 0.771056235    | community | 1472       | 724    | 36          | 0       | 760     | 95.26315789       | 85.44704804               |

From the generated leaderboard.csv, it seems that there are still 45 items that have not been evaluated.

I've retested multiple times, but there are still over 40 that haven't been evaluated.

I conducted the evaluation using the following code
```
alpaca_eval --model_outputs "mistra-dual.json" --output_path "mistra-dual" ----annotators_config "alpaca_eval_gpt4" --reference_outputs "model_outputs.json"
```

Hi, Have you solved this problem? I also faced the same problem.

I found this issue too.  And I modified the prompt, seems not happen any more.

the same problem.