zhiminy/EvalEng huggingface dataset:
Repository: tatsu-lab/alpaca_eval
Issue: The response was filtered due to the prompt triggering Azure OpenAI's content management policy
Root cause: External Dependency Breakage

This error is triggered when I evaluate with alpaca_eval version1. baseline is text_davinci_003 and evaluatior is gpt4. It was evaluating fine up until this point, and in the last couple of days it's been a problem
script:
alpaca_eval --model_outputs /data/local/user/fanyi/github/Humback/results//model_outputs.json --annotators_config 'alpaca_eval_gpt4_turbo_fn'
error:
openai.BadRequestError: Error code: 400 - {'error': {'message': "The response was filtered due to the prompt triggering Azure OpenAI's content management policy. Please modify your prompt and retry. To learn more about our content filtering policies please read our documentation: https://go.microsoft.com/fwlink/?linkid=2198766", 'type': None, 'param': 'prompt', 'code': 'content_filter', 'status': 400, 'innererror': {'code': 'ResponsibleAIPolicyViolation', 'content_filter_result': {'hate': {'filtered': False, 'severity': 'safe'}, 'jailbreak': {'filtered': False, 'detected': False}, 'self_harm': {'filtered': False, 'severity': 'safe'}, 'sexual': {'filtered': True, 'severity': 'medium'}, 'violence': {'filtered': False, 'severity': 'safe'}}}}}

I could really use your help.

same here
Repository: tatsu-lab/alpaca_eval
Issue: can't match the win rate posted on the leaderboard
Root cause: Documentation Deficiency

Hi team,

I'm observing a significant difference in win rates when using GPT4o Mini with greedy decoding, achieving only 30.33%, compared to the leaderboard sample which reaches a 43% win rate using the same annotator and reference output (GPT Turbo). I'm using the default reference output and standard procedures, but I'm unclear about the exact inference strategy and prompt used for the leaderboard-generated file. Could you please provide details on the inference strategy, the specific prompts, and any additional parameters or settings that might be contributing to the higher win rate in the leaderboard sample? Understanding these factors will help me align my setup more closely with the leaderboard results.

Thank you for your assistance!
Repository: tatsu-lab/alpaca_eval
Issue: alpaca eval config file
Root cause: Documentation Deficiency

What is the difference between the following two config evaluations? My openai url model cannot use logprob_parser, can I use the second one to evaluate the performance of my model's output on alpaca eval2？The max_tokens in the second config is set to 50. Does this mean a 50x difference in evaluation cost?

gpt4_turbo_logprob:
  prompt_template: "gpt4_turbo_clf/basic_clf_prompt.txt"
  fn_completions: "openai_completions"
  completions_kwargs:
    model_name: "gpt-4-1106-preview"
    max_tokens: 1
    temperature: 1 # temperature should be applied for sampling, so that should make no effect.
    logprobs: true
    top_logprobs: 5
  fn_completion_parser: "logprob_parser"
  completion_parser_kwargs:
    numerator_token: "A"
    denominator_tokens: [ "A", "B" ]
    is_binarize: false
  completion_key: "completions_all"
  batch_size: 1


gpt4_turbo:
  prompt_template: "chatgpt/basic_prompt.txt"
  fn_completions: "openai_completions"
  completions_kwargs:
    model_name: "gpt-4-1106-preview"
    max_tokens: 50
    temperature: 0
  completion_parser_kwargs:
    outputs_to_match:
      1: '(?:^|\n) ?Output \(a\)'
      2: '(?:^|\n) ?Output \(b\)'
  batch_size: 1
Repository: tatsu-lab/alpaca_eval
Issue: TypeError: 'NoneType' object is not iterable
Root cause: Validation Gap

```
Traceback (most recent call last):
  File "/home/miniconda3/envs/vllm/bin/alpaca_eval", line 8, in <module>
    sys.exit(main())
  File "/home/miniconda3/envs/vllm/lib/python3.10/site-packages/alpaca_eval/main.py", line 608, in main
    fire.Fire(ALL_FUNCTIONS)
  File "/home/miniconda3/envs/vllm/lib/python3.10/site-packages/fire/core.py", line 135, in Fire
    component_trace = _Fire(component, args, parsed_flag_args, context, name)
  File "/home/miniconda3/envs/vllm/lib/python3.10/site-packages/fire/core.py", line 468, in _Fire
    component, remaining_args = _CallAndUpdateTrace(
  File "/home/miniconda3/envs/vllm/lib/python3.10/site-packages/fire/core.py", line 684, in _CallAndUpdateTrace
    component = fn(*varargs, **kwargs)
  File "/home/miniconda3/envs/vllm/lib/python3.10/site-packages/alpaca_eval/main.py", line 155, in evaluate
    annotations = annotator.annotate_head2head(
  File "/home/miniconda3/envs/vllm/lib/python3.10/site-packages/alpaca_eval/annotators/pairwise_evaluator.py", line 237, in annotate_head2head
    out = self.__call__(df_to_annotate, **decoding_kwargs)
  File "/home/miniconda3/envs/vllm/lib/python3.10/site-packages/alpaca_eval/annotators/base.py", line 218, in __call__
    df_annotated = self._annotate(curr_df_to_annotate, **decoding_kwargs)
  File "/home/miniconda3/envs/vllm/lib/python3.10/site-packages/alpaca_eval/annotators/base.py", line 336, in _annotate
    curr_annotated = self.annotators[annotator](
  File "/home/miniconda3/envs/vllm/lib/python3.10/site-packages/alpaca_eval/annotators/base.py", line 718, in __call__
    completions = self.fn_completions(prompts=prompts, **self.completions_kwargs, **decoding_kwargs)
  File "/home/miniconda3/envs/vllm/lib/python3.10/site-packages/alpaca_eval/decoders/openai.py", line 153, in openai_completions
    completions = list(
  File "/home/miniconda3/envs/vllm/lib/python3.10/site-packages/tqdm/std.py", line 1181, in __iter__
    for obj in iterable:
  File "/home/miniconda3/envs/vllm/lib/python3.10/concurrent/futures/_base.py", line 621, in result_iterator
    yield _result_or_cancel(fs.pop())
  File "/home/miniconda3/envs/vllm/lib/python3.10/concurrent/futures/_base.py", line 319, in _result_or_cancel
    return fut.result(timeout)
  File "/home/miniconda3/envs/vllm/lib/python3.10/concurrent/futures/_base.py", line 458, in result
    return self.__get_result()
  File "/home/miniconda3/envs/vllm/lib/python3.10/concurrent/futures/_base.py", line 403, in __get_result
    raise self._exception
  File "/home/miniconda3/envs/vllm/lib/python3.10/concurrent/futures/thread.py", line 58, in run
    result = self.fn(*self.args, **self.kwargs)
  File "/home/miniconda3/envs/vllm/lib/python3.10/site-packages/alpaca_eval/decoders/openai.py", line 235, in _openai_completion_helper
    for i, choice in enumerate(choices):
TypeError: 'NoneType' object is not iterable
```

I have tried to print the choices. Most of the time, it works well but is stuck in some requests. Can you provide any suggestions for solving it?
Repository: tatsu-lab/alpaca_eval
Issue: Question about best_of when use_beam_search=True
Root cause: Documentation Deficiency

When `use_beam_search=True`, I noticed that the `best_of` parameter is not specified in the model configs. However, according to [the vLLM source code (0.5.0)](https://github.com/vllm-project/vllm/blob/8f89d72090da70895d77d32248ea8504f7daba50/vllm/sampling_params.py#L252), `best_of` should be set when using beam search.

Questions:
1. Does alpaca_eval have a default setting for the `best_of` parameter?
2. Is there a default value for `n` when using beam search?
Repository: tatsu-lab/alpaca_eval
Issue: openai.BadRequestError
Root cause: Validation Gap

openai.BadRequestError: Error code: 400 - {'error': {'message': "'' is not one of ['system', 'assistant', 'user', 'function'] - 'messages.2.role' (request id: 202501241541378273314282HgWcVi0) (request id: 20250124154137826274406FGEN1VWG) (request id: )", 'type': 'invalid_request_error'}}

?????
Repository: tatsu-lab/alpaca_eval
Issue: when model output is like deepseek-r1 ,wondering if it is better to omit content inside <think>
Root cause: Documentation Deficiency

When the model output contains content inside <think> tags, I'm not sure whether it's better to omit this content or keep it.

Here's why I'm asking. In the future, we'll need ways to measure the performance of models similar to DeepSeek in general aspects (have rather long thinking, this brings length bias maybe but people read the thinking process when waiting...).

Curious to know your thoughts on this. How can we better measure this kind of model output?

Looking forward to your insights. Thanks!
Repository: tatsu-lab/alpaca_eval
Issue: Why is the output “[\n {‘model’: ‘model_1’, ‘rank’: 1},\n {‘model’: ‘model_2’, ‘rank’: 2}\n]” and the final result is “preference”: 2.0, when these two do not correspond? What is the reason for this?
Root cause: Algorithmic Error

![Image](https://github.com/user-attachments/assets/e9de151a-68f7-41c2-9e2d-a60358d4dc5e)
Repository: tatsu-lab/alpaca_eval
Issue: When fn_completions is set to huggingface_local_completions, the model will be reloaded for each chunk causing OOM error
Root cause: Algorithmic Error

This can lead to OOM error and inefficiencies as reloading model checkpoints takes a significant amount of time. Currently a quick workaround is increasing the chunk size to the dataset size, but I wanted to raise this issue in case anyone else also faces this.
Repository: tatsu-lab/alpaca_eval
Issue: What models are used when GPT-4-1106-preview is inaccessible?
Root cause: Documentation Deficiency

Hi everyone,

I’m new to NLP and currently reviewing papers like SimPO that use AlpacaEval2 for evaluation. I have two questions:

1. Is GPT-4-1106-preview the default judge model in AlpacaEval2?
 Many recent papers (e.g., SimPO) seem to rely on GPT-4 for evaluation. Is it specifically the gpt-4-1106-preview version, or another variant?

2. If GPT-4-1106-preview is unavailable, what are the alternatives?
For fairness and reproducibility, what models do researchers typically use instead? 

Would appreciate any insights or references to papers addressing this! Thanks!
Repository: tatsu-lab/alpaca_eval
Issue: Cannot get length controlled win rate
Root cause: Validation Gap

Hi,

I have installed alpaca_eval-0.6.6 patsy-1.0.1 using `pip install git+https://github.com/tatsu-lab/alpaca_eval`.
However, when I ran the following code, I got nan for length_controlled_winrate.

```
ANNOTATORS_CONFIG = "weighted_alpaca_eval_gpt4_turbo"

df_leaderboard, annotations = evaluate(  # type: ignore
    model_outputs=response_df,
    annotators_config=ANNOTATORS_CONFIG,
    is_return_instead_of_print=True,
    output_path="pneai_base",
).

```
What is the correct way to get length_controlled_winrate?

Thanks.
Repository: tatsu-lab/alpaca_eval
Issue: Error when using DeepSeek as a judge in alpaca_eval
Root cause: Validation Gap

I can't use chatgpt api. I tried to use DeepSeek as a judge with alpaca_eval. Since the library does not provide built-in support, I followed the existing code patterns and wrote my own evaluators_configs. With this setup, I was able to successfully run the evaluation and generate the expected JSON output.

However, when I proceed to the metrics computation step, I always encounter an error. The evaluation itself runs fine, but calculating the metrics fails.

Would it be possible to add support for DeepSeek as a judge？What should I do?

error:
```
  File "D:\project\Py\LLM-eval\.venv\Lib\site-packages\alpaca_eval\metrics\glm_winrate.py", line 89, in get_length_controlled_winrate
    assert len(df["generator_2"].unique()) == 1
```
but no generator_2 found

example in json file :
<img width="420" height="267" alt="Image" src="https://github.com/user-attachments/assets/10b683d2-3c40-4ea4-a4fb-f690bbe1b00e" />
Repository: tatsu-lab/alpaca_eval
Issue: Falcon support for generation
Root cause: External Dependency Breakage

Running
```
alpaca_eval evaluate_from_model --model_configs 'falcon-7b-instruct'
```
Gives the following warning
> ```The model 'RWForCausalLM' is not supported for text-generation. Supported models are ['BartForCausalLM', 'BertLMHeadModel', 'BertGenerationDecoder', 'BigBirdForCausalLM', 'BigBirdPegasusForCausalLM', 'BioGptForCausalLM', 'BlenderbotForCausalLM', 'BlenderbotSmallForCausalLM', 'BloomForCausalLM', 'CamembertForCausalLM', 'CodeGenForCausalLM', 'CpmAntForCausalLM', 'CTRLLMHeadModel', 'Data2VecTextForCausalLM', 'ElectraForCausalLM', 'ErnieForCausalLM', 'GitForCausalLM', 'GPT2LMHeadModel', 'GPT2LMHeadModel', 'GPTBigCodeForCausalLM', 'GPTNeoForCausalLM', 'GPTNeoXForCausalLM', 'GPTNeoXJapaneseForCausalLM', 'GPTJForCausalLM', 'LlamaForCausalLM', 'MarianForCausalLM', 'MBartForCausalLM', 'MegaForCausalLM', 'MegatronBertForCausalLM', 'MvpForCausalLM', 'OpenLlamaForCausalLM', 'OpenAIGPTLMHeadModel', 'OPTForCausalLM', 'PegasusForCausalLM', 'PLBartForCausalLM', 'ProphetNetForCausalLM', 'QDQBertLMHeadModel', 'ReformerModelWithLMHead', 'RemBertForCausalLM', 'RobertaForCausalLM', 'RobertaPreLayerNormForCausalLM', 'RoCBertForCausalLM', 'RoFormerForCausalLM', 'RwkvForCausalLM', 'Speech2Text2ForCausalLM', 'TransfoXLLMHeadModel', 'TrOCRForCausalLM', 'XGLMForCausalLM', 'XLMWithLMHeadModel', 'XLMProphetNetForCausalLM', 'XLMRobertaForCausalLM', 'XLMRobertaXLForCausalLM', 'XLNetLMHeadModel', 'XmodForCausalLM'].```

It then proceeds with generation. We need to investigate if this is a bug or not. We should probably just rewrite the inference code to not use the HF generation pipeline and roll our own loop.

that's a known warning that has no impact on generations (see https://huggingface.co/tiiuae/falcon-40b/discussions/8 https://github.com/huggingface/transformers/issues/24598 ), and the way we generate for falcon is actually the one recommended by the authors on HF. A PR was merged to avoid this warning (https://github.com/huggingface/transformers/pull/24603 ) so it will disappear once transformers is updated on pypi

Besides if really necessary I would rather not reinvent the wheel for inference and use what HF has. This decreases maintenance on our side (e.g. if libraries or models change) , makes the code more readable, and ensures that we have any new feature that HF incorporates. If someone has a different inference pipeline, they can always use theirs and then use `evaluate` instead of `evaluate_from_model`
Repository: tatsu-lab/alpaca_eval
Issue: Unexpected keyword argument "leaderboard_mode" in print_leaderboard() when running alpaca_eval make_leaderboard
Root cause: Interface Contract Mismatch

When I run `alpaca_eval make_leaderboard`, setting `--is_return_instead_of_print` to `false` results in the following error:

```
TypeError: print_leaderboard() got an unexpected keyword argument 'leaderboard_mode'
```

In `main.py`, `utils.print_leaderboard` is called with the argument `leaderboard_mode=None`.
However, in my environment this appears to be an unexpected keyword argument for `print_leaderboard()`.

I suspect this is a naming mismatch: the correct keyword might be `leaderboard_mode_or_models` (or the function signature may have changed), so `make_leaderboard` is passing an outdated keyword name.
Repository: tatsu-lab/alpaca_eval
Issue: [style] fix ill-formatted logging message

Some of the log messages are single multi-line strings (for example [this](https://github.com/tatsu-lab/alpaca_eval/blob/f8e16ccbe9463894a99b0f3939e92a0b955e0bef/src/alpaca_eval/annotators/pairwise_evaluator.py#L126)). These multiline strings don't display nicely on console due to implicit tabs. Can reformat to use newlines.
Repository: tensorflow/model-analysis
Issue: Adding new metric spec causing OOM
Root cause: Resource Mishandling

**System information** 
- Have I specified the code to reproduce the issue
(Yes/No): yes
- Environment in which the code is executed (e.g., Local
(Linux/MacOS/Windows), Interactive Notebook, Google Cloud, etc): ubuntu 18.04
- TensorFlow
version (you are using): - TFX Version: 0.26.1- Python version:3.6.7 tensorflow version:2.3.2

**Describe the current behavior**
when do as the tutorial, I was adding a extra tfma.EvalConfig to evaluator as
```
eval_config = tfma.EvalConfig(
		model_specs=[
			# This assumes a serving model with signature 'serving_default'. If
			# using estimator based EvalSavedModel, add signature_name: 'eval' and
			# remove the label_key.
			tfma.ModelSpec(label_key='Label',
			               model_type=constants.TF_GENERIC
			               )
		],
		metrics_specs=[
			tfma.MetricsSpec(
				# The metrics added here are in addition to those saved with the
				# model (assuming either a keras model or EvalSavedModel is used).
				# Any metrics added into the saved model (for example using
				# model.compile(..., metrics=[...]), etc) will be computed
				# automatically.
				metrics=[
					tfma.MetricConfig(class_name='ExampleCount'),
					tfma.MetricConfig(
						class_name='BinaryAccuracy',
						threshold=tfma.MetricThreshold(
							value_threshold=tfma.GenericValueThreshold(
								lower_bound={'value': 0.5}),
							change_threshold=tfma.GenericChangeThreshold(
								direction=tfma.MetricDirection.HIGHER_IS_BETTER,
								absolute={'value': -1e-10})))
				]
			)
		],
		slicing_specs=[
			# An empty slice spec means the overall slice, i.e. the whole dataset.
			tfma.SlicingSpec(),
			# Data can be sliced along a feature column. In this case, data is
			# sliced along feature column trip_start_hour.
			# tfma.SlicingSpec(feature_keys=['trip_start_hour'])
		])
```
calls the function in tfma/metrics/metrics_spec.py: 477 _keys_and_metrics_from_specs(metrics_specs)

this functions will call the from_config() in tensorflow/python/keras/engine/base_layer.py :697

looks like this call will form a new layer. and this operation will take all the gpu memory and causeing OOM for the following running.

**Describe the expected behavior**
Well,  another OOM issue， looks like tf need a emergency surgery concerning the memory managment

**Standalone code to reproduce the issue** Providing a bare minimum test case or


**Name of your Organization (Optional)**

**Other info / logs** Include any logs or source code that would be helpful to
just normal OOM error
Repository: tensorflow/model-analysis
Issue: TFMA UI widgets fail to render in nbconvert HTML conversion
Root cause: Unimplemented Feature Gap

### System information

-   **Have I written custom code (as opposed to using a stock example script
    provided in TensorFlow Model Analysis)**: N/A
-   **OS Platform and Distribution (e.g., Linux Ubuntu 16.04)**: CentOS 7
-   **TensorFlow Model Analysis installed from (source or binary)**: 0.27.0
-   **TensorFlow Model Analysis version (use command below)**: 0.27.0
-   **Python version**: 3.7
-   **Jupyter Notebook version**: JupyterLab 2.2.9
-   **Exact command to reproduce**: `nbconvert --to html`

### Describe the problem

Currently, TFDV, Tensorboard and other TFX UI elements that are rendered inline in a Jupyter notebook convert and are usable interactively in an HTML export/render of the notebook. This is highly useful when e.g. publishing a static HTML snapshot of a notebook for sharing with others.

However, the TFMA output widget seems to render as a blank output in this mode. This presents a UX gap in the E2E sharing of TFX pipelines inside Jupyter notebooks.

It is potentially related to #112 (which supercedes #56).

### Source code / logs

```
eval_result = tfma.load_eval_result(eval_result_uri)
tfma.view.render_slicing_metrics(eval_result)
```


It seems like the rendered view (its html) is not saved in the notebook. The output of the corresponding cell is an empty array in the notebook file.
Repository: tensorflow/model-analysis
Issue: Model analysis Tensorflow requirement issue
Root cause: Validation Gap

The requirment for Tensorflow in model analysis at:
https://github.com/tensorflow/model-analysis/blob/67a4d9601b6e735307480c53a1227c691efc6d20/setup.py#L303
is incorrect. A coma is missing before "!=2.3.*" as it is at TFX install requirments:
https://github.com/tensorflow/tfx/blob/21e5af7f9543e108328c1a83b9132acda4875d93/tfx/dependencies.py#L88
This missing come reinstalls an already installed version of Tensorflow 2.4.0 and installs a 2.4.1 version of it

@akewan , thanks for reporting this bug.A PR has been raised to address it.Thanks. 

This is been addressed in 
https://github.com/tensorflow/model-analysis/commit/19d49dc274c38201b6cb3ea9d6c7b9c5f006a244
Repository: tensorflow/model-analysis
Issue: More elaborate documentation for TFMA
Root cause: Documentation Deficiency

### System information

-   **Have I written custom code (as opposed to using a stock example script
    provided in TensorFlow Model Analysis)**: No
-   **OS Platform and Distribution (e.g., Linux Ubuntu 16.04)**: Linux-4.9.0-4-amd64-x86_64-with-Ubuntu-18.04-bionic
-   **TensorFlow Model Analysis installed from (source or binary)**: source
-   **TensorFlow Model Analysis version (use command below)**: 0.29.0
-   **Python version**: 3.6.9
-   **Jupyter Notebook version**: 6.2.0

### Describe the problem
I have been trying to apply TensorFlow Model Analysis to a model of mine in which the dataset exists of images. I have found however that the documentation is (in my opinion) quite lacking in quite a few parts of tfma.x
I have also found that most example notebooks use text data, I started by following along this example [here](https://www.tensorflow.org/responsible_ai/model_remediation/min_diff/tutorials/min_diff_keras). But the problem is that my image data doesn't get decoded before being evaluated, resulting in problems. Are there different tools I should be using when working with images when I want to evaluate my models to prevent this from happening?

So in general I would like a few more example notebooks using tfma and evaluating model fairness on an image dataset and would like to see the tfma documentation be expanded since this is lacking in quite a few parts.
Repository: tensorflow/model-analysis
Issue: AttributeError: partially initialized module error, test.py
Root cause: Documentation Deficiency

Following instructions from the [Getting Started with TFMA](https://www.tensorflow.org/tfx/tutorials/model_analysis/tfma_basic#load_the_files) I stumbled on an error.
<details>
<summary>Error</summary>
<br>
<code>
AttributeError: partially initialized module 'apache_beam.coders.coders' has no attribute 'VarIntCoder' (most likely due to a circular import)
</code>
</details>

I've created file 'test.py' and tried the example code:
<details>
<summary>example code</summary>
<br>
<code>
import sys 
import tensorflow as tf
print('TF version: {}'.format(tf.__version__))
import apache_beam as beam
print('Beam version: {}'.format(beam.__version__))
import tensorflow_model_analysis as tfma
print('TFMA version: {}'.format(tfma.__version__))
</code>
</details>

According to this [answer](https://stackoverflow.com/a/58586917) from Stackoverflow the problem is in the python file name 'test.py' colliding with beam and TFMA modules.

<details>
<summary>after name change/modules version</summary>
<br>
<code>
TF version: 2.5.0
Beam version: 2.30.0
TFMA version: 0.31.0
</code>
</details>

This looks like a beam issue. I'm not sure what we can do in TFMA.

@Shandelier , closing this as there has been no activity on this issue for sometime.Please feel free to reopen based on the comment by @mdreves .Thanks.
Repository: tensorflow/model-analysis
Issue: Incorrect Documentation for MetricConfig
Root cause: Documentation Deficiency

### Describe the problem
It seems like the `config` does not rely on the `__init__` method; and relies on the `get_config` and `from_config` method of the (custom) metrics class.

This distinction is important as TFX's `Evaluator` serialize and deserialize the metric and `__init__` is not used during the final metrics evaluation. Not sure if the issue should be raised with TFX or TFMA.

https://github.com/tensorflow/model-analysis/blob/4ea7d16125d26aa469dc2c5199bbb5f588586fef/tensorflow_model_analysis/proto/config.proto#L407-L414

**Link to TFX Issue**: https://github.com/tensorflow/tfx/issues/4022