zhiminy/EvalEng huggingface dataset:
Repository: prometheus-eval/prometheus-eval
Issue: Best way to reuse the same model for direct assessment and pairwise ranking
Root cause: Documentation Deficiency

Hello! Followed up on my previous issue, I am now able to load the model on A10G in `bfloat16` format. 

However, after I ran the judge snippet for direct assessment, I was not able to load another judge for pairwise because of OOM issues. 

Do you have suggestions on how to either
1) Clean up the memory/ garbage collection before importing for pairwise
2) Are there tricks to reuse the same model weights or the judge object in memory but for pairwise evaluation?

Thank you 

Hi @vionwinnie ! Thanks for the good question! Actually you don't have to load a model again because our model is trained to do both task inherently. But to share some tricks to avoid CUDA OOM errors:

- Decrease `max_model_len`
- Decrease `gpu_memory_utilization`
- Use quantized version of Prometheus (Not supported in `prometheus-eval` library yet)

Because prometheus-eval library is built on top of `vllm`, I definitely recommend you to check the parameters of the [LLM class](https://docs.vllm.ai/en/latest/offline_inference/llm.html).
Repository: prometheus-eval/prometheus-eval
Issue: Use of customized score/eval rubrics
Root cause: Documentation Deficiency

Thank you so much for this great and interesting work!
I have a question regarding the use of rubrics. Have you tested how robust model is towards using customized evaluation rubrics?

Hi @xinyazha92! Thank you for your interest in our work :)

This is a great question! We are conducting experiment about this, and the results are currently not in our Prometheus 2 paper yet. 

Are you curious about using Prometheus 2 as an evaluator for tasks like Math, Coding, or some kind of other fields not mentioned in the paper?

Thank you for your reply. By customization, my question is more about using customized rating rubrics/criteria that are not included in training data such as the following:

###Evaluation rubrics:
    1- Very bad: Given the context, the Assistant's response is either completely irrelevant and does not follow Human's instructions at all or it contains low-quality response such repetitive contents. 
    2- Bad: The response is somewhat relevant or only partially following instructions, but the contents are not informative or helpful.
    3- Good: The response is relevant and helpful. Instructions are followed. However, the fluency or length of the response could be better.
    4- Very Good: The response is very helpful and strictly follows instructions. The contents are very detailed and informative.



Hi @scottsuk0306! I'm curious on using Prometheus 2 as an evaluator for coding tasks, with rubrics like: code smels, code bugs, technical debt, etc. Do you think it would work well?

@xinyazha92 I think your rubric looks fine! I don't have the numbers but I've seen numerous community examples with customized score rubric.

Hey @nfrnunes, I recommend you to tryout [Prometheus 2 BGB](https://huggingface.co/prometheus-eval/prometheus-bgb-8x7b-v2.0) where we trained it on various domains including reasoning tasks (ex. math and coding)!
Repository: prometheus-eval/prometheus-eval
Issue: Which directory is the relevant code for weight merging located in?

Hello, this work is great. But I can not find the relevant code for weight merging 

Hi, @ychyhy! The instructions for merging is [here](https://github.com/prometheus-eval/prometheus-eval/tree/main/train/recipes/prometheus-v2.0) where we use [Lazy Mergekit notebook](https://colab.research.google.com/drive/1obulZ1ROXHjYLn6PPZJwRR6GzgQogxxb?usp=sharing) and [safetensors-merge-supermario](https://github.com/martyn/safetensors-merge-supermario). Let me know if you have further questions about it!
Repository: prometheus-eval/prometheus-eval
Issue: Use prometheus LLM or other os LLMs through API like Ollama
Root cause: Unimplemented Feature Gap

### Discussed in https://github.com/prometheus-eval/prometheus-eval/discussions/29

<div type='discussions-op-text'>

<sup>Originally posted by **arthurGrigo** May 20, 2024</sup>
It would be great if one could deploy the judge LLM with a tool like Ollama or use GPT-4 via its API to allow a comparison. This would also allow to remove the LLM dependencies from the lib and allow a simpler installation. I struggle to install prometheus-eval on windows using anaconda.</div>
Repository: prometheus-eval/prometheus-eval
Issue: Feature Request: Release quantized version of Prometheus models
Root cause: Unimplemented Feature Gap

- Support quantization to enable working with Promethues 2 in google colab
- Good reference: [AutoGPTQ](https://github.com/AutoGPTQ/AutoGPTQ), [AutoGPTQ Notebook](https://colab.research.google.com/drive/1lSvVDaRgqQp_mWK_jC9gydz6_-y6Aq4A?usp=sharing), [llama.cpp](https://github.com/ggerganov/llama.cpp)
- Release the quantized model in Huggingface and Ollama

Hi, I'm highly interested in this too!

Hey @ilyalasy, thank you for the interest! Are you willing to help for this? I have very little background in quantization..

There is this huggingface [space](https://huggingface.co/spaces/ggml-org/gguf-my-repo), which supposed to be easy way to quantize any model to gguf. I have never used it, but seems very straightforward, so you should give it a try.
Repository: prometheus-eval/prometheus-eval
Issue: small bug in relative_grade judge 
Root cause: Validation Gap

In PrometheusEval.relative_grade, _check_inputs takes in `responses` instead of `responses_A` or `resposnes_B`, leading to a "Referenced before assignment..." error:

```
    def relative_grade(
        self,
        *,
        instructions: List[str],
        responses_A: List[str],
        responses_B: List[str],
        rubric: List[str] | str,
        reference_answers: List[str] = None,
        params: Dict[str, Any] = {},
    ) -> Tuple[List[str], List[int]]:
        """
        Grades a batch of responses relatively based on the provided instructions and paired responses.

        :param instructions: List of instructions for each paired responses.
        :param responses_A: List of first responses in each pair.
        :param responses_B: List of second responses in each pair.
        :param params: Additional parameters for the model completion requests. Refer to the vllm SamplingParmas class.
        :return: A tuple containing lists of feedbacks and scores.
        """

        instructions, responses, rubric, reference_answers = self._check_inputs(
            instructions, responses, rubric, reference_answers
        )
       ...
```

Thanks for catching this @lithafnium, just fixed the error and released a new version :)
Repository: prometheus-eval/prometheus-eval
Issue: What are the rubricks for evaluation?
Root cause: Documentation Deficiency

It seems like the rubric can be important for getting the scores on the benchmarks shown in the paper. However, I can't find them anywhere in the repo or paper. It would be nice to know some of them :D

Hi @IsThatYou, you can refer to the predefined rubrics [here](https://github.com/prometheus-eval/prometheus-eval/blob/main/libs/prometheus-eval/prometheus_eval/prompts.py).

For replicating the scores specifically in the Prometheus 2 papers, we have [data loader](https://github.com/prometheus-eval/prometheus-eval/blob/main/eval/benchmark/data_loader.py) class for loading the relevant benchmark with scoring rubrics, but it is not documented yet.
Repository: prometheus-eval/prometheus-eval
Issue: Error trying to load with VLLM
Root cause: Documentation Deficiency

I'm trying to run/load prometheus on Amazon Sagemaker Stuidio notebooks but keep running into errors.

If I load it using VLLM
`model = VLLM(model="prometheus-eval/prometheus-7b-v2.0")`
`ValueError: The model's max seq len (32768) is larger than the maximum number of tokens that can be stored in KV cache (18672). Try increasing `gpu_memory_utilization` or decreasing `max_model_len` when initializing the engine.`

From what I found out, the reason is due to Mistral having 32k context length which is longer than the KV cache can handle. Do you suggest which value of gpu_memory_utilization or max_model_len to try out?

If loading with OpenAI endpoint:
`model = LiteLLM('openai/prometheus-eval/prometheus-7b-v2.0') # VLLM endpoint`
`litellm.exceptions.BadRequestError: litellm.BadRequestError: BadRequestError: OpenAIException - Error code: 400 - {'error': {'message': 'invalid model ID', 'type': 'invalid_request_error', 'param': None, 'code': None}}`

...and HF endpoint:
`model = LiteLLM('huggingface/prometheus-eval/prometheus-7b-v2.0') # Huggingface TGI`
`litellm.exceptions.APIError: litellm.APIError: HuggingfaceException - response is not in expected format - [{'error': 'Rate limit reached. Please log in or use a HF access token'}]`

Thanks.


I figured it out myself by setting VLLM params to:
`model = VLLM(model="prometheus-eval/prometheus-7b-v2.0",
                 max_model_len=23000,
                 gpu_memory_utilization=0.9)`
Repository: prometheus-eval/prometheus-eval
Issue: How to load "prometheus-8x7b-v2.0"
Root cause: Environment Incompatibility

I have tried for gpu nums of 2,4,8 on A100 with below code
```
from prometheus_eval.vllm import VLLM
model_path = "prometheus-eval/prometheus-8x7b-v2.0"
model = VLLM(model=model_path, tensor_parallel_size=8)
```
but can not load model successfully, 
my environment are as belows:
```
python=3.10.14
vllm=0.5.1
torch=2.3.0
transformers=4.43.3
```
Could you provide your environment info if this MoE model can be loaded by vllm?

Hi @bittersweet1999, we used the following parameters throught the experiments (using 4 A100 gpus):

```python
from prometheus_eval.vllm import VLLM
model_path = "prometheus-eval/prometheus-8x7b-v2.0"
model = VLLM(model=model_path, max_model_len=8192, gpu_memory_utilization=0.9, tensor_parallel_size=4)
```

Let me know if you have further issues.

Thanks for your response, I use the code above to load the model, however the same error as below
![图片](https://github.com/user-attachments/assets/ec54f6e2-d467-42e8-8532-87eb772dc18e)
so I thinks this may caused by environment, can you provide me your version of vllm and transformers?

oh this should be a compile error, I will check my environment first, thank you
Update GCC version solved this problem
Repository: prometheus-eval/prometheus-eval
Issue: Finetune for structured output

Most open-source libraries and projects using LLMs now prefer JSON-structured output to facilitate easier parsing and validation. This preference became a challenge when we tried to add support for Prometheus_eval models in [ragas](https://github.com/explodinggradients/ragas). Moving forward, we strongly recommend fine-tuning your models to produce JSON output to enhance usability in future versions.

I am currently struggling to parse JSON output from the model.

@shahules786 Thanks for the suggestion! 

Would you mind sharing what key-values you might need in the JSON-structured output? For instance, perhaps the "feedback" and "judgment"?

@SeungoneKim In your case I think the keys would be score of type int and feedback of type string. It seems pretty straightforward. This could enable a lot more people to use these models with trust - parsing errors are always a problem while evaluating or in general building LLM software without type annotations and formating.
Repository: prometheus-eval/prometheus-eval
Issue: Support of VLM-as-a-Judge
Root cause: Unimplemented Feature Gap

We're planning to add VLM-as-a-Judge functionality to prometheus-eval.

* References: [https://arxiv.org/abs/2401.06591](https://arxiv.org/abs/2401.06591)

The high-level idea is that using gpt-4v or gpt-4o, the judge VLM would receive an image as input in addition to the components that is currently passed to judge LMs.

The judge VLM would then be able to evaluate image-grounded text responses generated by different VLMs (i.e., VLMs evaluating VLMs).
Repository: prometheus-eval/prometheus-eval
Issue: prometheus-eval lib is always returning score 5
Root cause: Algorithmic Error

Prometheus model is scoring correctly between 1 and 5, but the lib is always returning score 5. The problem is the regex is matching the prompt template ([here for instance](https://github.com/prometheus-eval/prometheus-eval/blob/517c188222e5553fb14d799c3b5052cf2a441f57/libs/prometheus-eval/prometheus_eval/prompts.py#L18)), instead of Prometheus' generation.

Since Prometheus seems to reliably follow instruction to add score at the end of the response, I'd propose matching regex from end of string and longer term explore structured response.

@rafaelliu Thanks for catching this :) I also strongly agree about supporting structured responses to ensure robustness of llm-as-a-judge.
Repository: prometheus-eval/prometheus-eval
Issue: llama3 evaluator

curious if you're also gonna train an 8b llama3 evaluator. also wonder if you can implement 4 bit eval, as hf transformers supports, so we can use mixtral evaluator at home. 

Thanks for your interest in what we're working on! We have plans to train an 8B LLaMA 3 evaluator within 2-3 weeks. Stay tuned!

As for the 4-bit idea, that's a great shout! We're definitely looking into it. But one issue is that our package `prometheus-eval` is based on the `vllm` which doesn't support `bitsandbytes` quantization yet (referring to https://github.com/vllm-project/vllm/issues/4033). 

For quantization methods like GPTQ and AWQ, we will upload a quantized version of Prometheus with them soon 😄 



Sounds great thanks for sharing!
Repository: prometheus-eval/prometheus-eval
Issue: doc link not working for batch predictions

This link in the README leads to a 404 error: https://prometheus-eval.github.io/prometheus-eval/docs/advanced-usage.html#53-handling-large-datasets

Hi @wise-east , thank you for bringing this to our attention! I've removed the link as the documentation is temporarily deprecated. For now, we're focusing on maintaining the README.md for usage instructions.
Repository: prometheus-eval/prometheus-eval
Issue: Plan to update eval benchmark documentation?

Hi prometheus team! 

Thanks for the great work. Wondering when will the README in this section get updated?
https://github.com/prometheus-eval/prometheus-eval/tree/main/eval

Thanks.

Hey @CharlieJCJ, thanks for your interest! The section you referred to is for "meta-evaluation", which is evaluating evaluation models.

You could simply run
```
python3 run_evaluate.py --model_name "prometheus-eval/prometheus-8x7b-v2.0" --eval_data_names ["hhh_alignment_eval", "vicuna_eval", "flask_eval"]
```

If you're finding for just evaluating language models by using evaluator LMs, you could refer to the README page.

Let me know if this doesn't answer your questions!
Repository: prometheus-eval/prometheus-eval
Issue: Hi. Is it support to korean language?

I wonder that is it support to korean language output by llm?

Hello @95jinchul , the model isn't specifically trained to evaluate Korean responses, but it does work to some extent. The trick here is to only convert the {instruction, response} into Korean and leave the {meta prompt, verbal feedback} in English.

Here's a new paper from our group exploring how English-centric judge/reward models perform on multilingual instances: https://arxiv.org/abs/2410.17578

Hope this helps!
Repository: prometheus-eval/prometheus-eval
Issue: Suggestion to remove reference answer from prompt if not provided
Root cause: Documentation Deficiency

Hi, I kindly suggest to add a conditional statement to remove the whole section of "reference answer" from the prompt if it is not provided. Some models including gpt-4o were using the empty reference answer for its scoring.

Hello @limshengxue, thanks for your suggestion!

If you use the `ABSOLUTE_PROMPT_WO_REF` prompt (link below), you could avoid including a reference answer!

https://github.com/prometheus-eval/prometheus-eval/blob/b7a431a553b320e0a7cc49c6c5d3c54b1b840d39/libs/prometheus-eval/prometheus_eval/prompts.py#L27

Thanks!
Repository: prometheus-eval/prometheus-eval
Issue: Why are results of task arithmetic and linear merging different in Prometheus 2.0 ?

Hi @scottsuk0306 @SeungoneKim, thank you very much for your works!

I noticed that results for task arithmetic and linear merging were different in the paper. However, both were fine-tuned from the same base model (i.e. Mistral) and the coefficients also summed to 1, why could the results be different ?
Repository: prometheus-eval/prometheus-eval
Issue: Error: Unable to run with LLM API
Root cause: Documentation Deficiency

### Issue

I tried to run Prometheus with the LLM APIs suggested on the README, but none worked. Also, I am confused about the function of the parameter 'api_base' in the class LiteLLM, yet it does not seem to be the placeholder for my API Key. I would greatly appreciate if you could write a detailed code documentation or Wiki.

### Error Messages
Code 1:
```
model = LiteLLM('openai/prometheus-eval/prometheus-7b-v2.0')
judge = PrometheusEval(model=model, absolute_grade_template=ABSOLUTE_PROMPT_WO_REF)
```

Result 1:
```
litellm.BadRequestError: OpenAIException - Error code: 400 - {'error': {'message': 'invalid model ID', 'type': 'invalid_request_error', 'param': None, 'code': None}}
```

-------------

Code 2:
```
model = LiteLLM("gpt-4o-mini", api_base = settings.OPENAI_API_KEY)
```

Result 2:
```
litellm.APIError: APIError: OpenAIException - Connection error.
```

----------

Code 3:
```
model = LiteLLM('huggingface/prometheus-eval/prometheus-7b-v2.0')
```

Result 3:
```
litellm.APIError: HuggingfaceException - response is not in expected format - [{'error': 'The model prometheus-eval/prometheus-7b-v2.0 is too large to be loaded automatically (14GB > 10GB).'}]
```

---------

Code 4:
```
model = AsyncLiteLLM('gpt-4-turbo', requests_per_minute=100)
```

Result 4:
```
asyncio.run() cannot be called from a running event loop
```

### Dependencies
[tool.poetry.dependencies]
python = "^3.10"
alembic = "^1.11.1"
loguru = "^0.7.0"
fastapi = {extras = ["all"], version = "^0.100.0"}
python-jose = {extras = ["cryptography"], version = "^3.3.0"}
passlib = {extras = ["bcrypt"], version = "^1.7.4"}
sqlalchemy = "^2.0.19"
pydantic = "^2.0.3"
asyncpg = "^0.28.0"
psycopg2-binary = "^2.9.9"
openai = "^1.37.1"
redis = "^5.0.8"
portkey-ai = "^1.7.2"
typing-extensions = "^4.12.2"
aioredis = "^2.0.1"
pytz = "^2024.2"
apscheduler = "^3.10.4"
transformers = "^4.47.0"
tensorflow = "^2.18.0"
tf-keras = "^2.18.0"
torch = "^2.5.1"
cloud-tpu-client = "^0.10"
prometheus-eval = "^0.1.20"
Repository: prometheus-eval/prometheus-eval
Issue: Context window for eval models
Root cause: Documentation Deficiency

Hi guys,

Was wondering how can I determine the context window for the following models:

- `kaist-ai/prometheus-13b-v1.0`
- `prometheus-eval/prometheus-7b-v2.0`

I am interested to know specifically how much could I fit into the model before I exceed its maximum context window.

Is this the place to look for:
```python
from transformers import AutoConfig

model_name = "prometheus-eval/prometheus-7b-v2.0"
config = AutoConfig.from_pretrained(model_name)
print("Context window (max_position_embeddings):", config.max_position_embeddings)
```
> Context window (max_position_embeddings): 32768

Or something around these lines:
```python
from transformers import AutoTokenizer

model_name = "prometheus-eval/prometheus-7b-v2.0"
tokenizer = AutoTokenizer.from_pretrained(model_name)

sample_text = "The quick brown fox jumps over the lazy dog. " * 1000  
words = sample_text.split()
word_count = len(words)
tokens = tokenizer.tokenize(sample_text)
token_count = len(tokens)

print("Sample text word count:", word_count)
print("Sample text token count:", token_count)

words_per_token = word_count / token_count
print("Average words per token:", words_per_token)

max_tokens = 32768
estimated_words = max_tokens * words_per_token
print("Estimated context window in words:", estimated_words)
```
> Sample text word count: 9000
> Sample text token count: 12001
> Average words per token: 0.7499375052078994
> Estimated context window in words: 24573.952170652447

More concretly I am interested to understand the authors decision in this paper: [Assisting in Writing Wikipedia-like Articles From Scratch with Large Language Models](https://arxiv.org/abs/2402.14207) to truncate its articles text before send to prometheus for rubric scores evaluation.

Here the snippet that is relevant:
> ## C.2 LLM Evaluator  
> We use Prometheus14 (Kim et al., 2023), a 13B open-source evaluator LLM that can assess longform text based on customized 1-5 scale rubric, to grade the article from the aspects of Interest level, Coherence and Organization, Relevance and Focus, and Coverage. Table 8 gives our grading rubric. While Prometheus is best used with a score 5 reference answer, we find adding the reference will exceed the context length limit of the model. Since Kim et al. (2023) show Prometheus ratings without reference also correlate well with human preferences, we omit the reference and trim the input article to be within 2000 words by iteratively removing contents from the shortest section to ensure the input can fit into the model’s context window.

Awesome work and thanks for the clarification guys!
Repository: prometheus-eval/prometheus-eval
Issue: Reference Free metrics

Hi! 

First of all congrats for your awesome work! 
I have a question related with reference-free metrics, do you also train the model for this type of metrics or you just use reference-based ones?

Finally, did you test the trained model on this setting?

Thanks!

Hi, @SSamDav! Thank you for the interest in our work.

The training data we used are all with reference answers, so it is encouraged to use Prometheus with giving the reference.

However, for the pairwise ranking (relative grading) benchmarks, since we didn't have reference answer for these benchmarks, the results in our paper for these are reference-free evaluations. I want to note that Prometheus achieves significant improvement compared to the base models in these benchmarks without the reference!
Repository: promptfoo/promptfoo
Issue: [Feature Request]More Results based on Same Value
Root cause: Unimplemented Feature Gap

Hi, good job! It is useful. Is it possible to generate more than one result based on same value? I want to test if a prompt is stable.

Thanks for the suggestion, I'll see if I can build in an option to do this.  How many times do you think you'd run it to decide if it's stable?

Right now, the easiest way to accomplish what you want is to include the prompt multiple times, either in the same file:

```
<prompt>
---
<prompt repeated>
---
...
```

Or by passing it multiple times:

```
promptfoo -p prompt.txt prompt.txt
```


Hi, I will want to test the same prompt more than three times. And your suggestion is useful for me either.

Closing this as the requirement can be met by adding the same prompt several times!