zhiminy/EvalEng huggingface dataset:
Repository: plurai-ai/intellagent
Issue: Database Not Loading Experiments in Experiments Report

I encountered an issue where the Experiments Report page does not display any experiments, even when a valid database is selected. The error message states:

"The database you selected does not contain any experiments. Please select another database."

This occurs even when trying with the saved database in this path"C:\Users\user\Desktop\VS_CODE\Intellagent\examples\airline\output\run_0\datasets\dataset__19_11_2024_11_09_37.pickle" for airlines

![Image](https://github.com/user-attachments/assets/ff086827-4244-49d1-9867-9bf612cee2b1)


Thank you for your time and support!
.

The database is indeed valid, but the message indicates that the experiments were not valid. 
Probably this is due to rate-limit issue. You can check it in the log file:
`results/<resulsts_folder>/experiments/<exp_name>/experiment.log`
See if there are records with:
`ERROR - Error in chain invoke:`
If the error is due to the rate limit, you should reduce the number of workers in the config file. 


Thank you for your response. 
I checked the log file at results\airline\experiments\dataset__13_02_2025_10_35_22__exp_1\experiment.log, but I did not find any errors.
Instead, my log contains the following messages:
2025-02-13 10:49:34,018 - INFO - Start running the simulator

2025-02-13 10:49:34,018 - INFO - Running batch 0...

2025-02-13 10:51:42,495 - INFO - Running batch 1...

2025-02-13 10:52:49,896 - INFO - Finish running the simulator

2025-02-13 10:52:50,817 - INFO - Analyzing the results

2025-02-13 10:53:02,874 - INFO - Finish running results analysis


Since there are no error messages indicating a rate-limit issue, could there be another reason why the experiments are not being recognized?

Please let me know if there are additional debugging steps I should take.

Thank you!

Thanks for checking.
Can you share the full `results\airline` folder so I can debug it?
There is no any personal information in these dump files... 


Sure, this my result folder

[airline.zip](https://github.com/user-attachments/files/18799328/airline.zip)

I put the folder inside `results` folder, and run:
`streamlit run simulator/visualization/Simulator_Visualizer.py`
And it works well (I'm getting the experiments with all the details). 
Observe that in the image you upload the dataset start with `12_02`, however the one that you sent me start from `13_02` so it seems like a different dataset. The best way is to delete all the content of the `results` folder, and then put there only airline folder you provide.

**An important remark**:  The vizualizer is looking for the last dataset in `results` folder. So if your output dir is a different, which is the case in your example: 
`C:\Users\user\Desktop\VS_CODE\Intellagent\examples\airline\output\run_0` 
You need to explicitly write the database location in the textbox. In your case it is:
`C:\Users\user\Desktop\VS_CODE\Intellagent\examples\airline\output\run_0\datasets\dataset__19_11_2024_11_09_37`
Repository: plurai-ai/intellagent
Issue: integrate tool calling agent that outputs python-style function calls
Root cause: Unimplemented Feature Gap

I am putting together a little robotics sandbox example for some testing I am doing (I'll be happy to submit PR when I get it together). I would like to test a tool calling agent (watt-ai/watt-tool-8B huggingface pipeline) that is fine tuned to respond with raw python function calls like this

```[weather_forecast(location="Berlin Germany", days=5)]```

I believe you require tool calls in OpenAI JSON format. If this is indeed the case, can you please tell me where in your code base would be best to adapt the python function call to a JSON tool call format? In the dialog_graph.py module?

Thank you.
 

This is the place that needs to be [adjust](https://github.com/plurai-ai/intellagent/blob/bce45c3eee74738cf4b78063df606a2032eab7fb/simulator/agents_graphs/langgraph_tool.py#L100)

So I think the best way is to [implement](https://github.com/langchain-ai/langchain/discussions/24435#discussioncomment-10344305) the bind_tools for your custom model

It's an interesting use case, so if you need any help with the implementation, we can discuss the details in our discord channel


Thanks, @Eladlev. And thanks for the reminder about the Discord channel.
Repository: plurai-ai/intellagent
Issue: Can the IntellAgent Framework Be Integrated for Evaluating Document Summarization Applications Which Uses Multiple LLM Models to Summarize?
Root cause: Documentation Deficiency

I am exploring the idea of integrating a document summarization application to assist in evaluating LLM (Large Language Model)-based applications with intellagent framework. The main objective is to assess the performance of these applications by leveraging multiple models for summarization and document analysis across various contexts.

I would like to understand if this integration is feasible within the **IntellAgent framework**, or if it is primarily designed for agent-based AI applications.

Your insights on this would be greatly appreciated, as they will be very helpful for moving forward.

Thank you in advance!

Modified ask

IntellAgent's main strength is with multi-turn conversation. 
I think you might want to check my other [repo](https://github.com/Eladlev/AutoPrompt), which should be a better fit for the summarization task evaluation. 
If you have specific questions on the integration (you can also ask in our discord channel)

Okay, thank you for your prompt response. I will go through the mentioned repo and If I have query will connect on discord. Thank you once again...
Repository: plurai-ai/intellagent
Issue: Support for non-English prompts, specifically Chinese language
GitHub labels: enhancement

I'm currently using Chinese prompts for my agent, but it seems the codebase primarily supports English. When I input Chinese prompts, the output shows mixed results - some scenarios in Chinese and others in English.

I would like to know if there's a quick way to switch the built-in prompts to other languages (such as Chinese). Is there any language localization support planned or available?

Any guidance on this would be appreciated. Thank you!

Hi, unfortunately, there is currently no support for multiple languages.
The issue is that the dataset is generated in English, which confuses the user agent and causes it to mix languages.
To fix this, we need to update various system prompts.
It's on our roadmap, and we hope to add this feature soon.

 I'll keep an eye on future updates.Thank you for the quick response and clarification!
Repository: plurai-ai/intellagent
Issue: Support for AWS Bedrock
Root cause: Unimplemented Feature Gap

Hi, is there any plan to support AWS Bedrock and its Converse API?
Thanks for your answer

Hi, since the API is supported by [langchain](https://python.langchain.com/api_reference/aws/chat_models/langchain_aws.chat_models.bedrock_converse.ChatBedrockConverse.html), it can be easily added.

Steps:
1. Add in [get_llm](https://github.com/plurai-ai/intellagent/blob/2680da4545b87717a8dbc869738238022ca84ee4/simulator/utils/llm_utils.py#L230):
```    
if config['type'].lower() == 'bedrock_converse':
        from langchain_aws import ChatBedrockConverse

        return ChatBedrockConverse(
            model=config['name'],
            temperature=temperature,
            model_kwargs=model_kwargs, timeout=timeout
        )
```
2. In the config file, modify the chatbot/InetellAgent system model type to `bedrock_converse` and the model name to your preferred model
Repository: plurai-ai/intellagent
Issue: Can we improve unit testing

Can you provide unit test files for each module? For example, I want to test the functionality of the env.py module.When I am reading the source code and running the functionality of a single module, I always encounter some path and resource configuration issues. If comprehensive unit testing can be provided, the program will be more robust and avoid unpredictable problems.

Definitely, we should add it.
In the short term, if you have a specific request for a few tests (such as configurations), please share the list and we’ll prioritize delivering those first.

Thank you for your reply. 
If I understand correctly, the directories`  ./gents_graphs`,` ./dataset`, and `./dialog` contain the core functionalities of the entire project. So I suggest that the modules under these directories should have unit testing, especially the node output of Langgraph should be as expected.

Wishing you a happy life.
Repository: plurai-ai/intellagent
Issue: design questions while reading the source code

hi，bros,I have several design questions while reading the source code。

In actual implementation, the data in `user_messages` is not read elsewhere except for the first entry `messages = [state["user_messages"][0]] + set_user_message(state).` This data seems useless— so why is it needed?

`    def get_user_node(self):
        def simulated_user_node(state):
            messages = [state["user_messages"][0]] + set_user_message(state)
            # Call the simulated user
            response = self.user.invoke(messages)
            # This response is an AI message - we need to flip this to be a human message
            user_thoughts = state['user_thoughts']
            if self.memory is not None:
                if response['thought'] is not None:
                    self.memory.insert_thought(state['thread_id'], response['thought'])
                    user_thoughts.append(response['thought'])
                self.memory.insert_dialog(state['thread_id'], 'Human', response['response'])

            result_state = {'user_thoughts': user_thoughts, 'critique_feedback': '', 'stop_signal': ''}
            if '###STOP' in response['response']:
                result_state['stop_signal'] = response['response']
            else:
                result_state.update({"chatbot_messages": [HumanMessage(content=response['response'])],
                                     'user_messages': [AIMessage(content=response['response'])]})
            return result_state`

Why is the returned 'user_messages': [AIMessage (content=response ['response'])] packaged as AIMessage ,if the message type in user_messages changes like HumanMessage,Will it affect any logic (except readability)? Why are AIMessage and HumanMessage hard coded in set_user_message()? Is there any special logic here?


In the simulated_user_node node,  'user_messages' are packaged as AIMessages, and chatbot_messages are packaged as HumanMessages,

and in the chat-bot_node node, the user_messages data is packaged as HumanMessage,

Why is it packaged like this and why is there such a distinction made? What are the side effects of swapping packaging for  'user_messages'?

1. Regarding `user_messages`: you’re right — the list is redundant. Only the user’s system prompt is needed.
2. I might be misunderstanding your point, but in the user-agent flow the `"user_message"` refers to the agent’s message, while the chatbot’s response is the external feedback.  
   - When sending historical messages **to the user agent**, the agent’s message should have the `"assistant"` role, and the chatbot’s feedback should have the `"user"` role.  
   - When sending messages **to the chatbot**, it’s the opposite: the user message is `"user"` and the chatbot messages are `"assistant"`.

   This is why the swap function is necessary.
Repository: prometheus-eval/prometheus-eval
Issue: Issue with examples
Root cause: Documentation Deficiency

Hi,

Thanks for this release and your paper. I noticed that there is an issue with the examples that are provided: `reference_answers` (plural) is the name of the argument instead of `reference_answer`. However, even with the correct name and a list, I wasn't able to get a feedback or a score.

Hi @mlabonne, Thank you for reporting this. I just fixed the issue you reported. Can you try again by downloading the new version (0.1.10) of our package, and use the updated example in our README.md?



Thanks it looks like it works now but it took 18 attempts to produce a feedback (it failed the first time). Is there something wrong with the way I use it (low seq length?)? https://colab.research.google.com/drive/1p3BlzQcRRX6ZWeGofyWf24LhJECPQCPc?usp=sharing

Yeah I think increasing max_model_len will solve the issue. We haven't tested prometheus model on max_model_len < 4096. This is because we found that the sum of input length and feedback it produces for the evaluation benchmarks (like HHH alignment, Auto-J, Flask) generally goes over 4096.

Thanks for the clarification!
Repository: prometheus-eval/prometheus-eval
Issue: About the evaluation on direct assessment benchmarks
Root cause: Documentation Deficiency

Hi authors,
Thanks for the great work! A few questions about the evaluation on the direct assessment benchmarks:
1. For MT-bench and FLASK, I saw that there are 3 "gpt4_score" and their corresponding feedbacks. I'm wondering how to calculate the scoring correlations against the scores. Do I need to sample 3 evaluations from the judge model and then compare them one by one?
2. Also for these datasets, I don't find the reference for each prompt. How do you conduct reference-based evaluation?

Would appreciate any clarification on these. Thank you!

Hi, @wangpf3! Thank you for the interest in our work! 

> 1. For MT-bench and FLASK, I saw that there are 3 "gpt4_score" and their corresponding feedbacks. I'm wondering how to calculate the scoring correlations against the scores. Do I need to sample 3 evaluations from the judge model and then compare them one by one?

Yes. For each instances, we first sampled 3 evaluations from the judge model and calculated the mean of three values. Then, we measured correlation with the mean of `gpt4_score`. 

> 2. Also for these datasets, I don't find the reference for each prompt. How do you conduct reference-based evaluation?

Actually, for these datasets, the `instruction` includes the reference answer. You can find the reference by searching `###Reference Answer (Score 5):`

I hope this answers your question! Let me know if you need more clarifications.

For your information, we're also writing the documentation for running these evals. It will be updated in `eval/README.md` in few days.

Got it! Thank you Juyoung for the info!
Repository: prometheus-eval/prometheus-eval
Issue: [Question] Support for including context (RAG)?
Root cause: Unimplemented Feature Gap

Hi, curious if there are any plans to support evaluating context along with question and reference answer?

Hi @austinmw ! I haven't thought about the idea you described, but I think it would be great if prometheus-eval supported this feature!

Just for reference, there were some approaches using Prometheus (version 1) for RAG evaluation. You can check it out here: [link](https://www.llamaindex.ai/blog/llamaindex-rag-evaluation-showdown-with-gpt-4-vs-open-source-prometheus-model-14cdca608277).

May I ask if you are interested in contributing to this feature? If yes, we can set up a new issue for feature implementation and discuss it further. 😊

@austinmw To add on to Juyoung's comments, I think it would be an interesting idea to add additional context to the {instruction}.

It's as if evaluating an LLM's response like it was in an open-book setting (while it could have been a closed-book setting actually).

We would be glad to discuss this in more detail if needed!

@austinmw I experimented with Prometheus 2 for RAG evaluation:
[Blog post](https://haystack.deepset.ai/blog/rag-evaluation-with-prometheus-2) - [Notebook](https://github.com/deepset-ai/haystack-cookbook/blob/main/notebooks/prometheus2_evaluation.ipynb)
Repository: prometheus-eval/prometheus-eval
Issue: Question about multi-turn evaluation

Hi, thank you for good work first.

Is there a template for multi-turn evaluation for relative comparison?

Hi @jackyoung96, thanks for your interest.

We don't have the template for multi-turn evaluation for relative comparison yet 😢 
Would you mind giving some kind of example responses (or multi-turn chat) that you want to evaluate?

Just to add on, you could add the whole interaction to {instruction}, this is how we evaluated on the MT-Bench for absolute grading & MT-Bench Human Judgment for relative grading!

> Just to add on, you could add the whole interaction to {instruction}, this is how we evaluated on the MT-Bench for absolute grading & MT-Bench Human Judgment for relative grading!

Thank you for the answer!!
@scottsuk0306 I just asked a example such as the MT-bench, so @SeungoneKim 's answer is enough for me. Appreciate @SeungoneKim
Repository: prometheus-eval/prometheus-eval
Issue: Speed-up strategies
Root cause: Documentation Deficiency

Hi, 
Thank you for your great work. I would like to look into ways in which I can improve the speed of feedback and score generation for pairwise ranking. Currently, generating scores for 3165 responses requires 3 GPU hours, using `"prometheus-eval/prometheus-7b-v2.0"` and `torch.float16`. While this is commendable, if there are ways to improve the speed of response generation, I would be happy to hear about and implement them.

Hi @adhirajghosh, I’m really excited to hear that you’re open to contributing! 😊

The time you described is actually way more longer than we expected though. When using `vllm`, it took us about 5 minutes on grading 1000 responses on 1 GPU (we tested on both A6000 and A100). I'm curious which code you're using for this.


Oh wow, that is a bit shocking. 
I too am using an [A100](https://developer.nvidia.com/blog/nvidia-ampere-architecture-in-depth/) GPU. The relevant code is taken and adapted from this repository's README:

#### Loading the model
`judge = PrometheusEval(model_id="prometheus-eval/prometheus-7b-v2.0", relative_grade_template=PREFIX_PROMPT, dtype = torch.float16)`

#### Specific instructions and rubrics (this one for image captioning)
`instruction_cap = "The ground truth captions of a provided image are listed here. They are separated by a semi-colon. Please carefully observe the image and come up with a caption for the image."`
`rubric_cap = "Is the answer well correlated with the ground truth captions?"`

#### I retrieve ground truth and model responses separately. Code should still be readable.
`data = {
                        "instruction": instruction_cap,
                        "response_A": response1,
                        "response_B": response2,
                        "reference_answer": f"Ground truth captions: {ground_truth}",
                        "rubric": rubric_cap,
                    }`

`feedback, score = judge.single_relative_grade(**data)`

Any insight is greatly appreciated. Thank you for looking into this!

Thanks for sharing. I just added about [batch grading in our README](https://github.com/prometheus-eval/prometheus-eval?tab=readme-ov-file#batch-grading). Can you check this and change your implementation accordingly? Curious how much speed up you can get! 

Oh and also please set `dtype = 'bfloat16'` :)

This was extremely helpful. There is a bit of variation per batch with different numbers of retries but overall, I am close to the 1000 responses in 5 minutes statistic you initially provided.

Thanks again for your solution. If can help contribute to this work, please do let me know!

Cheers!
Repository: prometheus-eval/prometheus-eval
Issue: feature request: use model from HuggingFace Inference endpoint
Root cause: Unimplemented Feature Gap

It would be very useful to be able to point to an external inference API endpoint (e.g. on HuggingFace), especially when the local computer running the experiment may not have the required resources to run the model.

This is e.g. possible with langchain with `HuggingFaceEndpoint` which can either point to the model id to download the weights, or to a URL to access the model remotely. A similar mechanism for prometheus-eval would be great.

Thanks for the suggestion @jgrivolla. We will definitely work on this before the end of May :)
Repository: prometheus-eval/prometheus-eval
Issue: Can't load the model on T4 Inference GPU
Root cause: Unimplemented Feature Gap

Hello! What is the recommended configuration to load the model to a smaller Inference GPU?
```
+---------------------------------------------------------------------------------------+
| NVIDIA-SMI 535.54.03              Driver Version: 535.54.03    CUDA Version: 12.2     |
|-----------------------------------------+----------------------+----------------------+
| GPU  Name                 Persistence-M | Bus-Id        Disp.A | Volatile Uncorr. ECC |
| Fan  Temp   Perf          Pwr:Usage/Cap |         Memory-Usage | GPU-Util  Compute M. |
|                                         |                      |               MIG M. |
|=========================================+======================+======================|
|   0  Tesla T4                       Off | 00000000:00:1E.0 Off |                    0 |
| N/A   30C    P8               8W /  70W |      2MiB / 15360MiB |      0%      Default |
|                                         |                      |                  N/A |
+-----------------------------------------+----------------------+----------------------+
                                                                                         
+---------------------------------------------------------------------------------------+
| Processes:                                                                            |
|  GPU   GI   CI        PID   Type   Process name                            GPU Memory |
|        ID   ID                                                             Usage      |
|=======================================================================================|
|  No running processes found                                                           |
+---------------------------------------------------------------------------------------+
```

This is the code I used to load the model: 
```
import torch
from prometheus_eval import PrometheusEval
from prometheus_eval.prompts import ABSOLUTE_PROMPT, SCORE_RUBRIC_TEMPLATE

judge = PrometheusEval(model_id="prometheus-eval/prometheus-7b-v2.0", 
                       absolute_grade_template=ABSOLUTE_PROMPT,
                       dtype=torch.float16,
                       )

```
Error message:
```
INFO 05-14 20:01:20 selector.py:32] Using XFormers backend.
INFO 05-14 20:01:21 weight_utils.py:199] Using model weights format ['*.safetensors']
INFO 05-14 20:02:00 model_runner.py:175] Loading model weights took 13.4966 GB
OutOfMemoryError: CUDA out of memory. Tried to allocate 224.00 MiB. GPU 
```
I am testing on A10G as well with bfloat16. But any suggestions re inference setup will be greatly appreciated

Hi @vionwinnie! We actually tried to load the model on T4 GPU (Google colab free instance) but failed even with changing parameters multiple times. I think you might want to consider quantization if you want to load it on small GPUs.

Able to load the 7B model on L4 GPU (24GB RAM)