Repository: plurai-ai/intellagent
Issue: Clarification on Scenario Redundancy, Recursion Limit Failures, and Chatbot System Evaluation in Intellagent Framework
Root cause: Documentation Deficiency

First of all, thank you again for developing and open-sourcing your framework.

In order to better understand and apply it, I ran through the full process in a simple scenario, **which is 'education' in example** and viewed all intermediate outputs in debug mode. However, I encountered the following issues and hope to receive your response. Please correct me if I have misunderstood anything.

1. In the generated events, there is a very high redundancy between the user's initial query and the scenario. I sampled 25 events, but there were only 3 different scenarios, with each scenario having the same initial user query. In other words, under 25 events, only 3 different user queries were tested. Logically, I think this is not what we expect for system evaluation. Diversity in scenarios and user queries is crucial to evaluating a chatbot. 
Additionally, I tested my own agent, whose main task is query filtering and intent classification, and I also observed very high redundancy in the scenarios. 
![Image](https://github.com/user-attachments/assets/faa59192-3f8c-4d27-a433-35e976a751c1)


2. During simulation, the default recursion_limit is set to 25, but a large number (almost all) of dialogs cannot be completed in 25 steps. If the dialog is not completed within 25 steps, these failed dialog details are not visible in the results (it seems that only successful ones, where the user ends the conversation with ###stop###, are saved). If I want to see these failed processes, do I need to integrate with something like LangSmith?
With such a high failure rate, I personally feel this is unreasonable. I'd like to inquire about the best practices for using this framework.
![Image](https://github.com/user-attachments/assets/64c4c298-949d-49d9-96c8-9d00988f5b62)


3. How can the Intellagent framework be used to evaluate our chatbot system? The chatbot system we wish to evaluate has an API that receives user requests and then returns a response, which is the chatbot's reply. The system may have a complex internal network, but my understanding is that Intellagent doesn't care about these internal details.
However, after going through the source code, I found that the chatbot is simulated using a langgraph AgentTool object, i.e., self.chatbot = AgentTools(llm=llm_chat, tools=self.env_tools, tools_schema=self.env_tools_schema), and the environment prompt (system input) is used as the first system message in the dialog. Does this mean that the Intellagent framework is currently evaluating the chatbot system represented by this fictitious AgentTool object? But our goal is to evaluate our own chatbot system. Where did I go wrong in my understanding?


btw, all processes use gpt4o as llm.

1. There was a bug in the agents without a DB, which caused all the scenarios inside the mini-batch to be exactly the same (thanks for informing us about the issue!) I fixed the issue. It's important to note that we are working on two features that will improve significantly the events diversity: 
- Adding 'personality', and other random context enrichment
- Adding the possibility of adding user data, and utilising this data 
2. The default `recursion_limit` is set to **35**. From our experiments, this was good enough (for sure it should be good enough for the education agent). You can easily change the parameter in the config file if your agent requires more loops with the user
3. Does your agent interact with a Database? If No you can easily integrate your agent by wrapping it as a custom LLM. 
If yes, then currently the system supports either vanilla tool-based agents or LangGraph-based agents (see [here](https://intellagent-doc.plurai.ai/customization/custom_environment/) ). We are working hard on integrating to other frameworks, and also integration via API. If you have some interesting business use-case, you can contact us and we might be able to provide access.

Hi @Eladlev

Thank you for the quick bug fixes and comments! I’m really looking forward to these two new features as they are exactly what we urgently need! Below, I’ll briefly share some of my thoughts and our requirements for your reference.

**Events Diversity**

1. Personality is very important, as you already mentioned, because we want our product to effectively serve users with different personality types. Different personality users may ask the same question in different ways, so adding personality to the user agent is something I strongly support.

2. Additionally, it would be very helpful if we could provide a set of questions or seed queries directly, which could be used as the initial user query for interacting with the chatbot. This would allow us to better evaluate the system in the context of our own use case. We’ve noticed that the scenarios and questions generated by the LLM may differ from the actual distribution of user queries, which could potentially affect the reliability of the evaluation.

**Integration via API as Chatbot**
I admit that Langgraph is a very flexible and powerful multi-agent platform. However, for users who focus on development efficiency or whose business logic isn’t very complex (such as us), we prefer to use simpler workflow+agent frameworks like Dify, or directly wrap the chatbot into an HTTP API to be used by the frontend or external systems. Therefore, the API-based chatbot is a critical need and can be applied to all scenarios.

_Regarding these two features, do you have an estimated time of arrival (ETA) or a rough timeline for when they might be supported?_



Besides, I still have a few more questions:

1. System Level Policies
Our system has strict requirements for the output format, which must adhere to a specific JSON structure. In other words, every chatbot’s responses should always follow this specified format, which I will indicate in the system input.
However, the problem is that this rule would be treated as a policy but could be randomly sampled. What we want is for each event and scenario to adhere to these "system level policies," meaning every scenario should follow all these system level policies and then randomly sample other scenario related policies. Similar policies include "the language used in the response should match the user's language," and so on.
Does the current framework support this kind of requirement?

2. Success Rate Report

![Image](https://github.com/user-attachments/assets/fa9cf77b-0896-4c25-bae2-e7c3b69a576a)
This is the result I got from running the education example. I don’t fully understand the result chart. What I’d prefer to know is the pass rate for all events, which events failed, and a further analysis of these failures. 
Does the current framework provide this kind of data?


3. Multilingual Support(optional)
I understand that we are mainly targeting English scenarios, but our users are Chinese-speaking. I can translate all the prompts into Chinese for use, but I still wanted to ask if there’s a better way to handle this.


Once again, thank you for your hard work! I’m a big fan of your framework and look forward to your reply!

1. Regarding timeline: Unfortunately I currently can't estimate the time it will take to be generally available 
2.  Custom metric: If you have a specific metric you would like to add (like a specific JSON structure), you can easily modify the code add this metric and run it on all cases. You can for example add it [here](https://github.com/plurai-ai/intellagent/blob/bce45c3eee74738cf4b78063df606a2032eab7fb/simulator/simulator_executor.py#L169)
3. The graph you provide describes the success rate as a function of the challenge complexity. We also provide in this dashboard a table with all the specific policies and their success rate and a table with all the scenarios and the analysis (which is exactly what you asked). So you actually have in this dashboard all the relevant information...
4. Multilingual: There is currently an issue since the events are written in English so the user chatbot might get confused and talk in English. We will refine the prompt and fix the issue