Repository: plurai-ai/intellagent
Issue: Evaluation of Workflow-Driven Multi-Prompt Conversational Chatbots

First of all, we sincerely appreciate your work in developing such an intelligent and elegant evaluation framework. It perfectly addresses our business needs for chatbot evaluation. We have already decided to adopt this framework as the foundation for evaluating all of our conversational agents!


**Currently, we have a key question that would like to seek your advice on.**
All of our conversational agents are implemented through workflows, passing through multiple stages such as initial user intent recognition, query rewriting, rag, tool calling and ultimately generating responses for users. Each stage may has its own dedicated prompt and corresponding rules.
However, after reading your paper, it seems that before evaluation, we can only provide a single system prompt to generate multiple policies, which are then used to create multiple scenarios to simulate dialogues, followed by evaluation. Given our scenario, where multiple intermediate prompts exist across different stages, should we consolidate these prompts into a single prompt before passing it to your framework for evaluation? 
What would be the best practice in this case?


If there is any misunderstanding or incorrect interpretation on our part, please correct us. Once again, we sincerely appreciate your hard work🎉.

Hi @ReycoLi, thanks for your feedback! We’re glad to hear you found the system useful for your product needs.

Regarding your question:
The way we think about it is that from a product perspective, the journey doesn’t begin with "multiple intermediate prompts exist across different stages". Instead, it starts with a product specification outlining the desired chatbot behavior and the policies it should follow. The system input doesn’t necessarily have to be a **system prompt**—it could be a lengthy, unstructured document detailing all product requirements. For instance, you could start with your help center documentation or simply combine all your component prompts.

It's important to note that some components like RAG, aren't natively supported by the current system. However, we have internal extensions of this framework that accommodate these components. If you’re interested in incorporating them, feel free to reach out!


Hi @Eladlev,

Thank you so much for your prompt response and for your detailed explanation and clarification! We truly appreciate your insights.

We have some detailed functional requirement documents, which I believe can serve as the system input, as you suggested.

Additionally, each of our workflows includes an RAG component, and I believe most modern chatbot systems heavily rely on RAG as well. That’s why I’m very interested in your internal extensions that support RAG or other commonly used components. If you could share more details, I would greatly appreciate it!

Regarding the RAG, we developed a method to extract challenging questions by decomposing the database into a graph structure (very similar to GraphRAG). We definitely intend to extend this framework also to support RAG components, but I'm not sure about the timeline (we have an extensive roadmap), and the exact scope. 

If you have a concrete business use case you can [contact us](https://plurai.ai/contact-us) and we can help with the customization (It's not only RAG, but also for function calling we have interesting components). Alternatively, if you like to contribute and add the RAG component to the open-source, we can [assist](https://discord.gg/YWbT87vAau) in guidance

I’m really looking forward to all your future extensions and updates, not just RAG.
I will take a deeper look into our specific requirements and thoroughly review the source code. If I have any questions, I’ll reach out again.Thanks again for your help and support!