Repository: promptfoo/promptfoo
Issue: Promptfoo-Crewai Integration
Root cause: Documentation Deficiency

**Using Promptfoo to Evaluate Agent-Task Prompts in CrewAI**
Currently, I am exploring Promptfoo implementation for prompts used within the CrewAI framework. As you may know, CrewAI uses an agent-task pair structure. In a modular setup, agent definitions include role, goal, and backstory (in agents.yaml), while tasks include description and expected_output (in tasks.yaml).

I intend to evaluate these prompts to benchmark the prompt versions within the agentic framework I’ve developed. However, I haven’t been able to find any examples that specifically target this use case.

I’ll share one sample example below, and I would appreciate your guidance on whether I’m approaching this correctly or if adjustments are needed. Anyway its worked without any issues

agents.yaml content 
```
trend_researcher:
  role: >
    Market Trend Analyst specializing in {topic}
  goal: >
    Analyze the {input_text} for emerging trends, market shifts, consumer behaviors, and business investments relevant to {topic}.
  backstory: >
    You're a data-driven strategist who specializes in identifying actionable trends from market reports and news content.
    Your insights help companies stay ahead of the curve by monitoring where the market is headed.
```
tasks.yaml
```
trend_identification_task:
  description: >
    Read the input_text carefully and identify the most important market trends, changes in consumer behavior, major industry shifts, or strategic investments related to the topic.
  expected_output: >
    A bullet-point list (maximum of 10 points) summarizing the most relevant market trends or insights identified in the input_text.
  agent: trend_researcher
```


**How i Implemented in my case**
promptfooconfig.yaml
```

description: "TEST"


providers:
  - ollama:llama3.1


prompts: |
  - Market Trend Analyst specializing in {{topic}}
  - Analyze the {{input_text}} for emerging trends, market shifts, consumer behaviors, and business investments relevant to {{topic}}.
  - You're a data-driven strategist who specializes in identifying actionable trends from market reports and news content.
    Your insights help companies stay ahead of the curve by monitoring where the market is headed.
  - Read the input_text carefully and identify the most important market trends, changes in consumer behavior, major industry shifts, or strategic investments related to the topic.
  - A bullet-point list (maximum of 10 points) summarizing the most relevant market trends or insights identified in the input_text.

  
tests:
  - vars:
      topic: Electric Vehicles
      input_text: |
        According to recent industry data, the global electric vehicle (EV) market saw a 35% year-over-year growth in 2024, driven largely by surging demand in Asia and Europe.
        China accounted for over 60% of global EV sales, while European countries saw a 20% increase due to new government subsidies and stricter emission policies.
        Major automakers are accelerating their EV lineup, with Ford, GM, and Volkswagen announcing over $50 billion in combined investment for battery innovation by 2026.
        Additionally, consumers are increasingly prioritising range efficiency and fast-charging capabilities when selecting EV models.
    assert:
      - type: llm-rubric
        provider: ollama:llama3.1
        value: |
          You are evaluating whether the prompt has correctly guided the model to produce a high-quality output.

          Evaluate based on the following:

          1. **Instruction Adherence** – Does the output use bullet points (10 max) and focus only on insights related to the topic?
          2. **Topic Awareness** – Does the model focus on "Electric Vehicles" and not drift into unrelated areas?
          3. **Prompt Effectiveness** – Is the model clearly acting as a trend analyst and summarizing insights from the input?
          4. **Depth and Specificity** – Are the insights concrete and grounded in the input, not generic?
          5. **Consistency** – Would this prompt likely work similarly well for different topics?

  - vars:
      topic: Artificial Intelligence in Healthcare
      input_text: |
        The healthcare industry is witnessing a rapid integration of artificial intelligence, with 2024 marking a 40% increase in AI-related healthcare startups compared to the previous year.
        Hospitals across the U.S. and Europe are deploying AI-powered diagnostic tools, particularly in radiology and pathology, to speed up analysis and improve accuracy.
        Major pharmaceutical companies like Pfizer and Novartis are investing in AI-driven drug discovery platforms to accelerate R&D.
        Patient engagement platforms using natural language processing and chatbots are also gaining popularity for pre-diagnosis and follow-up care.
        Concerns remain, however, around patient data privacy and the explainability of AI models, prompting new regulatory discussions.

    assert:
      - type: llm-rubric
        provider: ollama:llama3.1
        value: |
          You are evaluating whether the prompt has correctly guided the model to produce a high-quality output.

          Evaluate based on the following:

          1. **Instruction Adherence** – Does the output use bullet points (10 max) and focus only on insights related to the topic?
          2. **Topic Awareness** – Does the model focus on "Artificial Intelligence in Healthcare" and not drift into unrelated AI or medical areas?
          3. **Prompt Effectiveness** – Is the model clearly acting as a trend analyst and summarising insights from the input?
          4. **Depth and Specificity** – Are the insights concrete and grounded in the input, not generic AI statements?
          5. **Consistency** – Would this prompt likely work similarly well for different topics?



outputPath: evaluations/test.html
```
- Here i just pasted role, goal, backstory, description and expected output in an ordered manner
- It worked and was able to save the results properly

**Describe alternatives you've considered**
- It would be helpful if Promptfoo officially supported an appropriate method for evaluating modular prompts as used in frameworks like CrewAI. In these setups, a single operation or task is distributed across different components—such as role, goal, and backstory in agents.yaml, and description or expected_output in tasks.yaml.

Even though these prompt fragments are stored separately, they collectively define a complete prompt when assigned to a task (e.g., Research, Restructure, etc.). The challenge is how to evaluate the quality and effectiveness of these agent-task pairs, especially when they are dynamically composed.

@mldangelo 


If any contributions are needed from my side, I’d be happy to contribute —especially with some guidance or direction from the team.

Hey @HarikrishnanK9,

Thanks for your super detailed write-up! You’re absolutely right—at the moment, Promptfoo doesn’t have a built-in way to compose prompts directly from modular YAML sources like CrewAI’s agent and task definitions. Your approach (manually merging role, goal, backstory, description, and expected output) is correct and works well with Promptfoo’s current config system.

We agree that official support for modular, composable prompts—where you could reference/import fragments from multiple YAML files—would be a valuable feature, especially for agentic or task-oriented frameworks like CrewAI.

For now, as you mentioned, the main workaround is manual concatenation or writing a script to automate merging the prompt fragments into a single block for Promptfoo.

How can we make this easier?
We’d love your input on how you’d ideally want this to work in Promptfoo.
For example, would referencing YAML fragments by key or supporting file includes/templates be helpful?
If you have any concrete ideas or examples, please share them—they’ll help us design the most useful solution!

If you’re interested in contributing, we’re happy to help guide or collaborate on a PR or RFC for this feature—just let us know.

Thanks again for your thoughtful feedback and for documenting your process!

CrewAI guide is already live: [Red Teaming a CrewAI Agent](https://www.promptfoo.dev/docs/guides/evaluate-crewai/?utm_source=chatgpt.com).
PR [#5807](https://github.com/promptfoo/promptfoo/pull/5807?utm_source=chatgpt.com) adds the matching runnable example under examples/. Once that’s merged, we can close this out. @mldangelo 

@HarikrishnanK9 - good news! We now have official CrewAI support:

- **Documentation**: https://www.promptfoo.dev/docs/guides/evaluate-crewai/
- **Example**: You can scaffold a complete CrewAI + Promptfoo project with:

```bash
npx promptfoo@latest init --example crewai
```

The guide covers setting up a CrewAI agent as a custom provider, running evaluations, and red team testing. The example includes a full recruitment agent setup you can use as a reference.

Thanks for the detailed write-up and interest in this feature!