Repository: confident-ai/deepeval
Issue: RedTeamer is broken
Root cause: External Dependency Breakage

Hello everyone, I was doing some tests running red_teamer today and I noticed that we have a bug related to some possible API update.

This was the error I got:
`Traceback (most recent call last):
  File "/mnt/c/Users/AntonioRamos/OneDrive - Atacana Group Inc/Desktop/antonio/repositories/ai-evaluation-backend/src/./ai_evaluation/metrics/safety_and_security/prompt_injection.py", line 85, in scan_vulnerability
    results = self.red_teamer.scan(
  File "/home/antonio/.cache/pypoetry/virtualenvs/ai-evaluation-backend-1d9iZnIQ-py3.10/lib/python3.10/site-packages/deepeval/red_teaming/red_teamer.py", line 118, in scan
    return loop.run_until_complete(
  File "/home/antonio/.cache/pypoetry/virtualenvs/ai-evaluation-backend-1d9iZnIQ-py3.10/lib/python3.10/site-packages/nest_asyncio.py", line 98, in run_until_complete
    return f.result()
  File "/usr/lib/python3.10/asyncio/futures.py", line 201, in result
    raise self._exception.with_traceback(self._exception_tb)
  File "/usr/lib/python3.10/asyncio/tasks.py", line 234, in __step
    result = coro.throw(exc)
  File "/home/antonio/.cache/pypoetry/virtualenvs/ai-evaluation-backend-1d9iZnIQ-py3.10/lib/python3.10/site-packages/deepeval/red_teaming/red_teamer.py", line 298, in a_scan
    await self.attack_synthesizer.a_generate_attacks(
  File "/home/antonio/.cache/pypoetry/virtualenvs/ai-evaluation-backend-1d9iZnIQ-py3.10/lib/python3.10/site-packages/deepeval/red_teaming/attack_synthesizer.py", line 159, in a_generate_attacks
    attack_results = await asyncio.gather(*generate_tasks)
  File "/usr/lib/python3.10/asyncio/tasks.py", line 304, in __wakeup
    future.result()
  File "/usr/lib/python3.10/asyncio/tasks.py", line 232, in __step
    result = coro.send(None)
  File "/home/antonio/.cache/pypoetry/virtualenvs/ai-evaluation-backend-1d9iZnIQ-py3.10/lib/python3.10/site-packages/deepeval/red_teaming/attack_synthesizer.py", line 146, in throttled_generate_base_attack
    result = await self.a_generate_base_attacks(
  File "/home/antonio/.cache/pypoetry/virtualenvs/ai-evaluation-backend-1d9iZnIQ-py3.10/lib/python3.10/site-packages/deepeval/red_teaming/attack_synthesizer.py", line 257, in a_generate_base_attacks
    remote_attacks = self.generate_remote_attack(
  File "/home/antonio/.cache/pypoetry/virtualenvs/ai-evaluation-backend-1d9iZnIQ-py3.10/lib/python3.10/site-packages/deepeval/red_teaming/attack_synthesizer.py", line 505, in generate_remote_attack
    return GenerateBaselineAttackResponseData(**response).baseline_attacks
UnboundLocalError: local variable 'response' referenced before assignment`

The error occurred because something went wrong in the API call to retrieve the attack prompts. At this stage of the code I saw that we are passing the vulnerability incorrectly:
<img width="475" alt="Image" src="https://github.com/user-attachments/assets/ed6986f1-7163-43c3-b089-63267c9ec81c" />

This is because if I pass, for example, the "Guards" vulnerability, the API will return an error.
<img width="422" alt="Image" src="https://github.com/user-attachments/assets/899d5021-0f8f-4bee-8a68-2c6792ebc940" />

Now if I just pass "guards", it will return a valid result.
<img width="611" alt="Image" src="https://github.com/user-attachments/assets/2bad0704-b7f1-432d-92c0-c2af708bda8e" />

Could you guys check?


Experiencing the same issue through the Python library..
Setup: DeepEval 2.5.5, Python 3.11.10

@thekaasking @tonylincon1 the docs isn't completely migrated but red teaming is now handled by our latest package dedicated for redteaming: https://www.trydeepteam.com/docs/getting-started#detect-your-first-llm-vulnerability

It follows the exact same pattern as DeepEval but it makes everything easier to use, can you try it and let me know does it work for you? Just do
```
pip install deepteam
```
We will be moving the docs to point to DeepTeam for red teaming very soon

@thekaasking @tonylincon1 the package is here: https://github.com/confident-ai/deepteam

@penguine-ip Even using this other variation of the library that you sent me, I'm facing the same errors:

<img width="593" alt="Image" src="https://github.com/user-attachments/assets/ca7aa78a-1ad9-4ae9-881d-e5d53f6cbfd7" />

@tonylincon1 i cannot see your error message, you left out the most important part (the actual error)

also which version of deepeval + deepteam r u using?


hey @penguine-ip I'm facing a lot of similar issues using `deepteam`. Basically a lot of unfound `str` values for Vulnerability. Below is just one example.

```
{"detail": "Invalid value 'direct disclosure' for Vulnerability"}
```

Seems to be consistent across the board with most of the Vulnerabilities I tried (e.g. `Bias`, `PIILeakage`, `Misinformation`, `Toxicity`)