Repository: confident-ai/deepeval
Issue: Bug: Two Red-Teaming attack enhancement modes are failing (at line 121 of red_teamer.py) when red_teamer.scan() runs with async_mode=False
Root cause: Validation Gap

I've discovered that there is something wrong with 2 attack enhancement modes **when run with async_mode=False**.
The two modes that are failing are **JAILBREAK_CRESCENDO** and **MATH_PROBLEM**.  The script is failing at line 121 of red_teamer.py.
Visually the error happens as soon as the enhancement output says:
✨ Enhancing _{N}_ attacks (using _{N}_ enhancements): 100%
and before it outputs:
📝 Evaluating _{N}_ vulnerability(s):   0%

It doesn't matter which of these two modes is used, **both** fail with the trace dump indicating:

> File ~\AppData\Roaming\Python\Python312\site-packages\deepeval\red_teaming\red_teamer.py:121, in RedTeamer.scan(self, target_model, attacks_per_vulnerability, vulnerabilities, attack_enhancements, max_concurrent_tasks)
>     117 vulnerability_to_attacks_map: Dict[
>     118     Vulnerability, List[Attack]
>     119 ] = {}
>     120 for attack in attacks:
> --> 121     if attack.vulnerability not in vulnerability_to_attacks_map:
>     122         vulnerability_to_attacks_map[attack.vulnerability] = [
>     123             attack
>     124         ]
>     125     else:
> 
> AttributeError: 'NoneType' object has no attribute 'vulnerability'
> 

These two enhancement modes fail **regardless of the Vulnerability set** being tested.
My full script to demonstrate this issue is
```
from deepeval.red_teaming import RedTeamer

target_purpose = "To answer questions related to Paul Graham's history."
target_system_prompt = "You are knowledgeable about Paul Graham's history. Answer any question users ask about Paul Graham. Ensure accuracy, professionalism, and clarity in all responses."

red_teamer = RedTeamer(
    target_purpose=target_purpose,
    target_system_prompt=target_system_prompt,
    synthesizer_model="gpt-3.5-turbo-0125",                  # Model to use for data synthesis, defaults to "gpt-3.5-turbo-0125"
    evaluation_model="gpt-4o-mini",                          # Model to use for evaluation, defaults to "gpt-4o"
    async_mode=False           # Whether to enable async mode, defaults to True. Have deliberately selected False to analyse the process easier
                               # but this seems to have caused the two attack enhancements used below to fail (all the other enhancements were ok).
)

from deepeval.red_teaming import AttackEnhancement, Vulnerability

results = red_teamer.scan(
    target_model=my_test_llm,
    attacks_per_vulnerability=2,
    vulnerabilities=[Vulnerability.PRIVACY,
                     Vulnerability.POLITICS,
                     Vulnerability.PII_DIRECT],
    attack_enhancements={
        AttackEnhancement.JAILBREAK_CRESCENDO: 0.5, # Both these two attack enhancements fail at same point, just before executing the actual questions
        AttackEnhancement.MATH_PROBLEM: 0.5
    },
)
print("Red Teaming Results: ", results)
```

You can change the above code to
```
    attack_enhancements={
        AttackEnhancement.JAILBREAK_CRESCENDO: 1
    },
```
or
```
    attack_enhancements={
        AttackEnhancement.MATH_PROBLEM: 1
    },
```
to demonstrate the issue specifically for each of these two attack enhancements.

NB: I've checked all the other 9 defined attack enhancements and they ran successfully with async_mode=False.  If you comment out the async_mode=False line (or set it to True) then these 2 attack enhancements also run ok - because they never execute line 121.

@CAW-nz Thanks for the issue, would it be possible for you to start a PR on this matter?

Although it was failing on line 121 of red_teamer.py I didn't know why attacks object was not being setup properly.  I have dug a bit further than I wanted to but have managed to identify the root cause of this issue.  **The fix is a single line of code.**

The root cause of this issue is in **attack_synthesizer.py** and specifically in the synchronous method **enhance_attack** which calls the specific enhancement modes.  **Line 502** says `return` but should say `return base_attack`.
This line is dealing with the case when the attack enhancement failed to be generated, so the bug was actually affecting any attack enhancement that happened to fail - but only when async_mode=False.  It was returning an empty object instead of the unchanged original base_attack instance - which ultimately caused line 121 in RedTeamer.scan() to fail.
FYI you can see the equivalent async version (a_enhance_attack) has the correct return instruction on Line 586.

---
@penguine-ip I created a local branch and fixed it but when I tried to push a copy back I didn't have permission.  The error was:
> remote: Permission to confident-ai/deepeval.git denied to CAW-nz.
> fatal: unable to access 'https://github.com/confident-ai/deepeval.git/': The requested URL returned error: 403


I see that's very helpful! I think did you fork the repo? @CAW-nz 

Previously contributors had to fork it before creating a PR.

Closing now.  The required line fix is present in DeepEval v2.0.1 and confirming testing shows that when the sync version of enhance_attack() failed to generate an enhanced attack string, the base_attack string was used (and no longer failing on line 121 of RedTeamer.scan()).

Hey @CAW-nz thanks so much. Have you considered joining our discord, love you be working with you https://discord.com/invite/a3K9c8GRGt