Repository: confident-ai/deepeval
Issue: Feat: Add AdversarialRobustnessMetric
Root cause: Unimplemented Feature Gap

**Is your feature request related to a problem? Please describe.**

LLMs are becoming central to many applications, but their reliability is often undermined by their vulnerability to adversarial perturbations. Small, often imperceptible changes to the input—such as minor rephrasing or typos—can cause a model to produce incorrect or unexpected outputs. This poses a significant risk in high-stakes applications.

Currently, DeepEval offers a comprehensive suite of metrics for evaluating aspects like faithfulness, bias, and answer relevancy. However, it lacks a dedicated metric to systematically and quantitatively measure an LLM's robustness against these adversarial textual perturbations.

**Describe the solution you'd like**

I propose the addition of a new metric, `AdversarialRobustnessMetric`, to the DeepEval library. This metric would be based on the **RoMA (Robustness Measurement and Assessment) framework**, as detailed in the research paper **"Towards Robust LLMs: an Adversarial Robustness Measurement Framework" ([arXiv:2504.17723v1](https://arxiv.org/html/2504.17723v1))**.

The key features of this metric would be:

1.  **Black-Box Evaluation:** It will assess model robustness without requiring access to model parameters, making it compatible with any LLM.
2.  **Systematic Perturbation:** It will automatically generate two types of adversarial inputs to test resilience:
    *   **Semantic Perturbations:** Swapping words with synonyms from an embedding space to test resilience against rephrasing.
    *   **Orthographic Perturbations:** Introducing common typos to test resilience against character-level noise.
3.  **Quantitative Robustness Score:** By analyzing the distribution of the model's confidence scores across these perturbed inputs, the metric will produce a score from 0 to 1, quantifying the model's stability.

This solution would provide a standardized and scalable way for developers to benchmark the adversarial robustness of their LLMs, enabling them to build more reliable and safer applications.

**Describe alternatives you've considered**

1.  **Manual Adversarial Testing:** One could manually craft a few adversarial examples to test a model. However, this approach is not scalable, systematic, or quantifiable. It relies on ad-hoc testing and does not produce a reliable, comparable metric.
2.  **Formal Verification Methods:** As mentioned in the research paper, formal verification techniques are computationally intractable for modern LLMs due to their massive scale. They are not a practical alternative for the vast majority of developers.

The proposed RoMA-based metric is a superior alternative because it is systematic, scalable, and provides a quantitative score, making it a perfect fit.

**Additional context**

This feature would directly address a critical gap in LLM safety and evaluation. The methodology is well-documented in the linked research paper, providing a strong theoretical foundation for the metric.

I have already thought through the implementation plan, which involves creating a new `adversarial_robustness` module within the `deepeval/metrics` directory, and I am prepared to open a pull request with the implementation ASAP