Repository: Giskard-AI/giskard-oss
Issue: Implement a Gemini `LLMClient`
Root cause: Unimplemented Feature Gap

## 🚀 Feature Request

Create a Gemini client that extends `LLMClient`.

## 🔈 Motivation

This will allow to run the scan and tests using Gemini models

## 🛰 Alternatives

N/A

## 📎 Additional context

Example can be seen with [mistral](https://github.com/Giskard-AI/giskard/blob/294982ccab00fe24c60d24ce2da49eafd0a580a2/giskard/llm/client/mistral.py) and [openai](https://github.com/Giskard-AI/giskard/blob/294982ccab00fe24c60d24ce2da49eafd0a580a2/giskard/llm/client/openai.py).

Hello, is this implementation still needed ? I would like to contribute to this issue

Hello @marouanetalaa ,

yes we still need this implementation. It would be appreciated if you can contribute to this issue!

I assigned you to it, let me know if you need help or information.

Hey, I just tried replicating the changes in the pull request on my system and set my gemini api key and tried to run a scan. 

I get this for all the LLM Assisted Detectors 
```
2024-07-02 22:16:07,754 pid:17976 MainThread giskard.scanner.logger INFO     LLMStereotypesDetector: Generating test case requirements
2024-07-02 22:16:07,756 pid:17976 MainThread root         WARNING  Unsupported format 'json', ignoring.
2024-07-02 22:16:07,758 pid:17976 MainThread giskard.scanner.logger ERROR    Detector LLMStereotypesDetector failed with error: GenerationConfig.__init__() got an unexpected keyword argument 'seed' 
```

I have added these lines of code to setup the client

```
import google.generativeai as genai
from giskard.llm.client.gemini import GeminiClient

genai.configure(api_key=os.environ["GEMINI_API_KEY"])
giskard.llm.set_default_client(GeminiClient())
```

> Hey, I just tried replicating the changes in the pull request on my system and set my gemini api key and tried to run a scan.
> 
> I get this for all the LLM Assisted Detectors
> 
> ```
> 2024-07-02 22:16:07,754 pid:17976 MainThread giskard.scanner.logger INFO     LLMStereotypesDetector: Generating test case requirements
> 2024-07-02 22:16:07,756 pid:17976 MainThread root         WARNING  Unsupported format 'json', ignoring.
> 2024-07-02 22:16:07,758 pid:17976 MainThread giskard.scanner.logger ERROR    Detector LLMStereotypesDetector failed with error: GenerationConfig.__init__() got an unexpected keyword argument 'seed' 
> ```
> 
> I have added these lines of code to setup the client
> 
> ```
> import google.generativeai as genai
> from giskard.llm.client.gemini import GeminiClient
> 
> genai.configure(api_key=os.environ["GEMINI_API_KEY"])
> giskard.llm.set_default_client(GeminiClient())
> ```

Thanks for reporting the issue, I corrected this inside of #1975 

Hey, now I'm having this issue 

```
2024-07-03 20:40:01,087 pid:11428 MainThread giskard.scanner.logger INFO     Running detectors: ['LLMBasicSycophancyDetector', 'LLMCharsInjectionDetector', 'LLMHarmfulContentDetector', 'LLMImplausibleOutputDetector', 'LLMInformationDisclosureDetector', 'LLMOutputFormattingDetector', 'LLMPromptInjectionDetector', 'LLMStereotypesDetector', 'LLMFaithfulnessDetector']
Running detector LLMBasicSycophancyDetector…
2024-07-03 20:40:01,283 pid:11428 MainThread root         WARNING  Unsupported format 'json', ignoring.
2024-07-03 20:40:03,589 pid:11428 MainThread giskard.scanner.logger ERROR    Detector LLMBasicSycophancyDetector failed with error: 400 Please use a valid role: user, model.
Traceback (most recent call last):
  File "e:\Work\1. Projects\giskard_demo\.venv\Lib\site-packages\giskard\scanner\scanner.py", line 152, in _run_detectors
    detected_issues = detector.run(model, dataset, features=features)
                      ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "e:\Work\1. Projects\giskard_demo\.venv\Lib\site-packages\giskard\scanner\llm\llm_basic_sycophancy_detector.py", line 85, in run
    dataset1, dataset2 = generator.generate_dataset(
                         ^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "e:\Work\1. Projects\giskard_demo\.venv\Lib\site-packages\giskard\llm\generators\sycophancy.py", line 101, in generate_dataset
    out = self.llm_client.complete(
          ^^^^^^^^^^^^^^^^^^^^^^^^^
  File "e:\Work\1. Projects\giskard_demo\.venv\Lib\site-packages\giskard\llm\client\gemini.py", line 75, in complete
    contents=_format(messages),
             ^^^^^^^^^^^^^^^^^^
  File "e:\Work\1. Projects\giskard_demo\.venv\Lib\site-packages\google\generativeai\generative_models.py", line 331, in generate_content
    response = self._client.generate_content(
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "e:\Work\1. Projects\giskard_demo\.venv\Lib\site-packages\google\ai\generativelanguage_v1beta\services\generative_service\client.py", line 827, in generate_content
    response = rpc(
               ^^^^
  File "e:\Work\1. Projects\giskard_demo\.venv\Lib\site-packages\google\api_core\gapic_v1\method.py", line 131, in __call__
    return wrapped_func(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "e:\Work\1. Projects\giskard_demo\.venv\Lib\site-packages\google\api_core\retry\retry_unary.py", line 293, in retry_wrapped_func
    return retry_target(
           ^^^^^^^^^^^^^
  File "e:\Work\1. Projects\giskard_demo\.venv\Lib\site-packages\google\api_core\retry\retry_unary.py", line 153, in retry_target
    _retry_error_helper(
  File "e:\Work\1. Projects\giskard_demo\.venv\Lib\site-packages\google\api_core\retry\retry_base.py", line 212, in _retry_error_helper
    raise final_exc from source_exc
  File "e:\Work\1. Projects\giskard_demo\.venv\Lib\site-packages\google\api_core\retry\retry_unary.py", line 144, in retry_target
    result = target()
             ^^^^^^^^
  File "e:\Work\1. Projects\giskard_demo\.venv\Lib\site-packages\google\api_core\timeout.py", line 120, in func_with_timeout
    return func(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^
  File "e:\Work\1. Projects\giskard_demo\.venv\Lib\site-packages\google\api_core\grpc_helpers.py", line 78, in error_remapped_callable
    raise exceptions.from_grpc_error(exc) from exc
google.api_core.exceptions.InvalidArgument: 400 Please use a valid role: user, model.
LLMBasicSycophancyDetector: 0 issue detected. (Took 0:00:02.711737)
```

I think Gemini only accepts user or model as role.

Also https://ai.google.dev/gemini-api/docs/system-instructions?lang=python is a thing. How would we ideally use this within the GeminiClient?

> Hey, now I'm having this issue
> 
> ```
> 2024-07-03 20:40:01,087 pid:11428 MainThread giskard.scanner.logger INFO     Running detectors: ['LLMBasicSycophancyDetector', 'LLMCharsInjectionDetector', 'LLMHarmfulContentDetector', 'LLMImplausibleOutputDetector', 'LLMInformationDisclosureDetector', 'LLMOutputFormattingDetector', 'LLMPromptInjectionDetector', 'LLMStereotypesDetector', 'LLMFaithfulnessDetector']
> Running detector LLMBasicSycophancyDetector…
> 2024-07-03 20:40:01,283 pid:11428 MainThread root         WARNING  Unsupported format 'json', ignoring.
> 2024-07-03 20:40:03,589 pid:11428 MainThread giskard.scanner.logger ERROR    Detector LLMBasicSycophancyDetector failed with error: 400 Please use a valid role: user, model.
> Traceback (most recent call last):
>   File "e:\Work\1. Projects\giskard_demo\.venv\Lib\site-packages\giskard\scanner\scanner.py", line 152, in _run_detectors
>     detected_issues = detector.run(model, dataset, features=features)
>                       ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
>   File "e:\Work\1. Projects\giskard_demo\.venv\Lib\site-packages\giskard\scanner\llm\llm_basic_sycophancy_detector.py", line 85, in run
>     dataset1, dataset2 = generator.generate_dataset(
>                          ^^^^^^^^^^^^^^^^^^^^^^^^^^^
>   File "e:\Work\1. Projects\giskard_demo\.venv\Lib\site-packages\giskard\llm\generators\sycophancy.py", line 101, in generate_dataset
>     out = self.llm_client.complete(
>           ^^^^^^^^^^^^^^^^^^^^^^^^^
>   File "e:\Work\1. Projects\giskard_demo\.venv\Lib\site-packages\giskard\llm\client\gemini.py", line 75, in complete
>     contents=_format(messages),
>              ^^^^^^^^^^^^^^^^^^
>   File "e:\Work\1. Projects\giskard_demo\.venv\Lib\site-packages\google\generativeai\generative_models.py", line 331, in generate_content
>     response = self._client.generate_content(
>                ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
>   File "e:\Work\1. Projects\giskard_demo\.venv\Lib\site-packages\google\ai\generativelanguage_v1beta\services\generative_service\client.py", line 827, in generate_content
>     response = rpc(
>                ^^^^
>   File "e:\Work\1. Projects\giskard_demo\.venv\Lib\site-packages\google\api_core\gapic_v1\method.py", line 131, in __call__
>     return wrapped_func(*args, **kwargs)
>            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
>   File "e:\Work\1. Projects\giskard_demo\.venv\Lib\site-packages\google\api_core\retry\retry_unary.py", line 293, in retry_wrapped_func
>     return retry_target(
>            ^^^^^^^^^^^^^
>   File "e:\Work\1. Projects\giskard_demo\.venv\Lib\site-packages\google\api_core\retry\retry_unary.py", line 153, in retry_target
>     _retry_error_helper(
>   File "e:\Work\1. Projects\giskard_demo\.venv\Lib\site-packages\google\api_core\retry\retry_base.py", line 212, in _retry_error_helper
>     raise final_exc from source_exc
>   File "e:\Work\1. Projects\giskard_demo\.venv\Lib\site-packages\google\api_core\retry\retry_unary.py", line 144, in retry_target
>     result = target()
>              ^^^^^^^^
>   File "e:\Work\1. Projects\giskard_demo\.venv\Lib\site-packages\google\api_core\timeout.py", line 120, in func_with_timeout
>     return func(*args, **kwargs)
>            ^^^^^^^^^^^^^^^^^^^^^
>   File "e:\Work\1. Projects\giskard_demo\.venv\Lib\site-packages\google\api_core\grpc_helpers.py", line 78, in error_remapped_callable
>     raise exceptions.from_grpc_error(exc) from exc
> google.api_core.exceptions.InvalidArgument: 400 Please use a valid role: user, model.
> LLMBasicSycophancyDetector: 0 issue detected. (Took 0:00:02.711737)
> ```
> 
> I think Gemini only accepts user or model as role.
> 
> Also https://ai.google.dev/gemini-api/docs/system-instructions?lang=python is a thing. How would we ideally use this within the GeminiClient?

You're right, it will be more robust to switch to the `system_instruction`. However since we ask the user to provide the `GenerativeModel` we don't have access to it, I'll take a look on how to implement this properly.

I also comitted a fix for the role name, it was implemented but somehow the variable used in the end was the initial one