Leverage Gemini Pro Vision on Android to get better at Bananagrams 🍌
Tags: android, gemini, generativeai
Published: 2024-03-20 04:09:44

Bananagram is a cooler version of Scrabble! In a race to the finish, players build word grids with their letters, aiming to use them all first.

![The Bananagrams pouch with a grid of letters](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/cvhesnjxpqtbxzmxypaj.png)

But word games can be tough when you're playing in a language that you aren't native in. 
Except if you have an Android app leveraging Gemini Pro vision!


## What are we building

We will build Potassium (😁) an application that suggests words that can be spelled given the tiles that are available. To do this we'll leverage Gemini Pro vision to:
1. Analyze a picture of the tiles and extract the list letters available, 
2. List words that can be spelled with these letters.


![Image description](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/k4ulmkfu28udhja48ap1.png)




## Can Gemini Pro actually do this?

Experimenting with the ML model is key as crafting a prompt often requires multiple iterations before reaching a satisfying result. 

Let’s use [Google AI studio](https://aistudio.google.com/) to evaluate Gemini Pro vision capabilities.

Can the model create a list of the letters based on a picture of the tiles?

![Image description](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/t8yt0hrg76je8erlp5w2.png)

Then, can the model then return a list of words made with these letters?

![Image description](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/r5b5zxp5s2uk01hn3pqv.png)

## Add Gemini to your application

Now that we crafted a prompt that returns a _relatively_ satisfying response (some suggested words might or might not be valid Scrabble words), let’s create the app!
On the top left of Google AI studio, click on “Get API key” to get your Gemini API key. 

Then, click on Get code on the top right of Google AI studio to access the code snippet.


1. Add the Gradle dependencies to your app’s `build.gradle` file:

```
implementation("com.google.ai.client.generativeai:generativeai:0.1.1")
```

2. In your Kotlin code create a `GenerativeModel`:

Define the generationConfig that will be used by the model. e.g:
```kotlin
val generationConfig = generationConfig {
                temperature = 0.15f
                topK = 32
                topP = 1f
                maxOutputTokens = 4096
        }
```

The configuration is reflecting the adjustments you made in the "Run settings" section of the console. These parameters define the creativity and diversity of the text generated during inference.

`topK`: the Top-K value defines, out of the token generated by the model, the number (k) of tokens considered for the output.

`topP`: the Top-P value is used to define the cumulative probability of the k tokens (after normalization of their probability) considered for the output.

`temperature`: controls the level of randomness of the token selected for the output.

To learn more about the LLM sampling mechanism, Vibudh Singh [wrote a good explainer](https://ivibudh.medium.com/a-guide-to-controlling-llm-model-output-exploring-top-k-top-p-and-temperature-parameters-ed6a31313910). 

Then instantiate the `GenerativeModel`: 

```Kotlin
val model = GenerativeModel(
            "gemini-pro-vision",
            "your_gemini_key",
            generationConfig = generationConfig,
            safetySettings = listOf(
                SafetySetting(HarmCategory.HARASSMENT, BlockThreshold.MEDIUM_AND_ABOVE),
                SafetySetting(HarmCategory.HATE_SPEECH, BlockThreshold.MEDIUM_AND_ABOVE),
                SafetySetting(HarmCategory.SEXUALLY_EXPLICIT, BlockThreshold.MEDIUM_AND_ABOVE),
                SafetySetting(HarmCategory.DANGEROUS_CONTENT, BlockThreshold.MEDIUM_AND_ABOVE),
            ),
        )
```

3. You can then call the model as follow:
```Kotlin
viewModelScope.launch {
   val result = model.generateContent(
      content {
         image(bitmap)
         text("What are the letters displayed on the tiles? " +
            "And given these letter which Scrabble words can you spell with it?")
      }
   )
}
```

You’ll note that we pass both an `image` (as a `bitmap`) and a `text` as `content`.

To create your `bitmap`, you can simply access the camera using `rememberLauncherForActivityResult` in compose:

```Kotlin
val resultLauncher =
       rememberLauncherForActivityResult(ActivityResultContracts.StartActivityForResult()) { result: ActivityResult ->
            if (result.resultCode == Activity.RESULT_OK) {
                if (result?.data != null) {
                    bitmap = result.data?.extras?.get("data") as Bitmap
                }
            }
        }

[...]
Button (
  onClick = {
    resultLauncher.launch(cameraIntent)
  },
)
```

You'll find a very basic compose scaffolding in this [gist](https://gist.github.com/lethargicpanda/3614d643435b1665ccc17e94eb663908).