Java & GenAI : The Ultimate Developer's joy with Quarkus, Langchain4j and Ollama
Tags: genai, quarkus, langchain4j, ollama
Published: 2024-03-05 19:05:21

This is the companion blog for this video 
{% embed https://www.youtube.com/watch?v=TYH95_Vzvzw&ab_channel=SebastienBlanc %}.


In this short post I will show you how to get started with Quarkus, Langchain4j and Ollama to set up your local dev environment to developer GenAI applications. Ready ? Let's go ! 

## Installing Ollama

Ollama is one of those magic those that makes developers happy, basically you install it and then you can run LLMs locally, it has also a embedded web server so that you make requests to it, just like you would do with OpenAI. 

Head over to [Ollama](https://ollama.com/)and follow the installation instructions. 

Open a terminal and type : 

`ollama list`

It will list you the LLMs that are installed locally on your computer. You can easily add a new LLM by doing for instance :

`ollama pull mistral:latest`

Check [the model page](https://ollama.com/library) to see what is available, it's that easy ! 

## Creating the Quarkus project

Your Quarkus app will need for now only 2 extensions : `quarkus-resteasy-reactive` and `quarkus-langchain4j-ollama` , you can also use this [link](https://code.quarkus.io/?g=org.sebi&a=fun-with-ai&e=resteasy-reactive&e=io.quarkiverse.langchain4j%3Aquarkus-langchain4j-ollama&extension-search=origin:platform%20langchain) and click on the big blue button "Generate your application" 

Once unzipped, open your project in your favorite IDE. 

### First steps with Langchain4j and Quarkus

We need to define in the properties which LLM we will be using, open your `application.properties` and add this line : 

```bash
quarkus.langchain4j.ollama.chat-model.model-id=llama2:latest
```

Now let's create our first AI Service, create a new interface called `MyAIService.java` : 

```java

package org.sebi;

import dev.langchain4j.service.UserMessage;
import io.quarki
verse.langchain4j.RegisterAiService;

@RegisterAiService
public interface MyAIService {
     public String chat(@UserMessage String message);
}

```

The important thing to notice here is the `@RegisterAiService` annotation, this indicates to Quarkus that when calling the method `chat` it will be using the LLM that we defined in the properties. 

Run your app : `mvn quarkus:dev` , this will start your application in dev mode. 

Once it's running, press `d` in the terminal to open the devUI : 


![Image description](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/gvfgszg93kr9fcxnjsln.png)

You should see the langchain4j widget containing a link called `chat`, you can click on it and it will bring you to an interface where can prompt your LLM ! 

At any point, you can change the model in the properties, you can also enable logging for the outgoing requests to see what is exactly sent to the LLM : 

```bash
quarkus.langchain4j.ollama.chat-model.model-id=llama2:latest
quarkus.langchain4j.ollama.log-requests=true
```

That's pretty cool, isn't it ? 

### Exposing our AI Service as a rest endpoint

The DevUI is useful but now we want to make it possible to access our LLM from a rest endpoint, we already have the `resteasy-reactive` extension and you should also have already a resource class called `GreetingResource` , change the content with : 

```java
package org.sebi;

import jakarta.inject.Inject;
import jakarta.ws.rs.Consumes;
import jakarta.ws.rs.POST;
import jakarta.ws.rs.Path;
import jakarta.ws.rs.Produces;
import jakarta.ws.rs.core.MediaType;

@Path("/chat")
public class GreetingResource {

    @Inject
    MyAIService myAIService;

    @POST
    @Produces(MediaType.TEXT_PLAIN)
    @Consumes(MediaType.TEXT_PLAIN)
    public String hello(String body) {
        return myAIService.chat(body);
    }
}
```

Here we simply inject our AI Service and call it in our POST method. 

Try it out with this curl command : 

```bash
curl -X POST  -H 'Content-Type: text/plain' -d 'Tell me a joke' http://localhost:8080/chat
```

I got this one : 

```
Sure, here's one:

Why don't scientists trust atoms?
Because they make up everything!
```

### Adding a system prompt

Let's fine tune our prompt by providing a system message. Usually you can use this to define how the chat bot must behave. Go to your AI Service and add this annotation :

```java
     @SystemMessage("You are a funny bot and always reply with a poem!")   
     public String chat(@UserMessage String message);
```

Try your curl command again and it should reply with a joke in the form of a poem ! 

You can change as you want the system prompt to experiment, change the model, all of this without having to stop&start your app because you are running in dev mode ! 

That all for today, next time we will explore how to add memory to our bot !