Building a Retrieval-Augmented Generation Chatbot with SvelteKit and Xata Vector Search
Tags: ai, database, tutorial
Published: 2024-05-15 08:29:38

In this post, you'll create a Retrieval-Augmented Generation Chatbot using Xata, SvelteKit, LiteLLM, TailwindCSS and Vercel. You'll learn how to:

- Set up Xata
- Create a schema with different column types
- Using LiteLLM to Create Vector Embeddings
- Using Xata to Store Vector Embeddings with Metadata
- Using Xata Vector Search to Create Relevant Context
- Using Replicate Predictions to Prompt LLAMA 2 70B Chat Model
- Wiring Up Chatbot UI in SvelteKit

## Before you begin

### Prerequisites

You'll need the following:

- A [Xata](https://xata.io/) account
- [Node.js 18](https://nodejs.org/en/blog/announcements/v18-release-announce) or later
- An [OpenAI](https://platform.openai.com) account
- A [Replicate](https://replicate.com) account
- A [Vercel](https://vercel.com) Account

### Tech Stack

| Technology                                    | Description                                                                          |
| --------------------------------------------- | ------------------------------------------------------------------------------------ |
| [Xata](https://xata.io)                       | Serverless database platform for scalable, real-time applications                    |
| [SvelteKit](https://kit.svelte.dev)           | UI framework that uses a compiler to let you write breathtakingly concise components |
| [LiteLLM](https://github.com/BerriAI/litellm) | Call all LLM APIs using the OpenAI format                                            |
| [Replicate](https://replicate.com)            | Platform to run and fine-tune open-source models                                     |
| [TailwindCSS](https://tailwindcss.com)        | CSS framework for building custom designs                                            |
| [Vercel](https://vercel.com)                  | A cloud platform for deploying and scaling web applications                          |

## Setting up a Xata Database

After you've created a Xata account and are logged in, create a database.

![Create Xata database](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/1ptkzcatcpkjb344yjly.png)

The next step is to create a table, in this instance `vectors`, that will contain all the vector embeddings along with their metadata.

![Create vectors table](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/c32o9b75yrvqggh5lxg2.png)

Let’s move on to adding relevant columns in the table you've just created.

## Creating the Schema

In the `vectors` table, you want to store all the vector embeddings along with metadata (such as string content) so that you can use metadata of the relevant vector(s) to create context for prompting the LLAMA 2 70B Chat model.

First, click **+ Add column** and select `String`.

![Create string type column](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/k1ijg7zhybduay10la3q.png)

Proceed with adding the column name as `contents`. This column is responsible for storing the metadata as `String` type.

![Name the column as contents](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/dnbr5vds0j1hntrypywu.png)

Next, proceed to add another column name as `embedding`. This column is responsible for storing the vector embedding generated via OpenAI as `vector` type.

Among multiple [embedding models offered by OpenAI](https://platform.openai.com/docs/guides/embeddings), the `text-embedding-3-small` model is one of the newest and performant embedding models. The length of the vector embedding generated via the `text-embedding-3-small` model is 1536.

![Create vector type column](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/bn8njmnuklejq7oenet8.png)

Enter the default value of the length of the vector embedding equal to 1536.

![Name the column as embedding and set the default embedding vector length](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/hikart6r2fko1naqvtnk.png)

Lovely! With all that done, the final schema shall be shown as below 👇🏻

![View the database schema](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/kstmtptjbaykjxe078zj.png)

## Setting up the project

Clone the app repository and follow this tutorial; you can fork the project by running the following command:

```bash
git clone https://github.com/rishi-raj-jain/xata-rag-chatbot
cd xata-rag-chatbot
pnpm install
```

## Configure Xata with SvelteKit

To seamlessly use Xata with SvelteKit, install the Xata CLI globally:

```bash
npm install -g @xata.io/cli
```

Then, authorize the Xata CLI so it is associated with the logged in account:

```bash
xata auth login
```

![Create xata cli api key](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/zgl4cd35vu0a1zqc8crj.png)

Great! Now, initialize your project locally with the Xata CLI command:

```bash
xata init --db https://Rishi-Raj-Jain-s-workspace-80514q.us-east-1.xata.sh/db/rag-chatbot
```

Use the following answers to the Xata CLI one-time setup question prompts to integrate Xata with SvelteKit:

- `Yes` when prompted to add `.env` to `.gitignore`.
- `TypeScript` when prompted to select the language to generate code and types from your Xata database.
- `src/xata.server.ts` when prompted to enter the output path for the generated code by the Xata CLI.

![Set up xata with sveltekit](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/u6s4626qfbsmbm4prc9g.png)

## Using LiteLLM to Create Vector Embeddings

To use OpenAI Embedding models using LiteLLM, we need to make sure that the `OPENAI_API_KEY` exists as an environment variable. Refer to the following OpenAI doc on how to find your API Key: [Where do I find my OpenAI API Key?](https://help.openai.com/en/articles/4936850-where-do-i-find-my-openai-api-key)

```bash
# OpenAI API Key
OPENAI_API_KEY="sk-..."
```

With LiteLLM, you can call 100+ LLMs with the same OpenAI-like input and output. To create an embedding vector from a string content, we use the `embedding` method from `litellm` with `text-embedding-3-small` as the model. By default, the length of the embedding vector will be `1536` for text-embedding-3-small. Using the OpenAI output format, we obtain the embedding vector from the first object in the `data` array returned.

```typescript
// File: src/routes/api/chat/+server.ts

import { embedding } from 'litellm';

export async function POST({ request }: RequestEvent) {
  // Generate embeddings of a message using OpenAI via LiteLLM
  const embeddingData = await embedding({
    model: 'text-embedding-3-small',
    input: 'Rishi is enjoying using LiteLLM'
  });

  // Using the OpenAI output format, obtain the embedding vector stored in
  // the first object of the data array
  const getEmbeddingVector = embeddingData.data[0].embedding;
}
```

## Using Xata to Store Vector Embeddings with Metadata

We’ll use the Xata Serverless database platform (powered by PostgreSQL) to store the content from which the embedding vector was created, and the embedding vector itself. It’s as easy as inserting a record in your database. Here’s the code on how to add each vector with it’s content to the `vectors` table in your Xata database.

```typescript
// Upsert the vector with description to be used further
// as the context to upcoming questions
xata.db.vectors.create({
  contents: message,
  embedding: embeddingData.data[0].embedding,
}),
```

Storing vectors in a database helps us create context when generating chatbot responses on the fly including the latest content added. To add embedding vectors along with metadata dynamically, we create a [Server Side Rendered SvelteKit endpoint](https://kit.svelte.dev/docs/routing#server) to accept `messages` (a string array) in the POST request. Using messages, we create embeddings and insert them into our Xata database.

```typescript
// File: src/routes/api/context/warm/+server.ts

import { embedding } from 'litellm';
import { getXataClient } from '@/xata.server';

// Load the in-memory Xata client
const xata = getXataClient();

export async function POST({ request }: RequestEvent) {
  // Set of messages to create vector embeddings on
  const { messages = [] } = await request.json();

  // Call the OpenAI API to get emebeddings on the messages
  const generatedEmbeddings = await Promise.all(
    messages.map((input: string) => embedding({ input, model: 'text-embedding-3-small' }))
  );

  // Insert all of them into Xata Embedding with the content
  await Promise.all(
    generatedEmbeddings.map((embeddingData, index) =>
      // Upsert the vector with description to be further as the context to upcoming questions
      xata.db.vectors.create({
        contents: messages[index],
        embedding: embeddingData.data[0].embedding
      })
    )
  );
}
```

## Using Xata Vector Search to Create Relevant Context

We’ll use the Xata Vector Search to perform similarity search on your data based on the latest query’s dynamically generated embedding vector. It’s as easy as retrieving a record in your database. Here’s the code on how to find the 5 most similar embedding vectors to the latest user query vector in your Xata database.

```typescript
// Fetch the relevant set of records based on the embedding
const relevantRecords = await xata.db.vectors.vectorSearch(
  // Column Name
  'embedding',
  // Embedding Vector
  embeddingData.data[0].embedding,
  // Count of relevant vector embeddings
  { size: 5 }
);
```

With each relevant embedding vector, each record of your Xata database is returned. This allows us to obtain the metadata associated with these vectors and use them to create the context for the AI to answer with. In the code below, we simply create a string containing all the metadata from the relevant vectors returned in our search.

```typescript
// File: src/routes/api/context/warm/+server.ts

import { embedding } from 'litellm';
import { getXataClient } from '@/xata.server';

// Load the in-memory Xata client
const xata = getXataClient();

export async function POST({ request }: RequestEvent) {
  // Set of messages to create vector embeddings on
  const { messages = [] } = await request.json();

  // Get the latest question stored in the last message of the chat array
  const userMessages = messages.filter((i: any) => i.role === 'user');
  const input = userMessages[userMessages.length - 1].content;

  // Generate embeddings of the latest question using OpenAI
  const embeddingData = await embedding({ input, model: 'text-embedding-3-small' });

  // Fetch the relevant set of records based on the embedding
  const relevantRecords = await xata.db.vectors.vectorSearch('embedding', embeddingData.data[0].embedding, { size: 5 });

  // Combine all the metadata of the relevant vectors
  const systemContext = relevantRecords.records.map((i) => i.contents).join('\n');
}
```
## Using Replicate Predictions to Prompt LLAMA 2 70B Chat Model

To prompt Meta’s LLAMA 2 70B Chat model, we’ll use Replicate which allows us to run the model with just an API call. In our case, we make use of the `replicate` package, saving us from the need to write a fetch api by ourselves and handle streaming responses natively. We need to make sure that the `REPLICATE_API_TOKEN` is set as an environment variable. Go to your [Replicate’s Account Settings > API tokens](https://replicate.com/account/api-tokens) to obtain your Replicate API token.

```typescript
import Replicate from 'replicate'

// Instantiate the Replicate API
const replicate = new Replicate({
  auth: process.env.REPLICATE_API_TOKEN,
}
```

Using Vercel’s AI SDK baked-in `experimental_buildLlama2Prompt` function, we’re able to:

1.  Pass instructions so that the code blocks in the markdown-like responses generated by AI are prepended with \`\`\`. This'll help us highlight them in our frontend setup via a syntax highlighter. This is done via creating a message with the role of `assistant`and passing our relevant instructions in the`content` field.
2.  Pass the dynamically generated context (as string) for the AI to consider before generating responses. This is done via creating a message with the role of `system` and passing our relevant context string in the `content` field.
3.  Pass the whole conversation between user and chatbot as `messages`, which contains each message with its relevant `content` and `role` field.

````typescript
import { experimental_buildLlama2Prompt } from 'ai/prompts';

// Now use Replicate LLAMA 70B streaming to perform the autocompletion with context
const response = await replicate.predictions.create({
  // You must enable streaming.
  stream: true,
  // The model must support streaming. See https://replicate.com/docs/streaming
  model: 'meta/llama-2-70b-chat',
  // Format the message list into the format expected by Llama 2
  // @see https://github.com/vercel/ai/blob/99cf16edf0a09405d15d3867f997c96a8da869c6/packages/core/prompts/huggingface.ts#L53C1-L78C2
  input: {
    prompt: experimental_buildLlama2Prompt([
      {
        // create a system content message to be added as
        // the llama2prompt generator will supply it as the context with the API
        role: 'system',
        content: systemContext
      },
      {
        // create a system instruction
        // make sure to wrap code blocks with ``` so that the svelte markdown picks it up correctly
        role: 'assistant',
        content: `When creating repsonses sure to wrap any code blocks that you output as code blocks and not text so that they can be rendered beautifully.`
      },
      // also, pass the whole conversation!
      ...messages
    ])
  }
});
````

## Wiring Up Chatbot UI in SvelteKit

Using Vercel's AI SDK we’re able to save tremendous time in setting up a reactive chatbot UI in SvelteKit. The `useChat` utility from `ai/svelte` encapsulates the frontend logic to maintaining the array of messages exchanged between the user and the chatbot, submitting form responses to the relevant API endpoint with the messages exchanged, and stream content right into the UI from the streaming API response.

```
<script lang="ts">
  // File: src/routes/+page.svelte
  import { useChat } from 'ai/svelte';
  import MemoizedMD from '@/markdown/MemoizedMD.svelte';
  import ChevronRight from '@/icons/ChevronRight.svelte';
  import InputElement from '@/lib/components/ui/input/input.svelte';

  const { input, handleSubmit, messages } = useChat();

  function capitalizeFirstLetter(input: string) {
    return input.charAt(0).toUpperCase() + input.slice(1);
  }
</script>

<div class="flex flex-col items-center">
  <div class="relative flex w-full max-w-lg flex-col items-start overflow-hidden px-5">
    <form on:submit="{handleSubmit}" class="fixed bottom-4 flex w-[75vw] max-w-[500px] flex-row items-center space-x-2">
      <InputElement
        id="message"
        type="message"
        autocomplete="off"
        bind:value="{$input}"
        placeholder="What's your next question?"
        class="rounded border-black/25 placeholder:text-black/75 hover:border-black"
      />
      <button
        class="absolute right-3 flex size-6 flex-col items-center justify-center rounded-full border border-black/50 hover:bg-black hover:text-white"
        type="submit"
      >
        <ChevronRight />
      </button>
    </form>
    <div class="flex max-h-[90vh] w-full flex-col overflow-y-scroll">
      {#each $messages as message}
      <div class="mt-3 h-[1px] w-full bg-white" />
      <span class="max-w-max rounded border px-2 py-1 text-xs"> {capitalizeFirstLetter(message.role)} </span>
      <div class="mt-2 h-[1px] w-full bg-white" />
      <MemoizedMD message="{message.content}" />
      <div class="mt-3 h-[1px] bg-black/10" />
      {/each}
    </div>
  </div>
</div>
```
Using the `svelte-markdown` package, we’re able to visually format the responses returned by the chatbot API route for the user. Breaking HTML into components of their own tags (such as `<code>` and `<a>`), we’re able to selectively pass our own Svelte components containing their own processing logic. In this example, we create our custom `Code` svelte component to touch up the `<code>` element’s visual appearance.
```
<script lang="ts">
  // File: src/markdown/MemomizedMD.svelte
  import Code from './Code.svelte';
  import SvelteMarkdown from 'svelte-markdown';

  export let message: string;
</script>
{% raw %}
<SvelteMarkdown source={message} renderers={{ code: Code }} />
{% endraw %}
```
Using `svelte-highlight` package, we’ll specifically highlight the code blocks in the chatbot responses to make them more visually appealing. Within just a few lines, we’re able to create HTML classes for each code block specifically based on its language (by default, we’ve set it to `typescript`).

```
<script lang="ts">
  // File: src/markdown/Code.svelte
  import Highlight from 'svelte-highlight';
  import typescript from 'svelte-highlight/languages/typescript';

  export let lang: string;
  export let text: any;
</script>
{% raw %}
<Highlight language="{typescript}" code="{text}" />
{% endraw %}
```
To make sure the specific HTML classes have their respective CSS styles present, we use the `GitHub Light Theme` (exported by `svelte-highlight`) in the app’s parent [layout file](https://kit.svelte.dev/docs/advanced-routing#advanced-layouts-layout).

```
<script lang="ts">
  // File: src/routes/+layout.svelte
  import github from 'svelte-highlight/styles/github';
</script>

<svelte:head>
  {@html github}
</svelte:head>

<slot />
```
Here's a preview of what we've succesfully created with SvelteKit and Xata 👇🏻

![Conversational UI with Xata](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/fin0rfbgbvzvzs9nye2o.png)

## Deploy to Vercel

The repository, is now ready to deploy to Vercel. Use the following steps to deploy:

- Start by creating a GitHub repository containing your app's code.
- Then, navigate to the Vercel Dashboard and create a **New Project**.
- Link the new project to the GitHub repository you just created.
- In **Settings**, update the **Environment Variables** to match those in your local `.env` file.
- Deploy! 🚀

## More Information

For more detailed insights, explore the references cited in this post.

| Resource              | Link                                                     |
| --------------------- | -------------------------------------------------------- |
| GitHub Repo           | https://github.com/rishi-raj-jain/xata-rag-chatbot       |
| SvelteKit with Xata   | https://xata.io/docs/getting-started/sveltekit           |
| Xata Vector Search    | https://xata.io/docs/sdk/vector-search                   |
| How to Prompt Llama 2 | https://huggingface.co/blog/llama2#how-to-prompt-llama-2 |
| SvelteKit AI SDK      | https://sdk.vercel.ai/docs/guides/frameworks/sveltekit   |

## What’s next?

We'd love to hear from you if you have any feedback on this tutorial, would like to know more about Xata, or if you'd like to contribute a community blog or tutorial. Reach out to us on [Discord](https://discord.com/invite/kvAcQKh7vm) or join us on [X | Twitter](https://twitter.com/xata). Happy building 🦋