Build a Streaming AI Chat App with Vercel AI SDK and Next.js
Tags: nextjs, ai, gpt3, tutorial
Published: 2023-06-19 08:00:00

> Also available on my [blog](https://nikolasbarwicki.com/articles/build-a-streaming-ai-chat-app-with-vercel-ai-sdk-and-next-js)

## Introduction

Vercel recently released their **Vercel SDK** - an open-source library designed to help developers build AI-powered user interfaces in JavaScript and TypeScript. [This SDK](https://vercel.com/blog/introducing-the-vercel-ai-sdk) makes it easy to create conversational, streaming, and chat user interfaces with cutting-edge technologies.

### Key Features of Vercel AI SDK

- **Edge-ready**: Compatible with [Vercel's Edge Network](https://vercel.com/docs/concepts/edge-network/overview), enabling fast and scalable AI-powered apps.
- **Built-in Adapters**: First-class support for LangChain, OpenAI, Anthropic, and Hugging Face.
- **Streaming First**: [SWR](https://swr.vercel.app/)-powered React hooks for fetching and rendering streaming text responses.
- **Stream Callbacks**: Callbacks for saving completed streaming responses to a database in the same request.

To illustrate the versatility of the Vercel AI SDK, we'll build a streaming AI chat app with Next.js in this step-by-step guide.

## Experimenting with Large Language Models (LLMs)

Choosing the right [large language model](https://en.wikipedia.org/wiki/Large_language_model) (LLM) and fine-tuning prompts are essential for achieving the best results in AI-powered applications. Vercel's interactive online [prompt playground](https://play.vercel.ai) launched in April, provides an excellent opportunity to experiment with 20 open-source and cloud LLMs in real-time.

![Comparing LLMs in Vercel's AI Playground](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/gsv05h6dli5e59pjk1d2.png)

As an developer, you may want to **experiment with different prompts, model sizes, and generation parameters**, such as temperature and max tokens. Adjusting these factors can significantly change the output of the LLM, ultimately affecting the final user experience of your AI-powered chat app.

To enhance your understanding of the different LLMs and their distinct capabilities, take advantage of [Vercel SDK's official playground](https://sdk.vercel.ai/) to compare how each model responds to the same prompts. As a result, you'll gain insights into the ideal models and prompt engineering techniques to meet your specific use case requirements.

Emphasize the importance of experimenting with prompts and generation parameters to find the right balance between creativity, relevance, and processing speed. Putting effort into optimizing these factors will not only help you understand the strengths and weaknesses of different models but also pave the way to create more engaging and useful conversational UIs in your applications.

## Understanding Streaming in AI Chat Apps

[Streaming](https://sdk.vercel.ai/docs/concepts/streaming) conversational text UIs, like ChatGPT, have gained massive popularity over the past few months. In this section, we'll explore the benefits and drawbacks of streaming and blocking interfaces, and how to implement them using the Vercel AI SDK.

**Large Language Models (LLMs) are extremely powerful**, but they can be slow when generating long outputs. A traditional blocking UI can cause users to wait for several seconds before the entire LLM response is generated. This can result in a poor user experience in conversational applications like chatbots. Streaming UIs can help mitigate this issue by displaying parts of the response as they become available.

- **Blocking UI**: Waits until the full response is available before displaying it.
- **Streaming UI**: Transmits and displays parts of the response as they become available.

Streaming UIs can display the response **much faster than a blocking UI**, as they don't need to wait for the entire response before showing anything. However, streaming interfaces may not be necessary or beneficial in every case. If you can achieve your desired functionality using a smaller, faster model without streaming, you'll often have a simpler and more manageable development process.

Regardless of your model's speed, the Vercel AI SDK makes implementing streaming UIs as simple as possible.

## Building a Streaming AI Chat App with Next.js and Vercel AI SDK

In this tutorial, we're going to build a streaming AI chat app using Vercel AI SDK and Next.js. We'll provide a more detailed breakdown of each step, ensuring that you can follow along easily.

### Step 1: Set up a new Next.js project

First, let's create a new Next.js project using the latest version with the **app router**. Run the following command in your terminal:

```shell
pnpm dlx create-next-app ai-app
```

During the project's creation process, select TypeScript, ESLint, Tailwind CSS, the `src/` directory, and the App Router. Customize the default import alias if you prefer.

### Step 2: Install necessary dependencies

Navigate to the project's directory with `cd ai-app` and install the required dependencies:

```shell
pnpm i ai openai-edge
```

The `ai` package is the **Vercel AI SDK** that allows us to build edge-ready AI-powered streaming text and chat UIs. The `openai-edge` package is a TypeScript module for querying OpenAI's API using the fetch method, instead of Axios.

### Step 3: Clean up the project's boilerplate code

To start fresh, remove the initial code by updating the content of the `app/globals.css` and `app/page.tsx` files:

```css
/* app/globals.css */
@tailwind base;
@tailwind components;
@tailwind utilities;
```

```tsx
// app/page.tsx
export default function Home() {
  return <main>Hello World</main>
}
```

### Step 4: Add an environment file and API key

Create a new `.env.local` file using the `touch .env.local` command. Then, generate an OpenAI API key in the OpenAI dashboard and paste it into the `.env.local` file.

### Step 5: Create a Route Handler for the `/api/chat` endpoint

In this step, we will create an `app/chat/route.ts` file to handle the `/api/chat` endpoint. Perform the following actions:

1. Inside the `app/chat/route.ts` file, create an edge-friendly OpenAI API client using the API key from the `.env.local` file.
2. Set the runtime to `edge`.
3. Create a `POST` function that will handle the `/api/chat` endpoint.

```tsx
// app/chat/route.ts
import { Configuration, OpenAIApi } from 'openai-edge'
import { OpenAIStream, StreamingTextResponse } from 'ai'

const config = new Configuration({
  apiKey: process.env.OPENAI_API_KEY,
})
const openai = new OpenAIApi(config)

export const runtime = 'edge'

export async function POST(req: Request) {
  const { messages } = await req.json()

  const response = await openai.createChatCompletion({
    model: 'gpt-3.5-turbo',
    stream: true,
    messages,
  })

  const stream = OpenAIStream(response)
  return new StreamingTextResponse(stream)
}
```

You can pass multiple options to the `createChatCompletion` request, such as choosing a model, setting the temperature, max tokens, etc. Optional callbacks can save the result of a completion to a database after streaming it back to the user, see examples below:

```typescript
const stream = OpenAIStream(response, {
  onStart: async () => {
    // This callback is called when the stream starts
    // You can use this to save the prompt to your database
    await savePromptToDatabase(prompt)
  },
  onToken: async (token: string) => {
    // This callback is called for each token in the stream
    // You can use this to debug the stream or save the tokens to your database
    console.log(token)
  },
  onCompletion: async (completion: string) => {
    // This callback is called when the stream completes
    // You can use this to save the final completion to your database
    await saveCompletionToDatabase(completion)
  },
})
```

### Step 6: Design the app's user interface

To create the chat app's user interface, we will use the `useChat` function from the `ai/react` sub-path in the `app/page.tsx` file. This will allow us to create a simple and functional chat app.

```tsx
// app/page.tsx
'use client'

import { useChat } from 'ai/react'

export default function Chat() {
  const { messages, input, handleInputChange, handleSubmit } = useChat()

  return (
    <div className="stretch mx-auto flex w-full max-w-md flex-col py-24">
      {messages.map((m) => (
        <div key={m.id}>
          {m.role === 'user' ? 'User: ' : 'AI: '}
          {m.content}
        </div>
      ))}

      <form onSubmit={handleSubmit}>
        <label>
          Say something...
          <input
            className="fixed bottom-0 mb-8 w-full max-w-md rounded border border-gray-300 p-2 shadow-xl"
            value={input}
            onChange={handleInputChange}
          />
        </label>
        <button type="submit">Send</button>
      </form>
    </div>
  )
}
```

The `useChat` hook takes care of appending a user message to the chat history and triggering an API request to the configured endpoint. As new chunks of streamed messages are received, the hook updates the `messages` state and triggers a re-render for a seamless chat experience.

![Streaming Chat App UI Interface](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/xjo3rpy4k8sl712fpksy.png)

## Summary

Congratulations! You've successfully built a **streaming AI chat app** using the Vercel AI SDK and Next.js. Now, users can see the AI response **as soon as it's available**, without waiting for the entire response to be received, resulting in an improved chat experience.