Build a realtime closed-caption system in React, AssemblyAI and Ably
Tags: tutorial, react, ai, vite
Published: 2023-11-06 11:25:44


Closed-captioning for television systems was first demonstrated in the early 1970s. [Realtime closed-captioning](https://en.wikipedia.org/wiki/Closed_captioning) was developed later in the early 1980s. Here, [stenotype](https://en.wikipedia.org/wiki/Stenotype) operators who type at speeds of over 225 words per minute provide captions for live television programs, allowing the viewer to see the captions within two to three seconds of the words being spoken was developed later in the early 1980s.

As the internet has exploded the ability for anyone to create content, including live video and audio, has also grown. It remains important to consider making that content as accessible as possible not only allowing those with hearing impairment to consume your content, but also those learning to read, learning to speak a non-native language, or those in an environment where the audio is difficult to hear or is intentionally muted.

Sometimes, consumers of your content may simply prefer to read a transcript.

In this blog post, we will build a realtime browser-based closed-captioning system using React, [AssemblyAI’s](http://assembly.ai/) realtime transcription service, and [Ably](https://hubs.la/Q027mfl_0).  If you want to jump ahead and check out the completed code, check out [this Github repo](https://github.com/ablydevin/react-closed-captioning), otherwise let's get started!

## Prerequisites

Before you start coding, you’ll need to make sure you have the following prerequisites.

First, you’ll need to create an [AssemblyAI account](https://www.assemblyai.com/).  We’ll be using their realtime transcription services as part of our project. Realtime transcription is [priced at a per-second rate](https://www.assemblyai.com/pricing) so while you will need to add a payment method to your account to use realtime transcription you’ll only pay for what you use.

Next, you will need a [free Ably account](https://ably.com/sign-up). We’ll use Ably to broadcast the transcription to our application users.

Lastly, we will be building the server-side parts of the application in Node including using Nodes native fetch API which is [available in Node 18](https://nodejs.org/en/blog/announcements/v18-release-announce) or newer.

Once you’ve got those accounts, then head on to the rest of the blog post to start to build!

## Application setup

Start your application by using [Vite](https://vitejs.dev/) to scaffold a basic React and JavaScript application. Open a terminal and run Vites `create` command:

`npm create vite@latest`

Vite will ask you a few questions about your project including the project name, an application framework (select React), and which framework variant you want to use (select JavaScript).

After the scaffolding completes, navigate into the project directory and install the project dependencies using npm. Finally, run the project to verify that the scaffolded project works as expected:

```
cd react-closed-captioning
npm install
npm run dev
```

## Add application routing

Our closed captioning application will have two modes:

- A publisher mode that records and transcribes the audio.
- A subscriber mode that receives the transcription text.

We’ll need to allow a user to navigate between those modes which we do by adding routing using the `react-router-dom` package.

Start by creating components in the applications `src` directory for the publisher (`Publisher.jsx`) and subscriber (`Subscriber.jsx`) modes:

```javascript
import { useState } from "react";
import './App.css'

function Publisher() {}

export default Publisher
```

```javascript
import { useState } from "react";
import './App.css'

function Subscriber() {}

export default Subscriber
```

Next, install the `react-router-dom` package using NPM:

`npm install react-router-dom`

In the `main.jsx` file import the `BrowserRouter` object from react-router-dom and wrap the default `App` component with it:

```javascript
import React from 'react'
import { BrowserRouter } from 'react-router-dom'
import ReactDOM from 'react-dom/client'
import App from './App.jsx'
import './index.css'

ReactDOM.createRoot(document.getElementById('root')).render(
  <React.StrictMode>
    <BrowserRouter>
     <App />
    </BrowserRouter>
  </React.StrictMode>,
)
```

In `App.jsx`, add import statements for the `Routes`, `Route`, and `Link` objects as well as the Publisher and Subscriber components you created earlier:

```javascript
import { Routes, Route, Link } from 'react-router-dom';
import Publisher from './Publisher'
import Subscriber from './Subscriber'
```
Finally, add the route links inside of the App components Empty tag:

```javascript
<div style={{ textAlign: "center", marginBottom: 10, }}>
  <div>
    <Link to="/Publisher">Publisher</Link>
  </div>
  <div>
    <Link to="/Subscriber">Subscriber</Link>
  </div>
</div>
<div style={{ textAlign: "center" }}>
  <Routes>
    <Route path="/publisher" element={<Publisher />} />
    <Route path="/subscriber" element={<Subscriber />} />
  </Routes>
</div>
```
Restart the app if needed by running `npm run dev` or refresh the page you should see links to the Publisher and Subscriber views.

![assemblyai-ably-image](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/fjogr18lxhgwtzj0r439.png)

## Generate service authentication tokens

The two services we use in the application, Ably and AssemblyAI, both require the application to authenticate itself with their APIs. To do that we need to create API tokens using server-side code. This lets us keep our service keys safe on the server and instead create temporary authentication tokens our client code can use to authenticate with the services.

To create the server code for the application we’ll use a Vite plugin called `vite-plugin-api`. This plugin makes it easy to create API endpoints as part of our application.

Use NPM to install the plugin package, [express](https://expressjs.com/) web server package, and the [Ably client library](https://hubs.la/Q027mn--0):

```
npm install vite-plugin-api express ably
```

Add the plugin to your Vite configuration file:

```javascript
import { defineConfig } from 'vite'
import react from '@vitejs/plugin-react'
import { pluginAPI } from "vite-plugin-api";

// https://vitejs.dev/config/
export default defineConfig({
  plugins: [react(),
    pluginAPI({
      // Configuration options go here
    }),
  ],
})
```
The application will store those sensitive Ably and AssemblyAI service keys as environment variables. Vite makes it easy to access environment variables stored in a local file named `.env.local`. Create this file in the root of your project directory and then add the following keys:

```
VITE_ABLY_API_KEY=[YOUR_ABLY_API_KEY]
VITE_ASSEMBLYAI_API_KEY=[YOUR_ASSEMBLYAI_API_KEY]
```
Make sure you replace the placeholder values with your actual Ably and AssemblyAI keys.

Next, inside of your project’s `src` directory create a new directory named `api/tokens` and add two files to it: `realtime.js` and `transcription.js`.

Inside of `realtime.js` add the following code which uses the Ably client library to generate and return a new Ably authentication token:

```javascript
import Ably from "ably/promises";

export const GET = async (req, res) => {
  const client = new Ably.Rest(import.meta.env.VITE_ABLY_API_KEY);
  const tokenRequestData = await client.auth.createTokenRequest({ clientId: 'ably-demo' });
  console.log(`Request: ${JSON.stringify(tokenRequestData)}`)
  return res.json(tokenRequestData);
}
```
Inside of `transcription.js` add the following code which uses Nodes `fetch` function to fetch and return a new AssemblyAI authentication token:

```javascript
export const GET = async (req, res) => {
  const response = await fetch("https://api.assemblyai.com/v2/realtime/token", {
    method: "POST",
    body: JSON.stringify({ expires_in: 3600 }),
    headers: {
      Authorization: import.meta.env.VITE_ASSEMBLYAI_API_KEY,
   "Content-Type": "application/json",
    },
  });
  const token = await response.json();
  console.log(token);
  return res.json(token);
};
```
Test your new API endpoints by loading the realtime ([http://localhost:5173/api/token/realtime/](http://localhost:5173/api/tokens/realtime/)) and transcription ([http://localhost:5173/api/token/realtime/](http://localhost:5173/api/tokens/transcription/)) routes in your browser. You should see responses similar to those seen in the images below:

![Testing your API](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/3xmiy1p86s7t1d2lktmi.png)


![API Token Responses](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/kuji395tvf5dzy3f0ghi.png)

## Create the publisher component

Now that we can generate tokens, we’ll implement the publisher component of the application. Remember that the publisher's job is to acquire realtime audio, send it to AssemblyAI for transcription, and then broadcast the transcription results to subscribers.

Start by adding some markup to the Publisher component:

```javascript
function Publisher() {
  return (
    <div>
      <p id="real-time-title">Click start to begin recording!</p>
      <button id="start">Start</button>
      <button id="stop">Stop</button>
      <p id="message"></p>
    </div>
  );
}
```
This markup adds two buttons to start and stop publishing and an area to display the transcription.

![Two buttons to start and stop publishing](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/iph3wzznjo6b7982v7s6.png)

Next, add a variable to store an audio recorder object (which we will create in a future step) and button click handler functions for the start and stop buttons.

```javascript
const texts = {};
let recorder = null;

async function start(e) {}
async function stop(e) {}
```
Connect those handler functions to the buttons using their `onClick` event:

```javascript
<button id="button" onClick={start}>Start</button>
<button id="button" onClick={stop}>Stop</button>
```

### Open an AssemblyAI WebSocket connection

AssemblyAI uses WebSockets as the transport for receiving raw audio and returning its transcription results. To use the AssemblyAI WebSocket endpoint in our React application we’ll use the `react-use-websocket` library. This library gives us a React-idiomatic way of using WebSockets.

Install `react-use-websocket` using NPM:

```
npm install react-use-websocket
```
Once installed, add another import statement into `Publisher.jsx` to import the `useWebSocket` object from the library:

```javascript
import useWebSocket from "react-use-websocket";
```
Add two React state properties, one to store the WebSocket URL and a second to toggle opening and closing the WebSocket connection:

```javascript
const [socketUrl, setSocketUrl] = useState('');
const [shouldConnect, setShouldConnect] = useState(false);
```

Notice that the default for the `shouldConnect` property is `false`. By default, the `react-use-websockets` library will immediately try to open a WebSocket connection. In our application, however, we only want to open that connection when the Start button is clicked.

Next, use the `useWebSocket` hook to configure the WebSocket connection.

```javascript
const { sendJsonMessage, lastMessage, readyState } = useWebSocket(socketUrl,
{
  onOpen: () => {
    console.log('AssemblyAI WebSocket opened');
  }
}, shouldConnect);
```
The `useWebSocket` hook accepts three parameters.

- url - The WebSocket URL we want to connect to.
- options - An object containing WebSocket configuration options.
- connect - A boolean value allowing us to control the library's automatic connection behavior.

Notice that our options object is defining a function for the `onOpen` event. We’ll use this event in just a moment to start recording realtime audio once the WebSocket connection opens.

In the start click handler add code that fetches a new AssemblyAI authentication token, appends that token to the WebSocket URL, and then toggles the `shouldConnect` value. This last step will trigger the `useWebSocket` hook to try to open a connection to the WebSocket URL.

```javascript
async function start(e) {
  const response = await fetch("/api/tokens/transcription");
  const { token } = await response.json();
  setSocketUrl(`wss://api.assemblyai.com/v2/realtime/ws?sample_rate=16000&token=${token}`);
  setShouldConnect(true);
}
```

### Capturing audio in the browser

If the WebSocket connection opens successfully we can start to capture realtime audio and send it to AssemblyAI for transcription. To capture audio we’ll use the browsers built in getUserMedia function. The `getUserMedia` function is part of the [MediaDevices](https://developer.mozilla.org/en-US/docs/Web/API/MediaDevices) set of APIs that browsers implement in order to provide developers with safe and standardized ways to access local media hardware devices like microphones, cameras, and speakers.

Calling getUserMedia, in our case, asking only to access audio, returns a [MediaStream](https://developer.mozilla.org/en-US/docs/Web/API/MediaStream) with tracks containing the requested types of media. Once we have the MediaStream we’ll use the [RecordRTC](https://recordrtc.org/) library to transform the raw audio track into a format that we can pass to AssemblyAI. RecordRTC is an abstraction over the native [MediaRecoder](https://developer.mozilla.org/en-US/docs/Web/API/MediaRecorder) APIs and provides a number of capabilities that simplify recording audio in the browser.

Install `recordrtc` using NPM:

```
npm install recordrtc
```
Once installed add an import statement importing the `RecordRTC` and `StereoAudioRecorder` objects from the library:

```javascript
import RecordRTC, { StereoAudioRecorder } from "recordrtc";
```

Inside the `onOpen` event handler call `getUserMedia` to access an audio MediaStream. Pass that stream to a new instance of a RecordRTC object. Based on the audio requirements of [AssemblyAI’s realtime streaming API](https://www.assemblyai.com/docs/guides/real-time-streaming-transcription#audio-requirements), configure the instance of the RecordRTC object to encode 250ms of audio as a single channel track using the [pcm](https://en.wikipedia.org/wiki/Pulse-code_modulation) codec at a sample rate of 16000Hz.

```javascript
onOpen: () => {
  console.log("AssemblyAI WebSocket opened");
  navigator.mediaDevices
      .getUserMedia({ audio: true })
      .then((stream) => {
        recorder = new RecordRTC(stream, {
        type: "audio",
        mimeType: "audio/webm;codecs=pcm", // endpoint requires 16bit PCM audio
        recorderType: StereoAudioRecorder,
        timeSlice: 250, // set 250 ms intervals of data that sends to AAI
        desiredSampRate: 16000,
        numberOfAudioChannels: 1, // realtime requires only one channel
        bufferSize: 16384,
        audioBitsPerSecond: 128000,
        ondataavailable: (blob) => {
          const reader = new FileReader();
          reader.onload = () => {
            const base64data = reader.result;
       // audio data must be sent as a base64 encoded string
           sendJsonMessage({
             audio_data: base64data.split("base64,")[1],
          });
        };
        reader.readAsDataURL(blob);
      },
    })
    recorder.startRecording();
  })
}
```
Each time RecordRTC records 250ms of data it calls the `ondatavailable` event passing the recorded audio as a [`Blob`](https://developer.mozilla.org/en-US/docs/Web/API/Blob). Use this event to base64 encode the audio and use the `sendJsonMessage` function of `useWebSockets` to send the encoded data to AssemblyAI.

Finally, after creating the new `RecordRTC` object, call the `startRecording` function which triggers the recording of audio and the process of converting and sending it to AssemblyAI.

### Pause the recorder

Starting the recording and transcription process is great, but at some point, you’ll probably want to stop it as well. To do that use the stop button click handler to toggle the shouldConnect value to false, closing the WebSocket connection and pause the RecordRTC recorder.

```javascript
async function stop(e) {
  setShouldConnect(false);
  if (recorder)
    recorder.pauseRecording()
}
```
Let’s make sure the application is working as expected. Go ahead and try pressing the Start button. You may be prompted to give your application permission to access the microphone. If you are, go ahead and give permission.

### Transcription using AssemblyAI

At this point our application is now capable of sending audio to AssemblyAI for transcription. AssemblyAI will perform that transcription and begin sending it back via the same WebSocket connection transcription data.

Start by adding a new state property to store transcribed text:

```javascript
const [transcription, setTranscription] = useState('');
```
And use that state property to display the text on our page:

```javascript
<p id="message">{transcription}</p>
```
As AssemblyAI transcribes the audio we’re sending, it uses the open WebSocket connection to send back to us messages containing partial transcription results. Those messages look roughly like this:

```
{
  "message_type":"PartialTranscript",
  "created":"2023-10-13T18:36:02.574861",
  "audio_start":0,
  "audio_end":2230,
  "confidence":0.805910792864326,
  "text":"this is a test",
  "words":[
    {"start":1430,"end":1465,"confidence":0.999489806225809,"text":"this"},
    {"start":1470,"end":1505,"confidence":0.676980979301034,"text":"is"},
    {"start":1910,"end":1945,"confidence":0.763494430232322,"text":"a"},
    {"start":1950,"end":1985,"confidence":0.783677955698138,"text":"test"}
  ]
}
```
As AssemblyAI continues to receive additional audio, it may send back partial responses containing some or all of the same audio samples. This means that we have to create a way to detect if we need to replace an existing partial response or if we’ve received a final response for a particular time slide of audio.

We’ll do that by using the `audio_start` value contained in each message and creating a function called `processPartialTranscription` that can store and update messages based on that value.

```javascript
const processPartialTranscription = (source) => {
  let msg = "";
  const res = JSON.parse(source.data);
  if (res.text) {
    texts[res.audio_start] = res.text;
    const keys = Object.keys(texts);
    keys.sort((a, b) => a - b);
    for (const key of keys) {
      if (texts[key]) {     
        msg += ` ${texts[key]}`;
      }
    }
  }
  return msg;
}
```
Each message is added to a dictionary using the audio_start value as the key. If that key already exists then its content is overwritten. Once the data is added to the dictionary it is sorted by its keys (in other words in time order). The sorted keys are looped over to create a single transcription result string.

To receive the WebSocket messages from AssemblyAI the useWebSockets hook gives us access to the last incoming message through the `lastMessage` property. We can use an Effect based on the `lastMessage` property to update the transcription text.

Import `useEffect` from React:

```javascript
import { useState, useEffect } from "react";
```
Create a new Effect which checks if lastMessage is not empty and if not passes the last message through the processing function before setting the value to the `transcription` property:

```javascript
useEffect(() => {
  if (lastMessage !== null) {
    setTranscription(processPartialTranscription(lastMessage));
  }
}, [lastMessage]);
```
Run the app. Start the recording and watch as text transcription shows up.


## Broadcasting transcription messages

So far in our application we’ve done everything as the Publisher, meaning the person creating the audio is also receiving the transcription text. But closed-captioning is about broadcasting the audio transcription to a wide audience.

AssemblyAI only gives us a single WebSocket connection. To fan out to many transcription consumers we’ll use Ably.

Start by importing the Ably client library into the App component:

```javascript
import { AblyProvider } from "ably/react";
import * as Ably from 'ably/promises';
```

Create a new connection to Ably:

```javascript
const client = new Ably.Realtime.Promise({ authUrl:'/api/tokens/realtime' });
```
Enclose the Routes component with an `AblyProvider`, setting its `client` property to the Ably client you just created. The AblyProvider holds and manages the Ably WebSocket connection for you and makes it accessible to any child component:

```javascript
<AblyProvider client={ client }>
  <Routes>
    <Route path="/publisher" element={<Publisher />} />
    <Route path="/subscriber" element={<Subscriber />} />
  </Routes>
</AblyProvider>
```
To broadcast the transcribed messages from the Publisher to Subscribers we need to access an Ably Channel and publish messages to it. The Ably client library includes a set of React hooks that make using Ably in React applications simple.

In the Publisher component import the `useChannel` object:

```javascript
import { useChannel } from "ably/react";
```
Within the component use the `useChannel` hook to access an Ably channel:

```javascript
const { channel } = useChannel('closed-captions');
```
Finally, publish messages to that channel by calling the channels `publish` method from inside of the `processPartialTranscription` function:

The application should now begin to publish the JSON formatted transcription data that AssemblyAI is sending the publisher. You can validate that this is happening correctly by logging into the Ably dashboard and connecting to the `closed-captions` channel using the Dev Console.

![Connecting to the closed-captions channel](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/a2cahsw8et6pyxquf8k7.png)

## Create a subscriber component

With messages being published to Ably, we can shift gears to the Subscriber component. We’ll want the application to consume and display the messages being published into the `closed-captions` channel.

Import the `useState` hook from React and Ably `useChannel` hook into the Subscriber component:

```javascript
import { useState } from 'react';
import { useChannel } from "ably/react";
```
Add a variable to hold partial snippets and a state value to hold the current transcription message:

```javascript
const texts = {};

const [transcription, setTranscription] = useState('');
```
Add some basic HTML markup that defines two buttons and a place to display the transcription text:

```javascript
return (
  <>
    <div>
      <p id="real-time-title">Listening for transcription messages!</p>
      <p id="message">{transcription}</p>
    </div>
  </>
);
```
Add the `processPartialTranscription` method:

```javascript
const processPartialTranscription = (source) => {
 let msg = "";

 const res = JSON.parse(source.data);
 if (res.text) {
  texts[res.audio_start] = res.text;
  const keys = Object.keys(texts);
  keys.sort((a, b) => a - b);
  for (const key of keys) {
   if (texts[key]) {     
    msg += ` ${texts[key]}`;
   }
  }
 }
```
`useChannel` has a callback function that gets called when new messages are received. Use that to receive and process incoming transcription messages and update the `transcriptions` state value:

```javascript
const { channel } = useChannel("closed-captions", (message) => {
 setTranscription(processPartialTranscription(message));
});
```
That’s it! Open the application in two separate browser windows, one with the Publisher route loaded and the other with the Subscriber route loaded. In the publisher view, start the recorder and watch as transcription text begins to show in both the publisher and subscriber views.

## Wrap up

Congrats on building this realtime closed-captioning application! In building his application you learned how easy it is to capture realtime audio from the browser, send it to AssemblyAI for transcribing and broadcast the transcription to an audience.

Combining AssemblyAI’s realtime transcription with Ably’s realtime message broadcasting gives you a powerful set of tools for your developer toolbelt.

If you’re looking to take this application further, one idea is to allow users of your application to select a language in which to receive the transcribed text, using a service like [https://languageio.com/](https://languageio.com/) to translate the text in realtime.

I'm always interested in how developers are combining tools like AssemblyAI or Language.io  with Ably.  Tell me how you are mashing tools together by hitting me up on [X (Twitter)](http://twitter.com/ablydevin)