Configure Symbl.ai to Listen for Action Items in a WebSocket Call
Tags: javascript, ai, machinelearning, nlp
Published: 2021-05-28 21:59:35

Symbl.ai, a conversation intelligence platform for developers to create new experiences around conversation data, empowers developers to extend beyond mere automated speech recognition to contextual insights. Contextual insights are the results of Symbl.ai's Natural Language Processing algorithms to recognize speaker intentions. Among a few of the most common intentions speakers express in speech are follow-ups, questions or action items. 

In short, action-items as a conversational entity recognized by [Symbl.ai][Symbl.ai]'s platform are a reflection of a speaker's call to action during his or her conversation at any time. 

## First Step 

The first step to access action items as [Symbl.ai][Symbl.ai]'s contextual insights is to sign up. Register for an account at [Symbl.ai][Symbl.ai] (i.e., platform.symbl.ai). Grab both your `appId` and your `appSecret`. With both of those you receive your `x-api-key`.

If you would like to sign your JSON Web Tokens with your `appId` together with `appSecret` in a `cURL` command executed in the terminal, here is a code snippet. 
```
curl -k -X POST "https://api.symbl.ai/oauth2/token:generate" \
     -H "accept: application/json" \
     -H "Content-Type: application/json" \
     -d "{ \"type\": \"application\", \"appId\": \"<appId>\", \"appSecret\": \"<appSecret>\"}"
```

If you would like to authenticate into [Symbl.ai][Symbl.ai]'s developer platform to signing the JWTs in [Postman][postman], the added benefit is that Symbl.ai's public workspace includes almost all of its APIs, including a section called "Labs" for its most experimental developer APIs. 

## Second Step 

The second step is to familiarize yourself with [Symbl.ai][Symbl.ai]'s [documentation][documentation] on its [Streaming API][Streaming API]. To review, loop through each of the following items from [Symbl.ai][Symbl.ai]'s [live speech-to-text][live speech-to-text] tutorial: 1) creating a WebSocket, 2) setting its listeners, 3) creating an audio stream, or 4) handling the audio stream. If you wouldn't like to review these concepts, then paste the following copied code directly into your console: 

```js
/**
 * The JWT token you get after authenticating with our API.
 * Check the Authentication section of the documentation for more details.
 */
const accessToken = ""
const uniqueMeetingId = btoa("user@example.com")
const symblEndpoint = `wss://api.symbl.ai/v1/realtime/insights/${uniqueMeetingId}?access_token=${accessToken}`;

const ws = new WebSocket(symblEndpoint);

// Fired when a message is received from the WebSocket server
ws.onmessage = (event) => {
  // You can find the conversationId in event.message.data.conversationId;
  const data = JSON.parse(event.data);
  if (data.type === 'message' && data.message.hasOwnProperty('data')) {
    console.log('conversationId', data.message.data.conversationId);
  }
  if (data.type === 'message_response') {
    for (let message of data.messages) {
      console.log('Transcript (more accurate): ', message.payload.content);
    }
  }
  if (data.type === 'message' && data.message.hasOwnProperty('punctuated')) {
    console.log('Live transcript (less accurate): ', data.message.punctuated.transcript)
  }
  console.log(`Response type: ${data.type}. Object: `, data);
};

// Fired when the WebSocket closes unexpectedly due to an error or lost connetion
ws.onerror  = (err) => {
  console.error(err);
};

// Fired when the WebSocket connection has been closed
ws.onclose = (event) => {
  console.info('Connection to websocket closed');
};

// Fired when the connection succeeds.
ws.onopen = (event) => {
  ws.send(JSON.stringify({
    type: 'start_request',
    meetingTitle: 'Websockets How-to', // Conversation name
    insightTypes: ['question', 'action_item'], // Will enable insight generation
    config: {
      confidenceThreshold: 0.5,
      languageCode: 'en-US',
      speechRecognition: {
        encoding: 'LINEAR16',
        sampleRateHertz: 44100,
      }
    },
    speaker: {
      userId: 'example@symbl.ai',
      name: 'Example Sample',
    }
  }));
};

const stream = await navigator.mediaDevices.getUserMedia({ audio: true, video: false });

/**
 * The callback function which fires after a user gives the browser permission to use
 * the computer's microphone. Starts a recording session which sends the audio stream to
 * the WebSocket endpoint for processing.
 */
const handleSuccess = (stream) => {
  const AudioContext = window.AudioContext;
  const context = new AudioContext();
  const source = context.createMediaStreamSource(stream);
  const processor = context.createScriptProcessor(1024, 1, 1);
  const gainNode = context.createGain();
  source.connect(gainNode);
  gainNode.connect(processor);
  processor.connect(context.destination);
  processor.onaudioprocess = (e) => {
    // convert to 16-bit payload
    const inputData = e.inputBuffer.getChannelData(0) || new Float32Array(this.bufferSize);
    const targetBuffer = new Int16Array(inputData.length);
    for (let index = inputData.length; index > 0; index--) {
        targetBuffer[index] = 32767 * Math.min(1, inputData[index]);
    }
    // Send audio stream to websocket.
    if (ws.readyState === WebSocket.OPEN) {
      ws.send(targetBuffer.buffer);
    }
  };
};


handleSuccess(stream);
```

## Third Step 

The third step  [Symbl.ai][Symbl.ai]'s WebSocket in your browser. Open up an instance of Chrome. Open up the console. Paste the above copied code directly into your console. After hitting enter, your WebSocket's messages start to pile up. 

## Fourth Step 

After having run the [Streaming API][Streaming API] in the browser, you receive [Symbl.ai][Symbl.ai]'s transcription from automated speech recognition in real-time. However, [Symbl.ai][Symbl.ai] enables you as a developer to extend far beyond merely automated speech recognition to contextual insights. In the code you run in the browser, you configure the WebSocket's event listener to capture contextual insights in real-time. 

Navigate to the event listener `ws.onmessage`. Inside of the event listener, the `ws.onmessage` method provides you the ability to track events such as those events that are sent or received by the WebSocket. In the event listener there is a stream of raw data following on the WebSocket's protocol. 

```js
 if (data.type === 'topic_response') {
    for (let topic of data.topics) {
      console.log('Topic detected: ', topic.phrases)
    }
  }
  if (data.type === 'insight_response') {
    for (let insight of data.insights) {
      console.log('Insight detected: ', insight.payload.content);
       if (insight.type === "action_item" ) {
          console.log("Insight detected is an Action Item!!!")
      }
    }
  }
```

After adding the new log to your WebSocket's `ws.onmessage` method, the following is the full code: 

```js
/**
 * The JWT token you get after authenticating with our API.
 * Check the Authentication section of the documentation for more details.
 */
const accessToken = ""
const uniqueMeetingId = btoa("user@example.com")
const symblEndpoint = `wss://api.symbl.ai/v1/realtime/insights/${uniqueMeetingId}?access_token=${accessToken}`;

const ws = new WebSocket(symblEndpoint);

// Fired when a message is received from the WebSocket server
ws.onmessage = (event) => {
  // You can find the conversationId in event.message.data.conversationId;
  const data = JSON.parse(event.data);
  if (data.type === 'message' && data.message.hasOwnProperty('data')) {
    console.log('conversationId', data.message.data.conversationId);
  }
  if (data.type === 'message_response') {
    for (let message of data.messages) {
      console.log('Transcript (more accurate): ', message.payload.content);
    }
  }
if (data.type === 'topic_response') {
    for (let topic of data.topics) {
      console.log('Topic detected: ', topic.phrases)
    }
  }
  if (data.type === 'insight_response') {
    for (let insight of data.insights) {
      console.log('Insight detected: ', insight.payload.content);
       if (insight.type === "action_item" ) {
          console.log("Insight detected is an Action Item!!!")
      }
    }
  }
  if (data.type === 'message' && data.message.hasOwnProperty('punctuated')) {
    console.log('Live transcript (less accurate): ', data.message.punctuated.transcript)
  }
  console.log(`Response type: ${data.type}. Object: `, data);
};

// Fired when the WebSocket closes unexpectedly due to an error or lost connetion
ws.onerror  = (err) => {
  console.error(err);
};

// Fired when the WebSocket connection has been closed
ws.onclose = (event) => {
  console.info('Connection to websocket closed');
};

// Fired when the connection succeeds.
ws.onopen = (event) => {
  ws.send(JSON.stringify({
    type: 'start_request',
    meetingTitle: 'Websockets How-to', // Conversation name
    insightTypes: ['question', 'action_item'], // Will enable insight generation
    config: {
      confidenceThreshold: 0.5,
      languageCode: 'en-US',
      speechRecognition: {
        encoding: 'LINEAR16',
        sampleRateHertz: 44100,
      }
    },
    speaker: {
      userId: 'example@symbl.ai',
      name: 'Example Sample',
    }
  }));
};

const stream = await navigator.mediaDevices.getUserMedia({ audio: true, video: false });

/**
 * The callback function which fires after a user gives the browser permission to use
 * the computer's microphone. Starts a recording session which sends the audio stream to
 * the WebSocket endpoint for processing.
 */
const handleSuccess = (stream) => {
  const AudioContext = window.AudioContext;
  const context = new AudioContext();
  const source = context.createMediaStreamSource(stream);
  const processor = context.createScriptProcessor(1024, 1, 1);
  const gainNode = context.createGain();
  source.connect(gainNode);
  gainNode.connect(processor);
  processor.connect(context.destination);
  processor.onaudioprocess = (e) => {
    // convert to 16-bit payload
    const inputData = e.inputBuffer.getChannelData(0) || new Float32Array(this.bufferSize);
    const targetBuffer = new Int16Array(inputData.length);
    for (let index = inputData.length; index > 0; index--) {
        targetBuffer[index] = 32767 * Math.min(1, inputData[index]);
    }
    // Send audio stream to websocket.
    if (ws.readyState === WebSocket.OPEN) {
      ws.send(targetBuffer.buffer);
    }
  };
};


handleSuccess(stream);
```

# What's Next? 

If you would like to add a listener for real-time sentiment analysis to your [Symbl.ai][Symbl.ai] configurations, [Symbl.ai][Symbl.ai] provides you the ability to listen to polarity scores on sentiments from messages in real-time. A [basic knowledge of WebSockets][basic knowledge of WebSockets] is a first step. After [logging sentiments][logging sentiments], the next step is create a way to [capture the data in real-time][capture the data in real-time]. If you would like to skip these blogs, feel free to download the code from [Symbl.ai's GitHub][Symbl.ai's GitHub] where you can find the [real-time sentiment analysis][real-time sentiment analysis] repo with instructions. 

Sentiment analysis is only one way to handle the conversation data from Symbl.ai's enabled voice. Another way to handle Symbl.ai's enabled voice is to connect an API. In particular, `action_item`s enable developers to create automated workflows from the detected insights in real-time so that those detected insights appear in external Third Party SaaS dashboards in real-time. 

Imagine, for instance, creating JIRA tickets in real-time through a `POST` request after a [Symbl.ai][Symbl.ai]'s insight for an `action_item` is detected in a live conversation. With Symbl.ai, you as a developer are empowered to connect, transform or visualize conversations in ways not yet imagined or implemented in real-life, since Symbl.ai's core product extends far beyond merely automated speech recognition. 

# Join Our Community 

[Symbl.ai][Symbl.ai] invites developers to reach out to us via email at [`developer@symbl.ai`](mailto:developer@symbl.ai), join our [Slack][Slack] channels, participate in our [hackathons][hackathon], fork our [Postman public workspace][Postman public workspace], or `git clone` our repos at [Symbl.ai's GitHub]. 

[hackathon]: developer.symbl.ai/hackathon?utm_source=dev.to&utm_medium=referral&utm_campaign=devrel

[Postman public workspace]: https://www.postman.com/symbldotai/workspace/symbl-ai/overview?utm_source=dev.to&utm_medium=referral&utm_campaign=devrel

[live speech-to-text]: https://docs.symbl.ai/docs/streamingapi/tutorials/receive-ai-insights-from-your-web-browser?utm_source=dev.to&utm_medium=referral&utm_campaign=devrel

[basic knowledge of WebSockets]: https://medium.com/codex/basic-javascript-code-for-live-conversational-ai-with-its-websockets-api-449d787ddd7a?utm_source=dev.to&utm_medium=referral&utm_campaign=devrel

[logging sentiments]: https://javascript.plainenglish.io/real-time-sentiment-analysis-api-in-javascript-with-symbl-ai-9e39d50f0654?utm_source=dev.to&utm_medium=referral&utm_campaign=devrel

[capture the data in real-time]: https://dev.to/eric_giannini/real-time-sentiment-analysis-on-messages-14n3?utm_source=dev.to&utm_medium=referral&utm_campaign=devrel

[real-time sentiment analysis]: https://github.com/symblai/real-time-sentiment-analysis-with-websockets?utm_source=dev.to&utm_medium=referral&utm_campaign=devrel

[Streaming API]: https://docs.symbl.ai/docs/streamingapi/introduction/?utm_source=dev.to&utm_medium=referral&utm_campaign=devrel

[documentation]: https://docs.symbl.ai/docs/?utm_source=dev.to&utm_medium=referral&utm_campaign=devrel

[Symbl.ai]: https://symbl.ai?utm_source=dev.to&utm_medium=referral&utm_campaign=devrel

[Symbl.ai's GitHub]: https://github.com/symblai?utm_source=dev.to&utm_medium=referral&utm_campaign=devrel

[signup]: https://platform.symbl.ai/?_ga=2.63499307.526040298.1609788827-1505817196.1609788827?utm_source=dev.to&utm_medium=referral&utm_campaign=devrel

[slack]: https://join.slack.com/t/symbldotai/shared_invite/zt-4sic2s11-D3x496pll8UHSJ89cm78CA?utm_source=dev.to&utm_medium=referral&utm_campaign=devrel

[postman]: https://docs.symbl.ai/docs/developer-tools/postman/?utm_source=dev.to&utm_medium=referral&utm_campaign=devrel