Text to Speech + Image  — A Talkie in JS
Tags: react, webdev, rxjs, javascript
Published: 2020-07-07 16:41:45

In the previous part we created a website where users can generate GIF animations using Emoji, domain-specific language (DSL) and a Canvas. In this post we'll upgrade our animations to [talkies](https://en.wikipedia.org/wiki/Sound_film)!

![gif](https://dev-to-uploads.s3.amazonaws.com/i/u8rjn8djzjxh9v0n6g3k.gif)

<figcaption>example animation we achieved in part I</figcaption>

## Intro

I thought that it'd be funny to create animations where Emoji can talk. I already had Emoji moving around and displaying phrases as text. Obviously it was missing sound. In this article I'll show you how I added it!

> **tl;dr:** try this [animation](https://framd.cc/watch?v=5efb0e57edbeeb0007617386)
> ⚠️ warning: contains sound!

## Text-to-Speech

Accidentally I stumbled upon ["Text To Speech In 3 Lines Of JavaScript"](https://dev.to/asaoluelijah/text-to-speech-in-3-lines-of-javascript-b8h) article (thanks, @asaoluelijah!) and that _**"3 lines"**_ quickly migrated to my project.

```js
const msg = new SpeechSynthesisUtterance();
msg.text = 'Hello World';
speechSynthesis.speak(msg);
// ☝️ You can run this in the console, BTW
``` 

<figcaption>example taken from Asaolu Elijah's article. Go read it for more API details</figcaption>

_Surely "**3 lines**" turned out to be **80**. But I'll get to that later._

Text-to-Speech — is a part of browser Web Speech API that allows us to read text out loud and recognize speech.

**But before we can go further with adding Text-to-Speech to animation, I need to show you how I rendered animation in the first place.**

### Animation and RxJS

After parsing DSL and rendering it to canvas (see [part I](https://dev.to/kosich/text-to-gif-animation-react-pet-project-devlog-5eel)), I had an array of frames:

```js
[ { image: 'http://.../0.png' 
  , phrases: [ 'Hello!' ]
  , duration: 1000
  }
, { image: 'http://.../1.png' 
  , phrases: [ 'Hi!' ]
  , duration: 1000
  }
]
```

Each frame had a rendered `image`, `phrases` within it and frame `duration`.

To show the animation I used a React component with RxJS stream inside:

```jsx
import React, { useState, useEffect } from 'react';

function Animation({ frames }) {
  // state for current frame
  const [frame, setFrame] = useState(null);

  useEffect(() => {
    // turn array intro stream of arrays
    const sub = from(frames).pipe(
      // with each frame delayed by frame.duration
      delayWhen(frame => timer(frame.duration)),
      // mapped to an Image
      map(frame => <img src={frame.image} />)
    )
    .subscribe(setFrame);

    return () => sub.unsubscribe(); // teardown logic
  }, [frames]);

  return frame;
}
```
<figcaption>to simplify things, I'll use pseudocodish JS around the article</figcaption>

Here I use a `useEffect` hook to create a RxJS Observable and a subscription to it. The `from` function will iterate over the rendered `frames` array, `delayWhen` will delay each frame by `frame.duration` and `map` will turn each frame into a new `<img />` element. And I can easily loop the animation by simply adding a `repeat()` operator.

_Note that subscription has to be cancelled at some point (specially the endless `repeat()`): the component might be destroyed or the `frames` might change. So the function passed to `useEffect` hook needs to return a teardown callback. In this case I unsubscribe from the animation observable, effectively terminating the flow._

**With that covered, we can now discuss the Text-to-Speech!**

### Text-to-Speech and RxJS

Now I needed to pronounce the text using Speech API, but that `frame.duration` delay I used wouldn't work: I had to wait until the phrase is spoken and only then switch to the next frame. Also, if user edits the scenario or navigates away — I need to stop current synthesis. Happily, RxJS is ideal for such things!

First I needed to create an Observable wrapper around Speech Synthesis API:

```js
export function speak(text) {
  return new Observable((observer) => {
    // create and config utterance
    const utterance = new SpeechSynthesisUtterance();
    utterance.text = text;

    // subscribe our observer to utterance events
    utterance.onend = () => observer.complete();
    utterance.onerror = (err) => observer.error(err);

    // start the synthesis
    speechSynthesis.speak(utterance);

    return () => {
      speechSynthesis.cancel();
    }
  });
}
```

<figcaption>this is a shortened version, see <a href="https://github.com/kosich/rxjs-tts/blob/master/src/speak.ts">sources for more</a></figcaption>

When utterance will end Observable will complete, thus letting us chaining the synthesis. Also, if we unsubscribe from Observable — the synthesis will be stopped.

I've actually decided to publish this Observable wrapper as an **npm** package. There's a link in the footer 👇!

Now we can safely compose our phrases and be notified when they end:

```js
concat(
  speak('Hello'),
  speak('World')
)
  .subscribe({
    complete(){ console.log('done'); }
  });
```

_Try this code online at https://stackblitz.com/edit/rxjs-tts?file=index.ts_

And to integrate the Text-to-Speech back into our Animation component:

```jsx
from(frames).pipe(
  concatMap(frame => {
    // concat all phrases into a chain
    const phrases$ = concat(
        EMPTY,
        ...frame.phrases.map(text => speak(text))
    );
    
    // we'll wait for phrase to end
    // even if duration is shorter
    const duration$ = merge(
        phrases$,
        timer(frame.duration)
    );

    // to acknowledge the duration we need to merge it
    // while ignoring it's values
    return merge(
        of(<img src={frame.image} />),
        duration$.pipe(ignoreElements())
    );
  })
)
```

Thats it! Now our Emoji can walk and talk!

Turn the volume up and try this ["Dancing"](https://framd.cc/watch?v=5efb0e57edbeeb0007617386) animation

{% vimeo 436201143 %}
<figcaption>a recorded version of <a href="https://framd.cc/watch?v=5efb0e57edbeeb0007617386">"Dancing"</a></figcaption>

And surely try [creating your own](framd.cc) 🙂

## Outro

It was pretty simple, huh?

But there was a hidden trick: previously the web app was hosted on GitHub pages and users shared their animations using downloaded GIFs. But GIF cannot contain sound, you know... so I needed another way for users to share animations.

In the next article I'll share details on how I migrated the create-react-app to NextJS/Vercel platform and added MongoDB to it.

Have a question or idea? Please, share your thoughts in the comments!

**Thanks for reading this and see you next time!**

❤️ 🦄 📖

## Links

- Web Speech API  
      
    https://developer.mozilla.org/en-US/docs/Web/API/Web_Speech_API

- RxJS Text-to-Speech wrapper npm package

    `npm i rxjs-tts`

    https://github.com/kosich/rxjs-tts

- My twitter (in case you want to follow 🙂)

    https://twitter.com/kddsky