Build Your First Voice-Driven Web Application
Tags: webdev, typescript, angular, javascript
Published: 2020-07-24 16:14:45

Some time ago I had the goal to implement a web application to be able to write what I'm speaking automatically and allow to execute actions through my voice commands. I thought it was a good idea to provide a way to listen to some feedback from the application too. After doing quick research I discovered a couple of Web APIs to solve this problem.

In this post, I will explain how you can use modern Web APIs to add the nice capability to speak to your web application and allow it to respond to you. We will implement the app from scratch.

### What is an API?
API is the acronym for _Application Programming Interface_. According to [MDN](https://developer.mozilla.org/en-US/docs/Learn/JavaScript/Client-side_web_APIs/Introduction) website:

> APIs are constructs made available in programming languages to allow developers to create complex functionality more easily. 

In simple words, APIs provide a _way_ to create complex applications without having to learn or implement the details. 

![Alt Text](https://dev-to-uploads.s3.amazonaws.com/i/qjyk805punnh19iz6f2n.png)
<figcaption>API Diagram - Camera example</figcaption>

### Web APIs

Have you ever used _fetch_ or a _Service Worker_? Maybe you used or accessed the [DOM](https://developer.mozilla.org/en-US/docs/Web/API/Document_Object_Model) from JavaScript?

Well, you can accomplish complex tasks based on those features since they are part of an extensive list of [Web APIs](https://developer.mozilla.org/en-US/docs/Web/API). These APIs are not part of JavaScript, however, you can use them through this programming language(or any other JavaScript-based library/framework).

On the other hand, you may need to make sure a Web API is fully supported by your web browser before to start building an application based on it. For example, if you are planning to work with _fetch_, you can see what browsers or JavaScript engines [support it](https://caniuse.com/#search=fetch).

### The Web Speech API

![Alt Text](https://dev-to-uploads.s3.amazonaws.com/i/6pivk9bd7ylpqg9ndfsj.png)
<figcaption>Speech Recognition through the Web Speech API</figcaption>

As you can see in the previous image, this Web API can help you with the following:
* Generate speech-to-text output
* Uses speech recognition as input
* Supports continuous dictation(You can write a complete letter)
* Control interface for web browsers

For more details, please see the [Web Speech API](https://wicg.github.io/speech-api/) specification.

### The SpeechSynthesis Interface

![Alt Text](https://dev-to-uploads.s3.amazonaws.com/i/hak7o6576607w1pcihjv.png)
<figcaption>Text to Speech through the SpeechSynthesis interface</figcaption>

You got the idea with the above image. The _Web Speech Synthesis_ interface can generate a text-to-speech output.

Please refer to the [specification](https://wicg.github.io/speech-api/#tts-section) to learn more about this interface.

### Watch The Video
{% youtube PIbMsHciovQ %}
<figcaption>Demo: Voice-Driven Web Application</figcaption>

### Implement the Web Application

The application will be based on HTML, CSS, and TypeScript as the programming language. We'll use the latest Angular version through Angular Material components.
Also, We will define a _reactive programming_ approach using _Observables_ and the _AsyncPipe_ from Angular. Finally, we will provide a _Strategy pattern_  implementation among other features.

#### Creating the project
Let's create the web application from scratch using the latest [Angular CLI](https://cli.angular.io/):


```bash
ng new web-speech-angular --routing --style css --prefix wsa --strict
```


* `--routing`: Generates a _routing_ module for the project. 
* `--style`: The file extension for style files.
* `--prefix`: Set a prefix for the component selectors
* `--strict`: Available from [Angular 10](https://blog.angular.io/version-10-of-angular-now-available-78960babd41). Enable a stricter type checking and build optimization options.

#### Adding Angular Material
Adding Angular Material would be simple at this point:

```bash
ng add @angular/material
```

Now, we can follow the [Overall structural guidelines](https://angular.io/guide/styleguide#overall-structural-guidelines) from Angular to generate `shared` and `material` modules:

```bash
ng generate module shared --module app
ng generate module shared/material --module shared
```

These commands will generate the following structure in your project:

```text
|- src/
    |- app/
        |- shared/
            |- material/
                |- material.module.ts
            |- shared.module.ts
```


#### Adding the `web-speech` module
It's time to add a new module to define the components needed to display the controls of the app.

```bash
ng generate module web-speech --module app
ng generate component web-speech
```

Now we will have the following structure:

```text
|- src/
    |- app/
        |- shared/
        |- web-speech/
            |- web-speech.module.ts
            |- web-speech.component.ts|html|css
```

#### Adding the `web-apis` directory
Let's create a new folder to group services related to the Web APIs we are going to use. Also, let's define some TypeScript files for the languages, Notifications, Errors and Events to be supported by the new service.

```bash
ng generate service shared/services/web-apis/speech-recognizer
```

After run the previous command and create the model files, the structure will be as follows:

```text
|- src/
    |- app/
        |- shared/
            |- shared.module.ts
            |- services/
                |- web-apis/
                    |- speech-recognizer.service.ts
            |- model/
                |- languages.ts
                |- speech-error.ts
                |- speech-event.ts
                |- speech-notification.ts
        |- web-speech/
            |- web-speech.module.ts
            |- web-speech.component.ts|html|css
```

#### Modeling Notifications, Events and Errors
Since the current specification is written in JavaScript, we can provide some TypeScript code to take advantage of the _typing_. This is even more important since the project has been configured with the `strict` mode enabled for TypeScript.

```ts
// languages.ts
export const languages = ['en-US', 'es-ES'];
export const defaultLanguage = languages[0];

// speech-error.ts
export enum SpeechError {
  NoSpeech = 'no-speech',
  AudioCapture = 'audio-capture',
  NotAllowed = 'not-allowed',
  Unknown = 'unknown'
}

// speech-event.ts
export enum SpeechEvent {
  Start,
  End,
  FinalContent,
  InterimContent
}

// speech-notification.ts
export interface SpeechNotification<T> {
    event?: SpeechEvent;
    error?: SpeechError;
    content?: T;
}
```

Pay attention to `SpeechError` enum. The string keys match with actual values from the [SpeechRecognitionErrorEvent](https://wicg.github.io/speech-api/#speechreco-error) specification.

#### Creating the `SpeechRecognizerService` (Asynchronous Speech Recognition)

The main goal would be to define an abstraction of the functionality we'll need for the application:

* Define a basic configuration for the `SpeechRecognizerService`(a `webkitSpeechRecognition` instance which is supported by Google Chrome).
* Define a language configuration.
* _Catch_ interim and final results.
* Allow start and stop the recognizer service.

The following code provides an implementation for those requirements:

```ts
// speech-recognizer.service.ts
@Injectable({
  providedIn: 'root',
})
export class SpeechRecognizerService {
  recognition: SpeechRecognition;
  language: string;
  isListening = false;

  constructor() {}

  initialize(language: string): void {
    this.recognition = new webkitSpeechRecognition();
    this.recognition.continuous = true;
    this.recognition.interimResults = true;
    this.setLanguage(language);
  }

  setLanguage(language: string): void {
    this.language = language;
    this.recognition.lang = language;
  }

  start(): void {
    this.recognition.start();
    this.isListening = true;
  }

  stop(): void {
    this.recognition.stop();
  }
}
```

Now it is time to provide a _Reactive Programming_ oriented API to use _Observables_ for a continuous data flow. This will be helpful to "catch" the inferred text while the user is continuously talking(We won't need to pull values every time to see if there is something new).

```ts
export class SpeechRecognizerService {
  // previous implementation here...

  onStart(): Observable<SpeechNotification<never>> {
    if (!this.recognition) {
      this.initialize(this.language);
    }

    return new Observable(observer => {
      this.recognition.onstart = () => observer.next({
        event: SpeechEvent.Start
      });
    });
  }

  onEnd(): Observable<SpeechNotification<never>> {
    return new Observable(observer => {
      this.recognition.onend = () => {
        observer.next({
          event: SpeechEvent.End
        });
        this.isListening = false;
      };
    });
  }

  onResult(): Observable<SpeechNotification<string>> {
    return new Observable(observer => {
      this.recognition.onresult = (event: SpeechRecognitionEvent) => {
        let interimContent = '';
        let finalContent = '';

        for (let i = event.resultIndex; i < event.results.length; ++i) {
          if (event.results[i].isFinal) {
            finalContent += event.results[i][0].transcript;
            observer.next({
              event: SpeechEvent.FinalContent,
              content: finalContent
            });
          } else {
            interimContent += event.results[i][0].transcript;
            observer.next({
              event: SpeechEvent.InterimContent,
              content: interimContent
            });
          }
        }
      };
    });
  }

  onError(): Observable<SpeechNotification<never>> {
    return new Observable(observer => {
      this.recognition.onerror = (event) => {
        const eventError: string = (event as any).error;
        let error: SpeechError;
        switch (eventError) {
          case 'no-speech':
            error = SpeechError.NoSpeech;
            break;
          case 'audio-capture':
            error = SpeechError.AudioCapture;
            break;
          case 'not-allowed':
            error = SpeechError.NotAllowed;
            break;
          default:
            error = SpeechError.Unknown;
            break;
        }

        observer.next({
          error
        });
      };
    });
  }  
}
```

In the previous code we're writing _wrapper functions_ that return _Observables_ to manage the following event handlers:

```js
recognition.onstart = function() { ... }
recognition.onend = function() { ... }
recognition.onresult = function(event) { ... }
recognition.onerror = function(event) { ... }
```

To understand better how these functions work, please see the API Specification for [SpeechRecognition Events](https://wicg.github.io/speech-api/#speechreco-events), the [SpeechRecognitionResult](https://wicg.github.io/speech-api/#speechreco-result) and the [SpeechRecognitionErrorEvent](https://wicg.github.io/speech-api/#speechreco-error).

#### Working on the `WebSpeechComponent`
Since we have the `SpeechRecognizerService` already available, it's time to define the Angular Component:

```ts
// web-speech-component.ts
import { ChangeDetectionStrategy, Component, OnInit } from '@angular/core';
import { merge, Observable, Subject } from 'rxjs';
import { map, tap } from 'rxjs/operators';
import { defaultLanguage, languages } from '../shared/model/languages';
import { SpeechError } from '../shared/model/speech-error';
import { SpeechEvent } from '../shared/model/speech-event';
import { SpeechRecognizerService } from '../shared/web-apis/speech-recognizer.service';

@Component({
  selector: 'wsa-web-speech',
  templateUrl: './web-speech.component.html',
  styleUrls: ['./web-speech.component.css'],
  changeDetection: ChangeDetectionStrategy.OnPush,
})
export class WebSpeechComponent implements OnInit {
  languages: string[] = languages;
  currentLanguage: string = defaultLanguage; // Set the default language
  totalTranscript: string; // The variable to accumulate all the recognized texts

  transcript$: Observable<string>; // Shows the transcript in "real-time"
  listening$: Observable<boolean>; // Changes to 'true'/'false' when the recognizer starts/stops
  errorMessage$: Observable<string>; // An error from the Speech Recognizer
  defaultError$ = new Subject<undefined>(); // Clean-up of the previous errors

  constructor(private speechRecognizer: SpeechRecognizerService) {}

  ngOnInit(): void {
    // Initialize the speech recognizer with the default language
    this.speechRecognizer.initialize(this.currentLanguage);
    // Prepare observables to "catch" events, results and errors.
    this.initRecognition();
  }

  start(): void {
    if (this.speechRecognizer.isListening) {
      this.stop();
      return;
    }

    this.defaultError$.next(undefined);
    this.speechRecognizer.start();
  }

  stop(): void {
    this.speechRecognizer.stop();
  }

  selectLanguage(language: string): void {
    if (this.speechRecognizer.isListening) {
      this.stop();
    }
    this.currentLanguage = language;
    this.speechRecognizer.setLanguage(this.currentLanguage);
  }
}
```

In essence, the previous code shows how to define the main attributes and functions to accomplish:

* Allow switching the language for Speech Recognition.
* Know when the _SpeechRecognizer_ is "listening".
* Allow start and stop the _SpeechRecognizer_ from the component context.

The question now is: How can we get the transcript(what the user is talking in text) and how may I know when the Speech service is listening? Also, How we know if there is an error with the microphone or the API itself?

The answer is: Using the _Observables_ from the `SpeechRecognizerService`. Instead of using _subscribe_, let's get and assign the _Observables_ from the service, which will be used through the _Async Pipes_ in the template later.

```ts
// web-speech.component.ts
export class WebSpeechComponent implements OnInit {
  // Previous code here...
  private initRecognition(): void {

    // "transcript$" now will receive every text(interim result) from the Speech API.
    // Also, for every "Final Result"(from the speech), the code will append that text to the existing Text Area component.
    this.transcript$ = this.speechRecognizer.onResult().pipe(
      tap((notification) => {
        if (notification.event === SpeechEvent.FinalContent) {
          this.totalTranscript = this.totalTranscript
            ? `${this.totalTranscript}\n${notification.content?.trim()}`
            : notification.content;
        }
      }),
      map((notification) => notification.content || '')
    );

  // "listening$" will receive 'true' when the Speech API starts and 'false' when it's finished.
    this.listening$ = merge(
      this.speechRecognizer.onStart(),
      this.speechRecognizer.onEnd()
    ).pipe(
      map((notification) => notification.event === SpeechEvent.Start)
    );

  // "errorMessage$" will receive any error from Speech API and it will map that value to a meaningful message for the user
    this.errorMessage$ = merge(
      this.speechRecognizer.onError(),
      this.defaultError$
    ).pipe(
      map((data) => {
        if (data === undefined) {
          return '';
        }
        let message;
        switch (data.error) {
          case SpeechError.NotAllowed:
            message = `Cannot run the demo.
            Your browser is not authorized to access your microphone.
            Verify that your browser has access to your microphone and try again.`;
            break;
          case SpeechError.NoSpeech:
            message = `No speech has been detected. Please try again.`;
            break;
          case SpeechError.AudioCapture:
            message = `Microphone is not available. Plese verify the connection of your microphone and try again.`;
            break;
          default:
            message = '';
            break;
        }
        return message;
      })
    );
  }
}
```

#### The Template for the `WebSpeechComponent`
As we said before, the component's template will be powered by _Async Pipes_:

```html
<section>
  <mat-card *ngIf="errorMessage$| async as errorMessage" class="notification">{{errorMessage}}</mat-card>
</section>
<section>
  <mat-form-field>
    <mat-label>Select your language</mat-label>
    <mat-select [(value)]="currentLanguage">
      <mat-option *ngFor="let language of languages" [value]="language" (click)="selectLanguage(language)">
        {{language}}
      </mat-option>
    </mat-select>
  </mat-form-field>
</section>
<section>
  <button mat-fab *ngIf="listening$ | async; else mic" (click)="stop()">
    <mat-icon class="soundwave">mic</mat-icon>
  </button>
  <ng-template #mic>
    <button mat-fab (click)="start()">
      <mat-icon>mic</mat-icon>
    </button>
  </ng-template>
</section>
<section *ngIf="transcript$ | async">
  <mat-card class="notification mat-elevation-z4">{{transcript$ | async}}</mat-card>
</section>
<section>
  <mat-form-field class="speech-result-width">
    <textarea matInput [value]="totalTranscript || ''" placeholder="Speech Input Result" rows="15" disabled="false"></textarea>
  </mat-form-field>
</section>
```

At this point, the app is ready to enable the microphone and listen your voice!

#### Adding the `SpeechSynthesizerService`(Text-to-Speech)

Let's create the service first:

```bash
ng generate service shared/services/web-apis/speech-synthesizer
```

Add the following code into that file.
```ts
// speech-synthesizer.ts
import { Injectable } from '@angular/core';

@Injectable({
  providedIn: 'root',
})
export class SpeechSynthesizerService {
  speechSynthesizer!: SpeechSynthesisUtterance;

  constructor() {
    this.initSynthesis();
  }

  initSynthesis(): void {
    this.speechSynthesizer = new SpeechSynthesisUtterance();
    this.speechSynthesizer.volume = 1;
    this.speechSynthesizer.rate = 1;
    this.speechSynthesizer.pitch = 0.2;
  }

  speak(message: string, language: string): void {
    this.speechSynthesizer.lang = language;
    this.speechSynthesizer.text = message;
    speechSynthesis.speak(this.speechSynthesizer);
  }
}
```

Now the application will be able to talk to you. We can call this service when the application is ready to perform a voice-driven action. Also, we can confirm when the actions have been performed or even ask for parameters.

The next goal is to define a set of voice commands to perform actions over the application.

### Define the Actions through Strategies

Let's think about the main actions to be performed by voice commands in the application:
* The app can change the default theme by any other theme available from Angular Material.
* The app can change the title property of the application.
* At the same time, we should be able to attach every _final result_ over the existing Text Area component.

There are different ways to design a solution to this context. In this case, let's think to define some _strategies_ to change the theme and title of the application.

For now, _Strategy_ is our favorite keyword. After taking a look in the world of [Design Patterns](https://sourcemaking.com/design_patterns) it's clear we can use the [Strategy Pattern](https://sourcemaking.com/design_patterns/strategy) for the solution.

![Alt Text](https://dev-to-uploads.s3.amazonaws.com/i/xhlwzddx43fxd3scomk2.png)
<figcaption>Applying the Strategy Pattern</figcaption>

#### Adding the `ActionContext` Service and the Strategies
Let's create the `ActionContext`, `ActionStrategy`, `ChangeThemeStrategy` and `ChangeTitleStrategy` classes:

```bash
ng generate class shared/services/action/action-context
ng generate class shared/services/action/action-strategy
ng generate class shared/services/action/change-theme-strategy
ng generate class shared/services/action/change-title-strategy
```

```ts
// action-context.ts
@Injectable({
  providedIn: 'root',
})
export class ActionContext {
  private currentStrategy?: ActionStrategy;

  constructor(
    private changeThemeStrategy: ChangeThemeStrategy,
    private changeTitleStrategy: ChangeTitleStrategy,
    private titleService: Title,
    private speechSynthesizer: SpeechSynthesizerService
  ) {
    this.changeTitleStrategy.titleService = titleService;
  }

  processMessage(message: string, language: string): void {
    const msg = message.toLowerCase();
    const hasChangedStrategy = this.hasChangedStrategy(msg, language);

    let isFinishSignal = false;
    if (!hasChangedStrategy) {
      isFinishSignal = this.isFinishSignal(msg, language);
    }

    if (!hasChangedStrategy && !isFinishSignal) {
      this.runAction(message, language);
    }
  }

  runAction(input: string, language: string): void {
    if (this.currentStrategy) {
      this.currentStrategy.runAction(input, language);
    }
  }

  setStrategy(strategy: ActionStrategy | undefined): void {
    this.currentStrategy = strategy;
  }

  // Private methods omitted. Please refer to the repository to see all the related source code.
```

```ts
// action-strategy.ts
export abstract class ActionStrategy {
  protected mapStartSignal: Map<string, string> = new Map<string, string>();
  protected mapEndSignal: Map<string, string> = new Map<string, string>();

  protected mapInitResponse: Map<string, string> = new Map<string, string>();
  protected mapFinishResponse: Map<string, string> = new Map<string, string>();
  protected mapActionDone: Map<string, string> = new Map<string, string>();

  constructor() {
    this.mapFinishResponse.set('en-US', 'Your action has been completed.');
    this.mapFinishResponse.set('es-ES', 'La accion ha sido finalizada.');
  }

  getStartSignal(language: string): string {
    return this.mapStartSignal.get(language) || '';
  }

  getEndSignal(language: string): string {
    return this.mapEndSignal.get(language) || '';
  }

  getInitialResponse(language: string): string {
    return this.mapInitResponse.get(language) || '';
  }
  getFinishResponse(language: string): string {
    return this.mapFinishResponse.get(language) || '';
  }
  abstract runAction(input: string, language: string): void;
}
```

```ts
// change-theme-strategy.ts
@Injectable({
  providedIn: 'root',
})
export class ChangeThemeStrategy extends ActionStrategy {
  private mapThemes: Map<string, Theme[]> = new Map<string, Theme[]>();
  private styleManager: StyleManager = new StyleManager();

  constructor(private speechSynthesizer: SpeechSynthesizerService) {
    super();
    this.mapStartSignal.set('en-US', 'perform change theme');
    this.mapStartSignal.set('es-ES', 'iniciar cambio de tema');

    this.mapEndSignal.set('en-US', 'finish change theme');
    this.mapEndSignal.set('es-ES', 'finalizar cambio de tema');

    this.mapInitResponse.set('en-US', 'Please, tell me your theme name.');
    this.mapInitResponse.set('es-ES', 'Por favor, mencione el nombre de tema.');

    this.mapActionDone.set('en-US', 'Changing Theme of the Application to');
    this.mapActionDone.set('es-ES', 'Cambiando el tema de la Aplicación a');

    this.mapThemes.set('en-US', [
      {
        keyword: 'deep purple',
        href: 'deeppurple-amber.css',
      }
    ]);
    this.mapThemes.set('es-ES', [
      {
        keyword: 'púrpura',
        href: 'deeppurple-amber.css',
      }
    ]);
    }

  runAction(input: string, language: string): void {
    const themes = this.mapThemes.get(language) || [];
    const theme = themes.find((th) => {
      return input.toLocaleLowerCase() === th.keyword;
    });

    if (theme) {
      this.styleManager.removeStyle('theme');
      this.styleManager.setStyle('theme', `assets/theme/${theme.href}`);
      this.speechSynthesizer.speak(
        `${this.mapActionDone.get(language)}: ${theme.keyword}`,
        language
      );
    }
  }
}
```

```ts
// change-title-strategy.ts
@Injectable({
  providedIn: 'root',
})
export class ChangeTitleStrategy extends ActionStrategy {
  private title?: Title;

  constructor(private speechSynthesizer: SpeechSynthesizerService) {
    super();
    this.mapStartSignal.set('en-US', 'perform change title');
    this.mapStartSignal.set('es-ES', 'iniciar cambio de título');

    this.mapEndSignal.set('en-US', 'finish change title');
    this.mapEndSignal.set('es-ES', 'finalizar cambio de título');

    this.mapInitResponse.set('en-US', 'Please, tell me the new title');
    this.mapInitResponse.set('es-ES', 'Por favor, mencione el nuevo título');

    this.mapActionDone.set('en-US', 'Changing title of the Application to');
    this.mapActionDone.set('es-ES', 'Cambiando el título de la Aplicación a');
  }

  set titleService(title: Title) {
    this.title = title;
  }

  runAction(input: string, language: string): void {
    this.title?.setTitle(input);
    this.speechSynthesizer.speak(
      `${this.mapActionDone.get(language)}: ${input}`,
      language
    );
  }
}
```

Pay attention to the uses of `SpeechSynthesizerService` and the places where this service has been called. The moment you use the `speak` function, the app will use your speakers to answer you.

### Source Code and Live Demo

#### Source Code
Find the complete project in this GitHub repository: [https://github.com/luixaviles/web-speech-angular](https://github.com/luixaviles/web-speech-angular). Do not forget to give it a star ⭐️ or send a Pull Request if you decide to contribute with more features.

#### Live Demo
Open your Chrome web browser and go to [https://luixaviles.com/web-speech-angular/](https://luixaviles.com/web-speech-angular). Review the notes inside the app and test it in English or even Spanish.

#### Final Words
Even though the demo has been written using Angular and TypeScript, you can apply these concepts and Web APIs with any other JavaScript framework or library.

You can follow me on [Twitter](https://twitter.com/luixaviles) and [GitHub](https://github.com/luixaviles) to see more about my work.

Thank you for reading!  
— [Luis Aviles](https://luixaviles.com/)