React Native Speech Recognition Tutorial
Tags: 100daysofcode, challenge, reactnative, mobile
Published: 2023-02-28 05:00:11

Native apps of Google and Apple process voice data on the device. However, they do not offer it to other developers. Luckily, Picovoice does. On day 40, we'll go over how to process voice data on device by using [Picovoice React Native SDK](https://picovoice.ai/docs/quick-start/picovoice-react-native/). Picovoice SDK combines [Porcupine Wake Word](https://picovoice.ai/platform/porcupine/) and [Rhino Speech-to-Intent](https://picovoice.ai/platform/rhino/) engines, enabling commands like `"Alexa, set timer for 5 minutes"` but even better! We'll use a custom hotword instead of Alexa, and voice commands will be processed with zero latency.

{% embed https://www.youtube.com/watch?v=Eal9vRYUBuk&feature=youtu.be&themeRefresh=1 %}
1. Install the latest Picovoice packages:
```
npm i @picovoice/react-native-voice-processor
npm i @picovoice/porcupine-react-native
npm i @picovoice/rhino-react-native
npm i @picovoice/picovoice-react-native
```
**2.Initialize the Speech Recognition Platform**
First, grab your `AccessKey` for free from the [Picovoice Console](https://console.picovoice.ai/).
To keep things straightforward,  we’re going to use the pre-trained hotword model Pico Clock and the pre-trained context model Clock for this tutorial. You can [download pre-trained models here](https://github.com/Picovoice/picovoice/tree/master/demo/react-native-clock). However, you can also train custom wake words and contexts on the [Picovoice Console](https://console.picovoice.ai/). 

Now you should have an `AccessKey`, a `Porcupine model` (.ppn file), and a `Rhino model` (.rhn file). 
Let's initialize a `PicovoiceManager` in your React Native app.

```
import {PicovoiceManager} from '@picovoice/picovoice-react-native';

async createPicovoiceManager() {
    const accessKey = "..."; // your Picovoice AccessKey
    try {
        this._picovoiceManager = await PicovoiceManager.create(
            accessKey,
            '/path/to/keyword.ppn',
            this._wakeWordCallback,
            '/path/to/context.rhn',
            this._inferenceCallback,
            (error) => {
              this._errorCallback(error.message);
            }
        );
    } catch (err) {
        // handle error
    }
}

_wakeWordCallback() {
    // wake word detected!
}

_inferenceCallback(inference) {
    // `inference` has the following fields:
    // (1) isUnderstood
    // (2) intent
    // (3) slots      
}
```
**3. Get Permission to Record Audio**
To get the permission on `iOS`, open your `Info.plist` and add the following line:
```
<key>NSMicrophoneUsageDescription</key>
<string>[Permission explanation]</string>
```
For Android, open your `AndroidManifest.xml` and add the following line:
```
<uses-permission android:name="android.permission.RECORD_AUDIO" />
```
Then check for permission before proceeding with audio capture:

```
let recordAudioRequest;
if (Platform.OS == 'android') {
    // For Android, we need to explicitly ask
    recordAudioRequest = this._requestRecordAudioPermission();
} else {
    // iOS automatically asks for permission
    recordAudioRequest = new Promise(function (resolve, _) {
    resolve(true);
    });
}

recordAudioRequest.then((hasPermission) => {
    if (!hasPermission) {
        console.error('Required microphone permission was not granted.');        
        return;
      }
    
    // start feeding Picovoice
    this._picovoiceManager?.start().then((didStart) => {
    if (didStart) {
      // let app know we're ready to go
    }
  });

});

async _requestRecordAudioPermission() {
    const granted = await PermissionsAndroid.request(
    PermissionsAndroid.PERMISSIONS.RECORD_AUDIO,
    {
        title: 'Microphone Permission',
        message: '[Permission explanation]',
        buttonNeutral: 'Ask Me Later',
        buttonNegative: 'Cancel',
        buttonPositive: 'OK',
    }
    );
    return (granted === PermissionsAndroid.RESULTS.GRANTED)
  }
```

Once `.start()` called, Picovoice is listening for the hotword “PicoClock” and follow-up commands.

**4. Controlling the App With Voice Inputs**
In the source code, you can find a simple clock app with three main components: a clock that shows the time, a timer, and a stopwatch.
Let's connect these three components to the Voice User Interface (VUI):

```
_wakeWordCallback(keywordIndex){    
  // turn mic blue to show we're listening
  this.setState({    
    isListening: true
  });
}

_inferenceCallback(inference) {
  var tab = this.state.activeTab;  
  if (inference.isUnderstood) {         
    if (inference.intent === 'clock') {
      // show clock
      tab = 'clock';
    } else if (inference.intent === 'timer') {
      // control timer operation
      this._performTimerCommand(inference.slots);
      tab = 'timer';
    } else if (inference.intent === 'setTimer') {
      // set timer duration
      this._setTimer(inference.slots);
      tab = 'timer';
    } else if (inference.intent === 'alarm') {
      // control alarm operation
      this._performAlarmCommand(inference.slots);
      tab = 'clock';
    } else if (inference.intent] === 'setAlarm') {
      // set alarm time
      this._setAlarm(inference.slots);
      tab = 'clock';
    } else if (inference.intent === 'stopwatch') {
      // control stopwatch operation
      this._performStopwatchCommand(inference.slots);
      tab = 'stopwatch';
    }
  }
  
  // change active tab and show we've stopped listening
  this.setState({
    activeTab: tab,
    isListening: false,
  });
}
```
Then connect each `intent` to a specific action taken in the app and pass in the `intent’s` `slots` as arguments.

```
_setTimer(slots) {
  var hours = 0;
  var minutes = 0;
  var seconds = 0;
  
  // parse duration
  if (slots['hours'] != null) {
    hours = Number.parseInt(slots['hours']);
  }
  if (slots['minutes'] != null) {
    minutes = Number.parseInt(slots['minutes']);
  }
  if (slots['seconds'] != null) {
    seconds = Number.parseInt(slots['seconds']);
  }
  
  // set timer
  this.setState({
    timerCurrentTime: moment.duration({
      hour: hours,
      minute: minutes,
      second: seconds,
      millisecond: 0,
    }),
    isTimerRunning: true,
  });
}
```
Voila! Once you connect all the functions with the VUI, you have a hands-free and cross-platform clock app.

Resources:
[Original article](https://medium.com/picovoice/add-voice-recognition-to-react-native-without-adding-the-cloud-af9e299336e4)
[Source Code](https://github.com/Picovoice/picovoice/tree/master/demo/react-native-clock)
[Picovoice React Native SDK](https://picovoice.ai/docs/quick-start/picovoice-react-native/)