Build Simple CLI-Based Voice Assistant with PyAudio, Speech Recognition, pyttsx3 and SerpApi
Tags: python, api, tutorial, programming
Published: 2022-11-28 14:05:38

## Intro

As you saw by the title, this is a demo project that shows a very basic voice-assistant script that can answer your questions in the terminal based on Google Search results. 

You can find the full code in the GitHub repository: [`dimitryzub/serpapi-demo-projects/speech-recognition/cli-based/`](https://github.com/dimitryzub/serpapi-demo-projects/tree/4789cf3c2a5cb7af6c29ba13a4222a0eec05eadf/speech-recognition/cli-based)

The follow-up blog post(s) will be about:
- Web-based solution using [Flask](https://flask.palletsprojects.com/en/2.2.x/), some HTML, CSS and Javascript.
- Android & Windows based solution using [Flutter](https://flutter.dev/) and [Dart](https://dart.dev/).

## What we will build in this blog post

💡Click on the image to open demo video.

<div align="center">
  <a href="https://www.youtube.com/watch?v=sY1NgZGCcFE">
    <img src="https://img.youtube.com/vi/sY1NgZGCcFE/0.jpg"  style="width:100%;" alt="SerpApi Voice Assistant Demo Usage Example">
  </a>
</div>


## Prerequisites

First, let's make sure we are in a different environment and properly install the libraries we need for the project. The hardest (possibly) will be to install `pyaudio`. 

### Virtual Environment and Libraries Installation

Before we start installing libraries, we need create and activate new environment for this project:

```bash
# if you're on Linux based systems
$ python -m venv env && source env/bin/activate
$ (env) <path>

# if you're on Windows and using Bash terminal
$ python -m venv env && source env/Scripts/activate
$ (env) <path>

# if you're on Windows and using CMD
python -m venv env && .\env\Scripts\activate
$ (env) <path>
```

||Explanation|
|-|-|
|`python -m venv env`|tells Python to run module ([`-m`](https://docs.python.org/2/using/cmdline.html#cmdoption-m)) [`venv`](https://docs.python.org/3/library/venv.html) and create a folder called `env`.|
|`&&`|Stands for AND.|
|`source <venv_name>/bin/activate`|will activate your environment and you'll be able to install libraries only in that environment.|

Now install all needed libraries:

```bash
pip install rich pyttsx3 SpeechRecognition google-search-results
```


Now to [`pyaudio`](https://pypi.org/project/PyAudio/). Please, keep in mind that `pyaudio` may throw an error while installing. An additional research may be needed on your end.

If you're on Linux, we need to [install some development dependencies to use `pyaudio`](https://stackoverflow.com/a/50409644/15164646):

```bash
$ sudo apt-get install -y libasound-dev portaudio19-dev
$ pip install pyaudio
```

If you're on Windows, it's simpler (tested with CMD and Git Bash):

```bash
pip install pyaudio
```

## Full Code

```python
import os
import speech_recognition
import pyttsx3
from serpapi import GoogleSearch
from rich.console import Console
from dotenv import load_dotenv

load_dotenv('.env')
console = Console()

def main():
    console.rule('[bold yellow]SerpApi Voice Assistant Demo Project')
    
    recognizer = speech_recognition.Recognizer()

    while True:
        with console.status(status='Listening you...', spinner='point') as progress_bar:
            try:
                with speech_recognition.Microphone() as mic:
                    recognizer.adjust_for_ambient_noise(mic, duration=0.1)
                    audio = recognizer.listen(mic)
                    
                    text = recognizer.recognize_google(audio_data=audio).lower()
                    console.print(f'[bold]Recognized text[/bold]: {text}')
                    
                    progress_bar.update(status='Looking for answers...', spinner='line')
                    params = {
                        'api_key': os.getenv('API_KEY'),
                        'device': 'desktop',
                        'engine': 'google',
                        'q': text,
                        'google_domain': 'google.com',
                        'gl': 'us',
                        'hl': 'en'
                    }

                    search = GoogleSearch(params)
                    results = search.get_dict()
                    
                    try:
                        if 'answer_box' in results:
                            try:
                                primary_answer = results['answer_box']['answer']
                            except:
                                primary_answer = results['answer_box']['result']
                            console.print(f'[bold]The answer is[/bold]: {primary_answer}')
                            
                        elif 'knowledge_graph' in results:
                            secondary_answer = results['knowledge_graph']['description']
                            console.print(f'[bold]The answer is[/bold]: {secondary_answer}')
                        else:
                            tertiary_answer = results['answer_box']['list']
                            console.print(f'[bold]The answer is[/bold]: {tertiary_answer}')
                        progress_bar.stop() # if answered is success -> stop progress bar.
                        
                        user_promnt_to_contiune_if_answer_is_success = input('Would you like to to search for something again? (y/n) ')
                        
                        if user_promnt_to_contiune_if_answer_is_success == 'y':
                            recognizer = speech_recognition.Recognizer()
                            continue # run speech recognizion again until `user_promt` == 'n'
                        else:
                            console.rule('[bold yellow]Thank you for cheking SerpApi Voice Assistant Demo Project')
                            break
                    except KeyError:
                        progress_bar.stop()
                        
                        error_user_promt = input("Sorry, didn't found the answer. Would you like to rephrase it? (y/n) ")
                        
                        if error_user_promt == 'y':
                            recognizer = speech_recognition.Recognizer()
                            continue # run speech recognizion again until `user_promt` == 'n'
                        else:
                            console.rule('[bold yellow]Thank you for cheking SerpApi Voice Assistant Demo Project')
                            break
                            
            except speech_recognition.UnknownValueError:
                progress_bar.stop()
                user_promt_to_continue = input('Sorry, not quite understood you. Could say it again? (y/n) ')
                
                if user_promt_to_continue == 'y':
                    recognizer = speech_recognition.Recognizer()
                    continue # run speech recognizion again until `user_promt` == 'n'
                else:
                    progress_bar.stop()
                    console.rule('[bold yellow]Thank you for cheking SerpApi Voice Assistant Demo Project')
                    break

                
if __name__ == '__main__':
    main()
```

### Code Explanation

Import libraries:

```python
import os
import speech_recognition
import pyttsx3
from serpapi import GoogleSearch
from rich.console import Console
from dotenv import load_dotenv
```

|Library|Purpose|
|-|-|
|[`rich`](https://github.com/Textualize/rich)|Python library for beautiful formatting in the terminal.|
|[`pyttsx3`](https://github.com/nateshmbhat/pyttsx3)|Python's Text-to-speech converter that works in offline.|
|[`SpeechRecognition`](https://github.com/Uberi/speech_recognition)|Python library to convert speech to text.|
|[`google-search-results`](https://github.com/serpapi/google-search-results-python)|SerpApi's Python API wrapper that parses data from 15+ search engines.|
|[`os`](https://docs.python.org/3/library/os.html)|To read secret environment variable. In this case it's SerpApi API key.|
|[`dotenv`](https://github.com/theskumar/python-dotenv)|To load your environment variable(s) (SerpApi API key) from `.env` file. `.env` file could renamed to any file: `.napoleon` `.` (dot) represents a environment variable file.|

Define `rich` [`Console()`](https://rich.readthedocs.io/en/stable/console.html). It will be used to prettify terminal output (animations, etc):

```python
console = Console()
```

Define `main` function where all will be happening:

```python
def main():
    console.rule('[bold yellow]SerpApi Voice Assistant Demo Project')

    recognizer = speech_recognition.Recognizer()
```

At the beginning of the function we're defining `speech_recognition.Recognizer()` and `console.rule` will create the following output:

```
───────────────────────────────────── SerpApi Voice Assistant Demo Project ─────────────────────────────────────
```

The next step is to create a while loop that will be constantly listening for microphone input to recognize the speech:

```python
while True:
    with console.status(status='Listening you...', spinner='point') as progress_bar:
        try:
            with speech_recognition.Microphone() as mic:
                recognizer.adjust_for_ambient_noise(mic, duration=0.1)
                audio = recognizer.listen(mic)
                
                text = recognizer.recognize_google(audio_data=audio).lower()
                console.print(f'[bold]Recognized text[/bold]: {text}')
```


|Code|Explanation|
|-|-|
|[`console.status`](https://github.com/Textualize/rich/blob/5f4e93efb159af99ed51f1fbfd8b793bb36448d9/rich/console.py#L1144-L1163)|A `rich` progress bar, it's used only for cosmetic purpose.|
|`speech_recognition.Microphone()`|To start picking input from the microphone.|
|[`recognizer.adjust_for_ambient_noise`](https://github.com/Uberi/speech_recognition/blob/353d4ef74a55082de1f32796601899506d1b8bc7/speech_recognition/__init__.py#L560-L567)|Intended to calibrate the energy threshold with the ambient energy level.|
|[`recognizer.listen`](https://github.com/Uberi/speech_recognition/blob/353d4ef74a55082de1f32796601899506d1b8bc7/speech_recognition/__init__.py#L636-L649)|To listen for actual user text.|
|[`recognizer.recognize_google`](https://github.com/Uberi/speech_recognition/blob/353d4ef74a55082de1f32796601899506d1b8bc7/speech_recognition/__init__.py#L859-L874)|Performs speech recognition using Google Speech Recongition API. `lower()` is to lower recognized text.|
|`console.print`|A `rich` `print` statement that allows to use text modification, such as adding **bold**, *italic* and so on.|

`spinner='point'` will produce the following output ([use `python -m rich.spinner` to see list of `spinners`](https://github.com/Textualize/rich/blob/5f4e93efb159af99ed51f1fbfd8b793bb36448d9/rich/console.py#L1157)):

![rich-loading-progress](https://user-images.githubusercontent.com/78694043/203987922-937460b4-2754-429a-af82-f8f7491cb33e.gif)

After that, we need to initialize SerpApi search parameters for the search:

```python
progress_bar.update(status='Looking for answers...', spinner='line') 
params = {
    'api_key': os.getenv('API_KEY'),  # serpapi api key   
    'device': 'desktop',              # device used for 
    'engine': 'google',               # serpapi parsing engine: https://serpapi.com/status
    'q': text,                        # search query 
    'google_domain': 'google.com',    # google domain:          https://serpapi.com/google-domains
    'gl': 'us',                       # country of the search:  https://serpapi.com/google-countries
    'hl': 'en'                        # language of the search: https://serpapi.com/google-languages
    # other parameters such as locations: https://serpapi.com/locations-api
}

search = GoogleSearch(params)         # where data extraction happens on the SerpApi backend
results = search.get_dict()           # JSON -> Python dict
```

`progress_bar.update` will, well, update `progress_bar` with a new `status` (text printed in the console), and `spinner='line'` will produce the following animation:

![rich-loading-line-progress](https://user-images.githubusercontent.com/78694043/204004803-6a8412eb-70f8-4060-82be-5df9ce076133.gif)


After that, the data extraction happens from Google search using SerpApi's [Google Search Engine API](https://serpapi.com/search-api). 

The following part of the code will do the following:

![image](https://user-images.githubusercontent.com/78694043/204011946-9f4ff6ea-20f3-4021-b57b-b783f7c2c71a.png)

```python
try:
    if 'answer_box' in results:
        try:
            primary_answer = results['answer_box']['answer']
        except:
            primary_answer = results['answer_box']['result']
        console.print(f'[bold]The answer is[/bold]: {primary_answer}')

    elif 'knowledge_graph' in results:
        secondary_answer = results['knowledge_graph']['description']
        console.print(f'[bold]The answer is[/bold]: {secondary_answer}')
    else:
        tertiary_answer = results['answer_box']['list']
        console.print(f'[bold]The answer is[/bold]: {tertiary_answer}')
    progress_bar.stop()  # if answered is success -> stop progress bar

    user_promnt_to_contiune_if_answer_is_success = input('Would you like to to search for something again? (y/n) ')

    if user_promnt_to_contiune_if_answer_is_success == 'y':
        recognizer = speech_recognition.Recognizer()
        continue         # run speech recognizion again until `user_promt` == 'n'
    else:
        console.rule('[bold yellow]Thank you for cheking SerpApi Voice Assistant Demo Project')
        break
except KeyError:
    progress_bar.stop()  # if didn't found the answer -> stop progress bar

    error_user_promt = input("Sorry, didn't found the answer. Would you like to rephrase it? (y/n) ")

    if error_user_promt == 'y':
        recognizer = speech_recognition.Recognizer()
        continue         # run speech recognizion again until `user_promt` == 'n'
    else:
        console.rule('[bold yellow]Thank you for cheking SerpApi Voice Assistant Demo Project')
        break
```


The final step is to handle error when no sound was picked up from the microphone:

```python
# while True:
#     with console.status(status='Listening you...', spinner='point') as progress_bar:
#         try:
            # speech recognition code
            # data extraction code
        except speech_recognition.UnknownValueError:
                progress_bar.stop()         # if didn't heard the speech -> stop progress bar
                user_promt_to_continue = input('Sorry, not quite understood you. Could say it again? (y/n) ')

                if user_promt_to_continue == 'y':
                    recognizer = speech_recognition.Recognizer()
                    continue               # run speech recognizion again until `user_promt` == 'n'
                else:
                    progress_bar.stop()    # if want to quit -> stop progress bar
                    console.rule('[bold yellow]Thank you for cheking SerpApi Voice Assistant Demo Project')
                    break
```

`console.rule()` will provide the following output:

```
───────────────────── Thank you for cheking SerpApi Voice Assistant Demo Project ──────────────────────
```


Add `if __name__ == '__main__'` idiom which protects users from accidentally invoking the some script(s) when they didn't intend to, and call the `main` function which will run the whole script:

```python
if __name__ == '__main__':
    main()
```
## Links

- [`rich`](https://github.com/Textualize/rich)
- [`pyttsx3`](https://github.com/nateshmbhat/pyttsx3)
- [`SpeechRecognition`](https://github.com/Uberi/speech_recognition)
- [`google-search-results`](https://github.com/serpapi/google-search-results-python)
- [`os`](https://docs.python.org/3/library/os.html)
- [`python-dotenv`](https://github.com/theskumar/python-dotenv)


<p style="text-align: center;">Join us on <a href="https://twitter.com/serp_api">Twitter</a> | <a href="https://www.youtube.com/channel/UCUgIHlYBOD3yA3yDIRhg_mg">YouTube</a></p>

<p style="text-align: center;">Add a  <a href="https://github.com/serpapi/public-roadmap/issues">Feature Request</a>💫 or a <a href="https://github.com/serpapi/public-roadmap/issues">Bug</a>🐞</p>