How To Build an AI-Powered Voice Assistant With Twilio, Laravel, and OpenAI
Tags: ai, openai, laravel, php
Published: 2024-06-14 17:06:30

Voice assistants, such as Amazon Alexa and Apple's Siri have become integral to people’s lives, as they're so helpful with mundane tasks, such as setting reminders and turning on smart home devices. However, most voice assistants struggle with complex questions and queries, leaving users disappointed.

In this tutorial, you will learn how to build an AI-powered voice assistant that can understand and respond to complex questions using Twilio Programmable Voice and OpenAI.

## Prerequisites
To complete this tutorial, you will need the following:
- A Twilio account, free or paid. If you don't have one, [create a Twilio account for free](https://www.twilio.com/try-twilio).
- A phone number you own. [Add it to your Twilio account](https://help.twilio.com/articles/223180048-How-to-Add-and-Remove-a-Verified-Phone-Number-or-Caller-ID-with-Twilio?_ga=2.5842992.288769677.1718382100-1858507213.1708114003&_gl=1*p29jxb*_gcl_au*NzAwOTQ1MDkzLjE3MTU5NjkwMjI.*_ga*MTg1ODUwNzIxMy4xNzA4MTE0MDAz*_ga_RRP8K4M4F3*MTcxODM4MjA5OC44My4xLjE3MTgzODIxNDcuMC4wLjA.#h_01GQT9YZMY444KNH3M5AK065GX).
- PHP 8.2 (or higher) installed
- [Composer](https://getcomposer.org/) installed globally
- An OpenAI account. [Create a free account here](https://openai.com/).
- [ngrok](https://ngrok.com/) and a free ngrok account
- Basic knowledge of the [Laravel](https://laravel.com/docs/10.x/installation) framework would be nice, although it’s not required

### Build the AI-powered voice assistant 

#### Create a new Laravel project

To create a new Laravel project using Composer, you need to run the command below in your terminal.
```bash
composer create-project laravel/laravel voice-assistant
```
Next, navigate to the project’s working directory and start the application development server by running the commands below in the terminal.
```bash
cd voice-assistant
php artisan serve
```
Once the application server is up, open http://localhost:8000/ in your browser to access the default Laravel welcome page, as shown in the image below.

![Laravel Welcome Page](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/d3qoq7s1jt2fz4emrnqs.png)

The next step is to open the project in your preferred IDE or text editor.
#### Install the Twilio PHP Helper Library
The [Twilio PHP Helper Library](https://www.twilio.com/docs/libraries/reference/twilio-php/) provides functionality for interacting with Twilio's APIs. With this library, you can easily interact with [Twilio Programmable Voice](https://www.twilio.com/docs/voice) in the application. Twilio Programmable Voice uses [Twilio Markup language(TwiML)](https://www.twilio.com/docs/voice/twiml), to specify the desired behaviour when receiving an incoming call or SMS. 
In your project’s working directory, run the command below in a new terminal window or tab to install the library.
```bash
composer require twilio/sdk
```
#### Retrieve your Twilio API credentials
You will need your **Account SID** and **Auth Token** to interact with Twilio Programmable Voice using the Twilio PHP Helper Library. You can find them in the **Account Info** panel on your [Twilio Console dashboard](https://console.twilio.com/?_ga=2.75769622.288769677.1718382100-1858507213.1708114003&_gl=1*135cmdu*_gcl_au*NzAwOTQ1MDkzLjE3MTU5NjkwMjI.*_ga*MTg1ODUwNzIxMy4xNzA4MTE0MDAz*_ga_RRP8K4M4F3*MTcxODM4MjA5OC44My4xLjE3MTgzODIxNDcuMC4wLjA.), as shown in the image below. Copy the respective values.

![Twilio Dashboard](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/2ph8jtd10btvb8mo8hrg.png)

#### Store the Twilio access token in your .env file
After retrieving your Twilio access tokens, you need to ensure that they are stored securely in your project using environment variables.
In your project’s root directory, locate the _.env_ file. This file is used to securely store sensitive data and other configuration data. Update it with the following:
```text
TWILIO_SID=<your_twilio_account_sid>
TWILIO_AUTH_TOKEN=<your_twilio_account_auth_token>
TWILIO_PHONE_NUMBER=<your twilio_account_phone_number>
```
Then, replace the placeholders with your corresponding Twilio access token values which you just copied.
#### Install the OpenAI Laravel Helper Library
OpenAI is a leading AI research lab known for creating advanced language models trained on extensive amounts of data. The OpenAI Laravel package allows you to easily use OpenAI models in your application. Install the package by running the command below.
```bash
composer require openai-php/laravel
```
Finally, set up the package configurations by executing the below command:
```bash
php artisan openai:install
```
This command will create a new file called _openai.php_ in the _config_ folder. The file contains the configurations required to connect to the OpenAI API. 
To use the OpenAI API, you'll need an API key for authentication. [Log in to your OpenAI account](https://platform.openai.com/docs/overview) and navigate to the **API Keys** section. Click **Create new secret key** to generate your key.

![OpenAI Dashboard](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/r5isu84a4t2r27xncxwy.png)
After retrieving the key, store it and add the below line to your _.env_ file
```text
OPENAI_API_KEY=<your_openAI_api_key>
```
Remember also to replace the placeholder with your corresponding API Key.
#### Update the routes
Run the command below in your terminal.

```php
php artisan install:api
```
This command will create a new file named _api.php_ in the _routes_ folder. Add the below `import` statement at the beginning of the new file.
```php
use App\Http\Controllers\VoiceController;
```
Then, update the file contents to include the following routes.
```php
Route::post('/voice', [VoiceController::class, 'voiceInput']);
Route::post('/chat', [VoiceController::class, 'speechtoText']);
```
Here you are defining the routes for the voice and chat endpoints.

**Note**: In Laravel, API [routes](https://laravel.com/docs/11.x/routing) defined in the api.php file have a prefix of `api/`. This means the route defined as `/voice` will be accessible at `/api/voice`. The same applies to `/chat`.

#### Create the controller
The next step is to create a controller class that will house all the application logic. To create the controller, run the command below in your terminal.
```bash
php artisan make:controller VoiceController
```
This command will create a new file called _VoiceController.php_ in the _app/Http/Controllers_ folder.
Open the _VoiceController.php_ file and replace the content with the following.

```php
<?php

namespace App\Http\Controllers;

use Illuminate\Http\Request;
use Twilio\TwiML\VoiceResponse;
use OpenAI\Laravel\Facades\OpenAI;

class VoiceController extends Controller
{
    private $response;

    public function __construct()
    {
        $this->response = new VoiceResponse();
    } 

    public function voiceInput(Request $request)
    {
        $gather = $this->response->gather(['input' => 'speech', 'action' => '/api/chat']);
        $gather->say('Hello, I am your personal assistant. How can I help you today?');

        return $this->response;
    }

    public function speechtoText(Request $request)
    {
        $result = OpenAI::chat()->create([
            'model' => 'gpt-3.5-turbo',
            'messages' => [
                ['role' => 'user', 'content' => $request->SpeechResult],
            ],
        ]);
        $this->response->say($result->choices[0]->message->content);
        $this->response->gather(['input' => 'speech', 'action' => '/api/chat']);

        return $this->response;
    }
}
```
In the code above, the necessary imports required for the voice assistant are added. In the `voiceInput()` function, the TwiML (Twilio Markup Language) `gather()` method listens for the caller's input, and specifies the action to take on receiving it. Next, the TwiML `say()` method is used to prompt the caller. The `gather()` method converts the audio received into text and adds the transcribed audio to the `$request` object using the parameter `speechResult`. 
After gathering the user's input, the `$request` object is sent as a POST request to the `/api/chat` endpoint, corresponding to the `speechToText()` function. The `speechToText()` function retrieves the transcribed audio from the request using the `speechResult` parameter and then forwards this transcribed audio to OpenAI Chat API for processing. 
OpenAI's Chat API requires a role parameter containing a message with a system role and content for the intended actions. The model must also be configured, it is set to `gpt-3.5-turbo` in this tutorial. Finally, TwiML is used to return the response from OpenAI to the user in audio format.
### Set up the Ngrok server
To make your voice assistant accessible from the internet, you can use a tool called [Ngrok](https://ngrok.com/). Ngrok allows you to expose your local Laravel server to the internet.
Run the command below in your project's root directory.
```bash
ngrok http 8000 
```
This command creates a secure tunnel that allows you to access your local Laravel server running on port 8000 from a public URL provided by Ngrok.
After executing the command, you should see the following in your terminal.

![Ngrok](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/19c48g9arq4mhgh82x2o.png)

Copy the Forwarding URL displayed here.
### Configure TwiML
#### Create a New TwiML App
Next, you need to create a new TwiML app. This is to ensure reusability across multiple phone numbers. You would be able to easily add the created TwiML app to any number in your Twilio account.
In your [Twilio Console](https://console.twilio.com/?_ga=2.5818416.288769677.1718382100-1858507213.1708114003&_gl=1*k7cuvg*_gcl_au*NzAwOTQ1MDkzLjE3MTU5NjkwMjI.*_ga*MTg1ODUwNzIxMy4xNzA4MTE0MDAz*_ga_RRP8K4M4F3*MTcxODM4MjA5OC44My4xLjE3MTgzODIxNDcuMC4wLjA.), navigate to **Explore Products > Voice > Manage > TwiML apps**, and select **Create new TwiML App**.

![twilio console](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/iskvpxtem45dgge4etah.png)

Then, fill out the form, as in the screenshot above, and replace `<your_ngrok_address>` with the Ngrok Forwarding URL you copied earlier, append "**/api/voice**" to the end of it, and click **Create**.
#### Add TwiML to your number
You must also add the created TwiML app to your phone number to ensure that Twilio executes your code when a call is made to/from the phone number.
In the Twilio Console, navigate to **Phone Numbers > Manage > Active numbers**, and click on your Twilio number.

![twilio console](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/ag1qsmd1bib268rpecwp.png)

Then, in the **Voice Configuration** section for your phone number, set **Configure with** to **TwiML App**, set **TwiML App** to your TwiML app, and then click **Save**.

![twilio console](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/owc1bjnkl3z5tjux255i.png)

### Test the application
It’s finally time to see your AI-powered voice assistant in action! You can test it by calling your Twilio phone number from your verified phone number.
When you successfully set up the system, a call will be made to your Twilio phone number. A voice prompt will ask for your request and the system will intelligently respond to it.
### What’s next for building an AI-powered voice assistant with Twilio, Laravel, and OpenAI?
Congratulations on successfully creating your very own AI-powered voice assistant using Twilio Programmable Voice and OpenAI! 

This comprehensive tutorial guided you through setting up the necessary tools, integrating Twilio and OpenAI, and building a voice assistant using the Laravel framework. Now, your voice assistant is ready to respond intelligently to any questions you might have.

So, what's next on your journey? Here are a few suggestions:

- **Integration with External APIs**: You can improve your voice assistant by integrating with external APIs. For example, you could integrate with e-commerce platforms to enable voice-based shopping experiences.
- **Interaction History**: You can also implement a feature that allows your voice assistant to remember previous interactions, and extract context from it. This would allow your voice assistant to provide more tailored and relevant responses.

Cheers to building and learning!