Let’s build our smart robot NPC demo 🤖
Step 0: Get the project files
You can find the 👉 complete Unity project here
Step 1: Install Unity Sentis
The Sentis Documentation 👉 https://docs.unity3d.com/Packages/com.unity.sentis@latest
Open the Jammo project
Click Sentis Pre-Release package or go to Window > Package Manager, click the + icon, select “Add package by name…” and type “com.unity.sentis”
Set the version to 1.3.0
- Press the Add button to install the package.
Step 2: Install Sharp Transformers 💪
Sharp Transformers is a Unity plugin of utilities to run Transformer 🤗 models in Unity games.
We need to do it for the tokenization step.
Go to “Window” > “Package Manager” to open the Package Manager.
Click the ”+” in the upper left-hand corner and select “Add package from git URL”.
Enter the URL of this repository and click “Add”: https://github.com/huggingface/sharp-transformers.git
Step 3: Build the inference process 🧠
As described in Sentis documentation, to use Sentis to run a neural network in Unity, we need to follow these steps.
Use the Unity.Sentis namespace.
Load a neural network model file.
Create input for the model.
Create an inference engine (a worker).
Run the model with the input to infer a result.
Get the result.
In our case, we do all of this in SentenceSimilarity.cs file, that will be attached to our robot.
In awake, we:
Load our neural network.
Create an inference engine (a worker).
Create an operator, that will allow us to perform operations with tensors.
We have three functions:
- Encode: takes our player input (text), tokenizes it, and embeds it.
- SentenceSimilarityScores: calculate the similarity scores between the input embed (what the user typed) and the comparison embeds (the robot action list)
- RankSimilarityScores: get the most similar action and its index given the player input
Step 4: Build the Robot Behavior 🤖
We need to define the behavior of our robot.
The idea is that our robot has different possible actions and the choice of the actions will depend on most similar actions.
We need first to define the Finite State Machine, a simple classical AI technique where each State defines a certain behavior.
Then, we’ll make the utility function that will select the State hence the series of actions to perform.
The State Machine 🧠🤖
In a state machine, each state represents a behavior, for instance, moving to a column, saying hello, etc. Based on the state the agent is it will perform a series of actions.
In our case, we have 7 states:
The first thing we need to do is create an enum called State that contains each of the possible States:
Because we need to constantly check the state, we define the state machine into the Update() method using a switch system where each case is a state.
For each state case, we define the behavior of our agents, for instance in our state Hello, the robot must move towards the player, face him correctly, and then launch its Hello animation, then go back to an Idle State.
We have now defined the behavior for each different State. The magic here will come from the fact that’s the language model that will define what State is the closest to the Player input. And in the utility function, we call this state.
Let’s define the Utility Function 📈
Our action list looks like this:
The sentence is what will be embedded in the AI model.
The verb is the State
Noun (if any) is the object to interact with (Pillar, Cube, etc.)
This utility function will select the Verb and Noun associated with the sentence having the highest similarity score with the player input text.
But first, to get rid of a lot of strange input text, we need to have a similarity score threshold.
For example, if I say “Look at all the rabbits”, none of our possible actions are relevant. Hence instead of choosing the action with the highest score, we’ll call the State Puzzled which will animate the robot with a perplexed animation.
If the score is higher, then we’ll get the verb corresponding to a State and the noun (goalObject) if any.
We set the state corresponding to the verb. That will activate the behavior corresponding to it.
And that’s it, now we’re ready to interact with our robot!
Step 5: Let’s interact with our Robot 🤖
In this step, you just need to click on the play button in the editor. And you can prompt some orders and see the results!
That’s nice! But the best way to learn is to try things, break things and modify the demo.
(Optional) Improving the game: let’s add a new action
Adding a new action is quite easy. Let’s take an example:
Copy the YellowPillar game object and move it
Change the name to GreenPillar
Create a new material and set it to green
Place the material on GreenPillar
Now that we’ve placed the new game object, we need to add this possibility into the sentences and click on Jammo_Player.
In the list of actions click on the plus button and fill in this new action item:
- Add Go to the green column
- GoTo
- GreenColumn
And that’s it! You can easily iterate and add more actions to your game.
Again, don’t forget to test the similarity threshold, to see if you need to increase it or decrease it.
< > Update on GitHub