How to Add Chat with Face Filters to Your Multiplayer Unity Game
Tags: unity3d, facefilter, banuba
Published: 2020-08-19 17:16:01


# How to Add Chat with Face Filters to Your Multiplayer Unity Game

Do you have any thoughts about real time engagement in your game? Undoubtedly, having video chat during a co-op or multiplayer online game adds jokes and laughs to the experience. How about extend the fun with a face filter? It will definitely increase a player’s session time during this COVID-19 lock down.

In this tutorial, we will make use of an existing project, written by Joel Thomas and apply a face filter feature to the player. To follow what has been done, please check out the tutorial [here](https://bit.ly/2CvfARA). Make sure you can download[ the full project from GitHub](https://github.com/icywind/agora-partychat-demo) to start. For comparison to the original project, consider the following preview:

![](https://cdn-images-1.medium.com/max/2000/1*AZEF1ssKToJkxaEwNfVl8Q.gif)

## Prerequisites

* [Create an Agora.io developer account here](https://www.agora.io/en/blog/how-to-get-started-with-agora?utm_source=medium&utm_medium=blog&utm_campaign=unity_party_chat) to get your AppID.

* [Create a Photon developer account](https://dashboard.photonengine.com/en-US/Account/SignUp) for their AppID.

* Clone [my GitHub project](https://github.com/icywind/agora-partychat-demo) as the starter project, which contains Viking demo’s assets, Agora Video SDK and Photon SDK.

* [Banuba Face SDK for Unity](https://assetstore.unity.com/packages/add-ons/machinelearning/face-ar-plugin-for-unity-153466).

* Unity 2019.3 or up.
> *You need to have a Banuba client token to run the project. Contact [info@banuba.com](mailto:info@banuba.com) for a trial token and SDK package if you don’t want to pay up front at the Asset Store.*

## Getting Started

First, we will use the Banuba Face SDK to start the AR face masking. Import the package that you downloaded from the Asset store or the unzipped SDK to the Unity project directory. You should see the current project directory structure and find the **BanubaClientToken** file in the Resource folder. Open the file in text editor and paste your SDK token key there.

![](https://cdn-images-1.medium.com/max/2000/1*iHW3fzhotYpZaVoWQLB32g.png)

Next, enter your Photon project info in *Window > Photon Unity Networking > PUN Wizard > Setup Project.* Paste your AppID or email you used for your Photon account.

![](https://cdn-images-1.medium.com/max/2000/1*FAzyyyOD9YpUUFwW3K9ZHA.png)
> Note that the GitHub project already contains a copy of the Agora Video SDK. You may choose update to the latest from the Asset Store. It is recommended that you delete the folder Assets/AgoraEngine/Plugins for a clean replacement.

### Create BNB_AR Prefab

Open the **Sample** scene from BanabuFaceAR/Scenes, and do the following:

1. Create an empty game object, name it “BNBARStuff”.

1. Drag and drop items *Camera, Faces and Surface *into **BNBARStuff.**

1. Change layers of **BNBARStuff** to “default” recursively.

1. Update the Camera component to culling mask on “default” only.

1. Drag the **BNBARStuff** object into th Prefabs folder to make it a prefab.

1. Open the prefab, rename *Camera* to “ARCamera”, and rename *Surface* to “ARSurface”. Change **ARCamera’s** culling mask to Default only. Uncheck *Default* from MainCamera’s culling mask list.

![](https://cdn-images-1.medium.com/max/2000/1*xNaafDIxKDetYAQCb3cHdg.gif)

If your Banuba SDK has a FPS game object under *Surface*, you can remove it from the prefab. Also update the code in CameraController to do a null check:

    **if (fpsTextObj != null)** fpsTextObj.text = “FPS: “ + fps;

Do NOT save the Sample scene. Open **VikingsScene** from DemoVikings and set up the BNBARStuff object using the prefab:

1. Drag and drop the **BNBARStuff** prefab to the *Hierarchy.*

1. Enable the one of the filters under *Faces* (e.g. *Afro*).

1. Highlight the Camera under **BNBARStuff** and play the app in Editor.

√ You can observe yourself with the AR filter applied in the Camera preview.

![](https://cdn-images-1.medium.com/max/2000/1*yJT5c6VZc01RdJTibEN-3w.png)

Save the **VikingsScene** scene if everything looks good so far.

### Create Render Texture

Create a render texture to host the *ARCamera* input:

1. Create a RenderTexture; name it “ARRenderTexture”.

1. Assign **ARRenderTexture** into *ARCamera’s* Target Texture field.

![](https://cdn-images-1.medium.com/max/2000/1*4m5RTQXSZGuwfd2txnnjlg.png)

### Create Local View Prefab

In the* Viking* scene, create a UI hierarchy as following:

1. Create a new Image, change its width to 140 and height to 140; name it “ViewContainerWithMask”. Add a Mask component to this object.

1. Add a new game object under **ViewContainerWithMask**, name it “AvatarRenderTarget”. Make sure its scale is set to 1.

1. Add a RawImage component to **AvatarRenderTarget**. Drag and drop the **ARRenderTexture** into the Texture field of the **RawImage** component.

1. Add a button that has no text and no image as a child object of **AvatarRenderTarget**, set Normal Color and Pressed Color’s alpha to 0 (transparent). Set a low alpha (e.g. 35) for the other states. This button will invoke the appearance of the **FilterDropdown**.

1. Attach this “AvatarViewController.cs” script to **AvatarRenderTarget** object.

<iframe src="https://medium.com/media/6079c4036e8cb7679ece68b3a33e837a" frameborder=0></iframe>

5. Assign **ViewContainerWithMask** object into the “Mask RectTan” field of the **AvatarViewController** component on the **AvatarRenderTarget** object.

Verify you set up as describe in Step 1–4. Drag the game object ViewContainerWithMask to the Prefab folder. Delete it from the *Vikings* Scene. Save.

![](https://cdn-images-1.medium.com/max/2000/1*4w_c8kxjKpKWXnZm6ExePg.gif)

### Set up dropdown list for the masks

In the **VikingsScene**, create a UI Dropdown and place it to the upper right corner. Make sure you set the Canvas scaler to “Scale with Screen”. Assign the dropdown’s layer to “UI” and name it “FilterDropdown”. Hide this from the scene for now, since it will be activated later only when user taps the self-video display.

![](https://cdn-images-1.medium.com/max/2000/1*IgjLULRTPnuey1ZgTDoCQw.png)

## Code Modification

### Spawn with Banuba FaceFilter enabled Prefab

In AgoraVideoChat.cs, update the *CreateUserVideoSurface* method with the following code snippet:

<iframe src="https://medium.com/media/5cd5f2688c3ebe6c75cb89d652be88d6" frameborder=0></iframe>

### Update Prefab Field Assignments

Open Assets/DemoVikings/Resources/Charprefab, updated the fields:

1. Assign the prefab *ViewContainerWithMask* into the “AvatarVideoPrefab” field (see line 2 in the above code snippet).

1. Update the AppID and Channel ID.

![](https://cdn-images-1.medium.com/max/2060/1*pp5aJG8ccif5_pRR9T_gLg.png)

### Send the Frames as Custom Source Video

Add the following code snippet to the end of the **AgoraVideoChat.cs** script. The code runs at every frame to capture what is displaying on the **ARCamera** and sends it as a video frame into the channel.

<iframe src="https://medium.com/media/5278e41d3ba3da89b91ec30ee32740b3" frameborder=0></iframe>

### Last Check

Since we created a bigger rectangle for Avatar image view (140 pt), we should also update Asset/Prefabs/UserVideo’s RectTransform to 140x140. Also, in **AvatarViewController**, we provided an option for zoom level. You may want to try 1.5x zooming for better view of yourself in that window. Here is the look after everything connects up:

![](https://cdn-images-1.medium.com/max/2000/1*Hev2HVPGc4BN64zKV2xEjA.png)

That’s it! With a face mask on, you look much cooler than before! Have fun playing with friends and talking to them at the same time with your cool new look!

## In Summary

* We connected to Agora’s network to display our video chat channel.

* We enabled other users to join our party, see their faces, and talk with them in real time.

* We took it one step further and let user to pick a face mask provided by the Banuba ARFace Filter SDK.

If you have any questions or hit a snag in the course of building your own networked group video chat, please feel free to reach out directly or via the Agora Slack Channel!

Check out the link to the full project (it is the *CompletedProject* branch of the same repo):[UnityVikingGameChatDemo](https://github.com/icywind/UnityVikingGameChatDemo/tree/CompletedProject)