Building a realtime React VR app
Tags: reactvr, pusher
Published: 2018-02-23 01:30:16

---
title: Building a realtime React VR app
published: true
description: 
tags: #react-vr #pusher
canonical_url: https://blog.pusher.com/building-a-realtime-react-vr-app/
---

React VR is a library that allows you to write virtual reality apps for the web using JavaScript and React on top of the [WebVR API](https://webvr.info). This specification is now supported by the latest (or in some cases experimental) versions of browsers such as Chrome, Firefox, and Edge, and you don’t need to have a headset to access VR in a browser.

[WebVR Experiments](https://experiments.withgoogle.com/webvr) is a site that showcases some projects that show you what’s possible with WebVR. One project that caught my attention was [The Musical Forest](https://forest.webvrexperiments.com/), made by the awesome people of Google Creative Lab using [A-Frame](https://aframe.io/), a web framework for WebVR developed by the [Mozilla VR team](https://vr.mozilla.org/).

In the Musical Forest, hitting a shape triggers a note, and using WebSockets, people can play music together in realtime. However, due to all the features and technologies used, the app is somewhat complicated (you can find the source code [here](https://github.com/googlecreativelab/webvr-musicalforest)). So, why not use React VR and Pusher to create a simplified version of it?

Here’s how the React VR/Pusher version looks:

{% youtube vm5vKh7h0b4 %}

A user can enter a channel identifier in the URL. When a 3D shape is hit, a sound will play and a Pusher event will be published so other users in the same channel can receive the event and the app can play that sound too.

We’ll have a Node.js backend to publish the events, so you should be familiar with JavaScript and React. If you’re not very familiar with some VR concepts or React VR, here’s a [guide](https://www.pluralsight.com/guides/front-end-javascript/getting-started-with-react-vr) to get you started.

For reference (or if you just want to try the app), you can find the React VR project [here](https://github.com/eh3rrera/reactvr_musical) and the Node.js backend [here](https://github.com/eh3rrera/server_pusher_reactvr_musical).


## Setting up the React VR project

Let’s start by installing (or updating) the React VR CLI tool:
```
npm install -g react-vr-cli
```

Next, create a new React VR project:
```
react-vr init musical-exp-react-vr-pusher
```

Now go to the directory it created and execute the following command to start the development server:
```
cd musical-exp-react-vr-pusher
npm start
```

In your browser, go to [http://localhost:8081/vr/](http://localhost:8081/vr/). Something like the following will show up:
![Hello World no VR button](https://dl.dropboxusercontent.com/s/gx8cx6m0q0g5gxf/creating-a-musical-experience-with-react-vr-and-pusher-first-no-vr.png)

If you’re using a compatible browser, you should also see the View in VR button to view the app with a headset:
![Hello World VR button](https://dl.dropboxusercontent.com/s/xrq8g51rr7cfevh/creating-a-musical-experience-with-react-vr-and-pusher-first-vr.png)

Now let’s start coding our app.


## Creating the background

We’re going to use an [equirectangular image](https://www.flickr.com/groups/equirectangular/) as the image background. The main characteristic of this type of images is that the width must be exactly twice the height, so open your favorite image editing software and create an image of size 4096x2048 with a gradient or color of your choice:
![Gradient](https://dl.dropboxusercontent.com/s/v3qbhdalrjc083i/creating-a-musical-experience-with-react-vr-and-pusher-background.png)

Create a new folder called `images` inside the `static_assets` directory in the root of your app and save your image there. Now open the file `index.vr.js` and replace the content of the `render` method with the following:
```js
render() {
  return (
    <View>
      <Pano source={asset('images/background.jpg')} />
    </View>
  );
}
```

When you reload the page (or if you enable [hot reloading](https://facebook.github.io/react-vr/docs/dev-tools.html#hot-reloading)), you should see something like this:
![Background](https://dl.dropboxusercontent.com/s/kyas3gr52x7pqq2/creating-a-musical-experience-with-react-vr-and-pusher-background.gif)

Now, to simulate a tree, we’re going to use a [Cylinder](https://facebook.github.io/react-vr/docs/cylinder.html). In fact, we’ll need a hundred of this to create a forest around the user. In the original Musical Forest, we can find the algorithm to generate the trees around the users in the [js/components/background-objects.js](https://github.com/googlecreativelab/webvr-musicalforest/blob/master/js/components/background-objects.js) file. If we adapt the code into a React component for our project, we can get this:
```js
import React from 'react';
import {
  View,
  Cylinder,
} from 'react-vr';

export default ({trees, perimeter, colors}) => {
  const DEG2RAD = Math.PI / 180;
                
  return (
    <View>
      {Array.apply(null, {length: trees}).map((obj, index) => {
          const theta = DEG2RAD * (index / trees) * 360;
          const randomSeed = Math.random();
          const treeDistance = randomSeed * 5 + perimeter;
          const treeColor = Math.floor(randomSeed * 3);
          const x = Math.cos(theta) * treeDistance;
          const z = Math.sin(theta) * treeDistance;

          return (
            <Cylinder
              key={index}
              radiusTop={0.3}
              radiusBottom={0.3}
              dimHeight={10}
              segments={10}
              style={{
                color: colors[treeColor],
                opacity: randomSeed,
                transform: [{scaleY : 2 + Math.random()}, {translate: [x, 3, z]},],
              }}
            />
          );
      })}
    </View>
  );
}
```

This functional component takes three parameters:

- `trees`, which indicates the number of trees the forest will have
- `perimeter`, a value to control how far the trees will be rendered from the user
- `colors`, an array with values of [colors](https://facebook.github.io/react-vr/docs/colors.html) for the trees.

Using `Array.apply(null, {length: trees})`, we can create an array of empty values to which we can apply the map function to render an array of cylinders with random colors, opacities and positions inside a [View](https://facebook.github.io/react-vr/docs/view.html) component.

We can save this code in a file called `Forest.js` inside a `components` directory and use it inside of  `index.vr.js` in the following way:
```js
...
import Forest from './components/Forest';

export default class musical_exp_react_vr_pusher extends React.Component {
  render() {
    return (
      <View>
        <Pano source={asset('images/background.jpg')} />
                
        <Forest trees={100} perimeter={15} colors={['#016549', '#87b926', '#b1c96b']} 
        />
      </View>
    );
  }
};

...
```

In the browser, you should see something like this:
![Forest](https://dl.dropboxusercontent.com/s/xpt6nah4sqbzr2o/creating-a-musical-experience-with-react-vr-and-pusher-trees.gif)

Great, our background is complete, now let’s add the 3D objects to play the sounds.


## Creating the 3D shapes

We are going to have six 3D shapes and each will play six different sounds when clicked. Also, a little animation when the cursor enters and exits the shape will come in handy.

To do that, we’ll need a [VrButton](https://facebook.github.io/react-vr/docs/vrbutton.html), an [Animated.View](https://facebook.github.io/react-vr/docs/animated.html), and a [Box](https://facebook.github.io/react-vr/docs/box.html), a [Cylinder](https://facebook.github.io/react-vr/docs/cylinder.html), and a [Sphere](https://facebook.github.io/react-vr/docs/sphere.html) for the shapes. However, as each shape is going to be different, let’s just encapsulate in a component what is the same. Save the following code in the file `components/SoundShape.js`:
```js
import React from 'react';
import {
  VrButton,
  Animated,
} from 'react-vr';

export default class SoundShape extends React.Component {
        
  constructor(props) {
    super(props);
    this.state = {
      bounceValue: new Animated.Value(0),
    };
  }
  
  animateEnter() {
    Animated.spring(  
      this.state.bounceValue, 
      {
        toValue: 1, 
        friction: 4, 
      }
    ).start(); 
  }

  animateExit() {
    Animated.timing(
      this.state.bounceValue,
      {
        toValue: 0,
        duration: 50,
      }
    ).start();
  }
  
  render() {
    return (
      <Animated.View
        style={{
          transform: [
            {rotateX: this.state.bounceValue},
          ],
        }}
      >
        <VrButton
          onEnter={()=>this.animateEnter()}
          onExit={()=>this.animateExit()}
        >
          {this.props.children}
        </VrButton>
      </Animated.View>
    );
  }
};
```
When the cursor enters the button area, `Animated.spring` will change the value of `this.state.bounceValue` from `0` to `1` and show a bouncy effect. When the cursor exits the button area, `Animated.timing` will change the value of `this.state.bounceValue` from `1` to `0` in `50` milliseconds. For this to work, we wrap the `VrButton` with an `Animated.View` component that will change the `rotateX` transform of the `View` on each state change.

In `index.vr.js`, we can add a `SpotLight` (you can add any other type of light you want or change its properties) and use the `SoundShape` component to add a cylinder this way:
```js
...
import {
  AppRegistry,
  asset,
  Pano,
  SpotLight,
  View,
  Cylinder,
} from 'react-vr';
import Forest from './components/Forest';
import SoundShape from './components/SoundShape';

export default class musical_exp_react_vr_pusher extends React.Component {
  render() {
    return (
      <View>
        ...

        <SpotLight intensity={1} style={{transform: [{translate: [1, 4, 4]}],}} />
                
        <SoundShape>
          <Cylinder
            radiusTop={0.2}
            radiusBottom={0.2}
            dimHeight={0.3}
            segments={8}
            lit={true}
            style={{
              color: '#96ff00', 
              transform: [{translate: [-1.5,-0.2,-2]}, {rotateX: 30}],
            }}
          />
        </SoundShape>
      </View>
    );
  }
};
...
```
Of course, you can change the properties of the shapes or even replace them with [3D models](https://facebook.github.io/react-vr/docs/model.html).

Let’s also add a pyramid (which is a cylinder with a zero op radius and four segments):
```js
<SoundShape>
  <Cylinder
    radiusTop={0}
    radiusBottom={0.2}
    dimHeight={0.3}
    segments={4}
    lit={true}
    style={{
      color: '#96de4e',
      transform: [{translate: [-1,-0.5,-2]}, {rotateX: 30}],
    }}
  />
</SoundShape>
```

A cube:
```js
<SoundShape>
  <Box
    dimWidth={0.2}
    dimDepth={0.2}
    dimHeight={0.2}
    lit={true}
    style={{
      color: '#a0da90', 
      transform: [{translate: [-0.5,-0.5,-2]}, {rotateX: 30}],
    }}
  />
</SoundShape>
```

A box:
```js
<SoundShape>
  <Box
    dimWidth={0.4}
    dimDepth={0.2}
    dimHeight={0.2}
    lit={true}
    style={{
      color: '#b7dd60',
      transform: [{translate: [0,-0.5,-2]}, {rotateX: 30}],
    }}
  />
</SoundShape>
```

A sphere:
```js
<SoundShape>
  <Sphere
    radius={0.15}
    widthSegments={20}
    heightSegments={12}
    lit={true}
    style={{
      color: '#cee030',
      transform: [{translate: [0.5,-0.5,-2]}, {rotateX: 30}],
    }}
  />
</SoundShape>
```

And a triangular prism:
```js
<SoundShape>
  <Cylinder
    radiusTop={0.2}
    radiusBottom={0.2}
    dimHeight={0.3}
    segments={3}
    lit={true}
    style={{
      color: '#e6e200',
      transform: [{translate: [1,-0.2,-2]}, {rotateX: 30}],
    }}
  />
</SoundShape>
```

After you add the necessary imports, save the file and refresh your browser. You should see something like this:
![Shapes](https://dl.dropboxusercontent.com/s/i7qj34285dcc3z0/creating-a-musical-experience-with-react-vr-and-pusher-shapes.gif)

Now let’s add some sounds!


## Adding sound

For audio, React VR supports `wav`, `mp3`, and `ogg` files, among others. You can find the complete list [here](https://github.com/facebook/react-vr/blob/master/ReactVR/js/Audio/getSupportedFormats.js).

You can go to [Freesound](https://www.freesound.org) or other similar sites to get some sound files. Download the ones you like and place them in the directory `static_assets/sounds`. For this project, we’re going to use six animal sounds, [something like a bird](https://www.freesound.org/people/funkymuskrat/sounds/217050/), [another bird](https://www.freesound.org/people/dobroide/sounds/59367/), [another bird](https://www.freesound.org/people/dobroide/sounds/34162/), [a cat](https://www.freesound.org/people/NoiseCollector/sounds/4911/), [a dog](https://www.freesound.org/people/NoiseCollector/sounds/4911/), and [a cricket](https://www.freesound.org/people/Adam_N/sounds/164482/) (as a quick note, I had to re-save this file lowering its bitrate so it can be played by React VR).

For our purposes, React VR give us three options to play a sound:

- [VrSoundEffects](https://github.com/facebook/react-vr/blob/master/Libraries/Utilities/VrSoundEffects.js)
- The [onClickSound](https://facebook.github.io/react-vr/docs/vrbutton.html#onclicksound) event of a VrButton
- A [Sound](https://facebook.github.io/react-vr/docs/sound.html) component

However, only the Sound component supports 3D/positional audio so the left and right balance of the sound will change as the listener moves around the scene or turns their head. So let’s add it to our `SoundShape` component along with an `onClick` event to the `VrButton`:
```js
...
import {
  ...
  Sound,
} from 'react-vr';

export default class SoundShape extends React.Component {
  ...
  render() {
    return (
      <Animated.View
        ...
      >
        <VrButton
          onClick={() => this.props.onClick()}
          ...
        >
          ...
        </VrButton>
        <Sound playerState={this.props.playerState} source={this.props.sound} />
      </Animated.View>
    );
  }
}
```
We’re going to use a [MediaPlayerState](https://facebook.github.io/react-vr/docs/mediaplayerstate.html) to control the playing of the sound. Both will be passed as properties of the component.

This way, let’s define an array with this information in `index.vr.js`:
```js
...
import {
  ...
  MediaPlayerState,
} from 'react-vr';
...

export default class musical_exp_react_vr_pusher extends React.Component {
        
  constructor(props) {
    super(props);
        
        this.config = [
          {sound: asset('sounds/bird.wav'), playerState: new MediaPlayerState({})},
          {sound: asset('sounds/bird2.wav'), playerState: new MediaPlayerState({})},
          {sound: asset('sounds/bird3.wav'), playerState: new MediaPlayerState({})},
          {sound: asset('sounds/cat.wav'), playerState: new MediaPlayerState({})},
          {sound: asset('sounds/cricket.wav'), playerState: new MediaPlayerState({})},
          {sound: asset('sounds/dog.wav'), playerState: new MediaPlayerState({})},
        ];
  }
  
  ...
}
```
And a method to play a sound using the MediaPlayerState object when the right index is passed:
```js

...

export default class musical_exp_react_vr_pusher extends React.Component {
        
  ...
  
  onShapeClicked(index) {
    this.config[index].playerState.play();
  }
  
  ...
}
```
Now, we only need to pass all this information to our SoundShape components. So let’s group our shapes in an array and use a map function to generate the components:
```js
...

export default class musical_exp_react_vr_pusher extends React.Component {
        
  ...
  
  render() {
    const shapes = [
      <Cylinder
        ...
      />,
      <Cylinder
        ...
      />,
      <Box
        ...
      />,
      <Box
        ...
      />,
      <Sphere
        ...
      />,
      <Cylinder
        ...
      />
    ];
        
    return (
      <View>
        ...
         
        {shapes.map((shape, index) => {
          return (       
            <SoundShape 
              onClick={() => this.onShapeClicked(index)} 
              sound={this.config[index].sound} 
              playerState={this.config[index].playerState}>
                {shape}
            </SoundShape>
          );
        })}
        
      </View>
    );
  }
  
  ...
}
```

If you restart your browser and try, you should hear the sounds as you click on the shapes.

Now let’s add to our app multi-user support in realtime with Pusher. 

## Setting up Pusher

Create a free account at [https://pusher.com/signup](https://pusher.com/signup).
When you create an app, you'll be asked to enter some configuration options:
![Create Pusher App](https://dl.dropboxusercontent.com/s/t1ud5lyv508wgyz/creating-a-musical-experience-with-react-vr-and-pusher-set-up-pusher.png)

Enter a name, choose React as your front-end tech, and Node.js as the back-end tech. This will give you some sample code to get you started:
![Pusher Getting Started Code](https://dl.dropboxusercontent.com/s/3b5knaiaykyn5y4/creating-a-musical-experience-with-react-vr-and-pusher-app.png)

But don't worry, this won't lock you into this specific set of technologies as you can always change them. With Pusher, you can use any combination of libraries.

Next, copy your cluster ID (next to the app title, in this example `mt1`), App ID, Key, and Secret information as we'll need them next. You can also find them in the *App Keys* tab.

## Publishing the event

React VR acts as a [Web Worker](https://www.w3schools.com/HTML/html5_webworkers.asp) (you can know more about React VR architecture in this [video](https://www.youtube.com/watch?v=KHx-32m50c4)) so we need to include Pusher’s worker script in `index.vr.js` this way:
```js
...
importScripts('https://js.pusher.com/4.1/pusher.worker.min.js');

export default class musical_exp_react_vr_pusher extends React.Component {
  ...
}
```
We have two requirements that need to be taken care of. First, we need to be able to pass an identifier through the URL (like `http://localhost:8081/vr/?channel=1234`) so users can choose which channel they want to be in and share it with their friends. 

To address this, we need to read the URL. Luckily, React VR comes with the [native module Location](https://facebook.github.io/react-vr/docs/native-modules.html), which makes available to the React context the properties of the object `window.location`.

Next, we need to make a call to a server that will publish the Pusher event so all the connected clients can also play the event. However, we don’t want the client that broadcasts the event to receive it too, because in that case, the sound will be played twice, and there’s no point in waiting to receive the event to play the sound when you can play it immediately when the user clicks the shape.

Each Pusher connection is assigned a unique socket ID. To exclude recipients from receiving events in Pusher, we just need to pass to the server the socket ID of the client we want to be excluded a `socket_id` when this is triggering an event. (You can find more information [here](https://pusher.com/docs/server_api_guide/server_excluding_recipients).)

This way, adapting a little bit a function (`getParameterByName`) to [read the parameters of the URL](http://stackoverflow.com/a/901144/3593852), and saving the `socketId` when a successful connection is made to Pusher, we can address both requirements with this:
```js
...
import {
  ...
  NativeModules,
} from 'react-vr';
...
const Location = NativeModules.Location;

export default class musical_exp_react_vr_pusher extends React.Component {
  componentWillMount() {
    const pusher = new Pusher('<INSERT_PUSHER_APP_KEY>', {
      cluster: '<INSERT_PUSHER_APP_CLUSTER>',
      encrypted: true,
    });
    this.socketId = null;
    pusher.connection.bind('connected', () => {
      this.socketId = pusher.connection.socket_id;
    });
    this.channelName = 'channel-' + this.getChannelId();
    const channel = pusher.subscribe(this.channelName);
    channel.bind('sound_played',  (data) => {
      this.config[data.index].playerState.play();
    });
  }
  
  getChannelId() {
    let channel = this.getParameterByName('channel', Location.href);
    if(!channel) {
      channel = 0;
    }
          
    return channel;
  }
  
  getParameterByName(name, url) {
    const regex = new RegExp("[?&]" + name + "(=([^&#]*)|&|#|$)");
    const results = regex.exec(url);
    if (!results) return null;
    if (!results[2]) return '';
    return decodeURIComponent(results[2].replace(/\+/g, " "));
  }
  
  ...
}
```
If there isn’t a channel parameter in the URL, by default we assign the ID 0. This ID will be added to the Pusher channel to make it unique.

Finally, we just need to call an endpoint on the server side that will publish the event, passing the socket ID of the client and the channel where we’ll be publishing events:
```js
...
export default class musical_exp_react_vr_pusher extends React.Component {
  ...
  onShapeClicked(index) {
    this.config[index].playerState.play();
    fetch('http://<INSERT_YOUR_SERVER_URL>/pusher/trigger', {
      method: 'POST',
      headers: {
        'Accept': 'application/json',
        'Content-Type': 'application/json',
      },
      body: JSON.stringify({
        index: index,
        socketId: this.socketId,
        channelName: this.channelName,
      })
    });
  }
  ...
}
```

And that’s all the code of the React part. Now let’s take a look at the server.

## Creating the Node.js backend

Execute the following command to generate a `package.json` file:
```
npm init -y
```

Add the following dependencies:
```
npm install --save body-parser express pusher
```

And save the following code in a file:
```js
const express = require('express');
const bodyParser = require('body-parser');
const Pusher = require('pusher');

const app = express();
app.use(bodyParser.json());
app.use(bodyParser.urlencoded({ extended: false }));
/*
  The following headers are needed because the development server of React VR
  is started on a different port than this server. 
  When the final project is published, you may not need this middleware
*/
app.use((req, res, next) => {
  res.header("Access-Control-Allow-Origin", "*")
  res.header("Access-Control-Allow-Headers", 
             "Origin, X-Requested-With, Content-Type, Accept")
  next();
});

const pusher = new Pusher({
  appId: '<INSERT_PUSHER_APP_ID>',
  key: '<INSERT_PUSHER_APP_KEY>',
  secret: '<INSERT_PUSHER_APP_SECRET>',
  cluster: '<INSERT_PUSHER_APP_CLUSTER>',
  encrypted: true,
});

app.post('/pusher/trigger', function(req, res) {
  pusher.trigger(req.body.channelName, 
                 'sound_played', 
                 { index: req.body.index },
                 req.body.socketId );
  res.send('ok');
});

const port = process.env.PORT || 5000;
app.listen(port, () => console.log(`Running on port ${port}`));
```
As you can see, here we set up an Express server, the Pusher object, and the route `/pusher/trigger`, which just triggers an event with the index of the sound to be played and the socketID to exclude the recipient of the event.

And we’re done. Let’s test it.

## Testing

Execute the Node.js backend with:
```
node server.js
```

Update your server URL in `index.vr.js` (with your IP instead of `localhost`) and enter in your browser an address like [http://localhost:8081/vr/?channel=1234](http://localhost:8081/vr/?channel=1234) in two browser windows. When you click on a shape, you should hear the sound played twice (of course, it’s more fun doing this with a friend in another computer):

{% youtube c1lf8FearWQ%}

## Conclusion

React VR is a great library to create virtual reality experiences in an easy way, especially if you already know React/React Native. Pair it with Pusher and you’ll have powerful tools to program the next generation of web applications.

You can build a production release of this project to deploy it in any web server by following the steps on this page.

Also, you can customize this code by changing the colors, the shapes, the sounds, or add more functionality from the original Musical Forest. 

Finally, remember that you can find the code of the app in this GitHub repository.

This post first appeared on [Pusher's blog](https://blog.pusher.com/building-a-realtime-react-vr-app/).