Quicktour

Simulate’s API is inspired by the great Kubric API. The user can create a scene and add assets to it (objects, cameras, lights if needed). Once the scene is created you can save/share it and also render or do simulations using one of the backend rendering/simulation engines (at the moment Unity, Blender and Godot). The saving/sharing format is engine-agnostic and uses the standard glTF format for saving scenes.

Let’s do a quick exploration together.

To install and contribute (from CONTRIBUTING.md)

Create a virtual env and then install the code style/quality tools as well as the code base locally

pip install --upgrade simulate

Before you merge a PR, fix the style (we use isort + black)

make style

Project Structure

The Python API is located in src/simulate. It allows creation and loading of scenes, and sending commands to the backend.

The backend options (Unity, Godot, Blender) can be found in the integrations folder. The most fully-featured backend is Unity, located in integrations/Unity. The Unity editor isn’t required to run 🤗 Simulate, unless making changes to the backend, which requires Unity 2021.3.2f1.

Loading a scene from the hub or a local file

Loading a scene from a local file or the hub is done with Scene.create_from(), saving or pushing to the hub with scene.save() or scene.push_to_hub():

from simulate import Scene

scene = Scene.create_from('tests/test_assets/fixtures/Box.gltf')  # either local (priority) or on the hub with full path to file
scene = Scene.create_from('simulate-tests/Box/glTF/Box.gltf', is_local=False)  # Set priority to the hub file

scene.save('local_dir/file.gltf')  # Save to a local file
scene.push_to_hub('simulate-tests/Debug/glTF/Box.gltf')  # Save to the hub

scene.show()

Creating a Scene and adding/managing Objects in the scene

Basic example of creating a scene with a plane and a sphere above it:

import simulate as sm

scene = sm.Scene()
scene += sm.Plane() + sm.Sphere(position=[0, 1, 0], radius=0.2)

>>> scene
>>> Scene(dimensionality=3, engine='PyVistaEngine')
>>> └── plane_01 (Plane - Mesh: 121 points, 100 cells)
>>>     └── sphere_02 (Sphere - Mesh: 842 points, 870 cells)

scene.show()

An object (as well as the Scene) is just a node in a tree provided with optional mesh (as pyvista.PolyData structure) and material and/or light, camera, agents special objects.

The following objects creation helpers are currently provided:

Object3D any object with a pyvista.PolyData mesh and/or material
Plane
Sphere
Capsule
Cylinder
Box
Cone
Line
MultipleLines
Tube
Polygon
Ring
Text3D
Triangle
Rectangle
Circle
StructuredGrid

Most of these objects can be visualized by running the following example:

python examples/basic/objects.py

Objects are organized in a tree structure

Adding/removing objects:

Using the addition (+) operator (or alternatively the method .add(object)) will add an object as a child of a previous object.
Objects can be removed with the subtraction (-) operator or the .remove(object) command.
The whole scene can be cleared with .clear().

Accessing objects:

Objects can be directly accessed as attributes of their parents using their names (given with name attribute at creation or automatically generated from the class name + creation counter).
Objects can also be accessed from their names with .get(name) or by navigating in the tree using the various tree_* attributes available on any node.

Here are a couple of examples of manipulations:

# Add two copy of the sphere to the scene as children of the root node (using list will add all objects on the same level)
# Using `.copy()` will create a copy of an object (the copy doesn't have any parent or children)
scene += [scene.plane_01.sphere_02.copy(), scene.plane_01.sphere_02.copy()]

>>> scene
>>> Scene(dimensionality=3, engine='pyvista')
>>> ├── plane_01 (Plane - Mesh: 121 points, 100 cells)
>>> │   └── sphere_02 (Sphere - Mesh: 842 points, 870 cells)
>>> ├── sphere_03 (Sphere - Mesh: 842 points, 870 cells)
>>> └── sphere_04 (Sphere - Mesh: 842 points, 870 cells)

# Remove the last added sphere
>>> scene.remove(scene.sphere_04)
>>> Scene(dimensionality=3, engine='pyvista')
>>> ├── plane_01 (Plane - Mesh: 121 points, 100 cells)
>>> │   └── sphere_02 (Sphere - Mesh: 842 points, 870 cells)
>>> └── sphere_03 (Sphere - Mesh: 842 points, 870 cells)

Objects can be translated, rotated, scaled

Here are a couple of examples:

# Let's translate our floor (with the first sphere, its child)
scene.plane_01.translate_x(1)

# Let's scale the second sphere uniformly
scene.sphere_03.scale(0.1)

# Inspect the current position and scaling values
print(scene.plane_01.position)
>>> array([1., 0., 0.])
print(scene.sphere_03.scaling)
>>> array([0.1, 0.1, 0.1])

# We can also translate from a vector and rotate from a quaternion or along the various axes

Visualization engine

A default vizualization engine is provided with the vtk backend of pyvista.

Starting the vizualization engine can be done simply with .show().

scene.show()

You can find bridges to other rendering/simulation engines in the integrations directory.

Reinforcement Learning (RL) with 🤗 Simulate

🤗 Simulate is designed to provide easy and scalable integration with reinforcement learning algorithms. The core abstraction is through the RLEnv class that wraps a Scene. The RLEnv allows an Actuator to be manipulated by an external agent or policy.

It is core to the design of 🤗 Simulate that we are not creating Agents, but rather providing an interface for applications of machine learning and embodied AI. The core API for RL applications can be seen below, where 🤗 Simulate constrains the information that flows from the Scene to the external agent through an Actuator abstraction.

At release, we include a set of pre-designed Actors that can act or navigate a scene. An Actor inherits from an Object3D and has sensors, actuators, and action mappings.

Tips

If you are running on GCP, remember to not install pyvistaqt, and if you did so, uninstall it in your environment, since QT doesn’t work well on GCP.