How to Visualize Autonomous Driving Dataset (nuScenes)
Tags: ai, python, tutorial, opensource
Published: 2024-05-05 14:18:38

{% cta https://www.rerun.io/viewer?url=https://app.rerun.io/version/0.15.1/examples/nuscenes.rrd %} Try it in browser {% endcta %} {% cta https://www.rerun.io/examples/robotics/nuscenes %} Source Code {% endcta %} {% cta https://www.rerun.io/examples %} Explore Other Examples {% endcta %}

This example demonstrates the ability to read and visualize scenes from the nuScenes dataset, which is a public large-scale dataset specifically designed for autonomous driving. The scenes in this dataset encompass data collected from a comprehensive suite of sensors on autonomous vehicles. These include 6 cameras, 1 LIDAR, 5 RADAR, GPS and IMU sensors. Consequently, the dataset provides information about the vehicle's pose, the images captured, the recorded sensor data and the results of object detection at any given moment.

# Logging and visualizing with Rerun 

The visualizations in this example were created with the following Rerun code:

---

## Sensor calibration 

First, pinhole cameras and sensor poses are initialized to offer a 3D view and camera perspective. This is achieved using the Pinhole and Transform3D archetypes.

```python
rr.log(
    f"world/ego_vehicle/{sensor_name}",
    rr.Transform3D(
        translation=calibrated_sensor["translation"],
        rotation=rr.Quaternion(xyzw=rotation_xyzw),
        from_parent=False,
    ),
    timeless=True,
)

rr.log(
    f"world/ego_vehicle/{sensor_name}",
    rr.Pinhole(
        image_from_camera=calibrated_sensor["camera_intrinsic"],
        width=sample_data["width"],
        height=sample_data["height"],
    ),
    timeless=True,
)
```

--- 

## Timelines 

All data logged using Rerun in the following sections is initially connected to a specific time. Rerun assigns a timestamp to each piece of logged data, and these timestamps are associated with timelines.

```python
rr.set_time_seconds("timestamp", sample_data["timestamp"] * 1e-6)
```

---

## Vehicle pose 

As the vehicle is moving, its pose needs to be updated. Consequently, the positions of pinhole cameras and sensors must also be adjusted using Transform3D.

```python
rr.log(
    "world/ego_vehicle",
    rr.Transform3D(
        translation=ego_pose["translation"],
        rotation=rr.Quaternion(xyzw=rotation_xyzw),
        from_parent=False,
    ),
)
```

---

## LiDAR data 

LiDAR data is logged as `Points3D` archetype.

```python
rr.log(f"world/ego_vehicle/{sensor_name}", rr.Points3D(points, colors=point_colors))
```

---

## Camera data 

Camera data is logged as encoded images using ImageEncoded.

```python
rr.log(f"world/ego_vehicle/{sensor_name}", rr.ImageEncoded(path=data_file_path))
```

---

## Radar data 

Radar data is logged similar to LiDAR data, as `Points3D`.

```python
rr.log(f"world/ego_vehicle/{sensor_name}", rr.Points3D(points, colors=point_colors))
```

---

## Annotations 

Annotations are logged as Boxes3D, containing details such as object positions, sizes, and rotation.

```python
rr.log("world/anns", rr.Boxes3D(sizes=sizes, centers=centers, rotations=rotations, class_ids=class_ids))
```

---

## Setting up the default blueprint 

The default blueprint for this example is created by the following code:

```python
sensor_space_views = [
    rrb.Spatial2DView(
        name=sensor_name,
        origin=f"world/ego_vehicle/{sensor_name}",
    )
    for sensor_name in nuscene_sensor_names(nusc, args.scene_name)
]
blueprint = rrb.Vertical(
    rrb.Spatial3DView(name="3D", origin="world"),
    rrb.Grid(*sensor_space_views),
    row_shares=[3, 2],
)
```

We programmatically create one view per sensor and arrange them in a grid layout, which is convenient when the number of views can significantly vary from dataset to dataset. This code also showcases the row_shares argument for vertical containers: it can be used to assign a relative size to each of the container's children. A similar column_shares argument exists for horizontal containers, while grid containers accept both.

---

## Join us on Github
{% github https://github.com/rerun-io/rerun %}