How to Visualise MediaPipe’s Human Pose Tracking in 2D and 3D with Rerun
Tags: machinelearning, tutorial, ai, python
Published: 2024-04-27 14:13:04

{% cta https://www.rerun.io/viewer?url=https://app.rerun.io/version/0.15.1/examples/human_pose_tracking.rrd %} Try it in browser {% endcta %} {% cta https://www.rerun.io/examples/video-image/human_pose_tracking %} Source Code {% endcta %} {% cta https://www.rerun.io/examples %} Explore Other Examples {% endcta %}

Human pose tracking is a task in computer vision that focuses on identifying key body locations, analyzing posture, and categorizing movements. At the heart of this technology is a pre-trained machine-learning model to assess the visual input and recognize landmarks on the body in both image coordinates and 3D world coordinates. The use cases and applications of this technology include but are not limited to Human-Computer Interaction, Sports Analysis, Gaming, Virtual Reality, Augmented Reality, Health, etc.

In this example, the MediaPipe Pose Landmark Detection solution was utilized to detect and track human pose landmarks and produces segmentation masks for humans. Rerun was employed to visualize the output of the Mediapipe solution over time to make it easy to analyze the behavior.

---

# Logging and visualizing with Rerun 
The visualizations in this example were created with the following Rerun code.

---

## Timelines 
For each processed video frame, all data sent to Rerun is associated with the two timelines time and frame_idx.

```python
rr.set_time_seconds("time", bgr_frame.time)
rr.set_time_sequence("frame_idx", bgr_frame.idx)
```
---
## Video 
The input video is logged as a sequence of Image objects to the 'Video' entity.

```python
rr.log(
    "video/rgb",
    rr.Image(rgb).compress(jpeg_quality=75)
)
```
---


## Segmentation mask 
The segmentation result is logged through a combination of two archetypes. The segmentation image itself is logged as an SegmentationImage and contains the id for each pixel. The color is determined by the AnnotationContext which is logged with timeless=True as it should apply to the whole sequence.

### Label mapping

```python
rr.log(
    "video/mask",
    rr.AnnotationContext(
        [
            rr.AnnotationInfo(id=0, label="Background"),
            rr.AnnotationInfo(id=1, label="Person", color=(0, 0, 0)),
        ]
    ),
    timeless=True,
)
```

### Segmentation image
![Image description](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/xbnvr2v2ga7cjvo24s5h.gif)
```python
rr.log(
    "video/mask",
    rr.SegmentationImage(segmentation_mask.astype(np.uint8))
)
```
---



## Body Pose Points 
Logging the body pose landmarks involves specifying connections between the points, extracting pose landmark points and logging them to the Rerun SDK. 

The 2D and 3D points are logged through a combination of two archetypes. First, a timeless ClassDescription is logged, that contains the information which maps keypoint ids to labels and how to connect the keypoints. Defining these connections automatically renders lines between them. Mediapipe provides the POSE_CONNECTIONS variable which contains the list of (from, to) landmark indices that define the connections. Second, the actual keypoint positions are logged in 2D and 3D as Points2D and Points3D archetypes, respectively.

### Label mapping and keypoint connections
```python
rr.log(
    "/",
    rr.AnnotationContext(
        rr.ClassDescription(
            info=rr.AnnotationInfo(id=1, label="Person"),
            keypoint_annotations=[rr.AnnotationInfo(id=lm.value, label=lm.name) for lm in mp_pose.PoseLandmark],
            keypoint_connections=mp_pose.POSE_CONNECTIONS,
        )
    ),
    timeless=True,
)
```
---

### 2D points
The 2D points are visualized over the image/video for a better understanding and visualization of the body pose. 
![Image description](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/i56lx2dsyycno8br9o4m.gif)
```python
rr.log(
    "video/pose/points",
    rr.Points2D(landmark_positions_2d, class_ids=1, keypoint_ids=mp_pose.PoseLandmark)
)
```
---


### 3D points
The 3D points allows the creation of a 3D model of the body posture for a more comprehensive representation of the human pose.
![Image description](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/c4y723gb9imevlj6hf4p.gif)
```python
rr.log(
    "person/pose/points",
    rr.Points3D(landmark_positions_3d, class_ids=1, keypoint_ids=mp_pose.PoseLandmark),
)
```

---

## Join us on Github
{% github https://github.com/rerun-io/rerun %}