Using Motion Capture to Animate a Model
Tags: deeplearning
Published: 2022-03-21 02:52:16

It's so rewarding to set the model you've created into motion. If only there were an easy way to do this… well, actually there is!

I had long sought out this kind of solution, and then voila! I got my hands on motion capture, a capability from HMS Core 3D Modeling Kit, which comes with technologies like human body detection, model acceleration, and model compression, as well as a monocular human pose estimation algorithm from the deep learning perspective.

Crucially, this capability does NOT require advanced devices — a mobile phone with an RGB camera is good enough on its own. The camera captures 3D data from 24 key skeletal points on the body, which the capability uses to seamlessly animate a model.

What makes the motion capture capability even better is its straightforward integration process, which I'd like to share with you.
## Application Scenarios
Motion capture is ideal for 3D content creation for gaming, film & TV, and healthcare, among other similar fields. It can be used to animate characters and create videos for user generated content (UGC) games, animate virtual streamers in real time, and provide injury rehab, to cite just a few examples.
## Integration Process
### Preparations
Refer to the [official instructions](https://developer.huawei.com/consumer/en/doc/development/graphics-Guides/config-agc-0000001142925473?ha_source=hms1) to complete all necessary preparations.
### Configuring the Project
Before developing the app, there are a few more things you'll need to do: Configure app information in AppGallery Connect; make sure that the Maven repository address of the 3D Modeling SDK has been configured in the project, and that the SDK has been integrated.
**1.Create a motion capture engine**
```
// Set necessary parameters as needed.  
Modeling3dMotionCaptureEngineSetting setting = new Modeling3dMotionCaptureEngineSetting.Factory() 
    // Set the detection mode.  
    // Modeling3dMotionCaptureEngineSetting.TYPE_3DSKELETON_QUATERNION: skeleton point quaternions of a human pose.  
    // Modeling3dMotionCaptureEngineSetting.TYPE_3DSKELETON: skeleton point coordinates of a human pose.  
.setAnalyzeType(Modeling3dMotionCaptureEngineSetting.TYPE_3DSKELETON_QUATERNION 
                        | Modeling3dMotionCaptureEngineSetting.TYPE_3DSKELETON) 
.create(); 
Modeling3dMotionCaptureEngine engine = Modeling3dMotionCaptureEngineFactory.getInstance().getMotionCaptureEngine(setting);
```
**Modeling3dFrame** encapsulates video frame or static image data sourced from a camera, as well as related data processing logic.

Customize the logic for processing the input video frames, to convert them to the **Modeling3dFrame** object for detection. The video frame format can be NV21.

Use **android.graphics.Bitmap** to convert the input image to the **Modeling3dFrame** object for detection. The image format can be JPG, JPEG, or PNG.
```
// Create a Modeling3dFrame object using a bitmap.  
Modeling3dFrame frame = Modeling3dFrame.fromBitmap(bitmap); 
// Create a Modeling3dFrame object using a video frame.  
Modeling3dFrame.Property property = new Modeling3dFrame.Property.Creator().setFormatType(ImageFormat.NV21) 
    // Set the frame width.  
    .setWidth(width) 
    // Set the frame height.  
    .setHeight(height) 
    // Set the video frame rotation angle.  
    .setQuadrant(quadant) 
    // Set the video frame number.  
    .setItemIdentity(framIndex) 
    .create(); 
Modeling3dFrame frame = Modeling3dFrame.fromByteBuffer(byteBuffer, property);
```
**2.Call the asynchronous or synchronous API for motion detection**
Sample code for calling the asynchronous API **asyncAnalyseFrame**
```
Task<List<Modeling3dMotionCaptureSkeleton>> task = engine.asyncAnalyseFrame(frame); 
task.addOnSuccessListener(new OnSuccessListener<List<Modeling3dMotionCaptureSkeleton>>() { 
    @Override 
    public void onSuccess(List<Modeling3dMotionCaptureSkeleton> results) { 
        // Detection success.  
    } 
}).addOnFailureListener(new OnFailureListener() { 
    @Override 
    public void onFailure(Exception e) { 
        // Detection failure.  
    } 
});
```
Sample code for calling the synchronous API **analyseFrame**
```
SparseArray<Modeling3dMotionCaptureSkeleton> sparseArray = engine.analyseFrame(frame); 
for (int i = 0; i < sparseArray.size(); i++) { 
    // Process the detection result.  
};
```
**3.Stop the motion capture engine to release detection resources, once the detection is complete**
```
try { 
    if (engine != null) { 
        engine.stop(); 
    } 
} catch (IOException e) { 
    // Handle exceptions.  
}
```
## Result
![Result](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/x48rxv3h74lvmhbkwome.gif)
## References
3D Modeling Kit [Official Website](https://developer.huawei.com/consumer/en/hms/huawei-3d-modeling/?ha_source=hms1)
3D Modeling Kit [Development Guide](https://developer.huawei.com/consumer/en/doc/development/Media-Guides/introduction-0000001101263902?ha_source=hms1)
[Reddit](https://www.reddit.com/r/HMSCore/) for discussion with other developers
[GitHub](https://github.com/HMS-Core) for demos and sample codes
[Stack Overflow](https://stackoverflow.com/questions/tagged/huawei-mobile-services?tab=Votes) for solutions to integration issues
Follow our official account for the latest HMS Core-related news and updates.