Towards Clinical Phenotyping at Scale with Serious Games in Mixed Reality
Abstract
Context: Mental healthcare systems are facing an ever-growing demand for appropriate assessment and intervention.
Unfortunately, services are often centralized, overloaded, and inaccessible, resulting in greater institutional and social inequities.
Therefore, there is an urgent need to establish easy-to-implement methods for early diagnosis and personalized follow-up.
In recent years, serious games have started to offer such a clinical tool at scale.
Problem: There are critical challenges to the development of secure and inclusive serious games for clinical research.
First, the quality of the data and features analyzed must be well defined early in the research process in order to draw meaningful conclusions.
Second, algorithms must be aligned with the purpose of the research while not perpetuating bias.
Finally, the technologies used must be widely accessible and sufficiently engaging for users.
Focus of the paper: To tackle these challenges, we designed a participatory project that combines three innovative technologies: Mixed Reality, Serious Gaming, and
Machine Learning.
This work is a proof of concept that analyzes preliminary data with a focus on the identification of the players and the measurement of classical biases such as sex and environment of data collection.
Method: We co-developed with patients and their families, as well as clinicians, a serious game in mixed reality specifically designed for evaluation and therapeutic intervention in autism.
Preliminary data were collected from neurotypical individuals with a mixed reality headset.
Relevant behavioral features were extracted and used to train several classification algorithms for player identification.
Results: We were able to classify players above chance with slightly higher accuracy of neural networks.
Interestingly, the accuracy was significantly higher when players were separated by sex.
Furthermore, the uncontrolled condition showed better levels of accuracy than the controlled condition.
This could mean that the data are richer when the player interacts freely with the game.
Our proof of concept cannot exclude the possibility that this last result is linked to the experimental setup.
Future development will clarify this point with a larger sample size and the use of deep learning algorithms.
Implications: We show that serious games in mixed reality can be a valuable tool to collect clinical data.
Our preliminary results highlight important biases to consider for future studies, especially for the sex and context of data collection.
Next, we will evaluate the usability, accessibility, and tolerability of the device and the game in autistic children.
In addition, we will evaluate the psychometric properties of the serious game, especially for patient stratification.
This project aims to develop a platform for the diagnosis and therapy of autism, which can eventually be easily extended to other conditions and settings such as the evaluation of depression or stroke rehabilitation.
Such a tool can offer novel possibilities for the study, evaluation, and treatment of mental conditions at scale, and thus ease the burden on healthcare systems.
Introduction
With the recent increase in mental health awareness comes a pressing need for evaluation and intervention.
However, health services struggle to meet this demand, mainly due to centralized systems and a lack of trained professionals.
In mental health, the wide variability of symptoms between and within individuals makes standard psychometric tools unreliable.
Diagnosis and treatment often require multiple appointments with several specialized professionals.
In addition to overburdening health facilities, this exhausting and expensive process can critically affect access to the care system and further reinforce social and institutional inequalities.
For a condition that requires lifelong follow-up, such as autism, these implications can be even more critical.
Autism Spectrum Disorders (ASDs) involve a large variety of symptoms, including difficulties in social interaction and communication, as well as repetitive behaviors and restricted interests [10].
Although the severity of symptoms varies between individuals, some people may need assistance throughout their lives.
Early and intensive interventions have been effective in helping autistic children improve their communication and social skills[6] but can, in time, be very costly.
For an autistic person, uncovered insurance expenses care are estimated at $4,000 per year.[12]
The lifetime cost associated with autism has been estimated between $2.4 and $3.2 million per person[8, 9].
These numbers underline the urgency to develop less onerous and more broadly accessible alternatives.
In recent years, advances in technology have offered new ways to tackle these challenges using serious games.
In addition to being entertaining, they can offer a more affordable and accessible alternative to conventional methods [4].
This approach provides controlled and reproducible environments for the study, intervention, and assessment of mental health, while also adapting to specific needs and preferences.
In recent years, head-mounted displays have gained popularity in clinical settings for providing immersive and engaging experiences.
For Autism, Mixed Reality (MR), which is a hybrid between virtual reality and augmented reality, has the advantage of allowing social interaction with the subject.
Several studies have shown its potential to help people with cognitive disabilities learn new skills and become more autonomous[1].
These devices can record a great quantity of data generated by engaging with a video game (e.g. score, movements, reaction times).
Machine Learning analyses, which are sophisticated methods based on mathematical and probabilistic formulas, can then leverage those rich data to extract meaningful patterns.
For mental health research, deep phenotyping provides a promising approach to characterize and identify patients based on multimodal observations and brings us one step closer to precision medicine [13].
However, machine learning can introduce and perpetuate biases, for instance regarding sex in healthcare [3], and this becomes critical to carefully control for those effects.
The Human Dynamic Clamp (HDC) is a human-computer interface consisting of a virtual partner whose behavioral model is empirically supported by findings from social neuroscience and experimental psychology.
This neuroinspired avatar can interact with humans and responds in real-time.
A 2D version of the HDC paradigm has been validated in neurotypicals and has demonstrated its ability, in interaction, to transfer new behaviors to participants [5].
Early research with children and adolescents in this therapeutic setting has shown the ability of the system to unravel sensory-motor and social-cognitive skills of autism [2].
To make this tool more attractive to pediatric populations in a clinical context, a gamified 3D version of the paradigm was developed for MR.
Here, we evaluate the resulting video game with multiple algorithms using preliminary data collected from neurotypical adults.
The current preliminary results illustrate a proof-of-concept for how the combination of serious games, MR, and machine learning can help with clinical phenotyping.
Methods 2.1 Mixed Reality Setup
We used the HoloLens 2 MR helmet (Microsoft Corp., Redmond, WA).
It is currently the most comfortable and safe MR device on the market, certified for industrial and medical use.
The entire device is untethered, so users can move freely without wires or external packs.
It is a self-contained computer with Wi-Fi connectivity and multiple sensors that scan the environment and measure body gestures.
At device initialization, a 3D spatial mapping phase allows the environment to become interactive, allowing players to view and interact with virtual objects projected as holograms in the room.
The default interaction mechanism (air tap) built into HoloLens can be difficult to master, especially for people with motor disabilities.
Therefore, we adapted our game so that the player can interact with the environment using the index finger directly to pop balloons.
2.2 Serious Game
Pop’Balloons is our first prototype of a serious game for autistic children (Supplementary Figure S1).
As its name suggests, this serious game requires the player to move around the room to pop virtual balloons.
Pop’Balloons was first meant to assess the player’s motor skills.
Therefore, it includes a "motricity" mode with four levels of progressive difficulty, as described in the Supplementary
Material section.
When the game is launched, the main menu allows the user to create or modify player profiles, access the game’s parameters, or launch a new play session.
When the last option is selected, the game starts with a countdown.
A screen indicates to the player the number of balloons to burst.
When the countdown is over, the first virtual balloon appears in the room.
The player must then move and pop it.
The next one then appears outside the player’s field of vision.
Arrows and spatialized sound effects guide the player in their task.
There is no time limit to complete a level.
However, to access the next one, all balloons must be popped.
At the end of each level, a star-shaped balloon appears.
The player has to pop it to get bonus points.
If it does not burst after a few seconds, the balloon disappears by itself.
Finally, the serious game ends when all four levels are completed.
2.3 Data collection
Preliminary data were collected from n = 10 neurotypical participants (6 men).
Most of them completed the 4 levels of the game.
On average, the participants played the game 3.80 times +/- 1.08 and scored 113.76 +/- 24.56.
The first 6 participants played the game in a controlled condition, while the others played the game in an uncontrolled condition.
The main difference between these two conditions is the environment in which the game was played.
In the controlled condition, the game was played in a small empty meeting room (5m x 10m), while in the uncontrolled condition it was in an open space (hall of the student social club) with different playable areas across attempts.
Importantly, participants were instructed under the exact same conditions, so that the differences in the collected data maximally reflect the variability between individuals.
2.4 Digital phenotyping
The behavior of the player was divided into chunks of data for each level of the game, corresponding to the trajectories and reaction time of each balloon.
In total, 148 chunks of data were recorded by the MR headset, and 6 features were extracted from the motion trajectories as follows, for each chunk:
• The median of the deviation from the ideal trajectory over time.
This trajectory is computed by linear interpolation between each consecutive balloon.
Then, trajectory deviations are computed with the Euclidean distance between the player’s trajectory and the ideal one.
• The minimum angular velocity with respect to the x-axis (pitch) in rad/s. • The maximum angular velocity with respect to the y-axis (yaw) in rad/s. • The median of the proportion of time spent motionless, that is, the proportion of time samples where the head of the player keeps motionless w.r.t.
the whole balloon presentation.
More specifically, we computed the norm of the movement at a given time, then accumulated the time spent below a criterion of 0.005 times the maximum of the movement norm across the dataset.
• The user’s score, calculated by adding the points earned from popping the balloons.
The maximum score given by a balloon is 35 points when it explodes in 7 seconds.
After that, the score value of a balloon starts to decrease by 5 points each second.
A ten-point bonus is attributed if the final bonus balloon is popped in less than 7 seconds.
• The median velocity computed in an episode using the norm speed at each time step.
2.5 Machine Learning
Several classification algorithms were trained on player identification tasks using the Scikit-Learn library [11].
A one-vs-rest (OVR) strategy was applied for algorithms that are not inherently multiclass such as SVMs.
The effect of players’ sex and experimental conditions was studied by splitting the dataset into 5 groups: a player group with every participant, a male group with 6 players, a female group with 4 players, a controlled group with 6 players, and an uncontrolled group with 4 participants.
The classification accuracy was computed by cross-validation on 50 random subsets with 20% test data in each group.
The significance of our results was calculated with a null hypothesis (H0) by shuffling participants in the player group, shuffling gender in the male/female groups, and shuffling experimental conditions in
Results
The results show that the MLP classifier was the most accurate (mean = 34.13%, std = 8.18%, H0 = 9.27%, std = 5.47%) among all models when all players were considered (Figure 1).
Interestingly, accuracy was slightly improved for most classifiers when groups were separated by gender (Supplementary Figure S2), with the best accuracy achieved using the linear SVM (mean = 47.60%, std = 10.83%; H0 = 29.90%, std = 9.77%) for male players and the MLP classifier (mean = 59.09%, std = 15.45%; H0 = 24.55%, std = 12.76) for female players.
Furthermore, the uncontrolled condition showed the best levels of accuracy than the controlled condition for most classifiers (Supplementary
Figure S3)
and was the best for the SVM linear models (mean = 71.00%, std = 15.00%; H0 = 27.20%, std = 14.15%).
This could indicate that the data is of better quality when the player interacts with the game freely and in a natural environment.
Given the small dataset analyzed, we cannot rule out that these results may be related to the experimental setup.
These findings direct our hypothesis towards deep learning classification algorithms, although they alert to biases in the classifiers.
Supplementary
Tables describe all the results in detail.
[Figure]
Discussion
This study demonstrates how serious games in MR can capture the motor signature and cognitive profile of individuals, thus giving access to valuable measures for clinical use.
Our proof-ofconcept was able to successfully identify participants with high accuracy using several classification algorithms.
The preliminary results reveal considerable bias related to sex and environmental context.
When the groups were divided by sex, the females performed better in almost all models.
Interestingly, the accuracy levels were much higher when participants were in an uncontrolled context, implying that a natural environment promotes more natural behavior.
Environmental bias may also explain this trend.
Although more data are needed to draw an appropriate conclusion, the results highlight the need for a more thorough understanding of the impact of these variables on classification.
To assess the usability of the device and the game for autistic children, we will collect significant amounts of data so we can generate accessibility guidelines for an inclusive design of MR applications, especially serious games.
Furthermore, we will assess the psychometric characteristics of the serious game, particularly to evaluate the precision of clinical phenotyping in ecological tasks.
Ultimately, this will help in the stratification of patients based on clinically interpretable characteristics across different domains (e.g., sensorimotor, cognitive, social).
The logic of those subgroups of patients will then follow a more dimensional perspective on mental conditions (e.g., (RDoC) [7]) and depart from a monolithic taxonomy of mental conditions.
Hopefully, addressing these theoretical questions and practical challenges could help patients and their families, as well as clinicians and researchers.
Acknowledgments
We thank the ECP students who participated in the early stages of this project: Maxime Fetiveau,
Younes Laaboudi, Jules Massin, Hugo Perrin, Olivier Polidori, Seung Eun Yi, Matthieu Divet, Julien
Malle, Sarah Saidani, Paul Viossat, Zairan Wang, Xavier Tinel, Adil Dinia, Marianne Clary, Clémence
Kopff, Grégoire Martin, Jose Menoci Neto, Niraj Srinivas, Malek Adel, Charlotte Arnaud, Eva Chatry,
Mohamed Choraichi, Clémence Giffin, Nadir Larhdir, Simon Moliner and Zhuoying Wu.
We also thank the Institut Pasteur, especially Antonio Borderia, for the initial support of the project, and Actimage, especially Thomas Klein, Jérémy Boistiere and Alexis Giraudet, for the development of Pop’Balloons.
G.D. is funded by the Institute for Data Valorization (Grant CF00137433), Montréal, and the Fonds de Recherche
du Québec (Grant 285289).
This research was supported by funding from Fondation Orange, MITACS and Eidos Interactive Corp.
Appendix
Supplementary Tables
[Table] [Table]
DecisionTree
[Table] [Table]
[Table] [Table]
[Table] [Table]
[Table] [Table]
RandomForest
[Table] [Table]
DecisionTree
[Table]
KNeighbors
[Table]
AdaBoost
[Table] [Table]
[Table] [Table]
DecisionTree
[Table]
Supplementary
Supplementary Material
A Description of Levels
There are four levels in the Pop’Balloons video game:
• Level 1: Static balloons.
They do not move.
• Level 2: Floating balloons.
They slowly move left and right.
• Level 3:
Falling balloons.
They fall until they reach the ground.
If they touch the ground, they explode and you do not earn any points.
• Level 4 : Fast falling balloons.
They fall faster than the previous level.
B GitHub repositories
• Pop’Balloons Unity 3D source code: https://github.com/HolAutisme/PopBalloons • Scripts of the data analyses pipeline : https://github.com/ppsp-team/XReality4ASD
[Table]