  In recent years, single modality based gait recognition has been extensively
explored in the analysis of medical images or other sensory data, and it is
recognised that each of the established approaches has different strengths and
weaknesses. As an important motor symptom, gait disturbance is usually used for
diagnosis and evaluation of diseases; moreover, the use of multi-modality
analysis of the patient's walking pattern compensates for the one-sidedness of
single modality gait recognition methods that only learn gait changes in a
single measurement dimension. The fusion of multiple measurement resources has
demonstrated promising performance in the identification of gait patterns
associated with individual diseases. In this paper, as a useful tool, we
propose a novel hybrid model to learn the gait differences between three
neurodegenerative diseases, between patients with different severity levels of
Parkinson's disease and between healthy individuals and patients, by fusing and
aggregating data from multiple sensors. A spatial feature extractor (SFE) is
applied to generating representative features of images or signals. In order to
capture temporal information from the two modality data, a new correlative
memory neural network (CorrMNN) architecture is designed for extracting
temporal features. Afterwards, we embed a multi-switch discriminator to
associate the observations with individual state estimations. Compared with
several state-of-the-art techniques, our proposed framework shows more accurate
classification results.
