A Multi-modal Personality Prediction System (2020)

Abstract: Automatic prediction of personality traits has many real-life applications, e.g., in forensics, recommender systems, personalized services etc.. In this work, we have proposed a solution framework for solving the problem of predicting the personality traits of a user from videos. Ambient, facial and the audio features are extracted from the video of the user. These features are used for the final output prediction. The visual and audio modalities are combined in two different ways: averaging of predictions obtained from the individual modalities, and concatenation of features in multi-modal setting. The dataset released in Chalearn-16 is used for evaluating the performance of the system. Experimental results illustrate that it is possible to obtain better performance with a hand full of images, rather than using all the images present in the video

## introduction our personality impacts a lot on our lives, affecting our life choices, mental health, well-being, and desires. thus, automatic prediction of one's personality has many applications such as enhanced personal assistants, recommender system, job screening, forensics, psychological studies, etc. @xcite . the big-five personality traits @xcite ) are the most popular measures used in the literature. they are extraversion, neuroticism, agreeableness, conscientiousness, and openness. most of the methods have utilized the cnnbased architectures for extracting the features from the images, mostly the facial features @xcite , @xcite , @xcite . mainly, researchers have used the late fusion strategy (averaging the predictions from all the modalities) for combining the results obtained from different modalities (audio, and video) @xcite @xcite @xcite . there are very few works, which have employed early fusion strategy for developing the multimodal system @xcite . this has motivated us to develop a multimodal system which uses early fusion for combining the features generated from different modalities, and using the combined feature for final prediction. we have developed an audio-visual system, which extracts ambient features from video using resnet @xcite , facial features using mtcnn @xcite , and the audio features from the vggish cnn @xcite . finally, those features are concatenated and then fed to the fully connected layer followed by a sigmoid layer for the final prediction. we have used the chalearn-16 dataset for evaluating the performance of our system @xcite ). an accuracy of 91.43% on the test data has been achieved using our proposed system. the main contributions of this work are i) to the best of our knowledge, combined feature representation of images has been carried out for the first time for personality prediction. ii) the vggish cnn has been used for extracting the audio features, and then those are used for the prediction. it has also been performed for the first time. from the results, it can be established that only some of the images extracted from different parts of the video are capable for successful training and testing of the model with good performance. ## the proposed methodology our methodology consists of learning some features extracted from two different modalities, namely video and audio. they are discussed below: ## conclusion and future work personality prediction reveals the overall characteristics of a user. in this work, we have proposed a deep multi-modal system for personality prediction given a video. it extracts features from a video, and those are then used in the neural network setting for the final prediction. from the obtained experimental results, we can conclude the following : i) only handful of images from different parts of video are enough for successful training and testing with good performance, ii) averaging the predictions of different modalities yields better performance than the simple concatenation of the modalities in the multi-modal setting. we are planning to improve the fusion strategy for combining the different modalities. we will try to use emotion features and different types of attention mechanisms like weighted attention, selfattention etc. for combining the modalities.