Leveraging Multimodal Behavioral Analytics for Automated Job Interview Performance Assessment and Feedback (2020)

Abstract: Behavioral cues play a significant part in human communication and cognitive perception. In most professional domains, employee recruitment policies are framed such that both professional skills and personality traits are adequately assessed. Hiring interviews are structured to evaluate expansively a potential employee’s suitability for the position - their professional qualifications, interpersonal skills, ability to perform in critical and stressful situations, in the presence of time and resource constraints, etc. Candidates, therefore, need to be aware of their positive and negative attributes and be mindful of behavioral cues that might have adverse effects on their success. We propose a multimodal analytical framework that analyzes the candidate in an interview scenario and provides feedback for predefined labels such as engagement, speaking rate, eye contact, etc. We perform a comprehensive analysis that includes the interviewee’s facial expressions, speech, and prosodic information, using the video, audio, and text transcripts obtained from the recorded interview. We use these multimodal data sources to construct a composite representation, which is used for training machine learning classifiers to predict the class labels. Such analysis is then used to provide constructive feedback to the interviewee for their behavioral cues and body language. Experimental validation showed that the proposed methodology achieved promising results.

## introduction in the business world, interviews are a prerequisite to personnel recruitment for assessing the candidates through a structured interaction and discus- * equal contribution sion either on a one-to-one basis or by a panel of interviewers. it is an opportunity for the candidates to prove that they are qualified for the position, and for recruiters to assess the job-to-candidate fit. such recruiters are trained in evaluating a candidate's personality, thought patterns, behavior under stressful situations, and emotional intelligence through well-established metrics through technical analysis, psychometric testing, etc. several theoretical models @xcite have suggested the "big five trait taxonomy", based on which an individual's traits can be summarized, and scoring can be performed for choosing a candidate @xcite . the field of personality computing focuses on automatically analyzing such essential insights into the psyche of a person based on their behavior, verbal responses and non-verbal actions, speech patterns, body language, etc @xcite . studies have shown that nonverbal behaviors such as smiling, maintaining eye contact, and good posture all contribute significantly to interpersonal communication along with an indication as to the mental health and well-being of the participants @xcite . the significant difference between verbal and non-verbal communication is that the former is specific and interpreted, while the latter is subtle and implied. both channels of communication affect conversational dynamics and influence the relationship between individuals @xcite . automating such behavioral analysis can be beneficial while conducting mass recruitment drives, primarily when many of these are done online through videoconferencing systems. from the candidates' perspective, knowing the effect of their own verbal and non-verbal behavior in creating a favorable impression and increasing their chances of success in interviews is also of significant interest. these cues may be available through their interview videos and audio, speaking qualities, and the effect of content delivery. analysis of a single modality can often be insufficient in obtaining usable insights into how humans perceive and express information @xcite . modalities other than text can commonly present clues for the expression of sentiment and feelings @xcite . audio and visual features also aid in linguistic disambiguation as they provide additional details regarding the speakers' sentiment. when a person speaks with optimal vocal modulation or using appropriate gestures, it conveys a lot more than just the content. research on automated analysis of both verbal and non-verbal behavior cues in the case of job interviews has recently gained momentum, which we aim to analyze in this work. in this paper, we experiment with several linguistic, audio, and visual features extracted from recorded interviews to create feature vectors passed to different classifiers to score the candidate on specific predefined labels. the remainder of this paper is organized as follows: section 2 presents a discussion on existing works in the area of interest. the details of the proposed methodology and specifics of implementation are presented in section 3. in section 4, the experimental analysis and performance of the various models for each task are discussed, followed by the conclusion and references. ## related work over the last decade, behavioral patterns analysis using multimodal data has received significant research attention. @xcite conducted experiments for speech-based emotion analysis to compare speaker-dependent and speakerindependent techniques. to perform this analysis, several acoustic features such as fundamental frequency, duration, intensity are extracted to find hidden information. however, the speaker-dependent approach was not scalable and cannot be used in large-scale applications with several users. @xcite proposed a different approach for sentiment analysis of visual content using sentibank. sentibank is a visual concept detector library used to extract various concepts and adjective noun pairs (anp) from the visual content. while existing models predicted sentiments or emotions directly from low-level visual features, their approach used high-level visual features to better capture sentiments. they use images to extract mid-level semantic features and use a classifier to predict semantic features, which can be used to determine the relevance and importance of the image in determining emotion. nguyen et al. (2013) used real-time interview data to monitor and analyze body communication cues. the interviewees are seated in the videos, which lets them analyze both upper body movement and facial cues. various visual features are automatically extracted, and data is annotated to predict the personality and job interview ratings. this model shows the importance of using bodily gestures to predict the personality and give interview ratings. @xcite collated and used the mit interview dataset and trained lasso and svr models to predict several emotions and verbal/nonverbal cues like eyecontact, calm, speaking rate, authentic, focused, structured answers, smiled, friendly, engagement, etc. these labeled ratings were manually assigned by amazon turk workers, and the ground truth labels were derived by averaging the scores of 9 turk workers. @xcite presented a new technique for sentiment analysis in the telecommunication domain. they extracted and combined prosodic, lexical, and visual features from news videos and applied various computational methods to recognize real-time emotions from facial cues. the speech delivered by each participant is processed, parsed, and sentiment analysis is done on the corresponding text transcript. features such as visual power of perceived emotion, field sizes of members, voicing likelihood, sound intensity, the fundamental frequency of the speech, and the scores associated with the sentiment were defined and used. one of the limitations is that the poor audio quality of the chosen dataset resulted in inaccurate sentiment prediction. another drawback observed is due to the selected distance metric, the model does not map well to the intensity of the sentiment. in recent years, behavioral pattern analysis using multimodal data has received significant research attention. the rise in online video streaming and hosting websites such as youtube has facilitated an increase in sentiment expressions in multiple modalities @xcite . the availability of standard datasets containing videos annotated for emotion or sentiment has also has been conducive to this, as shown in @xcite . @xcite generated a multimodal corpus with structured interview responses, by manu-ally rating the interviewee's personality and performance for 12 structured interview questions which measure different types of job related skills. along with interviews, the interviewee's public speaking videos were also recorded and used to provide useful cues. they used visual, lexical and speech features, based on which they showed that using both non-verbal and verbal cues outperforms other cases. @xcite studied the emotions from facial expressions, reporting that standard facial expressions are sufficient to provide several clues to detect emotions. speech-based emotion analysis based on the identification of various acoustic features, such as the intensity of utterance, bandwidth, pitch, and duration, is also helpful. they achieved a 5-10% improvement in performance compared, however, the contextual relationship between utterances are considered and treated equally in this model. @xcite 's multimodal emotion recognition model extracts features from text and videos using a convolutional neural network architecture, incorporating all three modalities-visual, audio, and text. @xcite proposed a new approach for sentiment analysis from audio clips, which uses a hybrid of the keyword spotting system. the maximum entropy classifier was designed to integrate audio and text processing into a single system, and this model outperformed other conventional classifiers. @xcite developed a fusion technique for audio and video modalities using audio and video features to analyze spoken sentences for the sentiment. they did not consider the traditional transcription features to minimize human intervention. however, the model can be scaled and deployed in the real world effortlessly. they selected high dimensional features for the model to test their generalizability in the sentiment detection domain. @xcite presented a novel approach that uses deep learning to identify the sentiment of multimodal data. the modalities considered were images and text, and computer vision techniques were combined with text mining. their aim was to treat it as a study of emotion, one of the most exciting fields in psychology. they did this using a large social media dataset of tumblr posts, using which the emotion word tags attached by users was predicted, treating these as emotions reported by the user. their work combined image and text and proved that combining these two modalities conveyed more information about the sentiment that either of the modalities alone. based on the review of existing work, several limitations were observed. when features from different modalities are considered, it is crucial to find only those features that influence the label. thus, we aim to address the issue of feature selection by experimenting with different feature selection algorithms. many features are strongly correlated to each other, and considering these strongly correlated features together will not add a lot of value in predicting a label. identifying and removing such features that are strongly correlated to each other and considering only one such feature in predicting the label can be more beneficial. ## proposed approach in this section, the proposed model and its associated processes are described in detail. our approach is built on all three modalities, and the data modality-specific preprocessing techniques and various algorithms used to classify the data for each of the labels, are presented. for experimental validation, we used the mit interview dataset @xcite , which consists of recordings of 138 mock job interviews of 69 candidates preintervention and post-intervention. it contains amazon mechanical turk worker scores for each video, which when averaged gives the final score for each of the labels. it includes the audio files as well, which we use for audio analysis. in contrast to @xcite who used regression as their evaluation metric, we use classification, with the numbers 1 to 7 representing the level of performance. a score of 1 for any label is treated as very bad performance whereas a score of 7 is treated as exceptionally good. we use class labels as, our objective is to provide users with feedback based on the class label. ## experimental results and analysis for extensively evaluating the proposed multimodal analytics pipeline, various combinations of prosodic, visual, and lexical features were experimented with, and used to train the four different classifiers, discussed in section 3. each classifier is trained to predict from 9 different class labels -eye contact, speaking rate, engaged, pauses, calmness, not stresses, focused, authentic, and not awkward. experiments using the fused multimodal feature vector were performed, the results of which are tabulated in two models performed well on the dataset to achieve an accuracy of 78.57% on the calmness parameter -the support vector classifier with the benjamini-hochberg technique and the random forest classifier with k best feature selection technique. the random forest classifier and support vector classifier achieved an accuracy of 82.14% for not stressed label, while the random forest classifier with the family-wise error technique outperformed other variations for the focused label. for the authentic label, the best accuracy obtained was only 67.87% using the random forest classifier with the family-wise error technique of feature selection when lexical and facial features were used. for the not awkward label, the random forest classifier showed the best performance at 60.71%, though still low when compared to other class labels. random forest performed the best for 8 out of 9 labels. this is because it selects features that contribute the most to the classification as it considers the average of all predictions, canceling out the bias. however, mlp underperformed on most of the parameters. another objective was to check how the different modalities measure up when feature sets using any two modalities are created and ml models are trained using these features. this basically provides insights into which modalities provide an edge in capturing personality-specific traits. to assimilarly, we also considered each of the three modalities on their own, that is, the audio, video and lexical feature sets. again, the best performing classifier, random forest was trained separately on the one-modality feature vectors, and label predic-tion performance was observed. ## concluding remarks in this paper, approaches to automatically assess candidates' strengths and weaknesses during an interview, using the video, audio, and transcripts of the interview was presented. various preprocessing steps, including normalization and feature extraction for each of the three modalities was performed, followed by feature selection to select the best features from each modality. label classification was performed using four machine learning models -random forest, support vector classifier, multitask lasso, and multi-layer perceptron model on the optimal set of fused features and their variations. effect of various combinations of modalities and feature selection techniques are experimented with. the models were trained for prediction with respect to nine labels to evaluate the candidate. experiments revealed that the random forest classifier outperformed all other models for 8 out of the 9 labels considered. the current dataset has only 169 videos, making it difficult to get a very high accuracy for all the labels. the dataset could be expanded to include more interview videos that are scored by amazon turk workers. we also aim to improve the predictions by incorporating behaviors such as hand movements and body posture to get a refined understanding of the candidate's performance. the current model will be integrated into a web application that can be used as a feedback tool to train candidates for interviews by providing them with real-time feedback on their performance and pointers to manage their strengths and weaknesses. the scores can then be interpreted to give meaningful suggestions to the candidate for boosting their interview performance.