Pain Detection from Facial Videos Using TwoStage Deep Learning A new method to objectively measure pain using computer vision and machine learning technologies is presented. Our method seeks to capture facial expressions of pain to detect pain, especially when a patients cannot communicate pain verbally. This approach relies on using Facial musclebased Action Units (AUs), defined by the Facial Action Coding System (FACS), that are associated with pain. It is impractical to use human FACS coding experts in clinical settings to perform this task as it is too laborintensive and recent research has sought computerbased solutions to the problem. An effective automated system for performing the task is proposed here in which we develop an endtoend deep learningbased Automated Facial Expression Recognition (AFER) that jointly detects the complete set of painrelated AUs. The facial video clip is processed frame by frame to estimate a vector of AU likelihood values for each frame using a deep convolutional neural network. The AU vectors are concatenated to form a table of AU values for a given video clip. Our results show significantly improved performance compared with those obtained with other known methods.

TwoStream Attention Network for Pain Recognition from Video Sequences Several approaches have been proposed for the analysis of painrelated facial expressions. These approaches range from common classification architectures based on a set of carefully designed handcrafted features, to deep neural networks characterised by an autonomous extraction of relevant facial descriptors and simultaneous optimisation of a classification architecture. In the current work, an endtoend approach based on attention networks for the analysis and recognition of painrelated facial expressions is proposed. The method combines both spatial and temporal aspects of facial expressions through a weighted aggregation of attentionbased neural networksxe2x80x99 outputs, based on sequences of Motion History Images (MHIs) and Optical Flow Images (OFIs). Each input stream is fed into a specific attention network consisting of a Convolutional Neural Network (CNN) coupled to a Bidirectional Long ShortTerm Memory (BiLSTM) Recurrent Neural Network (RNN). An attention mechanism generates a single weighted representation of each input stream (MHI sequence and OFI sequence), which is subsequently used to perform specific classification tasks. Simultaneously, a weighted aggregation of the classification scores specific to each input stream is performed to generate a final classification output. The assessment conducted on both the BioVid Heat Pain Database (Part A) and SenseEmotion Database points at the relevance of the proposed approach, as its classification performance is on par with stateoftheart classification approaches proposed in the literature.