R EUSING P REPROCESSING D ATA AS A UXILIARY S UPERVISION IN C ONVERSATIONAL A NALYSIS
Anonymous authors
Paper under double-blind review
A BSTRACT
Conversational analysis systems are trained using noisy human labels and often require heavy preprocessing during multi-modal feature extraction.
Using noisy labels in single-task learning increases the risk of over-ﬁtting.
However, including auxiliary tasks could improve the performance of the primary task learning.
This approach is known as Primary Multi-Task Learning (MTL).
A challenge of MTL is the selection of relevant auxiliary tasks that avoid negative transfer.
In this paper, we explore how the preprocessed data used for feature engineering can be re-used as auxiliary tasks in Primary MTL, thereby promoting the efﬁcient use of data in the form of auxiliary supervision learning.
Our main contributions are 1) the identiﬁcation of sixteen relevant auxiliary tasks the method of distributing learning capacity between the primary and auxiliary tasks , and (3) the relative supervision hierarchy between the primary and auxiliary tasks
Exten sive experiments on IEMOCAP and SEMAINE data validate the improvements over single-task approaches and suggest that it may generalizes across multiple primary tasks.
I NTRODUCTION
The sharp increase in uses of video-conferencing creates both a need and an opportunity to better understand these conversations (Kim et al., 2019a).
In post-event applications, analyzing conversations can give feedback to improve communication skills (Hoque et al., 2013; Naim et al., 2015).
In real-time applications, such systems can be useful in legal trials, public speaking, e-health services, and more (Poria et al., 2019; Tanveer et al., 2015).
Analyzing conversations requires both human expertise and a lot of time, which is the problem that many multimodal conversational analysis systems are trying to solve with automation.
However, to build such systems, analysts often require a training set annotated by humans (Poria et al., 2019).
The annotation process is costly, thereby limiting the amount of labeled data.
Moreover, third-party annotations on emotions are often noisy.
Noisy data coupled with limited labeled data increases the chance of overﬁtting (James et al., 2013).
From the perspective of feature engineering to analyze video-conferences, analysts often employ pre-built libraries (Baltruˇsaitis et al., 2016; Vokaturi, 2019) to extract multimodal features as inputs to training.
This preprocessing phase is often heavy, and the resulting features are only used as inputs.
In this paper, we investigate how the preprocessed data can be re-used as auxiliary tasks in Primary Multi-Task Learning (MTL), thereby promoting a more productive use of data, in the form of auxiliary supervised learning.
Speciﬁcally, our main contributions are (1) the identiﬁcation of relevant auxiliary tasks, (2) the method of distributing learning capacity between the primary and auxiliary tasks, and (3) the relative supervision hierarchy between the primary and auxiliary tasks.
We demonstrate the value of our approach through predicting emotions on two publicly available datasets, IEMOCAP (Busso et al., 2008) and SEMAINE (McKeown et al., 2011).
R ELATED W ORKS AND H YPOTHESES
Multitask learning has a long history in machine learning (Caruana, 1997).
In this paper, we focus on Primary MTL, a less commonly discussed subﬁeld within MTL (Mordan et al., 2018).
Primary
MTL is concerned with the performance on one (primary) task – the sole motivation of adding auxiliary tasks is to improve the primary task performance.
In recent years, primary MTL has been gaining attention in computer vision (Yoo et al., 2018; Fariha, 2016;
Yang et al., 2018; Mordan et al., 2018; Sadoughi & Busso, 2018), speech recognition (Krishna et al., 2018; Chen & Mak, 2015; Tao & Busso, 2020; Bell et al., 2016; Chen et al., 2014), and natural language processing (Arora et al., 2019; Yousif et al., 2018; Zalmout & Habash, 2019; Yang et al., 2019; Du et al., 2017).
The addition of multiple tasks increases the risk of negative transfer (Torrey & Shavlik, 2010; Lee et al., 2016; 2018; Liu et al., 2019; Simm et al., 2014), which leads to many design considerations.
Two of such considerations are, identifying (a) what tasks are relevant and (b) how much of the model parameters to share between the primary and auxiliary tasks.
In addition, because we are performing Primary MTL, we have the third consideration of (c) whether we should prioritize primary supervision by giving it a higher hierarchy than the auxiliary supervision.
In contrast with previous works, our approach (a) identiﬁes sixteen relevant auxiliary targets, (b) dedicates a primary-speciﬁc branch within the network, and (c) investigates the efﬁcacy and generalization of prioritizing primary supervision across eight primary tasks.
Hypothesis H1:
The introduced set of auxiliary supervision features improves primary MTL.
We introduce and motivate the full set of sixteen auxiliary supervisions, all based on existing literature: these are grouped into four families, each with four auxiliary targets.
The four families are (1) facial action units, (2) prosody, (3) historical labels, and (4) future labels: (1) Facial action units, from the facial action coding system identiﬁes universal facial expressions of emotions (Ekman, 1997).
Particularly, AU 05, 17, 20, 25 have been shown to be useful in detecting depression (Yang et al., 2016a;
Kim et al., 2019b) and rapport-building (Anonymous, 2021).
(2) Prosody, the tone of voice – happiness, sadness, anger, and fear – can project warmth and attitudes (Hall et al., 2009), and has been used as inputs in emotions detection (Garcia-Garcia et al., 2017).
(3 and 4) Features at different historical time-points ( X t , X t − 1 , X t − 2 etc.) is a common practice in statistical learning, especially in time-series modelling (Christ et al., 2018).
Lastly, future laboratory test results are proven to be useful to train a pneumonia detection model (Caruana et al., 1996).
Inspired by their work, we propose using historical ( Y t − 4 , Y t − 3 , Y t − 2 , Y t − 1 ) and future ( Y t +4 , Y t +3 , Y t +2 , Y t +1 ) target labels as auxiliary targets.
Given that the actions and prosody families have been proven useful as inputs, they are extracted as features.
We propose to explore whether they can be reused as supervision (see Fig. 1).
Our hypothesis H1 is that re-using them as auxiliary supervision improves primary MTL.
This is related to using hints in the existing MTL literature, where the auxiliary tasks promotes the learning of the feature (Cheng et al., 2015; Yu & Jiang, 2016).
Hypothesis H2: When the primary branch is given maximum learning capacity, it would not be outperformed by models with primary branch having less than the maximum learning capacity.
Deeper models with higher learning capacity produce better results (Huang et al., 2019; Nakkiran et al., 2019; Menegola et al., 2017; Blumberg et al., 2018;
Romero et al., 2014).
Also, since the auxiliary branch is shared with the primary supervision, the auxiliary capacity should be limited to enforce information transfer and improve performance (Wu et al., 2020).
Therefore, given a ﬁxed learning capacity budget, our hypothesis H2 implies that that we should allocate the maximum learning capacity to the primary branch because we care only about the primary task performance.
Hypothesis H3:
Auxiliary supervision at the lower hierarchy yields better primary MTL as compared to ﬂat-MTL.
Having the auxiliary tasks at the same supervisory level as the primary task is inherently sub-optimal because we care only about the performance of the primary task (Mordan et al., 2018).
To prioritize the primary task, we could change the model architecture such that the auxiliary supervision is at the lower hierarchy than the primary supervision, which will be discussed in the next section.
M ODEL A RCHITECTURE 3. F LAT -MTL H IERARCHICAL A TTENTION M ODEL
We start with an introduction of the Hierarchical Attention Model (HAN) (Yang et al., 2016b).
We chose HAN because of its easy interpretability as it only uses single-head attention layers.
There are four parts to the HAN model, (1) text input, (2) word encoder, (3) talkturn encoder, and (4) the predictor.
In our application, we perform our predictions at the talkturn-level for both IEMOCAP and SEMAINE.
For notation, let s i represent the i -th talkturn and w it represent the t -th word in the i -th talkturn.
Each single talkturn can contain up to T words, and each input talkturn can contain up to L past talkturns to give content context (to discuss in section 4.2).
Text Input Given a talkturn of words, we ﬁrst convert the words into vectors through an embedding matrix W e .
Word encoder
The word encoder comprises of bidirectional
(Bahdanau et
We then into
[Equation]
Talkturn encoder With the current and past talkturn embeddings (content context, to discuss in section 4.2), the talkturn encoder aggregates them into a single talkturn in a similar fashion, as shown below.
[Equation]
The simplest way of adding the sixteen (four auxiliary targets from four families of auxiliary supervision) auxiliary task predictors would be to append them to where the primary task predictor is, as illustrated in Fig.
That way, all predictors use the same representation v .
We refer to this architecture as ﬂat-MTL because the auxiliary supervision is at the same level as the primary supervision.
We are unable to test H2 and H3 using this architecture.
3. HAN-ROCK
We adapted 1 the ROCK architecture (Mordan et al., 2018) which was built for Convolutional Neural
Networks (LeCun et al., 1995) found in ResNet-SSD
(He et al., 2016; Liu et al., 2016) to suit GRUs (Bahdanau et al., 2014) found in HAN (Yang et al., 2016b)
(see Fig. 3).
To study H3 , we bring the auxiliary task predictors forward, so that the auxiliary supervision is of a lower hierarchy than primary supervision (see Fig. 3).
It is of a lower hierarchy because the backpropagation from the primary supervision is able to temper the back-propagation from the auxiliary supervision but not vice-versa.
This also sets us up to study H2 , the impact of distributing different learning capacities to the auxiliary and primary branch.
Each of the auxiliary tasks has its own talkturn encoder but shares one word encoder in the auxiliary branch (to keep the network small).
Subscript a indicates whether the word encoder is for the primary or auxiliary branch:
[Equation]
Each task has its own talkturn encoder.
Subscript b indicate which of the seventeen tasks – the primary talkturn task or one of the sixteen auxiliary tasks – is the talkturn encoder is dedicated to:
- tary implementation the review phase.
−→ h abi =
−−−−→ GRU ab ( s ai ) , i ∈ [1 , L ] , a ∈ { pri, aux } , b ∈ { pri, aux 1 , aux 2 , ..., aux 16 } ←− h abi =
←−−−− GRU ab ( s ai ) , i ∈
[ L, 1] , a ∈ { pri, aux } , b ∈ { pri, aux 1 , aux 2 , ..., aux 16 }
[Equation]
The seventeen talkturn embeddings ( v ab ) goes through a concatenation, then the single head attention, aggregating talkturn embeddings across seventeen tasks into one talkturn embedding for the primary task predictor.
Subscript c pertains to the fusion module.
concatenation: v
[Equation]
v ab ) , a ∈ { pri, aux } , b ∈ { pri, aux 1 , aux 2 , ..., aux 16 } attention
[Equation]
overall primary talkturn vector: v = Σ c α c v c
E XPERIMENTS 4. D ATA AND P RIMARY T ASKS
We validate our approach using two datasets with a total of eight primary tasks: the IEMOCAP (Busso et al., 2008) and the SEMAINE (McKeown et al., 2011) datasets.
Both datasets are used in multimodal emotions detection research (Poria et al., 2019), and have ﬁne-grained emotions annotations at the talkturn-level.
We divided the datasets into train, development, and test sets in an approximate 60/20/20 ratio such that the sets do not share any speaker.
Please refer to Appendix A. for details on the splits.
The target labels of the eight primary tasks are all at the talkturn-level.
The four primary tasks of IEMOCAP consists of the four-class emotions classiﬁcation (angry, happy, neutral, sad), and three regression problems – valance (1-negative, 5-positive), activation (1-calm, 5-excited), and dominance (1-weak, 5-strong).
The primary tasks of SEMAINE consist of four regression problems – activation, intensity, power, valence.
We use two standard metrics, mean absolute error (MAE), and 4-class weighted mean classiﬁcation accuracy, MA(4), to evaluate the hypotheses.
4. I NPUT
Multimodal feature extraction is computed using the MONAH framework (Anonymous, 2021).
This framework uses a variety of pre-trained models to extract nine multimodal features, associated with the prosody of the speech and the actions of the speaker, and weaves them into a multimodal text narrative.
We refer the reader to Anonymous (2021) for the details and efﬁcacy of the MONAH framework.
The beneﬁt of the created narrative is that it describes what is said together with how it is said for each talkturn, giving richer nonverbal context to the talkturn (see Fig.
4 for an example).
In addition to nonverbal context, we concatenated a variable number of preceding talkturns to the current talkturn as content context.
Content context has been proven to be useful in DialogueRNN (Majumder et al., 2019).
The content-context size is tuned as a hyperparameter.
The resulting multimodal text narrative, consisting of both nonverbal and content context, is used as the sole input to the models.
The impact of progressively removing nonverbal and content context is not a contribution of this paper and hence not analyzed.
4. A UXILIARY T ARGETS
We ﬁrst clarify the method of extraction for the auxiliary families.
The OpenFace algorithm (Baltruˇsaitis et al., 2016) is used to extract the four continuous facial action units (AU) – AU 05, 17, 20, 25.
The Vokaturi algorithm (Vokaturi, 2019) is used to extract the four continuous dimensions in the tone of voice – happiness, sadness, anger, and fear.
As for historical and future features, we simply look up the target label for Y t − 4 , Y t − 3 , Y t − 2 , Y t − 1 , and Y t +4 , Y t +3 , Y t +2 , Y t +1 .
If any of the historical or future labels is not available, for example, Y t − 4 is not available for the third talkturn, we substitute with the next nearest non-missing label.
For all auxiliary targets that reused the input features (actions and prosody), we converted them into a percentile rank that has the range [0,1] using the values from the train partition.
This is a subtle but note-worthy transformation.
When reusing an input as an auxiliary target, it would be trivial if the input can easily predict the target.
For example, given the following MONAH transcript as input, “The woman sadly and slowly said no.”
It would be trivial to use a binary (quantized) auxiliary target of “was the tone sad?”
because we would only be training the model to look for the word “sadly”.
However, if the auxiliary target is a percentile rank (less quantized) of the sadness in tone, then the presence of the word “sadly” increases the predicted rank, but the model could still use the rest of the nonverbal cues (“slowly”) and what is being said (“no”) to predict the degree of sadness.
That way, representations learnt for the auxiliary tasks uses more of the input.
Additionally, this transformation also helped us side-step the decision to use the multimodal streams exclusively in the inputs or in the supervision (Caruana & De Sa, 1997; Caruana & de Sa, 1998) because we could use them in both.
Percentile rank also has the convenient property of having the range [0,1].
We scaled the percentile ranks so that they all have the same range as the primary task (see appendix A.2 for transformation details).
This is done to ensure that if we assigned equal loss weights to all tasks, the contribution of every task (both auxiliary and primary) is of the same order of magnitude, an important consideration for MTL (Gong et al., 2019; Hassani & Haley, 2019; Sener & Koltun, 2018).
4. M ODELS , TRAINING , AND HYPERPARAMETERS TUNING
The overall loss is calculated as the weighted average across all seventeen tasks: (1) we picked a random weight for the primary task from the range [0.50, 0.99], this ensures that the primary task has the majority weight.
For the remaining weights (1 - primary weight), we allocated them to the sixteen auxiliary tasks by: (a) random, (b) linearly-normalized mutual information, or (c) softmax-normalized mutual information.
(a) is self-explanatory.
As for (b) and (c), mutual information has been shown that it is a best predictor – compared to entropy and conditional entropy – of whether the auxiliary task would be helpful to primary MTL (Bjerva, 2017).
We computed the mutual information (vector m ) of each auxiliary variable with the primary target variable (Kraskov et al., 2004; Ross, 2014) using scikit-learn (Pedregosa et al., 2011).
Then, we linearly-normalized or softmax-normalized m to sum up to 1.
Finally, we multiplied the normalized m with the remaining weights from (2); this ensures that the primary weight and the sixteen auxiliary weight sum up to one.
(a), (b), and (c) have ten trials each during hyper-parameters tuning.
Two variants of the HAN architectures are used (Fig 2 and 3).
Glove word embeddings (300dimensions) are used to represent the words (Pennington et al., 2014).
Hyper-parameters tuning is crucial because different combinations of primary and auxiliary tasks require different sets of hyperparameters.
For hyperparameters tuning, we used random search (Bergstra & Bengio, 2012) with thirty trials.
We tuned the learning rate, batch size, L2 regularization, the number of GRUs assigned to the primary and auxiliary branches, the auxiliary weights assignment, the content-context size, and lastly the GRU dropout and recurrent dropout.
Appendix A.3 details the range of hyperparameters tuned.
Training is done on a RTX2070 or a V100 GPU, for up to 350 epochs.
Early stopping is possible via the median-stopping rule (Golovin et al., 2017) after the ﬁfth epoch and after every two epochs (i.e. at epoch number 5, 7, 9, ... , 349).
Appendix A.4 details the hyperparameters of models that performed the best on the development set.
For hypotheses testing, we bootstrapped conﬁdence intervals of the test set performance of both baseline and challenger models as well as the conﬁdence intervals of the differences in performances.
Please see appendix A.5 for details on the bootstrap procedure.
R ESULTS AND D ISCUSSION
The key takeaways are: ( H1 )
The introduced set of auxiliary supervision improves primary MTL signiﬁcantly in six of the eight primary tasks.
( H2 ) Maximum learning capacity should be given to the primary branch as a default.
( H3 ) HAN-ROCK is unlikely (in one of the eight tasks) to degrade primary MTL signiﬁcantly, and sometimes signiﬁcantly improves it (in four of the eight tasks).
( H1 ): To test H1 (whether the introduced set of auxiliary supervision improves Primary MTL), we ﬁrst train the model with all sixteen auxiliary targets (from families actions, prosody, historical, and future).
Then, to differentiate the effect from the historical and future supervision, we set the loss weights from historical and future targets to be zero; effectively, there is only supervision from eight auxiliary targets (actions and prosody).
Lastly, for the baseline model (no auxiliary supervision), we set the loss weights from all sixteen auxiliary targets to zero.
Given auxiliary supervision, the model signiﬁcantly outperforms the baseline of not having auxiliary supervision in six out of the eight primary tasks (Table 1).
Comparing the baseline model with the model with two auxiliary target families, they signiﬁcantly outperformed the baseline model in ﬁve out of eight primary tasks.
The addition of two auxiliary target families (historical and future labels) sometimes signiﬁcantly improved primary MTL (valence in IEMOCAP), but it also sometimes signiﬁcantly made it worse (activation and intensity in SEMAINE).
This shows that the value of auxiliary tasks, and the associated risk of negative transfer, depends on the auxiliary task.
[Table]
We found a supplementary ﬁnding whilst studying H1 : the improvement on lower-resource (lower counts) classiﬁcation labels is bigger than higher-resource labels (Table 2).
There is a statistically signiﬁcant improvement for the “sad” label, and no signiﬁcant changes for the rest of the labels.
[Table]
( H2 ): To test H2 (whether maximum learning capacity should be given to the primary branch), we let P represent the number of GRU assigned to the primary talkturn encoder, and A represent the number of GRU assigned to each of the sixteen auxiliary talkturn encoder.
We constrained P + A to be equal to 257.
During our experiments, we set P to 1, 64, 128, 192, and 256.
We set 256 as the baseline model because it is the maximum learning capacity we can give to the primary branch while giving 1 GRU ( = 257 − 256 ) to each of the sixteen auxiliary talkturn encoders.
In all primary tasks, the baseline model of assigning 256 GRUs to the primary branch is not signiﬁcantly outperformed by models that assigned 1, 64, 128, 192 GRUs (Table 3).
Generally, the performance decreased as the number of GRUs assigned to the primary talkturn encoder decreased from 256 to 1, and we observed signiﬁcantly worse performance in two out of eight tasks – in power and valence in SEMAINE.
Also, assigning 256 GRU to the primary talkturn encoders and 1 to each of the sixteen auxiliary talkturn encoders yields the smallest model 2 , and thus trains the fastest.
Therefore, we recommend that the maximum capacity be given to the primary branch as a default.
That said, the presence of an auxiliary branch is still important.
The baseline of H1 (no auxiliary supervision, Table 1) can be approximated as P = 256 + 16 × 1 , A =0 because the model architecture is the same, except that the loss weights of all auxiliary tasks are zero.
We compared the former to the baseline in Table 3, and found that four out of eight primary tasks have signiﬁcant improvements by changing the number of talkturn encoders assigned to each auxiliary task from zero to one .
[Table]
( H3 ): To test H3 (whether auxiliary supervision should be given a lower hierarchy), we compare the results from the ﬂat-MTL HAN architecture (baseline) against the HAN-ROCK architecture (Table 4).
Placing auxiliary supervision at the lower hierarchy signiﬁcantly improves primary MTL in four out of eight tasks.
In only one out of eight tasks (power in SEMAINE), auxiliary supervision signiﬁcantly degrades primary MTL.
[Table]
C ONCLUSION
We proposed to re-use feature engineering pre-processing data as auxiliary tasks in primary MTL.
Three hypotheses were tested for primary MTL.
The experimental results conﬁrm H1 – Introducing auxiliary supervision resulted in better performance in most primary tasks.
For H2 , maximum learning capacity should be given to the primary branch.
Lastly, for H3 , placing the auxiliary supervision in a lower hierarchy is unlikely to hurt performance signiﬁcantly, and it sometimes signiﬁcantly improves performance.
This is encouraging news for multi-modal conversational analysis systems as we have demonstrated how pre-processed data can be used twice to improve performance, once as inputs, and again as auxiliary tasks.
A challenge to be addressed is the apriori selection of the auxiliary targets.
Future research could investigate targets selection, including how to use a much larger range of auxiliary targets, how to decide the optimum number of auxiliary targets, and whether it is possible to perform these automatically in an algorithm.
A A PPENDIX A. D ATASET PARTITIONS
We detail the dataset partition in 5 for aid reproducibility.
[Table]
A. S CALING THE AUXILIARY TARGETS
We detail the operations in scaling the percentile scores that range [0,1] to various primary tasks.
For IEMOCAP Primary Tasks that are regression problems, we multiply the percentile score byand add 1 to obtain the range [1,5].
For IEMOCAP classiﬁcation task, we leave the auxiliary targets in the range of [0,1].
As for SEMAINE tasks, which are all regression problems, we multiply the percentile score by 2 and minus 1 to obtain the range [-1,1].
A. R ANGE OF H YPER PARAMETERS TUNED
We document the range of hyper parameters tuned in Table 6 to aid reproducibility.
A. H YPER PARAMETERS WITH BEST DEVELOPMENT SET PERFORMANCE
We document our best performing hyper parameters in Table 7 to aid reproducibility.
A. D ETAILS OF COMPUTING THE BOOTSTRAP CONFIDENCE INTERVAL
Baseline models for each hypothesis are detailed in section 2.
All non-baseline models are referred to as challenger models.
We created 1000 bootstrap samples of the test set performance by (1) resampling the development set performance, then (2) selecting the set of hyperparameters that resulted in the best development set performance, and (3) looking up the test set performance given the set of best-performing hyperparameters for the development set.
To judge whether the challenger outperforms the baseline, we computed the 95 percent conﬁdence interval by (1) performing
[Table]
element
[Table]
wise subtraction between the resampled test set performance of the baseline against the challenger, (2) removing the top and bottom 2.5 percent from the differences
[Table]
and (3) observing whether the remaining 95 percent conﬁdence interval includes zero.
If it does not include zero, then the difference is statistically signiﬁcant
[Table] [Table]
AU05 loss weight 0.
AU17 loss weight 0.
AU20 loss weight 0.
AU25 loss weight 0.
Happy tone loss weight 0. Sad tone loss weight 0.
Angry tone loss weight 0. Fear tone loss weight 0.
Y t − 1 loss weight 0.
Y t − 2 loss weight 0.
Y t − 3 loss weight 0.
Y t − 4 loss weight 0.
Y t +1 loss weight 0.
Y t +2 loss weight 0.
Y t +3 loss weight 0.
Y t +4 loss weight 0.