Learning Backchanneling Behaviors for a Social Robot via Data Augmentation from Human-Human Conversations
Anonymous Author(s)
AfﬁliationAddress email
Abstract: Backchanneling behaviors on a robot, such as nodding, can make talking to a robot feel more natural and engaging by giving a sense that the robotis actively listening.
For backchanneling to be effective, it is important that thetiming of such cues is appropriate given the humans’ conversational behaviors.
Recent progress has shown that these behaviors can be learned from datasets ofhuman-human conversations.
However, recent data-driven methods tend to overﬁtto the human speakers that are seen in training data and fail to generalize well topreviously unseen speakers.
In this paper, we explore the use of data augmentation for effective nodding behavior in a robot.
We show that, by augmentingthe input speech and visual features, we can produce data-driven models that aremore robust to unseen features without collecting additional data.
We analyze theefﬁcacy of data-driven backchanneling in a realistic human-robot conversationalsetting with a user study, showing that users perceived the data-driven model to bebetter at listening as compared to rule-based and random baselines.
Introduction
Social robots that can communicate with people naturally and effectively serve a wide variety ofimportant daily life contexts, from companion robots for older adults [1], robots that help childrenwith social-emotional development [2], or for robots that support teens to cope with stress and mentalhealth challenges [3].
Verbal communication is the heart of many such applications of social robots.
If social robots are to become a part of daily life, it is critical that talking to a robot feels naturaland engaging.
One way robots can remain active in a conversation while the human is speaking isthrough backchanneling .
Backchannels, such as smiles, nods or short utterances like “hmm”, arecues that humans provide to each other during a conversation to indicate to the speaker that they arestill actively listening and understanding [4].
When timed well, these behaviors make a conversationfeel smooth and natural [5].
When performed by a robot or virtual agent, non-verbal backchannelresponses have been shown to give users an increased perception of attentive listening, an increasedfeeling of rapport, and a desire to continue interacting [6, 7].
Research into the use of backchannels in human communication has found speaker cues that canelicit backchannel responses, such as periods of low pitch [8], rapidly changing intonations [9], andcertain patterns of speaker gaze [10].
Many early approaches to programming backchannel behaviorhave been based on rules involving these cues [8, 11, 12].
Others have taken data-driven approaches,learning backchanneling behavior from datasets of human-human conversations.
Over time, severaldata-driven approaches have been explored including decision trees [13, 14], sequential probabilisticmodels [15, 16, 17], and more recently, deep neural networks [18, 19, 20, 21].
Deep neural networkshave shown promising performance in predicting when to backchannel, however, these approachestypically suffer from the generalization problem in which the learned backchanneling policies tendto overﬁt to human speakers that are seen in training data and fail to generalize well to previouslyunseen speakers.
One of the major contributing factors to this problem is data scarcity.
Learningbackchanneling behavior requires annotated video datasets of human-human conversations which are expensive and time-consuming to collect.
As a result, the datasets used to train backchannelingalgorithms typically have a small number of human speakers.
For example, the Interactive Emotional
Dyadic Motion Capture Database (IEMOCAP) [22], which is the most commonly used dataset forlearning backchanneling behaviors, consists of conversations from just 10 unique speakers.
Due tothe challenges of collecting sufﬁciently large datasets, it is a crucial problem to efﬁciently learn amore generalized policy for backchanneling without collecting additional data.
Data augmentation is a technique that has contributed to signiﬁcant improvements in performanceunder conditions of data scarcity in domains such as image classiﬁcation [23, 24, 25, 26] and speechrecognition [27, 28, 29, 30].
With data augmentation, we produce new datapoints by augmentingexisting datapoints, for example by warping the original input.
This makes the training dataseteffectively larger without collecting additional data, as multiple augmented versions of a singleexample can be used to train a model.
In this paper, we present a data augmentation scheme to learn more robust non-verbal backchanneling behaviors for a social robot from human-human conversation data.
We introduce a methodthat warps and masks portions of each human speaker’s input features.
Our experiments show thatthe technique results in improved performance on the metrics of accuracy and F1-score.
Finally, weconducted a user study to examine the efﬁcacy of a data-driven model for online human-robot interaction.
As recent works have evaluated their models using pre-collected datasets, we contribute—toour knowledge–the ﬁrst user study to examine a neural network based approach to backchanneling.
Our results showed that, compared to rule-based and random baselines, the neural network approachelicits more natural and preferable interactions with a listening robot.
Related WorkModeling
Human Backchannel Behavior
The description, modeling, and understanding of human backchanneling behavior has been a subject of research since early comprehensive studies ofnon-verbal behavior in the 1970s [31, 4].
Early models sought to describe relationships betweenbackchannel behaviors and speaker cues spanning gesticulation, gaze, syntactic markers and paralanguage [5, 32].
The ability to automatically detect speaker cues, like periods of low pitch orpauses, led to the ﬁrst applications of rule-based backchanneling systems [8, 33, 11, 12].
The useof data driven methods such as decision trees [13, 14], sequential probabilistic models [15, 16, 17],and neural networks [18, 34, 19, 20, 21], has reduced the dependence on hand-designed rules.
Embodied Backchanneling Agents
The impact of backchanneling behaviors on interactions withvirtual animated agents has been extensively studied [35, 36, 37].
Despite promising applications ineducational or eldercare robotics, there are relatively few systems that have been designed to workwith a physical robot.
Lala et al.
[38] evaluated an integrated backchanneling dialogue agent on
ERICA, a realistic humanoid robot, ﬁnding that their approach was perceived as more natural andempathetic than rule-based alternatives.
Inoue et al.
[39] compared user responses to the system withthose generated by a human operator and saw that their system matched the human control in basiclistening skills but fell short in ratings for complex attributes like empathy.
Park et al.
[7] showedthat children perceived a toy robot listener using a rules-based backchannel generation approach asmore attentive compared to a random baseline.
Data Augmentation Data augmentation has been applied to a wide variety of problems to reduceoverﬁtting in neural networks.
Image augmentations, such as horizontal ﬂips and random croppingor translations, are commonly used in image classiﬁcation and detection models [23, 24, 25, 26].
Krizhevsky et al.
[24] notably achieved state-of-the-art results on image classiﬁcation and showedthat their augmentations reduced the error rate of their model by over 1%.
Additionally, speechaugmentations like speed perturbation, pitch adjustments, and additional noise are commonly usedfor automatic speech recognition [27, 28, 29, 30].
Recently, Park et al.
[29] introduced SpecAugment, which achieves state-of-the-art performance on several speech recognition tasks by maskingthe mel-spectogram representation of speech.
Although data augmentation shows promise in manydomains, it has not yet been applied to robot backchanneling, which we explore in this work.
[Figure]
Method
The goal of this work is to improve listening behaviors in a robot during a conversation with a human.
Following the recent work of Ruede et al.
[19], we model our policy as a recurrent neural networkthat maps sequences of input from the speaker to output actions on the robot.
In this paper, we focuson nodding as a backchannelling modality because of it’s prevalence and utility in human-humanconversations [40, 41, 42], but we expect that the techniques we present will generalize to otherbackchanneling modalities.
We aim to enable accurate prediction while maintaining the feasibilityof deploying the model on computationally constrained robotics platforms.
Acoustic Features To summarize the speaker’s acoustic features we extract the fundamental frequency (F0) and 14th order mel-frequency cepstrum coefﬁcients (MFCCs).
F0 is an estimation ofthe speaker’s pitch and the MFCCs represent the short-term power spectrum of the sound.
Both features are commonly used in speech processing and can be calculated efﬁciently without introducingsigniﬁcant latency.
We extract the features over 20ms windows and concatenate them to produce asingle acoustic feature representation.
This conﬁguration signiﬁcantly compresses the audio signalwhile maintaining the ability to distinguish prosodic cues and phonemes.
Visual Features As a summary visual feature we extract the speaker’s head pose, as an euler angle,using a fast facial landmark detector [43] and estimating the correspondence of points to a 3D headmodel by solving the perspective-n-point problem.
This was informed by work showing gaze as acue for backchannels [10].
To moderate noise inherent in this pipeline, we smooth the angle using a
Kalman ﬁlter.
Model We model the backchanneling policy as a long short term memory (LSTM) recurrent neuralnetwork [44], which is commonly used for modeling sequential data.
At each step, the networkreceives the last two seconds of features as input and produces a binary output (Figure 1).
Data Augmentation As we mentioned above, data scarcity is an issue when training neural networks for backchanneling tasks.
Therefore, given a training dataset of the aforementioned inputfeatures and a binary output, we propose using data augmentation to add one new example for everyexample in the dataset.
Following Park et al.
[29], we augment the MFCC features by warping them across time, maskingblocks of utterances over time, and masking blocks of consecutive frequency channels.
An exampleof these augmentations can be seen in Figure 2.
We augment the F0 features by warping them acrosstime and masking blocks of utterances over time.
Finally, we augment the visual summary head posefeature by randomly warping the head pose in space and time, and by masking random blocks ofhead poses over time.
These augmentations are intended to improve the model’s robustness to partialloss of information (frequency, segments of speech, or head pose information) and deformationsacross time.
Data Collection and Systematic Evaluation
To evaluate our data-driven backchanneling models, we gathered a dataset of human non-verbalbackchanneling behaviors.
While we considered existing datasets such as IEMOCAP [22] and
CCDb [45], no existing dataset met our criteria of being both unscripted and structured, with distinctspeaker and listener roles.
We believe these criteria will promote natural head nods that are moreappropriate for a listening robot.
This lead us to collect our own dataset to meet these criteria.
Using a custom peer-to-peer video chat tool, we recorded audio and video of conversational interactions between two people.
Each interaction was structured around a prompt, with one person taskedto respond to the prompt while the other person was listening and performing natural backchannels.
Every session was composed of one researcher and one participant selected from that researcher’ssocial network.
After obtaining consent, participants were given instructions on how to positionthemselves for optimal data collection—in a well-lit room, with no bright lights, and with their headin the middle of the frame.
The researcher and participant then engaged in 10 interactions wherethey took turns prompting each other and listening to the other talk for 1-2 minutes.
The promptsconsisted of two random prompts from each of the following categories:
• Happy (e g. “Tell me about your most fun memory from last summer”)
• Neutral (e.g. “Tell me about your day yesterday”)
• Teaching (e.g. “Teach me how to cook your favorite dish”)
• Frustrated (e.g. “Tell me about a pet peeve of yours”)
• Practice Talk (e.g. “You are giving a speech at your best friend, Sam’s, wedding tomorrow.
Practice that speech in front of me”).
These categories were chosen to reﬂect the general scenarios in which we envision users speakingto an actively listening social robot.
Accounting for interactions missing due to recording issues,we collected a total of 84 interactions from 12 participants (9 male, 3 female), 3 of whom wereresearchers.
Interactions lasted 97.2 seconds on average (SD=31.2).
4.1 Annotation
We annotated listener nod behavior using a semi-automatic eye-position based scheme.
The left eyekeypoint position was extracted across the duration of the video using [43].
The range of this rawpixel position was then normalized to be between 0 and 1, and smoothed.
The resulting signal wasthen processed to detect peaks and valleys.
Portions of the signal with 3 or more peaks or valleyswith no more than 50ms between them were marked as candidate nod windows.
These candidateswere then checked and adjusted manually using ANVIL [46].
After annotation was complete, anaverage of 21.5% of per-interaction listener time was labeled as nodding (SD=10.8).
4.2 Training
To train the RNNs we ﬁrst split the dataset up into examples and labels where each example is asequence of features extracted over the past 2 seconds and each label indicates whether the humansubject nodded at that point in time.
Because our dataset contains many more negative labels thanpositive labels, we account for the imbalance by assigning each class a weight that is inverselyproportional to its frequency:
α = n + + n −, w − = 1 n − · α and w + = 1 n + · α (where n − is thenumber of negative examples and n + is the number of positive examples).
We then split the datasetby individual speakers, and use 6-fold cross-validation to train the model.
In each fold, the examplesfrom two speakers are held out for validation and the remaining examples are used for training.
Wetrain the model by optimizing cross-entropy loss and perform early-stopping according to the loss.
[Table]
4.3 Results
We compare the LSTM-based approach, with and without data augmentation, with a rule-basedbaseline.
We chose Ward [11] as our baseline because it was the highest performing rule-basedapproach on our dataset.
Table 1 shows the performance of the three models in terms of accuracy,precision, recall, and F 1 -score.
Precision is the probability that predicted head nods match theactual annotated human behavior, while recall is the probability that an annotated human head nodis predicted by our model.
The F 1 -score, which is the harmonic mean of precision and recall, istypically used as the key evaluation metric for imbalanaced datasets like ours [47].
We observe that both LSTM models outperform the rule-based model and the LSTM model withdata augmentation achieves the highest accuracy, precision, recall, and F 1 -score among all of theapproaches.
Figure 3 shows a plot of the training and validation F 1 -scores across training epochs tofurther illustrate this reduction in the performance gap between training and validation.
Additionally,in Figure 4 we present examples of qualitative improvements from two unseen speakers.
[Figure]
User Evaluation
While the systematic evaluation demonstrated the capacity for data-driven models to accurately [Figure] predict backchanneling behaviors in human-human interactions, it does not capture the qualitative [Figure]
experience of interacting with a robot that uses the models to produce nodding behaviors.
We
[Figure] investigated the effectiveness of these models in generating backchanneling behaviors on a robotthrough a user study in which participants talked to a virtual robot.
We used a virtual robot to adaptto the COVID-19 pandemic.
5.1 Study Design and Procedure
Our study involved participants talking to a virtual Mayﬁeld Kuri robot that uses either a learnedmodel, a rule-based model, or a random baseline to determine when to nod.
Speciﬁcally, the threeconditions in our study were:
• Learned uses the best performing learned model, the LSTM model with data augmentation(Sec. 3), trained on the whole dataset.
The model takes in audio and visual features, andthe robot produces a nod whenever the model assigns a probability above 50% to the nodclass.• Ward [11] uses the aforementioned rule-based baseline.
The model uses a rolling windowto approximate the distribution of the speaker’s pitch and nods upon detection of: (1) aregion of pitch less than the 26th-percentile pitch level (2) continuing for at least 110 milliseconds (3) coming after at least 700 milliseconds of speech (4) no backchannels in thepreceding 800 milliseconds (5) after 700 millisecond
wait.• Random produces a nod with nod duration sampled from a normal distribution ﬁt to the durations in our collected dataset.
Wait times between nods are sampled from an exponentialdistribution ﬁt to the wait times in our collected dataset.
To allow for direct comparisons of the three conditions, we performed a within-participants study inwhich participants interacted with all three conditions.
Each condition was presented as a differentrobot, named Robot 1-3.
The order of conditions was counter-balanced.
Mirroring our data collection procedure, this evaluation took place through the same custom online video communicationinterface.
Participants were shown a simulated robot in a living space, intended to mimic a webcamdisplay of a physical robot (see Figure 5).
The robot exhibited naturalistic blinking and ﬁdgetingidle behavior, and nodded when the model speciﬁed.
After reading instructions, participants went through a familiarization phase where they talked toa robot that had the idle behavior but did not nod.
The interaction started with a question promptfrom the robot.
Participants were instructed to talk for 1-2 minutes responding to the prompt.
Thetime elapsed was displayed on the screen for guidance and participants could move on at any pointby pressing the “Complete session” button.
After familiarization, they were taken to the study,where they interacted with each of the three models.
The prompts for the study were taken from the
Frustrated category of the data collection (Sec. 4) and the order of the prompts was kept constant. After the interaction with each model, participants were taken to a questionnaire that asked themabout their perceptions of the robot and its listening and nodding behavior (Table 2). After interacting with all three models and completing all three questionnaires, participants wereasked to select the robot that seemed the most natural, that they enjoyed speaking to the most, [Table]
• The robot was listening carefully.
• The robot’s behavior was natural.
• The robot cared about what I was saying. • The robot did not care about what I was saying. • The robot was not listening to me.
• The robot understood what I was saying.
• I felt a close connection with the robot.
• Talking with the robot was similar to talking with
[Table]
friend.
• Talking with the robot was similar to talking with
[Table]
• Talking with
[Table]
was comforting
[Table]
the most, and
[Table]
nodded the least.
Finally, they were asked demographic questions.
Participants for this study were recruited on Amazon Mechanical Turk, limited to users who hada greater than 95% approval rate and who were in the U.S., and were compensated $3 for theirparticipation in this approximately 25 minute user study.
5.1.1 Results
Our study was completed by 102 participants (53 male, 46 female, 3 other; ages 18-75).
To groupthe Likert items we performed an exploratory factor analysis with promax rotation, yielding twofactors shown in the appendix Table 7.1. We interpret the ﬁrst factor as measuring the perceivednodding and listening skill of the robot. We interpret the second factor as measuring to what extenttalking with the robot elicited feelings of comfort or closeness. The factors show good reliabilitywith α = 0 .
88 and α = 0 . 92 respectively. We group the continuous frequency questions into a singlescale measuring the perceived frequency of mistakes. Figure 6 summarizes participant responses onthese scales.
Figure 7 shows the distribution of participant preferences in forced-choice questions.
We used repeated measures ANOVA to measure the statistical signiﬁcance of responses on the Likertand frequency scales.
Using a signiﬁcance threshold of α < 0 .
for all statistical tests, there wasa signiﬁcant effect on the perceived nodding skill [ F (2 , 204)
= 9 . 86 , p < 0 .
001 ] and the perceivedfrequency of mistakes [ F (2 , 204) = 4 .
56 , p < 0 .
Post hoc comparisons using the Tukey HSDtest indicated that the mean score of the learned model is signiﬁcantly higher than the random modelon both scales.
The learned model also outperforms the rule-based model, although that differenceis not signiﬁcant. [Figure]
[Figure] User responses to the forced choice questions indicate a strong preference for the learned and rulebased models over the random model.
The learned model was the most common choice for “Whichrobot made me felt most heard”, “Which robot did you enjoy speaking to the most?”, and “Whichrobot was the best listener?”.
Additionally, it was the least common choice for “Which robot did youenjoy speaking to the least?”.
While there was no signiﬁcant difference between the learned and rulebased models on our Likert and frequency scales, there is a clear preference for the learned modelover the rule-based model in the forced choice responses.
This could indicate that the differencesare too small to appear on the scales but still large enough to result in a clear preference or it couldindicate that it was difﬁcult for participants to express on a scale the differences they noticed wheninteracting with the virtual robot.
Some participants noticed the timing of nods, e.g., one said the learned model appeared “ betterprogrammed to nod at more appropriate intervals ”; another said the rule-based model would “ nodat more appropriate times and didn’t miss as many opportunities to nod ”.
While some participantswere speciﬁc about the reasoning for their choice, as in the above quotes, others could not articulatebehavioral reasons but rather answered based on how they felt after the interaction; e.g., commentsabout the learned model included that it “ seemed natural and normal ”, “ felt like he was reallylistening ”, and “ seemed more life like .”.
Some users stated that they could not discern a difference;e.g.
one user stated that there was “ no detectable difference “ and another stated “ it seemed like thisrobot had better timing when it came to nodding than before, but I’m not sure if that is actuallythe case or not ”.
Some users expressed optimism about the utility of a backchanneling robot; e.g.one user said “ I would use this to practice job interviews ” and another suggested to “ make theserobots some kind of therapy app ”.
Other participants felt that the experience of talking to the robotwas negative, highlighting important difﬁculties of social robot design; e.g. one person stated that“ Speaking to the robot felt dehumanizing. ”.
Discussion
In this paper, we apply data augmentation to the task of learning non-verbal backchanneling behaviors from human-human interactions.
Our experiments show that our data augmentation techniquereduces overﬁtting and improves backchanneling prediction performance for unseen speakers.
Dataaugmentation is not the only technique that has been developed to reduce overﬁtting, and an exciting direction for future work is to explore additional techniques such as pre-training and multi-tasklearning.
Combining these techniques with reinforcement learning as in Hussain et al.
[21] is anotherexciting future direction.
The results of our user study indicate that user perception of data-driven backchanneling behaviorsis positive, with the data-driven approach being preferred in forced choice questions and elicitingnodding behavior that appears signiﬁcantly more natural than a random baseline.
Due to COVID19 limitations our user study was conducted online with a virtual robot so we aim to repeat ourevaluation with a physical robot in future work.
Although our evaluation involved a virtual robot,the learned model was developed to work the same way on physical Kuri robots using on-boardsensors and head nods.
Future work could extend our approach to provide richer backchannelingwith additional outputs like robot facial expressions and vocalizations.
Because our work is focusedon augmenting the input features, we expect that the techniques we have presented will generalizewell to other backchanneling modalities.
Further investigation of backchanneling within the contextof larger human-robot interaction systems is another important direction for future work.
7.1 Likert item factor analysis loading matrix
[Table]