Model Transfer for Event tracking as Transcript Understanding for Videos of Small Group Interaction (2022)

Abstract: Videos of group interactions contain a wealth of information beyond the information directly communicated in a transcript of the discussion. Tracking who has participated throughout an extended interaction and what each of their trajectories has been in relation to one another is the foundation for joint activity understanding, though it comes with some unique challenges in videos of tightly coupled group work. Motivated by insights into the properties of such scenarios, including group composition and the properties of task-oriented, goal directed tasks, we present a successful proof-of-concept. In particular, we present a transfer experiment to a dyadic robot construction task, an ablation study, and a qualitative analysis.

## introduction the broad area of transcript understanding from video encompasses more than the information communicated through discussion, especially when the video captures small group interactions. in that case, each action is meaningful in the context of a broader task. from a social perspective, actions and reactions are meaningful in relation to one another. sequences of actions of an individual within an interaction are meaningful as an enactment of role taking within a group activity. building on recent work in multi-object tracking, which is a paradigm of interest in the computer vision community, this paper presents a proof-of-concept for model transfer for tracking the trajectories of participants within a small group activity. in particular, we target tightly coupled group work, which is challenging due to the close proximity of participants, intermittent motion, and periodic movement in and out of view. success tracking within such scenarios is a key enabler for joint activity understanding, which requires at the foundation tracking who has participated throughout an extended interaction and what each of their trajectories has been in relation to one another. our results demonstrate positive impact of three different enhancements motivated by consideration of the nature of tightly coupled collaborative group activities. in many contexts of learning and work, dyads and small groups work together to accomplish a goal. the ability to understand a video capturing this type of interaction has many real world applications. for example, video recordings of such interactions are very common forms of data for research on group learning, communication, and group work. real time understanding of group interactions has also been used to trigger support for group behavior in order to improve outcomes. facilitators overseeing multiple groups can use reports of this real time understanding to support decision making regarding how they divide their attention between groups. in the remainder of the paper, we first offer a review of related work both from the computer vision community and from the multi-modal learning analytics community. next, we present our technical approach, extending recent successes using deepsort for multiple object tracking (mot). we then present a successful experiment producing results demonstrating improvement over a stateof-the-art baseline, as well as an ablation study to investigate the individual effects of each enhancement and a qualitative analysis of those effects. ## background & related work from a technical perspective, the work reported in this paper has its roots in recent directions in the multi-object tracking (mot) literature. however, as the intended application is within areas of research and practice focused on supported group work and learning, we also review work from the field of learning analytics. ## method to perform tracking on videos in small group interaction, we explore the widely used online deep-sort algorithm developed for multiple-object tracking (mot) benchmark. we extend the deep-sort algorithm to improve transfer of the model from the task on which it was trained, namely tracking pedestrians walking on streets, to our group work setting. we begin with an explanation of the well-known deepsort algorithm and then discuss the extensions we have added. ## analysis we perform ablation of our model to assess the impact of each of our extensions. smooth vs non smooth: for the smoothing ablation, instead of adding the new appearance feature as discussed in 3.2, we average all appearance features so that each frame feature receives equal importance such that smoothing is not applied. without smoothing, abrupt changes in appearance or slight movement will likely change inference more drastically. we expect that without smoothing, the model is more likely to confuse ids when there is motion. quantitatively, in qualitative analysis of the smoothing ablation, we find errors that align with these expectations. note in figure 2a , the track in the purple bounding box is incorrectly switched when the individual leans forward in the non-smoothing model. however, with smoothing, the model correctly maintains their track. from this ablation, we conclude that smoothing helps decrease noise in abrupt changes of appearance in cases of obstruction and motion. limited vs unlimited number of tracks: we also examine the ablation that removed the limit of the number of tracks that can be created. by limiting the maximum number of tracks to the number of participants within the activity, we hypothesized that the model would maintain tracks more consistently with less likelihood of creating irrelevant tracks during motion. the results in this can be seen qualitatively in figure 2b . due to motion by the individual in the cyan bounding box, the ablation model mistakes motion for a new person and incorrectly creates a new yellow bounding box track around the individual. when tracks are limited, the model does not have the ability to create a new track and correctly maintains the identity of the moving individual. ## limitation this paper targets tightly coupled group work, which is a closed setting with a fixed finite number of participants. if this assumption were required to be lifted, then people in the background might introduce the potential for false positives. a direction that would be valuable to explore in that case would be taking depth-perception into account in order to properly distinguish those engaged in the task from people in the background. as people move, their appearance changes, which introduces challenges for the matching process. one possible direction would be to tune the reid model over the first few frames when a new track appears. in order to further extend capabilities to participants who are easily confused, for example because of wearing similar clothing, more sophisticated reid models might be used that treat different body parts of individuals separately. ## conclusion this paper presents a successful proof of concept for the transfer of models trained to track pedestrians to a scenario that features tightly coupled group work. with a small change to the original deepsort algorithm, using detectron instead of yolo, we are already able to achieve substantial improvement. additional extensions motivated by the characteristics of tightly coupled group work add further improvement. in future work we play to explore more sophisticated reid models for this purpose. while this study lays the foundation for joint activity understanding, much is left to be done to explore aspects other than participant trajectories, such as the interplay of participant emotions and joint eye gaze.