Multimodal large language models for inclusive collaboration learning tasks (2022)

Abstract: This PhD project leverages advancements in multimodal large language models to build an inclusive collaboration feedback loop, in order to facilitate the automated detection, modeling, and feedback for participants developing general collaboration skills. This topic is important given the role of collaboration as an essential 21st century skill, the potential to ground large language models within learning theory and real-world practice, and the expressive potential of transformer models to support equity and inclusion. We address some concerns of integrating advances in natural language processing into downstream tasks such as the learning analytics feedback loop.

## introduction collaboration, a coordinated process involving two or more individuals participating in a task in an interdependent way, is an important topic of study given its importance as a major 21st century skill @xcite @xcite . though collaboration as a general term is viewed as a learnable competency, notable distinctions emerge when examining how collaboration surfaces within relevant research. one semantic distinction is that the term collaboration is not explicitly defined, or is used interchangeably with concepts such as group collaboration, teamwork, collective problem solving, cooperation, and more @xcite . these inconsistencies in meaning make it challenging to connect various research agendas that purport the advantages of collaboration. another distinction to note is modalityrelated. some research does not make any modality distinctions when reporting the impacts of results, though much has viewed collaboration via online/computer-mediated interactions, both synchronous and asynchronous, while other research has examined co-located collaborative acts that happen face-to-face. despite semantic, modality, and other distinctions, various fields have advanced what we know about collaboration, specifically collaboration as a language-mediated process. scholars within the fields of nlp, cognitive science, and educational research have focused separately on verbal and written aspects of collaborative exchanges -speech, text-based outputs, and audio such as non-linguistic pauses -to better understand aspects of collaboration. recent nlp research, for example, has explored neural models equipped with dynamic knowledge graph embeddings, the use of large language models to model real world speech, and the development of collaboration datasets @xcite @xcite , while cognitive science has explored general modeling approaches for collaborative behavior and large language models as knowledge sources for intelligent agents @xcite @xcite . learning analytics, a subset of educational research that extracts diverse datastreams from the learning process to improve learning, has developed automated multimodal approaches to detect, model and provide feedback about collaborative learning exchanges @xcite @xcite . though these studies differ in their disciplinary perspectives, they view language as essential to individuals' application of collaborative behavior and researchers' understanding of said behavior. ## purpose of research project because language is grounded in experience @xcite , and collaboration is mediated through language, collaboration is an appropriate skill to be learned, practiced, and analyzed through languagemediated experiences and techniques. this dissertation project, situated at the intersection of nlp, cognitive science, and learning analytics, focuses on how we may support people in their development of complex, dynamic collaborative language skills. the project extends prior research, but also introduces unexplored areas such as multimodal language modeling and inclusive collaboration. therefore, the aim is to contribute to several open research questions related to how we may foster collaborative language, a proxy for overall collaboration skills, in people as an explicit act of learning. this project examines these critical gaps in current research to explore the ultimate question of: how can we use multimodal large language models to detect, model, and provide feedback on inclusive collaboration behavior? sub-questions include: this study explores the potentials of adopting multimodal nlp techniques within a learning analytics lens. multimodal nlp is an emerging area within nlp that stems from the development of the large language model, a massive-parameter pretrained model. large language models are an active area of development within nlp, and one set of researchers have demonstrated impressive semantic and generative capabilities @xcite , while others pose ethical, environmental, and interpretability concerns about unbounded scaling of model size @xcite @xcite . we focus on the potential of multimodal nlp, large language models that integrate multimodal (acoustic, image, tactile, and/or video) data beyond text-based language, and explore potentials of multimodal nlp for automated, fine-grained detection of collaborative processes that will support learners within and across experiences, an important downstream application of the technology @xcite @xcite . we also contribute to current critiques of performance-first modeling that may overlook important opportunities to create real world nlp models that reduce bias. this project operationalizes an inclusive collaboration index with the goal of general equity and inclusion over identityspecific bias mitigation. ## integrating inclusion into downstream nlp collaboration tasks within learning analytics @xcite , nlp @xcite , and general machine learning/ai applications @xcite , researchers have made arguments for more equitable, fair, and inclusive practices. this includes verifying that the research approach is informed by ethical and human-centered principles, developing research methods that detect/mitigate unethical outcomes, and/or our aim of proposing that research methods should translate ethically when used in real-world contexts. with the recent focus on equity and inclusion across our fields of interest, formal inclusion theories are stated as important to integrate as a future idealized goal, though we lack blueprints for what forms these integrations may take. within research across learning analytics, nlp, and machine learning, formal experiments provide empirical support for those methods with the most promise for identifying and reducing unwanted societal bias, ambiguity, and exclusion in datasets and models @xcite @xcite , though there is less support for what works as an embedded practice within downstream tasks that utilize these algorithms, datasets, and platforms. this study considers ethical research approaches and outcomes, but primarily focuses on the stated areas of potential development -the ethical deployment of our nlp and learning analytics research methods in downstream tasks situated within actual learning settings by detecting lack of inclusion and intervening. our focus is not yet to identify any causal relationship between one or more social identities and collaboration quality, but rather to detect inclusive collaboration of individuals and groups, and in the process identify any disparities in collaboration quality among individuals and within the group as a whole. in this sense, our work advances the concept of inclusion @xcite , defined as the degree to which diverse individuals demonstrate that they are part of the collaborative process. we recognize that this study falls short of addressing equity since equity examines outcomes at the societal rather than individual or group level, though we highlight that inclusion is an integral step on the way to equity and ethical treatment within collaborative experiences @xcite . ## methodology we have sub-divided the planned methodology into multiple tasks as: due to the multidisciplinary nature of collaboration, this study will incorporate methods that stem from four distinct fields -learning analytics, cognitive science, natural language processing, and inclusion theory -to create an inclusive view of learning to collaborate. from learning analytics, we get a roadmap for developing an automated feedback loop necessary for learning to collaborate, and a variety of methods for detecting collaborative behaviors and operationalizing them into signals for model building. from cognitive science, we have an example for linking psychological theory, model, and real world behaviors, as well as ongoing research on intelligent agents as used for understanding learning and adaptation through feedback. from natural language processing, we have access to the ability of large language models to parse and generate human language, as well as approaches for addressing inclusion in language model building. lastly, we operationalize tenets of inclusion theory in order to build a learning to collaborate model that detects linguistic bias, thus working towards a more inclusive collaboration environment. all aspects involving human subjects, including phase 1 data collected via amazon mechanical turk, phase 2 large language modeling, and phase 3 interventions will receive full approval of the university's institutional review board (irb) prior to launching the study. datasets are either open for research use and cited, or collected and stored as part of the irb approval process and regulations. ## initial results an initial pilot focused on the language modeling portion, and uses irb-approved data that takes place within recitations of a large, stem class. groups of 3 students participated in small group work for the duration of the 75 minute period, and were tasked with solving problems related to the lecture and readings. audio and video recordings were captured, cleaned, and processed. transcripts were generated by an automated speech recognition (asr) software and corrected by hand, and were then paired with video frames. a random sampling of the text-based dialogue and video frames were generated and then mapped to the inclusive collaboration framework by 2 expert annotators and an additional 5 general skill annotators. these data will serve as the evaluation set. bertbase and gpt-2 were finetuned on a randomized sample (80%) of the ami collaboration dataset, as well as dialogue (text-based) portions of the multi-party collaboration corpus. results indicate some marginal improvement between the finetuned models, and between bert and the larger gpt-2 model, but additional analysis and more thorough data preparation and testing are needed. the finetuned gpt-2 model performed better than chance on all except for the inclusion dimension. we anticipate that more thorough finetuning and integration of multimodal finetuning data should improve performance on multimodal classification tasks. ## conclusion as an essential 21st century skill, our aim is to utilize the potentials of multimodal large language models to advance our ability to detect and model collaborative behaviors, with the ultimate goal being to offer feedback to learners as they develop these important skills. importantly, we focus on the tenets of inclusive collaboration, so that collaborators are encouraged to have equitable and inclusive exchanges as they work with each other. this doctoral research project builds an automated end-to-end inclusive collaboration feedback loop, relying on advancements in large language modeling as it is used in downstream tasks, and grounding machine learning methods within theory and real-world practice.