Cross Encoding as Augmentation: Towards Effective Educational Text Classification (2023)

Abstract: Text classification in education, usually called auto-tagging, is the automated process of assigning relevant tags to educational content, such as questions and textbooks. However, auto-tagging suffers from a data scarcity problem, which stems from two major challenges: 1) it possesses a large tag space and 2) it is multi-label. Though a retrieval approach is reportedly good at low-resource scenarios, there have been fewer efforts to directly address the data scarcity problem. To mitigate these issues, here we propose a novel retrieval approach CEAA that provides effective learning in educational text classification. Our main contributions are as follows: 1) we leverage transfer learning from question-answering datasets, and 2) we propose a simple but effective data augmentation method introducing cross-encoder style texts to a bi-encoder architecture for more efficient inference. An extensive set of experiments shows that our proposed method is effective in multi-label scenarios and low-resource tags compared to state-of-the-art models.

## introduction due to the overwhelming amount of educational content available, students and teachers often struggle to find what to learn and what to teach. autotagging, or text classification in education, enables efficient curation of content by automatically assigning relevant tags to educational materials, which aids in both students' understanding and teachers' planning @xcite . however, applying auto-tagging for real-world education is challenging due to data scarcity. this is because auto-tagging has a potentially very large label space, ranging from subject topics to knowledge components (kc) @xcite @xcite . the resulting data scarcity decreases performance on rare labels during training @xcite @xcite . in this paper, we aim to solve the data scarcity problem by formulating the task as a retrieval problem following a recent proposal @xcite . this can utilize a language model's ability to understand the tag text, such that even for an unseen tag, the models would be able to capture the relationship between the terms in the input content and labels. however, performance in the auto-tagging context still critically depends on the amount of training data. to this end, we first propose to leverage the knowledge of language models that are fine-tuned on large question-answering datasets. our intuition is that question of finding an answer in a passage can be a direct (or indirect) summary of the passage @xcite , which can serve as an efficient proxy of the gold tag for educational content. the large question-answering datasets thus become a better prior for the tag spaces. specifically, we adopt a recent bi-encoder architecture, called dpr @xcite foot_0 , for transfer learning, which performs bert encoding over the input and candidate label separately and measures the similarity between the final representations. to the best of our knowledge, our work is the first to leverage transfer learning from qa models for text classification tasks. as a further innovation, we introduce a novel data augmentation method for training a bi-encoder architecture, named ceaa, which adds the crossencoder view of the input-label pair in the biencoder architecture, as shown in figure 1 . by capturing the full interaction between input and labels already during training time, the models can be further optimized to take advantage of tokenlevel interactions that are missing in traditional bi-encoder training. at the same time, the computational efficiency of the bi-encoder is maintained, which makes ceaa able to tackle large label spaces as opposed to existing solutions based on cross-encoder architectures @xcite @xcite . experiments show that ceaa provides significant boosts to performance on most metrics for three different datasets when compared to state-of-the-art models. we also demonstrate the efficacy of the method in multi-label settings with constraints of training only with a single label per context. ## related work text classification in the education domain is reportedly difficult as the tags (or, labels) are hierarchical @xcite @xcite , grow flexibly, and can be multi-labeled @xcite . though retrieval-based methods were effective for such long-tailed and multilabel datasets @xcite , they relied on vanilla bert @xcite ) models, leaving room for improvement, for which we leverage question-answering fine-tuned retrieval models @xcite . recently, @xcite proposed tagrec++ using a bi-encoder framework similar to ours, with an introduction of an additional crossattention block. however, this architecture loses the efficiency of the bi-encoder architecture in the large taxonomy space for the education domain. unlike tagrec++, our distinction is that we leverage the cross-attention only in training time via input augmentation. ## conclusion in this paper, we discuss the problem of 'autotagging' with regard to data scarcity due to its large label space -an issue that is critical in the education domain, but also for other domains with a multi-label structure such as jurisdictional or clinical contexts. we propose two innovations to address this problem: first, exploiting the knowledge of language models trained on large questionanswering datasets. second, applying a novel augmentation for bi-encoder architecture inspired by cross-encoders to better capture the full interaction between inputs and labels while maintaining the bi-encoder's efficiency. a set of experiments demonstrated the effectiveness of our approach, especially in the multi-label setting. future research will explore re-ranking scenarios in which the bi-encoder trained with our cross-encoding augmentation (ceaa) is re-used to effectively rerank the tags with cross-encoding mechanism as in @xcite .