Enhancing Textbooks with Visuals from the Web for Improved Learning (2023)

Abstract: Textbooks are one of the main mediums for delivering high-quality education to students. In particular, explanatory and illustrative visuals play a key role in retention, comprehension and general transfer of knowledge. However, many textbooks lack these interesting visuals to support student learning. In this paper, we investigate the effectiveness of vision-language models to automatically enhance textbooks with images from the web. We collect a dataset of e-textbooks in the math, science, social science and business domains. We then set up a text-image matching task that involves retrieving and appropriately assigning web images to textbooks, which we frame as a matching optimization problem. Through a crowd-sourced evaluation, we verify that (1) while the original textbook images are rated higher, automatically assigned ones are not far behind, and (2) the precise formulation of the optimization problem matters. We release the dataset of textbooks with an associated image bank to inspire further research in this intersectional area of computer vision and NLP for education.

## introduction students use textbooks as one of the primary mediums of learning. it is thus imperative that textbooks are designed to provide a rich learning and engaging environment. visuals enhance learning through a number of means, including the ability to retain information, as well as its ability to promote comprehension and knowledge transfer @xcite @xcite @xcite . automatic approaches for retrieving and assigning images from the web to textbook chapters can therefore assist textbook designers in the creation of better textbooks. however, this is a very challenging task as an ideal illustrative visual should not only be related to the textbook material, but also have pedagogical value (see figure 1 ). agrawal et al. @xcite already enhance textbooks with images from the internet. they use similarity scores of textual captions from images and the text present in the textbooks for image assignment. this approach requires the image captions to be available. beyond the problem of availability, the captions are also highly context dependent which reduces their utility in our setting. in this work, we aim to reinvigorate this area, and present and analyse a real dataset of textbooks. we do so by setting up the problem of image assignment using textbook and wikipedia images. in contrast to previous work, we rely on the recent advances in vision-language models, such as clip @xcite and dall-e @xcite . we analyze our dataset to gain insights into the organization of concepts and illustrative images within the textbooks. this analysis inspires the formulation of a new optimization problem focused on modeling of illustrations in textbooks. the solution to this problem maximizes the coverage of the illustrations while minimizing redundancy during image-to-paragraph assigments. our approach uses clip to retrieve and appropriately assign images to long-form text, particularly a textbook section. then, the overall assignment is obtained in following stages: 1) given a piece of text, produce a set of concepts that should be addressed by a visualization, 2) given an image and a concept, determine their mutual relevance, and 3) given a piece of text (e.g., a textbook section), produce an adequate assignment of images. each of the three sub-problems listed above can be solved with a variety of approaches -indeed we explore several variants, which we describe in section 4. because of the modularity of this approach, each of the sub-problems can be improved independently and adapted to the dataset in future work. overall, we contribute the following: • a dataset that contains text and images drawn from 35 textbooks covering math, business, social sciences, and science in addition to a secondary image bank of ∼312k images taken from wikipedia (section 3). enriching text with images: the task of textbook enrichment was first explored by @xcite , who assume that web images have associated relevant captions. we note that the caption of images are largely dependent on the context where the image was originally assigned. 1 the language of the caption may not even match the textbook's language. to alleviate all this, we do not assume that the images have associated relevant captions. @xcite ; @xcite ; @xcite also studied associating images with textual information. however, their primary goal has largely been to comprehend the image content, and thus differs from our objective. finally, there has been more past work on nlp applied to textbooks @xcite @xcite . however, the goal of these works also differ significantly from ours. ## dataset we now present the process of curation and structure of our dataset with an analysis. ## image retrieval and assignment we first describe an image retrieval model and then formalize our task and the optimization approach. clip @xcite is a stateof-the-art vision-language model trained on many image-caption pairs from the web by maximizing the dot-product similarity the image and caption encodings. we further fine-tune clip on image-text pairs from the openstax books dataset. ## human evaluation the goal of improving textbooks is to help students learn. testing a wide range of models directly by monitoring learning progress would require a very expensive long-term evaluation. instead we turn to an intrinsic crowd-sourced evaluation where we ask teachers what they think about the qualities of the assignment. ## conclusion and future work we presented a dataset and a new task of enriching textbooks with visuals from the web. we proposed several technical solutions for this problem using neural image retrievers combined with a new assignment optimization setup. annotations by workers in the education industry verified that, even though the human assignment is still of the highest quality, the automatic assignments are not far behind. there are multiple venues for making further progress on this problem. first, individual concept importances and text-image relevance models could be improved and plugged into the existing algorithms. the varying domains of images and lengthy textual descriptions of concepts present challenges that could pave the way for exploring new approaches for learning image-text associations. professional textbook designers can be included to further refine the assignment optimization objective and pose this as a human ai collaboration problem.