Problem-Oriented Segmentation and Retrieval: Case Study on Tutoring Conversations (2024)

Abstract: Many open-ended conversations (e.g., tutoring lessons or business meetings) revolve around pre-defined reference materials, like worksheets or meeting bullets. To provide a framework for studying such conversation structure, we introduce *Problem-Oriented Segmentation & Retrieval (POSR), the task of jointly breaking down conversations into segments and linking each segment to the relevant reference item. As a case study, we apply POSR to education where effectively structuring lessons around problems is critical yet difficult. We present *LessonLink*, the first dataset of real-world tutoring lessons, featuring 3,500 segments, spanning 24,300 minutes of instruction and linked to 116 SAT Math problems. We define and evaluate several joint and independent approaches for POSR, including segmentation (e.g., TextTiling), retrieval (e.g., ColBERT), and large language models (LLMs) methods. Our results highlight that modeling POSR as one joint task is essential: POSR methods outperform independent segmentation and retrieval pipelines by up to +76% on joint metrics and surpass traditional segmentation methods by up to +78% on segmentation metrics. We demonstrate POSR’s practical impact on downstream education applications, deriving new insights on the language and time use in real-world lesson structures.

## introduction across education, business, and science, many open-ended conversations like meetings or tutoring sessions are designed to address a set of predefined topics. as a prominent example, educators often shape their lessons around worksheet problems. structuring lessons effectively is critical but challenging, as educators must allocate the right 1 pronounced as "poser" (/"poz@r/), a perplexing problem. 2 our code and dataset are open-sourced at https:// github.com/rosewang2008/posr . a) it has a negative slope and a positive y-intercept. b) it has a negative slope and a negative y-intercept. c) it has a positive slope and a positive y-intercept. d) it has a positive slope and a negative y-intercept. ## problem-oriented segmentation and retrieval (posr) we define the task of problem-oriented segmentation and retrieval (posr) as jointly dividing a conversation transcript into segments and retrieving the relevant topic (e.g., problem) discussed in each segment. while segmentation and retrieval are individually challenging, posr jointly addresses them together to improve ecological validity and expose new system design tradeoffs. we hypothe- algorithm 1 posr vs. non-posr methods require: t, r if with posr then s1, . . . , sn ← segment(t, r) else s1, . . . , sn ← segment(t ) end if w1, . . . , wn ← retrieve([s1, . . . , sn ], r) ## the lessonlink dataset we introduce the lessonlink dataset as a concrete case study of posr. lessonlink contains real-world tutoring lesson transcripts segmented and linked with problems in sat® math worksheets. the dataset features 3,500 segments of over 24,300 minutes of instruction, featuring 1,300 unique speakers and 116 linked problems. ## results opus and posr gpt4 cost @xmath021 per 100 transcripts, while the best combined independent methods, opus+gpt4, cost $54 per 100 transcripts. this demonstrates the importance of posr of jointly modelling segmentation and retrieval for better accuracy and cost performance. however, there is still room for improvement such as future work on developing and improving open-sourced longcontext methods. according to the time-and line-based metrics for segmentation and srs are well-correlated across methods, indicating that accounting for time does not impact relative rankings. however, time-weighing is still important in accounting for errors in long segments: time-p k errors are lower than line-p k because it reduces the impact of oversegmentation whereas time-windowdiff amplifies errors from missing long segments. ## downstream applications there are several applications that posr enables for gaining insights into tutoring practices at scale. we illustrate two. one application is a language analysis to compare how tutors talk about the same problem with the long vs. short talk times (top and bottom quartile). we use the log odds ratio ## discussion and conclusion we introduce the problem-oriented segmentation and retrieval (posr), a task that jointly segments conversations and retrieves the problem discussed in each segment. we contribute the lessonlink dataset as a concrete case study of posr in education. lessonlink is the first large-scale dataset of tutoring conversations linked with worksheets, featuring 3,500 segments, 116 linked sat® math problems and over 24,300 minutes of instruction. to evaluate the joint performance and account for time in segmentation, we introduce the segmentation and retrieval score (srs) and time-based segmentation metrics for p k and windowdiff. our comprehensive evaluations highlight the importance of jointly modeling segmentation and retrieval, rather than treating them as independent tasks: posr methods significantly outperform the independent approaches as measured against the traditional segmentation, srs, and new time-based metrics. the llm-based posr methods achieve the best performance, but come at a higher cost, motivating future work on cost-effective solutions. we also demonstrate the potential of posr by showcasing downstream applications, such as a language analysis comparing tutoring strategies. in conclusion, our work establishes posr as an important task to study conversation structure. the lesson-link dataset and the proposed methods pave the way for further research in joint segmentation and retrieval, with broad implications for educational technology, conversational analysis, and beyond.