Activity focused Speech Recognition of Preschool Children in Early Childhood Classrooms (2022)

Abstract: A supportive environment is vital for overall cognitive development in children. Challenges with direct observation and limitations of access to data driven approaches often hinder teachers or practitioners in early childhood research to modify or enhance classroom structures. Deploying sensor based tools in naturalistic preschool classrooms will thereby help teachers/practitioners to make informed decisions and better support student learning needs. In this study, two elements of eco-behavioral assessment: conversational speech and real-time location are fused together. While various challenges remain in developing Automatic Speech Recognition systems for spontaneous preschool children speech, efforts are made to develop a hybrid ASR engine reporting an effective Word-Error-Rate of 40%. The ASR engine further supports recognition of spoken words, WH-words, and verbs in various activity learning zones in a naturalistic preschool classroom scenario. Activity areas represent various locations within the physical ecology of an early childhood setting, each of which is suited for knowledge and skill enhancement in young children. Capturing children’s communication engagement in such areas could help teachers/practitioners fine-tune their daily activities, without the need for direct observation. This investigation provides evidence of the use of speech technology in educational settings to better support such early childhood intervention.

## introduction the preschool classroom is a viable space for capturing young children's interactions with teachers and peers. the quality and number of interactions children experience is a key factor in child language development @xcite . however, for supporting teachers working with young children with or without developmental delays, the use of direct observations or manual video recording and coding is not a scalable endeavor @xcite . sensor-based monitoring tools in classrooms can assist teachers in creating and maintaining a rich learning environment for all children. feedback from these tools could allow teachers to better identify children who could benefit from further support. rich and frequently available data can not only help in creating better classroom structure, but also create opportunities to maximize children's communication and interaction @xcite . eco-behavioral observational assessment has often been used to measure moment-to-moment effects with multiple environmental events on specific behaviors and interactions that occur in an early childhood inclusive classroom @xcite . these assessment samples are centered around teacher and child behavior, and overall classroom learning context (e.g., the interactions between them) by adding situational or contextual factors. specifically for inclusive classrooms, a child's daily interaction can influence their development and by using an ecobehavioral assessment, conclusions can be drawn between environmental contexts and the interactions that occur within them @xcite . these findings can inform practitioners how to arrange their inclusive environments to best support language development of all children. the variety of spontaneous language in an inclusive preschool classroom comes from a variety of speakers and includes both adults and children. although the language environment analysis (lena) framework is used extensively by the early childhood research community for a digital measurement system that is automatic @xcite @xcite @xcite , lena does not possess an automatic speech recognition (asr) engine to convert the child speech-to-text, nor does it capture location in the classroom. apart from conversational speech, children's coordinated movement and location within classrooms also act as an acquisition context driver for critically important skills including language, cognition, and social communication @xcite @xcite . therefore, automatic location tracking within the classroom can provide the ability to monitor interventions while maximizing learning opportunities @xcite . our multi-disciplinary educational research project focuses on quantifying "learning" based on social engagement for use in classroom settings by teachers -and thus we are building a tool that captures the granularity of eco-behavioral observational assessment. it is based on spontaneous interactions between multiple teachers and preschool children (3 to 5 years) with and without developmental delays within naturalistic noisy preschool classroom environments. in this study, we present a translational framework to automatically track conversational speech of preschool children in various activity areas supported by speech technology based on asr which is fine-tuned specifically for preschool children taking into account their developing nature and developmental delays. ## speech and language development in young children right from their first babbles, children start developing various speech sounds (shriberg, 1993) until mid-elementary school. typically-developing children are expected to progressively acquire various speech sounds throughout early childhood (birth to 8 years). these development occurs primarily in three stages: (i) 'early' stage from 1 to 3 years, (ii) 'middle' stage from 3 to 6 1 2 , and (iii) 'late' stage from 5 to 7 1 2 . in the 'early' stage, speech sounds like m (nasal; "mama"), b (stop; "baby"), y (semivowel; "you"), n (nasal; "no"), w (semivowel; "we"), d (stop; "daddy"), p (stop; "pop"), hh (aspirate; "hi") are expected to be developed. while sounds like t (stop; "two"), ng (nasal; "running"), k (stop; "cup"), g (stop; "go"), f (fricative; "fish"), v (fricative, "van"), ch (affricate, "chew"), and jh (affricate, "jump") are expected to be acquired in the 'middle' stage. finally, in the 'late' stage, children develop slightly harder sounds like sh (fricative; "sheep"), s (fricative; "see"), th (fricative; "think,that", r (liquid; "red"), z (fricative; "zoo"), l (liquid; "like") and zh (fricative; "measure"). children may omit, substitute or have inconsistency in production of speech sounds while they are learning. apart from speech, language planning is also evolving, so word selection and grammar may have issues. not all children acquire these skills at a similar pace, especially those with developmental delays. ## challenges of developing automatic speech recognition systems for young children various developmental factors like articulation/pronunciation, motor skills, vocabulary, etc., makes the task of developing asr systems for children challenging than that for adults @xcite . also, children in early childhood (birth to 8 years) have significantly different speech and language skills as compared to their older peers. prior research from the speech technology community on children asr @xcite @xcite @xcite @xcite is captivating. but these focused on: (i) older children, including kindergarten (6-15 yrs), (ii) data collected using head-mounted microphones or close-proximity handheld smartphones in clean/controlled settings under adult supervision, and (iii) with just one speaker using prompts or read stimuli, and limited spontaneous (not scripted) speech. limited focus and data is available for processing of adult-child interactions in naturalistic preschool settings (3-5 yrs) while they are involved in various activities throughout the day. there is lack of publicly available young child speech corpora (primarily due to privacy/regulations). however, a recent study @xcite described various challenges in developing asr systems for single-word utterances read aloud by kindergarten (5-6 yrs) children achieving a word error rate (wer) of 25%. therefore, all these factors make the task of developing asr systems for spontaneous preschool children speech in naturalistic educational settings extremely challenging. 4 data and collection 4.1 participants & procedure a total of 33 children aged 3 to 5 years with and without developmental delays, and 8 adults teachers participated in this study. the data was collected in preschool classrooms (refer fig.1(b)) in a large urban community in a southern state in us, and in multiple sessions over several days in different classrooms with different groups of participants. data from each participant was linked to an anonymous id for privacy. all participants consented to the use of de-identified data for analysis. this study was approved by the institutional review board of both ku and utd for analysis. ## activity-area based child speech recognition and discussions all experiment results for this section are summarized in time (min) = total time spent by each child in that specific activity area wer (%) = word error rate of the asr model for all words spoken in that specific activity area words @xmath0; primarily due to no words spoken tences, wh-words and verbs are a few of the prominent language learning milestones established by the american speech-language-hearing association foot_0 , outlined by the american academy of pediatrics @xcite , and adopted as cdc's (centers for disease control and prevention) developmental milestones foot_1 program "learn the signs. act early." primary child #1 spends the most quality time in 'art/snack' area (wer: 17.39%), followed by close to 5 mins in 'computer'(wer: 37.5%) and 'dramatic play'(wer: 27.1%) areas. the amount of spoken words is relatively much higher in 'art/snack' area. child #1 spends less than a minute in 'block', 'manipulative', and 'story' areas, which is also reflected in the word spo-ken count. overall across all activity areas, primary child #1 spends much less time and spoke less as compared to child #2 and #3. primary child #2 and #3 spent more time in the classroom boundary, and therefore word counts spoken were much higher. primary child #2 spends quality time in 'art/snack' (wer: 53.11%), 'dramatic play' (24.93%), 'manipulative' (26.62%), and close to 4 mins in 'computer'(wer: 38.18%) and 'cozy/book'(wer: 47.61) areas. primary child #3 spends quality time in 'art/snack' (wer: 56.03%), 'block' (46.39%), 'dramatic play' (43.03%), and close to 4 mins in 'computer'(wer: 23.33%) areas. irrespective of the child, performance of the asr engine in detecting wh-words and verbs across all activity areas is quite good, given the naturalistic noisy dynamic learning environment. while areas like 'cozy/book' are more personal learning spaces. areas like 'dramatic play', 'manipulative', 'block', 'art/snack' alternatively encourage group activity. 'computer' and 'story' areas are more focused on listening or seeing. some observations here can be: (i) primary child #1 did not engage much in areas of group activitysignifying difficulty to engage in groups, (ii) primary child #1 and #3 produced higher wh-word counts (not shown in the table) in 'computer' and 'dramatic play' areas -signifying more curiosity. longitudinal data of the same group of children over a significant time period should help in better informed decisions. however, amendments to classroom structure and plan will be at the discretion of teachers. performance of the asr engine can help to monitor/track such elements in a naturalistic preschool classrooms. ## towards data-based inclusion planning in classrooms non-segregated or inclusive educational settings possess a design best suited to prepare young children with disabilities for kindergarten (us dept. health, 2015; @xcite . careful considerations regarding environmental factors are imperative for meaningful interactions between children in inclusive classrooms @xcite . high-quality inclusive classrooms can also foster and support friendship development between children with and without disabilities @xcite . through communication skills and social interactions, individuals can begin to form meaningful social relationships and friendships, which could promote positive psychological states (e.g., happiness and self-efficacy; @xcite . teachers and peers as communicators can play important roles for inclusive classrooms to support communication skills of children with disabilities and facilitating social interactions between one another. the quantity and quality of interactions significantly influence the language environment and communication opportunities for young children with disabilities @xcite . also, it may be more important for a child with autism spectrum disorder (asd) to spend quality time in activity areas that promote language and social engagament because of the social-communication and play limitations that accompany asd. using audio recorded by lena and real-time location using ubisense supported by advanced speech processing algorithms could provide teachers with information about "what" and "where" child interactions are taking place so that they may be better able to discern when to provide additional support. ## conclusion this study has provided evidence and lays the foundation of deploying sensor-based monitoring tools to acquire and interpret eco-behavioral data (speech and location) in naturalistic early childhood settings to better support teachers and child learning. this work tends to address a major challenge faced by early childhood educators in supporting children (with and without developmental delays) due to a lack of real-time data to inform daily practices and that lead to longer-term school readiness outcomes. another component in this study has addressed the development of asr systems for preschool children, which is a very low-resource scenario. both collection and transcription of such data is a major challenge, especially due to both noisy data and speech intelligibility of young children. future work will focus on analyzing more children with and without developmental delays, and also collection of such naturalistic data. future work will also consider speaker group classification (adult vs. child) using speaker-group diarization as compared to human transcriptions.