K id LM : Advancing Language Models for Children – Early Insights and Future Directions (2024)

Abstract: Recent studies highlight the potential of large language models in creating educational tools for children, yet significant challenges remain in maintaining key child-specific properties such as linguistic nuances, cognitive needs, and safety standards. In this paper, we explore foundational steps toward the development of child-specific language models, emphasizing the necessity of high-quality pre-training data. We introduce a novel user-centric data collection pipeline that involves gathering and validating a corpus specifically written for and sometimes by children. Additionally, we propose a new training objective, Stratified Masking, which dynamically adjusts masking probabilities based on our domain-specific child language data, enabling models to prioritize vocabulary and concepts more suitable for children. Experimental evaluations demonstrate that our model excels in understanding lower grade-level text, maintains safety by avoiding stereotypes, and captures children’s unique preferences. Furthermore, we provide actionable insights for future research and development in child-specific language modeling.

## introduction children constitute one in three internet users globally, according to a unicef study @xcite , with the average screen time for kids aged 8-12 estimated to be over five hours per day @xcite . this level of digital engagement presents both opportunities and challenges for enhancing children's learning experiences. large language models (llms) have significantly lowered the barriers to building educational tools and applications @xcite , with some studies suggesting these models enhance children's learning by facilitating engaging and emotionally responsive conversations @xcite ## kidlm construction our aim for kidlm is to create language models tailored for children by developing a high-quality, user-centric corpus. this involves meticulous data collection and verification to ensure reliability and relevance, along with a novel masking process to enhance the model's focus on kid-specific words. ## evaluation we evaluate our kidlm models based on the following two criteria: (1) how well does kidlm understand lower grade-level texts ( §3.1)? (2) how robust is kidlm in maintaining safety standards by avoiding the generation of stereotypes ( §3.2)? we compared our model with base lms to ensure a fair and consistent comparison, highlighting the impact of our high-quality pre-training data. ## analysis in this section, we provide a qualitative analysis of our model outputs in two key settings. first, we assess the preferred lexical simplification within context compared to human labels. second, we design probe tests categorized into diverse types ( to structure the analysis, we employ the "cloze test" @xcite to design queries, where certain words in a query are masked, and the model's task is to predict or fill in these blanks. formally, let @xmath0, q 2 , . . . , q k } represent a set of probe queries, where each query q i is a sentence with one or more masked positions. each query can be represented as: lexical simplification involves replacing a word in context with a simpler alternatives @xcite . to analyze the ability of our kidlm models to generate simpler words within a given context, we utilized the tsar-en dataset @xcite , annotated by mturk annotators who are required to be at least 18 years old. for each sentence, we selected the annotated complex word (highlighted in bold in preference probing involves creating a set of probe queries and using language models to predict preferences for these queries (appendix [ in we qualitatively analyze and interpret the model's preferred completions, but a critical question remains: how can we evaluate this with actual human feedback? in next section, we discuss future directions involving human-centered evaluations. alignment to children base llms pre-trained with unsupervised text corpora are typically inadequate as open-domain conversational assistants. fine-tuning is essential, but using existing sft data can compromise the kid-specific properties developed during pre-training stage ( human-centered evaluation current llm evaluation methods focus on developing datasets and benchmarks @xcite but often fail to address the 'sociotechnical gap' @xcite . assessing models in isolated 'lab settings' limits the incorporation of human factors @xcite . human-computer interaction (hci) offers diverse metrics to meet the evaluation needs of different stakeholders @xcite . interdisciplinary research between hci and nlp is essential for responsible, human-centered evaluation and auditing of llms @xcite . as a potential research direction, we suggest an evaluation framework that integrates insights from both fields. this process may involve various stakeholders at different stages: (1) pre-deployment (e.g., educators, psychologists, parents), and (2) postdeployment (e.g., children, parents, educators). ## related work children and language technology prior studies from the hci community have explored how technology can support children in learning and sharing their emotions @xcite , as well as enhancing parents' awareness of their children's emotional well-being @xcite . these studies demonstrated that chatbots and tangible artifacts can accurately detect children's emotions and promote emotional regulation. however, they often overlook children's perceptions and preferences regarding emotional communication @xcite and are limited by the technical constraints of rule-based chatbots @xcite . llms have simplified the development of educational tools and applications @xcite . research suggests these models can enhance children's learning through engaging, emotionally responsive interactions @xcite and support visual programming @xcite . however, significant risks include bias and toxicity from unvetted datasets @xcite , insufficient contextual appropriateness @xcite , and difficulty in maintaining lexical simplicity suitable for young users @xcite . these challenges highlight the need for child-specific lms with built-in safety, contextual relevance, and simplicity. @xcite employs a masking strategy that targets "entities" identified by a domain-specific pretrained named entity recognizer (ner) model. similarly, salient span masking @xcite uses an ner model to mask entities for opendomain qa tasks. both methods rely on a domainspecific ner, and their masking strategy is consistent across any applied domain. in contrast, selective masking @xcite tailors token masking during continued pre-training based on data and labels from the downstream task. meanwhile, difference masking @xcite automatically selects tokens for masking by identifying unique anchor words in the target domain data, distinguished from the general domain using a tf-idf-like scoring function. @xcite found that a 15% masking rate is not universally optimal for mlms, suggesting that larger models should adopt a higher rate when pre-training from scratch. moreover, @xcite introduced time-variant masking, adjusting the masking rate at different training stages to enhance pre-training efficiency. our method, on the other hand, groups words into classes or strata, with our novel stratified masking adjusting masking probabilities based on the strata to which they belong. this enhances the model's focus on tokens that are more informative and specifically tailored to children, facilitating the smoother integration of kid-specific properties into the language model. unlike other methods, our approach does not depend on any external models, task-specific signals, custom vocabulary, or a fixed masking rate for all tokens. the works related to domain adaptation of lms are in appendix d. ## conclusion in this paper, we take the important first steps toward designing child-specific language models to make nlp systems more accessible to children. we curated a high-quality pre-training corpus using our proposed user-centric data collection pipeline and introduced novel stratified masking to enhance the model's focus on tokens that are more informative and specifically tailored to children. experimental evaluations demonstrate that our model effectively understands lower grade-level text, maintains safety standards by avoiding the generation of stereotypes, and captures children's unique preferences. furthermore, based on our insights, we offer suggestions for future research and development.