Toward Real Time Word Based Prosody Recognition (2024)

Abstract: Prosodic salience is a heuristic based on word-level prosody in child-directed speech that is thought to serve as a cue for attentional focus. It has been used in the context of robotic language acquisition to extract the contextually most relevant words from a human tutor’s speech to ground them in a robot’s sensorimotor data. However, the pipeline for performing word-based prosody-recognition operated in a semi-automatic manner and required substantial manual effort. We describe our efforts to automate the existing pipeline by including real time prosody recognition, and a modern speech recognition and forced alignment model. The intention is to enable its use in real time for human-in-the-loop robotic language acquisition and other socially driven forms of online learning.

## introduction prosodic salience is a measure calculated from a speech signal's pitch, energy, and duration features, and can be used to identify the most relevant words of an utterance produced by caregivers in childdirected speech. this heuristic has demonstrated use in robotic language acquisition @xcite , and can facilitate a more effective language learning process for robots, drawing on insights from how human children acquire language. it has been used as part of the italk project @xcite to learn the names of, and interactions with, objects based on human tutors' linguistically unconstrained speech when trying to teach the robot the names of various objects after having been told to speak to the robot as if it were a 2-year old child. further research was performed by @xcite demonstrating that negation words, such as "no", are prosodically salient which may explain why it is typically amongst the first 10 words in english-speaking children's early active vocabularies @xcite . while the previous work shows prosodic salience to be useful for word-level language acquisition in developmental robotics, it may have a wider potential in speech interfaces. for instance, it might be used within dialogue systems in generating different responses depending on whether some word was produced meekly or with strong intonation -think of the difference between a meekly uttered "no" and a vehemently shouted one. however, for word-based prosody recognition, features must be aligned accurately to the correct segment of the speech signal that is representative of the word. for prosodic salience, this meant that a large quantity of manual effort was required in the past from human transcribers marking word boundaries (e.g. @xcite @xcite . hence for any meaningfully large corpus to be processed by this method, it would need to be scaled up by automating the alignment process with speech processing methods and speech recognition models. the present paper describes our efforts to automate the existing semi-automatic prosodyprocessing pipeline. ## discussion our results indicate that word boundary detection as performed by forced aligners, still remains an open problem when applied to child-directed speech and with respect to word-based prosody detection. we observed that once a boundary detection error occurs within an utterance, this type of error frequently propagates to the boundaries of subsequent words in that utterance. this subsequently renders word-based prosody detection difficult to impossible. however, given a correct set of correct word boundaries, current automatic prosody feature extraction tools such as opensmile appear to perform sufficiently well when compared semiautomatic prosody processing methods involving tools such as praat. because traditional fa performs poorly in non-standard domains, settling for a hybrid usage of hmm and attention models for speech alignment appears to be insufficient. purely attention based forced alignment models hold some promise for improvement. future work @xcite posit that forced aligners perform poorly on cds due to its atypical phonetics, resulting in what they call "catastrophically aligned words". in their work, a kaldi forced aligner was adapted to cds by treating it as a domain adaptation problem. we hence intend to tune neufa @xcite and similar attention-based aligners to chosen cds and rds corpora to adequately adapt it to the respective domains.