Recognition of tone and intonation is essential for speech recognition and language understanding . However , most approaches to this recognition task have relied upon extensive collections of manually tagged data obtained at substantial time and financial cost . In this paper , we explore two approaches to tone learning with substantially reductions in training data . We employ both unsupervised clustering and semi-supervised learning to recognize pitch accent in English and tones in Mandarin Chinese . In unsupervised Mandarin tone clustering experiments , we achieve 57 - 87 % accuracy on materials ranging from broadcast news to clean lab speech . For English pitch accent in broadcast news materials , results reach 78 % . In the semi-supervised framework , we achieve Mandarin tone recognition accuracies ranging from 70 % for broadcast news speech to 94 % for read speech , outperforming both Support Vector Machines ( SVMs ) trained on only the labeled data and the 25 % most common class assignment level . These results indicate that the intrinsic structure of tone and pitch accent acoustics can be exploited to reduce the need for costly labeled training data for tone learning and recognition . Tone and intonation play a crucial role across many languages . However , the use and structure of tone varies widely , ranging from lexical tone which determines word identity to pitch accent signalling information status . Here we consider the recognition of lexical tones in Mandarin Chinese syllables and pitch accent in English . Although intonation is an integral part of language and is requisite for understanding , recognition of tone and pitch accent remains a challenging problem . The majority of current approaches to tone recognition in Mandarin and other East Asian tone languages integrate tone identification with the general task of speech recognition within a Hidden Markov Model framework . In some cases tone recognition is done only implicitly when a word or syllable is constrained jointly by the segmental acoustics and a higher level language model and the word identity determines tone identity . Other strategies build explicit and distinct models for the syllable final region , the vowel and optionally a final nasal , for each tone . Recent research has demonstrated the importance of contextual and coarticulatory influences on the surface realization of tones . ( Xu , 1997 ; Shen , 1990 ) The overall shape of the tone or accent can be substantially modified by the local effects of adjacent tone and intonational elements . Furthermore , broad scale phenomena such as topic and phrase structure can affect pitch height , and pitch shape may be variably affected by the presence of boundary tones . These findings have led to explicit modeling of tonal context within the HMM framework . In addition to earlier approaches that employed phrase structure ( Fujisaki , 1983 ) , several recent approaches to tone recognition in East Asian languages ( Wang and Seneff , 2000 ; Zhou et al . , 2004 ) have incorporated elements of local and broad range contextual influence on tone . Many of these techniques create explicit context-dependent models of the phone , tone , or accent for each context in which they appear , either using the tone sequence for left or right context or using a simplified high-low contrast , as is natural for integration in a Hidden Markov Model speech recognition framework . In pitch accent recognition , recent work by ( Hasegawa-Johnson et al . , 2004 ) has integrated pitch accent and boundary tone recognition with speech recognition using prosodically conditioned models within an HMM framework , improving both speech and prosodic recognition . Since these approaches are integrated with HMM speech recognition models , standard HMM training procedures which rely upon large labeled training sets are used for tone recognition as well . Other tone and pitch accent recognition approaches using other classification frameworks such as support vector machines ( Thubthong and Kijsirikul , 2001 ) and decision trees with boosting and bagging ( Sun , 2002 ) have relied upon large labeled training setsthousands of instances -for classifier learning . This labelled training data is costly to construct , both in terms of time and money , with estimates for some intonation annotation tasks reaching tens of times realtime . This annotation bottleneck as well as a theoretical interest in the learning of tone motivates the use of unsupervised or semi-supervised approaches to tone recognition whereby the reliance on this often scarce resource can be reduced . Little research has been done in the application of unsupervised and semi-supervised techniques for tone and pitch accent recognition . Some preliminary work by ( Gauthier et al . , 2005 ) employs selforganizing maps and measures of f0 velocity for tone learning . In this paper we explore the use of spectral and standard k-means clustering for unsupervised acquisition of tone , and the framework of manifold regularization for semi-supervised tone learning . We find that in clean read speech , unsupervised techniques can identify the underlying Mandarin tone categories with high accuracy , while even on noisier broadcast news speech , Mandarin tones can be recognized well above chance levels , with English pitch accent recognition at near the levels achieved with fully supervised Support Vector Machine ( SVM ) classifiers . Likewise in the semi-supervised framework , tone classification outperforms both most common class assignment and a comparable SVM trained on only the same small set of labeled instances , without recourse to the unlabeled instances . The remainder of paper is organized as follows . Section 2 describes the data sets on which English pitch accent and Mandarin tone learning are performed and the feature extraction process . Section 3 describes the unsupervised and semisupervised techniques employed . Sections 4 and 5 describe the experiments and results in unsupervised and semi-supervised frameworks respectively . Section 6 presents conclusions and future work . We have demonstrated the effectiveness of both unsupervised and semi-supervised techniques for recognition of Mandarin Chinese syllable tones and English pitch accents using acoustic features alone to capture pitch target height and slope . Although outperformed by fully supervised classification techniques using much larger samples of labelled training data , these unsupervised and semi-supervised techniques perform well above most common class assignment , in the best cases approaching 90 % of supervised levels , and , where comparable , well above a good discriminative classifier trained on a comparably small set of labelled data . Unsupervised techniques achieve accuracies of 87 % on the cleanest read speech , reaching 57 % on data from a standard Mandarin broadcast news corpus , and over 78 % on pitch accent classification for English broadcast news . Semi-supervised classification in the Mandarin four-class classification task reaches 94 % accuracy on read speech , 70 % on broadcast news data , improving dramatically over both the simple baseline of 25 % and a standard SVM with an RBF kernel trained only on the labeled examples . Future work will consider a broader range of tone and intonation classification , including the richer tone set of Cantonese as well as Bantu family tone languages , where annotated data truly is very rare . We also hope to integrate a richer contextual representation of tone and intonation consistent with phonetic theory within this unsupervised and semisupervised learning framework . We will further explore improvements in classification accuracy based on increases in labeled and unlabeled training examples .

Challenge: Existing methods for tone and intonation recognitions require extensive collections of costly manually tagged data and there are few works on unsupervised and semi-supervised techniques.
Approach: They employ unsupervised and semi-supervised learning to recognize pitch accents in English and Mandarin Chinese with less training data.
Outcome: They show that the intrinsic structure of tone and pitch accent acoustics can reduce the need for training data, approaching 90% of supervised levels.