Designing measures that capture various aspects of language ability is a central task in the design of systems for automatic scoring of spontaneous speech . In this study , we address a key aspect of language proficiency assessment -syntactic complexity . We propose a novel measure of syntactic complexity for spontaneous speech that shows optimum empirical performance on real world data in multiple ways . First , it is both robust and reliable , producing automatic scores that agree well with human rating compared to the stateof-the-art . Second , the measure makes sense theoretically , both from algorithmic and native language acquisition points of view . Assessment of a speaker 's proficiency in a second language is the main task in the domain of automatic evaluation of spontaneous speech ( Zechner et al . , 2009 ) . Prior studies in language acquisition and second language research have conclusively shown that proficiency in a second language is characterized by several factors , some of which are , fluency in language production , pronunciation accuracy , choice of vocabulary , grammatical sophistication and accuracy . The design of automated scoring systems for non-native speaker speaking proficiency is guided by these studies in the choice of pertinent objective measures of these key aspects of language proficiency . The focus of this study is the design and performance analysis of a measure of the syntactic complexity of non-native English responses for use in automatic scoring systems . The state-ofthe art automated scoring system for spontaneous speech ( Zechner et al . , 2009 ; Higgins et al . , 2011 ) currently uses measures of fluency and pronunciation ( acoustic aspects ) to produce scores that are in reasonable agreement with human-rated scores of proficiency . Despite its good performance , there is a need to extend its coverage to higher order aspects of language ability . Fluency and pronunciation may , by themselves , already be good indicators of proficiency in non-native speakers , but from a construct validity perspective1 , it is necessary that an automatic assessment model measure higher-order aspects of language proficiency . Syntactic complexity is one such aspect of proficiency . By " syntactic complexity " , we mean a learner 's ability to use a wide range of sophisticated grammatical structures . This study is different from studies that focus on capturing grammatical errors in non-native speakers ( Foster and Skehan , 1996 ; Iwashita et al . , 2008 ) . Instead of focusing on grammatical errors that are found to be highly representative of language proficiency , our interest is in capturing the range of forms that surface in language production and the degree of sophistication of such forms , collectively referred to as syntactic complexity in ( Ortega , 2003 ) . The choice and design of objective measures of language proficiency is governed by two crucial constraints : 1 . Validity : a measure should show high discriminative ability between various levels of language proficiency , and the scores produced by the use of this measure should show high agreement with human-assigned scores . 2 . Robustness : a measures should be derived automatically and should be robust to errors in the measure generation process . A critical impediment to the robustness constraint in the state-of-the-art is the multi-stage au-tomated process , where errors in the speech recognition stage ( the very first stage ) affect subsequent stages . Guided by studies in second language development , we design a measure of syntactic complexity that captures patterns indicative of proficient and non-proficient grammatical structures by a shallow-analysis of spoken language , as opposed to a deep syntactic analysis , and analyze the performance of the automatic scoring model with its inclusion . We compare and contrast the proposed measure with that found to be optimum in Yoon and Bhat ( 2012 ) . Our primary contributions in this study are : • We show that the measure of syntactic complexity derived from a shallow-analysis of spoken utterances satisfies the design constraint of high discriminative ability between proficiency levels . In addition , including our proposed measure of syntactic complexity in an automatic scoring model results in a statistically significant performance gain over the state-of-the-art . • The proposed measure , derived through a completely automated process , satisfies the robustness criterion reasonably well . • In the domain of native language acquisition , the presence or absence of a grammatical structure indicates grammatical development . We observe that the proposed approach elegantly and effectively captures this presencebased criterion of grammatical development , since the feature indicative of presence or absence of a grammatical structure is optimal from an algorithmic point of view . Seeking alternatives to measuring syntactic complexity of spoken responses via syntactic parsers , we study a shallow-analysis based approach for use in automatic scoring . Empirically , we show that the proposed measure , based on a maximum entropy classification , satisfied the constraints of the design of an objective measure to a high degree . In addition , the proposed measure was found to be relatively robust to ASR errors . The measure outperformed a related measure of syntactic complexity ( also based on shallow-analysis of spoken response ) previously found to be well-suited for automatic scoring . Including the measure of syntactic complexity in an automatic scoring model resulted in statistically significant performance gains over the stateof-the-art . We also make an interesting observation that the impressionistic evaluation of syntactic complexity is better approximated by the presence or absence of grammar and usage patterns ( and not by their frequency of occurrence ) , an idea supported by studies in native language acquisition .

Challenge: Existing approaches for automatic scoring of English language ability only fluency and pronunciation missing higher order aspects of language ability.
Approach: They propose a maximum entropy classification-based measure of syntactic complexity for spontaneous speech by a shallow-analysis of spoken language with discriminative ability between proficiency levels.
Outcome: The proposed measure combined with an automatic scoring model outperforms the existing model and is shown to be relatively robust to ASR errors.