nature neurOSCIenCe a r t I C l e S To understand connected speech, listeners must construct a hierarchy of linguistic structures of different sizes, including syllables, words, phrases and sentences [1] [2] [3] . It remains puzzling how the brain simultaneously handles the distinct timescales of the different linguistic structures, for example, from a few hundred milliseconds for syllables to a few seconds for sentences [4] [5] [6] [7] [8] [9] [10] [11] [12] [13] [14] . Previous studies have suggested that cortical activity is synchronized to acoustic features of speech, approximately at the syllabic rate, providing an initial timescale for speech processing [15] [16] [17] [18] [19] . But how the brain utilizes such syllabic-level phonological representations closely aligned with the physical input to build multiple levels of abstract linguistic structure, and represent these concurrently, is not known. We hypothesized that cortical dynamics emerge at all timescales required for the processing of different linguistic levels, including the timescales corresponding to larger linguistic structures such as phrases and sentences, and that the neural representation of each linguistic level corresponds to timescales matching the timescales of the respective linguistic level. Although linguistic structure building can clearly benefit from prosodic 20,21 or statistical cues 22 , it can also be achieved purely on the basis of the listeners' grammatical knowledge. To experimentally isolate the neural representation of the internally constructed hierarchical linguistic structure, we developed new speech materials in which the linguistic constituent structure was dissociated from prosodic or statistical cues. By manipulating the levels of linguistic abstraction, we found separable neural encoding of each different linguistic level. RESULTS In the first set of experiments, we sought to determine the neural representation of hierarchical linguistic structure in the absence of prosodic cues. We constructed hierarchical linguistic structures using an isochronous, 4-Hz sequence of syllables that were independently synthesized (Fig. 1a,b, Supplementary Fig. 1 and Supplementary Table 1) . As a result of the acoustic independence between syllables (that is, no co-articulation), the linguistic constituent structure could only be extracted using lexical, syntactic and semantic knowledge, and not prosodic cues. The materials were first developed in Mandarin Chinese, in which syllables are relatively uniform in duration and are also the basic morphological unit (always morphemes and, in most cases, monosyllabic words). Cortical activity was recorded from native listeners of Mandarin Chinese using magnetoencephalography (MEG). Given that different linguistic levels, that is, the monosyllabic morphemes, phrases and sentences, were presented at unique and constant rates, the hypothesized neural tracking of hierarchical linguistic structure was tagged at distinct frequencies. The MEG response was analyzed in the frequency domain and we extracted response power in every frequency bin using an optimal spatial filter (Online Methods). Consistent with our hypothesis, the response spectrum showed three peaks at the syllabic rate (P = 1.4 × 10 −5 , paired one-sided t test, false discovery rate (FDR) corrected), phrasal rate (P = 1.6 × 10 −4 , paired one-sided t test, FDR corrected) and sentential rate (P = 9.6 × 10 −7 , paired one-sided t test, FDR The most critical attribute of human language is its unbounded combinatorial nature: smaller elements can be combined into larger structures on the basis of a grammatical system, resulting in a hierarchy of linguistic units, such as words, phrases and sentences. Mentally parsing and representing such structures, however, poses challenges for speech comprehension. In speech, hierarchical linguistic structures do not have boundaries that are clearly defined by acoustic cues and must therefore be internally and incrementally constructed during comprehension. We found that, during listening to connected speech, cortical activity of different timescales concurrently tracked the time course of abstract linguistic structures at different hierarchical levels, such as words, phrases and sentences. Notably, the neural tracking of hierarchical linguistic structures was dissociated from the encoding of acoustic cues and from the predictability of incoming words. Our results indicate that a hierarchy of neural processing timescales underlies grammar-based internal construction of hierarchical linguistic structure. npg
Recently, REF demonstrated strong evidence for a neural tracking of hierarchical linguistic structures in the brain.
Cortical tracking of hierarchical linguistic structures in connected speech
Nature Neuroscience
Nature Neuroscience
Medicine Biology