Towards a Standardized, Fine-grained Manual Annotation Protocol for Verbal Fluency Data (2020)

Abstract: We propose a new method for annotating verbal fluency data, which allows the reliable detection of the age-related decline of lexical access capacity. The main innovation is that annotators should inferentially assess the intention of the speaker when producing a word form during a verbal fluency test. Our method correlates probable speaker inten-tions such as “intended as a valid answer” or “intended as a meta-comment” with lin-guistic features such as word intensity (e.g. reduced intensity suggests private speech) and syntactic integration. The annotation scheme can be implemented with high reliabil-ity, and minimal linguistic training. When fluency data are annotated using this scheme, a relation between fluency and age emerges; this is in contrast to a strict implementation of the traditional method of annotating verbal fluency data, which has no way of deal-ing with score-confounding phenomena because it force-groups all verbal fluency pro-ductions –regardless of speaker intention— into one of three taxonomic groups (i.e. val-id answers, perseverations, and intrusions). The traditional lack of fine-grained annota-tion units is especially problematic when analyzing the qualitatively distinct fluency da-ta of older participants and may cause studies to miss the relation between lexical access capacity and age.

## introduction verbal fluency tasks (also known as verbal fluency tests; vfts) are hybrid neuropsychological tasks @xcite ) that assess both lexical access capacity -the capacity to retrieve lexical units from the mental lexicon @xcite -and executive function. in vfts, participants are required to provide as many (non-repeated) words as possible that correspond to the experimental criteria within a time-limit (typically, 60 seconds). these criteria are either semantic (e.g. name as many animals as possible) or phonemic (e.g. list words that begin with the letter p). vfts are amongst the most widely used language tasks in both research and clinical settings owing to their sensitivity to a wide variety of psycho-neurobiological phenomena -dementia (e.g. @xcite , bilingualism (e.g. @xcite , and aging (e.g. @xcite , to name but these. performance on these tasks has been used to characterize the fluency capacity of various groups, the classic performance measures being the number of correct answers, the number of intrusions (words that do not meet the experimental criteria), and the number of perseverations (correct answers that have been repeated) (e.g. @xcite . while the traditional three-way taxonomy (i.e. correct answers, perseverations, intrusions) is easy to use and is widespread in the fluency literature (e.g. @xcite @xcite , it has a serious shortcoming: it can only handle clean data. consider, for instance, the following illustrative example of a series of verbal productions drawn from the t fluency condition. font size is used to suggest speech intensity (volume), ellipsis is used to represent pauses, colons denote an elongated speech sound, dashes are used to denote temporal proximity, and forward slashes represent truncation; the data of interest is underlined: existing fluency protocols completely gloss over these types of phenomena (e.g. @xcite , but in our own fluency corpus we have observed that such phenomena are not unusual amongst cognitively healthy participants, especially older ones (see subsection 3 for some descriptive statistics). importantly, it is well-documented that older adults experience not only a decline in lexical access but also have different narrative styles compared to younger adults, particularly when the task is difficult @xcite . it is therefore possible that older adults' verbal fluency data differs qualitatively compared to that of younger adults; to our knowledge, no verbal fluency study has investigated this issue to date. the open question is whether there are sufficient instances of these score-confounding phenomena in the fluency data to impact performance scores (and thus study findings) in one or more fluency conditions, and more specifically whether the performance measures in these conditions vary with age when score-confounding phenomena are controlled for. what is needed to investigate this research question is to separate the data of interest from score-confounding phenomena using a standardized, fine-grained taxonomy that allows fluency annotators to make replicable, theoretically-justified categorization choices based on linguistic features that correlate with specific, contextually-plausible communicative intentions. in this paper, we briefly describe such a taxonomy and report promising preliminary findings bearing on a) the reliability of our protocol's transcription-annotation process, and b) the effect this taxonomy has on two classic measures of fluency performance in younger and older adults compared to a strict implementation of the traditional annotation scheme. ## creation of the fluency corpus to date, 38 depression-free (gds-15≥5; @xcite , cognitively-normal (@xmath0; @xcite , right-handed (@xmath1; @xcite native speakers of french (@xmath2.3 years; @xmath3.08; @xmath4; @xmath5; @xmath6.65 years; @xmath7.17) have been audio-recorded during the execution of 4 verbal fluency tasks (letters: t-n-p; category: animals). participants are classified into one of two age groups: younger (@xmath8; @xmath9.48 years; @xmath10.76; @xmath11; @xmath12.44 years; @xmath13.71; @xmath14. participants were recorded in a doublewalled soundproof room (génie audio. inc, canada) using a shure headset microphone (microflex beta 53) connected to a quartet usb audio interface (apogee electronics, santa monica, ca 90404, usa) that fed into an imac computer. this permitted the capture of very low-intensity phenomena (e.g. whispering). the recordings were made using sound studio 4 (felt tip inc., nyc, usa) at a sampling rate of 48 khz with 24 bits of quantization. per traditional procedure (e.g. @xcite , participants were instructed to produce as many words as possible that corresponded to the experimenterprovided criteria within 60 seconds, avoiding repetitions, proper names, and words that merely have different suffixes (e.g. pitch/pitched or cat/cats). the fluency tasks were administered as a part of a battery of tests within a large on-going project that was approved (#1733-2018) by the neurosciences and health ethics review board of the centre intégré universitaire de la santé et des services sociaux de la capitale nationale (translation: integrated health and social services centre of the national capital) in quebec city, canada. the raw fluency audio recordings were transcribed and annotated in praat @xcite using the extended version of the transcription-annotation scheme described below (see appendix b). ## towards a finer-grained annotation scheme: the linguistic correlates of intent our expanded fluency taxonomy rests on two fundamental assumptions. firstly, a fluency taxonomy should be comprised of categories that reflect speaker communicative intentions. simply put, the issue is not whether the fluency annotators can associate -using all knowledge at their disposal-a given phonetic sequence within the data with conceptual content and thereby categorize the sequence using some rule-scheme. the issue, fundamentally, is whether one can reasonably infer that the speaker associated the sequence with conceptual content at the moment of verbalization, and, what is more, whether they produced the sequence to satisfy the experimental criteria (intentionality). our second fundamental assumption is that speaker intent has linguistic correlates (cf. @xcite . using a corpus-driven approach @xcite , we associated probable pragmatic intentions (e.g. intent to satisfy the experimental criteria, etc.) with syntactic and phonetic features. the taxonomy's key categories are briefly described below; a full list can be found in appendix b. percentage frequency of occurrence of each of the annotation units is featured in parentheses in the manner that follows: relative frequency in the whole corpus, relative frequency in the fluency data of adults aged 28-54, and relative frequency in the fluency data of adults aged 65+. features that served to make annotation decisions are also included in each description. ## testing the new annotation scheme in the following, we assess our fluency annotation scheme's reliability as well as its effect on two classic measures of fluency performance. we also show that our fluency protocol is sensitive to age-related decline of lexical access, and that this decline emerges largely because of the way in which syllable play and self-talk are handled by our inferential annotation scheme. ## discussion and future work the traditional fluency taxonomy has been used for decades but it offers no formally codified means to avoid the force-grouping of data that have fundamentally different cognitive causes (e.g. correct answers and syllable play), and whose use may, crucially, differ across the adult lifespan. given that fluency scores are typically based on a mere 60 seconds of verbal production and that differences in fluency performance between groups are often modest to begin with (e.g. @xcite , the non-systematic treatment of score-confounding phenomena has the potential to alter individual performance scores (and thereby study results). the accuracy of the annotation process is thus not a trivial matter, particularly for studies with small effect sizes and/or studies with small sample sizes (where annotation choices carry comparatively greater statistical weight). if fluency studies are to be maximally consistent and their results maximally comparable, the handling of fluency data cannot fall under the scope of "researcher degrees of freedom" -undeclared "flexibility in data collection, analysis, and reporting" @xcite . a detailed, publicly accessible annotation protocol -one that weighs the linguistic evidence that bears upon the participant-speaker's communicative intentions-is required to separate the fluency data into finer-grained categories in a non-arbitrary manner so that cognitively similar phenomena of interest may be identified and isolated for the purpose of statistical analysis. the extended fluency taxonomy briefly described and tested here constitutes a promising first step towards such a standardized transcription-annotation fluency protocol. preliminary results suggest that a) the protocol is highly implementable, as evidenced by excellent agreement and reliability scores; b) it requires minimal linguistic training (as evidenced by its successful use by lg, a first year undergraduate linguistics student); and c) most importantly, it yields non-inflated fluency performance scores compared to a purely mechanistic implementation of the traditional fluency taxonomy. preliminary findings suggest that the adoption of such a protocol is particularly germane for the comparative analysis of the verbal fluency data of adult participants of different ages. based on our corpus here, we suggest that older adults tend to produce qualitatively distinct fluency data compared to younger adults (likely as a compensatory strategy for declining lexical access ability), hence why the two annotation schemes produce significantly dissimilar performance measures for this group). with a new, finer-grained annotation scheme to structure the data also come new, exciting research questions. does the use of fillers and self-talk in fluency data vary along sociodemographic and/or cognitive parameters? what proportion of fluency errors are cognitively normal participants aware of? how effective is syllable play as a word identification strategy and is its effectiveness moderated by age? do older participants who use lexical access strategies such as self-talk and syllable play achieve lesser or greater performance scores according to our fluency protocol? perhaps most importantly of all, might the controversy regarding the maintenance or decline of phonemic fluency performance over the lifespan (cf. @xcite be partly attributable to inconsistent annotation procedures between fluency studies? we intend to soon investigate these and other questions using a considerably expanded fluency corpus (120-150 participants).