Masking Morphosyntactic Categories to Evaluate Salience for Schizophrenia Diagnosis (2022)

Abstract: Natural language processing tools have been shown to be effective for detecting symptoms of schizophrenia in transcribed speech. We analyze and assess the contribution of the various syntactic and morphological categories towards successful machine classification of texts produced by subjects with schizophrenia and by others. Specifically, we fine-tune a language model for the classification task, and mask all words that are attributed with each category of interest. The speech samples were generated in a controlled way by interviewing inpatients who were officially diagnosed with schizophrenia, and a corresponding group of healthy controls. All participants are native Hebrew speakers. Our results show that nouns are the most significant category for classification performance.

## introduction psychotic disorders such as schizophrenia are characterized by several symptoms, such as delusions, hallucinations, and thought disorders. thought disorders are described as disturbances in the normal way of thinking, typically presented as various language impairments, such as disorganized speech, which is related to abnormal semantic associations between words @xcite , and poverty of speech, a thought disorder that is associated with impairments in lexico-semantic retrieval @xcite . disorganized speech is divided into several markers, such as derailment, characterized by the usage of unrelated concepts in a conversa-tion; tangentiality, which happens when providing oblique or irrelevant answers to a question; and incoherence, also known as "word salad", refers to speech that is incomprehensible at times due to multiple grammatical and semantic inaccuracies @xcite . the diagnosis of schizophrenia is mostly based on a professional psychiatric review. however, some studies show that a computational linguistic analysis may help with diagnosis. @xcite , for example, demonstrated that by using a discriminant function analysis of linguistic variables it is possible to predict diagnoses with an accuracy rate of 79%. there have been many attempts to study speech impairments that are related to thought disorders using a computational method. some of those studies analyze the frequency of using different part-of-speech categories, such as nouns and verbs. for example, @xcite reported a significantly lower frequency of adjectives in schizophrenic speech than in healthy control speech. on the other hand, they reported a higher frequency of verbs used by patients. @xcite measured a low frequency of adverbs in speech produced by patients with schizophrenia. @xcite analyzed speech produced by hebrew speaking patients with schizophrenia and reported low frequencies of words inflected in the third person or in the past tense. aligned with previous work, they also reported lower frequencies of adverbs. it has been shown @xcite that patients with schizophrenia are produc-ing grammatically simpler speech than healthy people. the results are not always consistent; @xcite , for example, reported high frequencies of adverbs and adjectives in schizophrenic speech, in contrast to the reports made by other works. until very recently, the large majority of those studies were conducted with english speaking patients. one of the most popular technologies in natural language processing (nlp) is language modelling. a language model is essentially a function that assigns a probability to a given sequence of words occurring in a sentence. there are different ways to fit a language model to a certain distribution, typically using massive collections of texts. an autoregressive model conditions the probability of a word on the text that has already been seen in direction of reading. on the other hand, masked language models (mlm) are given the full sentence, while learning to assign probability to a randomly chosen hidden (masked) word. such models are typically used as the basis for an algorithm that aims at solving a specific downstream task, such as sentiment analysis or document classification. in the first phase, the models are pre-trained for the word-probability assignment using a large unlabeled collection of texts, and later are fine-tuned on a labeled dataset for a specific downstream task. while the autoregressive models are more suitable for generation tasks, mlms are typically the best option for fine-tuning on classification tasks. this development of pre-trained language models provides us with the opportunity to examine the importance of certain morphosyntactic categories in speech of patients with schizophrenia, and compare it to that of a healthy control group. specifically, we fine-tune an mlm to classify transcribed speech segments into patient or control categories, and examine its performance under extreme situations of hiding (masking) words that belong to a specific syntactic or morphological category. while most existing techniques use some sort of counting method, in this study, we explore an alternative innovative way for assessing the salience of a specific category for detecting schizophrenic speech. we utilize the original masking technique of an mlm, by naturally masking out specific morphosyntactic categories and measure the performance of the model on a downstream classification task. the experimental results show a decrease in pre-diction accuracy once nouns are masked, suggesting that nouns are more informative than other categories we tested for differentiating between patients and controls. our participants are all native hebrew speakers. ## related work computational modeling has been studied in relation to cognitive disorders in order to fill the gap between theoretical models and biological evidence. @xcite reviews popular neural network models for autism spectrum disorder and schizophrenia, using different types of input. both disorders are characterized by an altered perception of the world. according to this review, models of schizophrenia mainly concentrate on positive symptoms, such as hallucinations and delusional behavior (e.g., @xcite ; @xcite ). however, there are also models that target other symptoms such as disturbances of attention @xcite and movement disorders @xcite . the use of computational linguistic models has been applied to studying language abnormalities related to mental illness, specifically schizophrenia. disorganized speech, including derailment, incoherence, and tangentiality, is among the common symptoms of schizophrenia being studied by researchers using computational methods (e.g., @xcite al. (2015); pauselli et al. (2018); iter et al. (2018); bar et al. (2019); just et al. (2020) ). @xcite reviews computational methods that perform linguistic analysis of psychosis, focusing on three language abnormalities: disorganized speech, poverty of speech, and flat affect. many studies have employed latent semantic analysis (lsa) and word embedding models (e.g., word2vec and glove) to measure disorganized speech. typically, the embeddings are used to measure semantic similarity between words in the sentence, or between entire sentences or paragraphs, to assess semantic cohesion as a predictor for disorganized speech. in several studies (e.g., @xcite al. (2007); iter et al. (2018); just et al. (2019) ), psychosis patients scored significantly higher on disorganization than controls. however, @xcite argues that the measures are not consistent across other studies. as mentioned in the previous section, most of those works analyze transcribed speech spoken in english, which is characterized by a relatively simple morphological system. some recent studies have been exploring similar techniques applied to other languages, such as german @xcite and hebrew @xcite . the latter have studied derailment, a symptom of thought disorder characterized by switching between topics and jumping from one disconnected thought to another. they measure derailment in speech through semantic similarity of adjacent words using their embeddings. it was found that patients with schizophrenia are more likely to derail than healthy controls, consistent with previous studies @xcite . further, they examine incoherence in schizophrenic patients, to see how they use adjectives and adverbs to describe specific nouns and verbs. their analysis makes use of a dependency parser for hebrew, which yields a word-dependency list for each sentence. using dependencies, they discovered that the adjectives and adverbs used by the controls are more similar to those commonly used to describe the same nouns and verbs. there are not many works that leverage language models to analyse text for detecting mental health symptoms, such as we do. in a recent work @xcite , bert @xcite , a large english language model, has been used to encode full sentences and compare the resulting embeddings of adjacent sentences for measuring tangentiality. their results reflect increased tangentiality among patients with schizophrenia. in our work, we use a language model as a tool for assessing the contribution of six morphosyntactic categories to the classification of transcribed speech into patients or controls. ## participants and data collection we interviewed 49 males, aged 18-60, divided into control and patient groups, all speaking hebrew as their first language. the patient group includes 23 inpatients from the be'er ya'akov-ness ziona mental health center in israel who were admitted following a diagnosis of schizophrenia. diagnoses were made by a hospital psychiatrist according to the dsm-5 criteria (american psychiatric association dsm-5 task force, 2013) and a full psychiatric interview. each participant was rewarded with approximately $8. the control group includes 26 men, mainly recruited via an advertisement that we placed on social media. exclusion criteria for all participants were as follows: additionally, the control group had to score below the threshold for subclinical diagnosis of depression and post-traumatic stress disorder (ptsd). most of the control participants scored below the threshold for anxiety. most of the patients scored above the threshold for borderline or mild psychosis symptoms on a standard measure. foot_0 see section 3.1 for more information about the measures we use in this study. the demographic characteristics of the two groups are presented in patients were interviewed in a quiet room at the department where they are hospitalized by one of our professional team members, and the control participants were interviewed in a similar room outside the hospital. each interview lasted approximately one hour. the interviews were recorded and later manually transcribed by a native hebrew speaking student from our lab. all participants were assured of anonymity, and told that they are free to end the interview at any time. after signing a written consent, each participant was asked to describe 14 images picked from the thematic appreciation test (tat) collection; the images were presented one by one. we used the tat images identified with the following serial numbers: 1, 2, 3bm, 4, 5, 6bm, 7gf, 8bm, 9bm, 12m, 13mf, 13b, 14, and 3gf. all images are black and white, including a mixture of men and women, children and adults. once all 18 components (14 image descriptions and 4 open questions) were answered, each participant was requested to fill in a demographic questionnaire as well as some additional questionnaires for assessing mental-health symptoms, which we describe next. ## discussion we notice that nouns and adverbs make the biggest impact on the performance of the classifier, suggesting that those syntactic categories are the most informative to the model. comparing with random masking of the same number of words, the accuracy drops significantly (@xmath0.0005) when nouns are being masked. with adverbs, the difference in accuracy is less significant (@xmath1.058). based on the numbers assembled in figure 1 , we cannot attribute our findings to the frequency of usage of those categories. whereas nouns are used more frequently than the other categories, adverbs are much less frequent. for adverbs, at least, we see a significant difference in the frequency of usage between the two groups; controls use them more. adverbs are typically used in tandem with a verb; however, it turns out that the patients use slightly more verbs than the controls, although to an insignificant degree. therefore, we believe that the significant difference in usage frequency of adverbs may be the reason for the impact that they make on classification performance. as for nouns, we see no evidence for a usage frequency difference between the two groups. we believe that the reason for the impact made by masking nouns on the classification performance might be related to the importance of nouns in the syntactic tool set of patients with schizophrenia. our results may suggest that the patients convey their messages more through nouns than through other linguistic categories. nouns are considered the backbone of a language; it has been shown that english-speaking children acquire knowledge of nouns before verbs @xcite . nouns are considered easier to learn than verbs, probably due to their imageability @xcite . therefore, we presume that focusing more on nouns when conveying a message may be an indicator of poverty of speech. the way patients use nouns is slightly different from how controls do. this difference makes it easier for the model to predict schizophrenic symptoms. the source of the difference may be related to the type of nouns that they choose to use in a sentence, the similarity among the nouns in a sentence, or their syntactic relations with other words in the sentence. since hebrew is a highly inflected language, it could also be that patients inflect nouns differently than controls. we plan to further investigate the source of the difference in follow up work. ## ethical considerations ziona mental health center. participants were guaranteed anonymity. the data was stored on a secured server, with limited access provided only to the authors of this paper. like with every other machine-learning model, there is a risk that the training data is unbalanced. specifically, we do not intentionally balance the dataset for ethnicity or political affiliation. moreover, this work is based on interviews with men only. additionally, the language model that we use, alephbert, was trained on large and less controlled datasets. that may introduce some additional aspects of bias. therefore, our study may harbor the danger of over-reliance on possibly biased machine tools. we do not mean to suggest that an algorithm can or should be used to diagnose schizophrenia automatically. this study should not be considered as a building block for an apparatus that takes automatic decisions about topics related to mental health. our intention is, rather, to use computational tools to identify and study the importance of various linguistic characteristics for diagnosing schizophrenia. like other machine-learning applications, explainability is currently a problematic issue (what is it about the usage of nouns that contributes significantly to the model's success in classification?), and undue reliance on machine classification should be eschewed. ## conclusions we studied the relative importance of several morphosyntactic categories for transcribed speech towards the classification task of distinguishing schizophrenia sufferers and controls. this was based on interviews of 23 male inpatients at a mental health center in israel, officially diagnosed with schizophrenia, as well as 26 control participants; all are native hebrew speakers. the interviews were manually transcribed and divided into indi-vidual responses that the participants provided for 18 discussion topics. four topics were open-ended questions, and the rest were tat images that were shown to the participants who were asked to describe the situation they see in the image. we trained a natural-language-processing classifier by fine-tuning alephbert, a relatively large hebrew language model, to distinguish between responses generated by patients and controls. to evaluate the contribution of different syntactic and morphological categories to the classification performance, we fine-tune the model each time by masking words of one specific category, and compare the classification performance with the same model trained on texts that were instead masked randomly for the same number of words. when the category-masked model performed more poorly than the randomly-masked model, we attribute it to an increased importance of the corresponding category. this new, masking method of evaluating the significance of linguistic features promises to be of use in many additional feature evaluation tasks. overall we examined six categories, and found (unsurprisingly) that nouns are the most important for distinguishing between patients and controls. we believe that it has to do with the idea of nouns being easier to capture in the mind due to their imageability. given that nouns are used in comparable frequency by patients and controls, our findings reveal that the patients use nouns in a different way than do controls. we plan to investigate this further by looking more closely at the potential sources for this difference, in order to check how they may be related to poverty of speech.