Fantastic Features and Where to Find Them: Detecting Cognitive Impairment with a Subsequence Classification Guided Approach (2020)

Abstract: Despite the widely reported success of embedding-based machine learning methods on natural language processing tasks, the use of more easily interpreted engineered features remains common in fields such as cognitive impairment (CI) detection. Manually engineering features from noisy text is time and resource consuming, and can potentially result in features that do not enhance model performance. To combat this, we describe a new approach to feature engineering that leverages sequential machine learning models and domain knowledge to predict which features help enhance performance. We provide a concrete example of this method on a standard data set of CI speech and demonstrate that CI classification accuracy improves by 2.3% over a strong baseline when using features produced by this method. This demonstration provides an example of how this method can be used to assist classification in fields where interpretability is important, such as health care.

## introduction in recent years, word and sentence embeddingbased methods have had a significant impact on the field of nlp @xcite @xcite . these approaches stand as an alternative to classical feature engineering approaches, where carefully crafted features, such as word length or part of speech tag, are extracted from text and used as input. despite the promise of embeddingbased methods, there are still several advantages to feature engineering. most notably, using embeddings as input can lead to issues with interpretability @xcite @xcite , which is especially important in a healthcare domain @xcite . meanwhile, feature engineering approaches directly lend themselves to easily inter-pretable models @xcite . as such, feature engineering remains an important practice for fields such as health care, where interpretability is imperative. an extensive body of work has been produced where ml methods and engineered features have been applied to cognitive impairment (ci) detection @xcite @xcite . in this work, we present a new feature engineering method that is guided by classifying subsets of a pause-centred speech sequence (subsequences), and inspired by literature suggesting that ci could be indicated by the words that subjects pause before @xcite @xcite . this approach aims to extract pauserelated information while minimizing the noise added from unrelated factors. this method generates interpretable and effective features, potentially saving time and resources spent on excess feature engineering. we validate this method by presenting a 2.3% accuracy increase over a strong baseline on ci vs healthy (hc) classification, matching the state of the art @xcite . in summary, our major contributions are: ## related work several authors report increases in ci detection performance by extracting acoustic features such as filled and unfilled pause counts, as well as average pause duration @xcite @xcite 2018 ) achieved an accuracy of 78% when classifying dementiabank (sec. 3.1) transcripts as ci or hc using an extended set of lexical features, which is to the best of our knowledge the state of the art (sota). we use their recorded performance as a benchmark when validating our approach. several authors have reported performance gains by using subsequences to aid with classification. these authors use subsequences only as a means to process full sequences @xcite , or they use the presence of common subsequences as a feature for longer text sequences @xcite . to the best of our knowledge, no prior work describes using subsequence classification to guide feature engineering. ## experimental method in this section, we define the data sets and methodology used in our experimental framework. ## proposed feature engineering approach our approach to engineering new transcript-level features involves three major steps: 1) subsequences of varying length centred around a token of interest, in our case a pause, must be extracted from each of the input transcripts and grouped into subsets based on maximum length. each of the tokens in these subsequences must have token-level features extracted. the tokenlevel features, as well as the central token, should be chosen based on in-domain knowledge. these transcript-level aggregates should only be extracted for the distances that produced the great to validate this method, we create five transcriptlevel feature sets: features aggregated from tokens at the d1 position in reference to a pause (f-d1), features aggregated from the d2 position (f-d2), features aggregated from the d3 position (f-d3), the combination of f-d1, f-d2, and f-d3 (f-c3), and the combination of f-d1 and f-d2 (f-c2). ## results in this section, we report the results for the subsequence and transcript classification experiments. ## discussion as shown in tab. 3 and tab. 2, features from tokens within 2 tokens of a pause were the most effective in enhancing both subsequence and transcript classification. to determine how these two tasks are connected, we conduct a statistical analysis on the token-level and transcript-level features. two sided t-tests between features extracted from tokens found at d1, d2, and d3 from different classes show similar patterns for features that are significantly different between classes for both the token and transcript-level. larger concentrations of distinguishing features are found at d1 and d2 than at d3. this could explain the effectiveness of features from the d2 position in both tasks (tab. 4). however, this pattern congruity does not explain why f-d3 features on their own are more effective than f-d1 features on their own. the trend that both the f-d3 and f-c3 feature sets produced greater transcript-level accuracy than the f-d1 feature set, and lower transcript-level accuracy than the f-d2 and f-c2 feature sets, is the same as the trend for the subsequence classification results reported in tab. 3. this indicates that subsequence classification may be able to provide better insight into potential transcript classification performance than traditional statistical testing. it is important to consider the implications of producing a model with the f-d2 feature set that achieved significantly higher accuracy than the most accurate model produced with the f-c3 feature set. as described in section 3.3, we perform feature selection using anova f-values for each of the aggregate feature sets. since f-d2 is a subset of f-c3, this implies that this more traditional feature selection method did not select a group of features from f-c3 that was more effective than the features from f-d2, even though it was able to select any and all of the features in f-d2. this serves as a testament to our feature engineering method, as it demonstrates that even popular feature selection methods are not able to completely remove the negative effects of engineering an excessive amount of ineffective features. following several other works that used the db data set @xcite @xcite , all of our experiments are conducted with k-fold cross validation. while the small size of the db data set helps to justify this as a validation procedure, optimizing a cross validated performance metric (accuracy, f1, etc.) may lead results using k-fold cross validation to be an overestimate of generalization performance. db-c2 produced a more accurate subsequence classifier than any other data subset of db. this suggests that the class distinguishing signal from the pause is strongest within a two token radius around the pause. beyond that radius, the signal may be obstructed by noise from other patterns in speech. however, in different data sets, a different subsequence length may present the strongest, least noisy signal. new aggregate features should be created for tokens within whichever range produces the best subsequence classification performance. however, our results do indicate that there is a strong link between how well features from certain token positions contribute to both subsequence and transcript classification. this may relate to the effect of noise on those token positions, which we use subsequence classification to identify. ## conclusion and future work in this work, we present two principle contributions. first, we describe a novel method for speech classification -subsequence classification -in which speech is modelled as a token of interest, such as a pause, along with surrounding tokens of context. secondly, we demonstrate how subsequence classification can be used to engineer features that extract distinguishing information while minimizing added noise, and consequently match sota performance on a standard data set of ci speech. future work should be done to understand why certain context lengths are more conducive for subsequence classification than others, and when that performance can transfer to effective transcriptlevel classification. finally, additional work should be done to develop techniques for finding tokens of interest, such as pauses, that can be exploited using our feature engineering technique.