Deep Natural Language Feature Learning for Interpretable Prediction (2023)

Abstract: We propose a general method to break down a main complex task into a set of intermediary easier sub-tasks, which are formulated in natural language as binary questions related to the final target task. Our method allows for representing each example by a vector consisting of the answers to these questions. We call this representation Natural Language Learned Features (NLLF). NLLF is generated by a small transformer language model (e.g., BERT) that has been trained in a Natural Language Inference (NLI) fashion, using weak labels automatically obtained from a Large Language Model (LLM). We show that the LLM normally struggles for the main task using in-context learning, but can handle these easiest subtasks and produce useful weak labels to train a BERT. The NLI-like training of the BERT allows for tackling zero-shot inference with any binary question, and not necessarily the ones seen during the training. We show that this NLLF vector not only helps to reach better performances by enhancing any classifier, but that it can be used as input of an easy-to-interpret machine learning model like a decision tree. This decision tree is interpretable but also reaches high performances, surpassing those of a pre-trained transformer in some cases. We have successfully applied this method to two completely different tasks: detecting incoherence in students’ answers to open-ended mathematics exam questions, and screening abstracts for a systematic literature review of scientific papers on climate change and agroecology.

## introduction and related work the use of ai models is becoming increasingly pervasive in today's society. as such, applications of these models have seen the light in domains where decisions can have dramatic consequences such as healthcare @xcite , 1 code available at https://github.com/ furrutiav/nllf-emnlp-2023 justice @xcite , or finances @xcite . this pervasiveness is intrinsically linked to the huge success of deep learning models, which have shown to scale particularly well to complex decision-making problems. however, this success comes at a cost. to solve complex (high-stakes) decisions, these models develop inner hidden representations which are hard to understand and interpret even by researchers implementing the models @xcite . regulations like the european union's "right to explanation" @xcite or @xcite requirements for safe and secure ai in safety-critical tasks fostered advances in explainability. therefore, there has been a growing interest in developing techniques allowing to explain the mechanisms subtending these so-called "black-box" models @xcite . notably, this endeavor has given rise to an entire research field called explainable ai (xai), which focuses on providing human-interpretable information on the models' behavior @xcite . explainable deep learning in natural language processing (nlp) can be decomposed in two categories: representational and practical. representational xai in nlp focuses on understanding the underlying structure of the representations. for instance, studies have shown that transformer-based architectures develop abstract symbolic, or compositional, representations @xcite . similarly, it has been shown that conceptual knowledge can have sparse representations, which can thus be located and edited to induce different predictions @xcite . practical methods can analyze the outputs of the models, for example when perturbing the inputs @xcite @xcite . recently, practical xai in nlp focuses on prompting to increase explainability. for instance, chain-of-thought prompting is a q: does the article assess agroecological practices' impact on climate change? a: yes contribution of crop residue, soil, and fertilizer nitrogen to nitrous oxide emissions varies with long-term crop rotation and tillage agriculture is an important contributor to n2o emissions -a potent greenhouse gas -with high peaks occurring when soil mineral nitrogen (n) is high (e.g., after mineralization of organic n and n fertilizer application). nitrogen dynamics in soil and consequently n2o emissions are affected by crop and soil management practices (e.g., crop rotation and tillage), an effect mostly assessed in the literature through comparisons of total n2o emission. hence, information is scarce on the effect of these management practices on specific n sources affecting n2o emissions (i.e., n fertilizer, soil, above and belowground crop residues) -a knowledge gap explored in this study with the use of n-15 tracers. the isotope approach enabled … (more) yes does the article analyze how agroecology affects nitrogen dynamics? yes does the study specifically evaluate greenhouse gas emissions? no does the article discuss certified organic production? … binary subtask questions decision tree @xmath0: feature extraction: grass tillage crop organic management practices agriculture convent* ch4 emission* conservation greenhouse model mitigation … linguistic features ## methods this section describes the different parts of the proposed method, which are summarized in figures 2 and 3 . subsection 2.1 shows how to utilize an instructed llm to generate natural language binary subtasks questions that can be useful to solve a more complex task. subsection 2.2 (in green in figure 3 ) explains the process to leverage the zero-shot ability of a llm in order to label examples regarding the binary subtasks. next, subsection 2.3 (in orange in figure 3 ) contains details on how we train a bert-like model @xcite in a natural language inference (nli) fashion to resolved those subtasks. finally, subsections 2.4 and 2.5 (in blue in figure 3 ) describe the process to generate interpretable representations and how to integrate them in an explainable model to solve the main task. ## conclusion and future work we proposed a new method to leverage low-level reasoning knowledge related to a more complex task from a large language model, and integrate this into a smaller transformer. the transformer has been trained in a way that it can be used for zeroshot inference with any low-level reasoning having the task formulated in natural language. this allows the practitioners to formulate easily their own features related about the task. the natural language learned feature vector can then be used as a representation in any other classifier. we show that it is easy to train an interpretable model like a decision tree, leading to both competitive results and interpretability. our method can be applied to any predictive task using text as input. future work should focus on investigating the potential impacts of this approach in real-world educational settings, and especially inspect the preferences of the practitioners regarding different explainability methods in order to help them taking complex decision @xcite .