Enabling Classifiers to Make Judgements Explicitly Aligned with Human Values (2023)

Abstract: Many NLP classification tasks, such as sexism/racism detection or toxicity detection, are based on human values. Yet, human values can vary under diverse cultural conditions. Therefore, we introduce a framework for value-aligned classification that performs prediction based on explicitly written human values in the command. Along with the task, we propose a practical approach that distills value-aligned knowledge from large-scale language models (LLMs) to construct value-aligned classifiers in two steps. First, we generate value-aligned training data from LLMs by prompt-based few-shot learning. Next, we fine-tune smaller classification models with the generated data for the task. Empirical results show that our VA-Models surpass multiple baselines by at least 15.56% on the F1-score, including few-shot learning with OPT-175B and existing text augmentation methods. We suggest that using classifiers with explicit human value input improves both inclusivity & explainability in AI.

## introduction the demand for responsible nlp technologyto make it more robust, inclusive and fair, as well as more explainable and trustworthy -has increased since pre-trained large-scale language models (llms) have brought about significant progress in making nlp tasks more efficient and broad-ranging @xcite @xcite @xcite @xcite . researchers have studied how to align machines with human values as one of the directions to achieve responsible ai technology by teaching machines about moral and social norms @xcite @xcite , ethics and common human values given the same content, va-model makes variable predictions based on explicitly provided human values. @xcite or human preferences @xcite . value-alignment of ai systems is not a trivial problem as human values are non-consensual by nature @xcite . values can be very diverse and most existing works have attempted to align machines with shared human values or average norms, or from a certain cultural perspective with crowd sourced annotations @xcite . these days, for instance, many societies agree that sexism should be eliminated, and we expect machines to be non-sexist, but different individuals and cultures may perceive sexism differently. as is shown in figure 1 , the same content can be considered to be sexist or non-sexist depending on the values provided to make the judgements. in this paper, we propose a value-aligned judgement task that separates the value definition process from the development of the models for more inclusive and explainable value-aligned nlp. our proposed task aims to build a single model to make dynamic judgements based on explicitly provided human values, requiring the model to understand the value and its corresponding entailment on the given content. the value is provided in the form of instructions, allowing coarse-to-fine customization. we start with value-aligned sexism classification as a proof of concept for the proposed approach, as sexism is one of the most representative examples of varying cultural perspectives. we also present value-aligned models (va-models) that leverage value-based knowledge from llms. llms are trained from vast amounts of human data with embedded human values @xcite . however, they are not controllable and it is difficult to fine-tune such large models with explicit value alignment. instead, we distill value-based training data from the llms using prompt-based data generation with example values, and build va-model by fine-tuning smaller classification models with the distilled data. experimental results show that our approach is more stable and accurate than directly applying few-shot learning on llms. moreover, our methodology avoids costly human labeling or crowd-sourcing of values, allowing easier extensions to other valuealigned tasks in different domains. we further investigate model performance using data generated from different scales and types of llms, and study the effect of data size for fine-tuning, and analyze the quality of the generated data. moreover, we study the generalization ability of va-models by testing its performance on unseen value sets. our contributions are as follows: 1) we introduce the value-aligned classification task, where we first define human values externally and then use them at the instruction level in an in-context learning paradigm and construct value-aligned classifiers to make predictions; 2) we propose to leverage prompt-based data generation to distill valuealigned knowledge from llms for smaller classification models; 3) experimental results indicate that our approach significantly outperforms strong baselines, including in-context few-shot learning with llms and existing text augmentation methods; 4) we systematically study factors that impact promptbased data generation and highlight research questions and challenges in the value-aligned judgement task through thorough analysis. to teach models value-alignment, the literature has focused on improving the model's reasoning ability relating to human values and morality @xcite @xcite . recently, @xcite proposed to finetune gpt-3 to adapt to a manually crafted valuestargeted dataset to arrive at a values-targeted model. however, in their approach, value alignment and definition are intertwined and entangled in an iterative process. we instead separate the value definition and alignment process models about valuealigned judgement with explicit value provision. prompt-based learning recently, llms have shown great performance on prompt-based learning @xcite , which doesn't require fine-tuning. instead, the model is directly fed a prompt that includes some examples, and the model can generate results as if it has "learned". studies on efficient prompt-learning/construction include @xcite ; @xcite ; @xcite ; @xcite . we consider the literature for prompt-construction in our methodology. knowledge distillation knowledge distillation is the transfer of knowledge from teacher to student distribution @xcite . recent works have attempted to perform distillation from llms by prompting for text generation to show that it outperforms existing text augmentation methods @xcite . @xcite retrieves commonsense knowledge symbolically in a text form from gpt-3 for downstream tasks with help of smaller filtering classifiers. we distill value-specific knowledge, not all abilities of general language model, from llms through value-aligned training data generation for training smaller value-aligned classifiers. this reduces the cost of human labeling and enables building smaller models specialized for value-aligned judgment task. ## methodology there is no existing resource for training valuealigned classification models. we therefore propose to leverage llms for generating synthetic training data. llms have been found to learn significant amounts of inherent knowledge as well as human values during pre-training @xcite @xcite . however, the direct usage of llms in zero-shot setting for nlp tasks can be unstable and still limited @xcite . the richly embedded knowledge in llms nevertheless makes them good resource generators. therefore, we attempt to build value-aligned models (va-models) through fine-tuning smaller models on the valuealigned training data generated by llm(s). our proposed method (figure 2 ) consists of two steps: 1) prompting human value-aligned contents from llms by providing explicit human values and instructions, and 2) fine-tuning smaller lms on the generated data to teach them about value-aligned judgements. formally, we build va-model (parameterized by θ) to maximize the following likelihood: ## experiments in this paper, we conduct value-aligned sexism classification. models are expected to label content with label choices sexist, non-sexist, na depending on explicitly provided values. ## discussion and broader impacts in the present section, we provide a discussion on the broader impacts of our proposed framework, including its potential benefits and risks of misuse. benefits our proposal entails the decoupling of the human value definition from the valuealignment task, with the former being defined through collaboration among society, ethicists, social scientists, and other relevant parties. meanwhile, the mechanics of value alignment should remain independent of the initial value definition to prevent engineers from directly defining values in the training data or code. this separation of tasks brings numerous benefits, such as the development of localized values that align with the values espoused by users and developers, accounting for cultural and historical differences. additionally, it enables accountability for whoever provides the value definition, whether from human experts or digital sources. ## conclusion and future work in this paper, we propose a task that focuses on teaching a model human value alignment knowledge. we also introduce value-aligned models (va-model) that generate value-aligned training data from llms by prompt-based data generation and fine-tune smaller classification models with the value-aligned generated training data. experimental results show that va-model generally outperforms strong baselines. further analysis illustrates that the generated data from larger llms helps increase the performance, and more generated data can cause performance reduction when the data size is too large. finally, we highlights several research challenges for future work: improvements in 1) the robustness of the model on diverse values, 2) the models' generalization ability for our value-aligned judgement task, 3) higher quality generated data with more human curation.