Paper: Concept Alignment as a Prerequisite for Value Alignment (ICLR-2024)

Reviewer comment (Evaluation):
While it is not expected that the paper would solve these scaling issues, a deeper discussion of these potential limitations would be useful to the reader.

Author response:
We thank the reviewer for their feedback. It is true that this is not intended to scale in its current form, but rather demonstrate concretely the limitations of value alignment as it is currently discussed in the literature, and to show how incorporating concept alignment could address those limitations. To understand the limitations of value alignment at scale, the AI safety community will need to study it across many different contexts and disciplines, of which this paper addresses only one.

What these limitations look like vary greatly depending on the context of the AI being used, and the degree of human-AI alignment needed in each of those contexts. For example, scaling up concept alignment to a healthcare diagnostic algorithm would not require agent-based RL at all, and would not be implemented in an IRL setting. Rather it would be about the medical imaging artifacts that lead to diagnostic decisions, the words human physicians use to describe those artifacts, and how to align those concepts to the diagnostic algorithm which analyzes the same medical image and suggests a course of action, like surgery. This type of concept alignment would be a necessary prerequisite to discussing surgery based on what the patient values most (mobility, lack of pain, effect on other organs, long-term psychological impact of the procedure). If the AI is not able to honestly and concretely align concepts with the patient or the doctor, and meaningfully explain why a diagnostic judgment was made, the resulting medical decision is likely to be value misaligned with the patient in critical ways.

In contrast, scaling up concept alignment to something like deep RL could be much more similar to our Bayesian IRL example, with the values explicitly tied to the reward functions. We anticipate and encourage this breadth of work studying concept alignment and the resulting effect on value alignment, or misalignment, as a critical step forward in AI safety discussions. 

We have significantly revised our “Societal Implications of Concept Misalignment” section and “Limitations and Future Work” section (6.2-6.3) to discuss these implications, per the reviewer’s suggestion. We have also highlighted how our motivations differ from the previous works with similar mathematical models mentioned in the review, and the updated paper now references these works, thank you for bringing them to our notice.

Reviewer comment (Experiments):
Did any of the human subjects experiments evaluate human inference with a subset of the trajectories? It seems possible that humans might make the same inferences about preferences and concepts without sufficient examples to rule out alternative hypotheses. For example, differences between experiments might cue them to provide different answers than they did in previous experiments.

Author response:
Answers to questions:
1. All human subjects were shown the full set of trajectories, which were designed to have sufficient examples to rule out alternative hypotheses. The human subjects and the IRL models were of course shown the same set of trajectories.