Original: Feature-directed Active Learning for Learning User Preferences
Revised: Feature-directed Active Learning for Learning User Preferences

Original: Abstract
Revised: Abstract

Original: Learning preferences of users over plan traces can be a chal- lenging task given a large number of features and limited queries that we can ask a single user.
Revised: Learning preferences of users over plan traces can be a chal- lenging task given a large number of features and limited queries that we can ask a single user.

Original: Additionally, the pref- erence function itself can be quite convoluted and non-linear.
Revised: Additionally, the pref- erence function itself can be quite convoluted and non-linear.

Original: Our approach uses feature-directed active learning to gather the necessary information about plan trace preferences.
Revised: Our approach uses feature-directed active learning to gather the necessary information about plan trace preferences.

Original: This data is used to train a simple feedforward neural network to learn preferences over the sequential data.
Revised: This data is used to train a simple feedforward neural network to learn preferences over the sequential data.

Original: We evaluate the impact of active learning on the number of traces that are needed to train a model that is accurate and interpretable.
Revised: We evaluate the impact of active learning on the number of traces that are needed to train a model that is accurate and interpretable.

Original: This evaluation is done by comparing the aforementioned feedfor- ward network to a more complex neural network model that uses LSTMs and is trained with a larger dataset without active learning.
Revised: This evaluation is done by comparing the aforementioned feedfor- ward network to a more complex neural network model that uses LSTMs and is trained with a larger dataset without active learning.

Original: Introduction
Revised: Introduction

Original: When we have a human-in-the-loop during planning, learning that person’s preferences over plan traces becomes an important problem.
Revised: When we have a human-in-the-loop during planning, learning that person’s preferences over plan traces becomes an important problem.

Original: These preferences can be used to choose a plan from amongst a set of plans that are comparable by the planner’s cost metrics.
Revised: These preferences can be used to choose a plan from amongst a set of plans that are comparable by the planner’s cost metrics.

Original: Such a plan would naturally be more desired by the human.
Revised: Such a plan would naturally be more desired by the human.

Original: The user may not like to constantly dictate their preferences, and may not always be in the loop during execution.
Revised: The user may not like to constantly dictate their preferences, and may not always be in the loop during execution.

Original: Thus, it is important for the user’s preference function to be learned well, and for the user to be able to verify them.
Revised: Thus, it is important for the user’s preference function to be learned well, and for the user to be able to verify them.

Original: For veriﬁcation, there ought to be a way to interpret how the model’s decisions were made, and verify how faithful the learned model is to the user’s preferences.
Revised: For veriﬁcation, there ought to be a way to interpret how the model’s decisions were made, and verify how faithful the learned model is to the user’s preferences.

Original: A user’s preferences function may be quite complex with dependencies over different subsets of features.
Revised: A user’s preferences function may be quite complex with dependencies over different subsets of features.

Original: The utility of some features maybe non-linear as well.
Revised: The utility of some features maybe non-linear as well.

Original: Such a preference function may require a fair amount of information to approximate.
Revised: Such a preference function may require a fair amount of information to approximate.

Original: We cannot expect a single user to give feedback over a large set of traces to get the relevant information.
Revised: We cannot expect a single user to give feedback over a large set of traces to get the relevant information.

Original: So Active learning, with a sufﬁciently expressive user interface for feedback, is essential to minimize queries and redundant information.
Revised: So Active learning, with a sufﬁciently expressive user interface for feedback, is essential to minimize queries and redundant information.

Original: In this work, our objective was to model the user’s preferences over plan traces.
Revised: In this work, our objective was to model the user’s preferences over plan traces.

Original: There do exist techniques that efﬁciently represent and reason about preference relationships.
Revised: There do exist techniques that efﬁciently represent and reason about preference relationships.

Original: CP-nets (Boutilier et al.
Revised: CP-nets (Boutilier et al.

Original: and Generalized additive independence(Braziunas and Boutilier 2006) models are typically used to represent preferences over sets of variables without consideration to the order in which they appear.
Revised: and Generalized additive independence(Braziunas and Boutilier 2006) models are typically used to represent preferences over sets of variables without consideration to the order in which they appear.

Original: While these models can be adapted to handle sequential data, they are not intended for it.
Revised: While these models can be adapted to handle sequential data, they are not intended for it.

Original: LTL rules, however, can capture trajectory preferences very well and are used in
Revised: LTL rules, however, can capture trajectory preferences very well and are used in

Original: PDDL 3.0 (Gerevini and Long 2005), and LPP (Bienvenu,
Revised: PDDL 3.0 (Gerevini and Long 2005), and LPP (Bienvenu,

Original: Fritz, and McIlraith 2006).
Revised: Fritz, and McIlraith 2006).

Original: However, it can be very hard for a user to express their preferences in this form.
Revised: However, it can be very hard for a user to express their preferences in this form.

Original: We discuss existing approaches in more detail and the differences with respect to our work under the related work section.
Revised: We discuss existing approaches in more detail and the differences with respect to our work under the related work section.

Original: In our approach to learning preferences, we want to efﬁciently identify the relevant features and the degree to which they affect the preference score of a plan.
Revised: In our approach to learning preferences, we want to efﬁciently identify the relevant features and the degree to which they affect the preference score of a plan.

Original: We thus employ a feature-directed active learning approach that speciﬁcally picks plan traces that are most informative about the feature’s effects on preference.
Revised: We thus employ a feature-directed active learning approach that speciﬁcally picks plan traces that are most informative about the feature’s effects on preference.

Original: After active learning, we encode a plan trace in terms of the relevant features it contains.
Revised: After active learning, we encode a plan trace in terms of the relevant features it contains.

Original: We gather a set of training data from active learning, along with the user’s preference score to help train a simple Neural Network (NN) that we call the FeatureNN model.
Revised: We gather a set of training data from active learning, along with the user’s preference score to help train a simple Neural Network (NN) that we call the FeatureNN model.

Original: We use a Neural Network as they can approximate complex functions to a good degree.
Revised: We use a Neural Network as they can approximate complex functions to a good degree.

Original: Our approach is in one way, related to Generalized Additive Independence in that we try to learn a utility function over pertinent features, but we do not explicitly deﬁne or restrict the form of any utility functions.
Revised: Our approach is in one way, related to Generalized Additive Independence in that we try to learn a utility function over pertinent features, but we do not explicitly deﬁne or restrict the form of any utility functions.

Original: Rather a simple one hidden-layer feed-forward neural network learns the functions, dependencies, and relative weights over the relevant features.
Revised: Rather a simple one hidden-layer feed-forward neural network learns the functions, dependencies, and relative weights over the relevant features.

Original: The FeatureNN then predicts a preference score for each plan reﬂecting the user’s preferences.
Revised: The FeatureNN then predicts a preference score for each plan reﬂecting the user’s preferences.

Original: We also compare the performance of the FeatureNN to another SequenceNN model that processes sequential data using an
Revised: We also compare the performance of the FeatureNN to another SequenceNN model that processes sequential data using an

Original: LSTM(Schmidhuber and Hochreiter 1997) module.
Revised: LSTM(Schmidhuber and Hochreiter 1997) module.

Original: The SequenceNN is not trained with data from active learning, but with a larger dataset of traces with ratings.
Revised: The SequenceNN is not trained with data from active learning, but with a larger dataset of traces with ratings.

Original: This is to evaluate how efﬁcient our active learning approach is with respect to the number of traces.
Revised: This is to evaluate how efﬁcient our active learning approach is with respect to the number of traces.

Original: Speciﬁcally, we compare the number of traces required by SequenceNN and FeatureNN for the same accuracy and interpretability.
Revised: Speciﬁcally, we compare the number of traces required by SequenceNN and FeatureNN for the same accuracy and interpretability.

Original: Neural networks, unlike linear functions, are not as easy to interpret.
Revised: Neural networks, unlike linear functions, are not as easy to interpret.

Original: Even simple NN with a single hidden layer can be a challenge.
Revised: Even simple NN with a single hidden layer can be a challenge.

Original: We help the user interpret the decisions of the neural network by showing how the preference score is affected by removing different features of a plan trace.
Revised: We help the user interpret the decisions of the neural network by showing how the preference score is affected by removing different features of a plan trace.

Original: This is similar to using Saliency Maps (Simonyan, Vedaldi, and
Revised: This is similar to using Saliency Maps (Simonyan, Vedaldi, and

Original: Zisserman 2013) in images to explain what parts of the image contributed to the classiﬁcation.
Revised: Zisserman 2013) in images to explain what parts of the image contributed to the classiﬁcation.

Original: In this way, we can explain to the user what plan features contributed to the preference value and by how much.
Revised: In this way, we can explain to the user what plan features contributed to the preference value and by how much.

Original: The difference in preference score should correspond to the user’s expectations as per their preference model.
Revised: The difference in preference score should correspond to the user’s expectations as per their preference model.

Original: The more similar the effect of changes are to the user’s preferences, the more interpretable the NN model is to the user as it approximates well their own preference function.
Revised: The more similar the effect of changes are to the user’s preferences, the more interpretable the NN model is to the user as it approximates well their own preference function.

Original: Such a method of explaining a decision(score) is also related to explaining using counterfactuals (Miller 2018).
Revised: Such a method of explaining a decision(score) is also related to explaining using counterfactuals (Miller 2018).

Original: Here the counterfactual is the plan trace without a speciﬁc feature.
Revised: Here the counterfactual is the plan trace without a speciﬁc feature.

Original: Additionally, when the speciﬁc features used to compute preferences comes from the user’s feedback (during active learning), this interpretability is obviously improved.
Revised: Additionally, when the speciﬁc features used to compute preferences comes from the user’s feedback (during active learning), this interpretability is obviously improved.

Original: We present our work by ﬁrst deﬁning the problem before delving into the methodology of our approach.
Revised: We present our work by ﬁrst deﬁning the problem before delving into the methodology of our approach.

Original: In the
Revised: In the

Original: Methodology section, we discuss the domain used, the user preference model, and the feature-directed active learning process.
Revised: Methodology section, we discuss the domain used, the user preference model, and the feature-directed active learning process.

Original: We also discuss the two neural network models used to learn the preference model, viz.
Revised: We also discuss the two neural network models used to learn the preference model, viz.

Original: the FeatureNN and the SequenceNN models.
Revised: the FeatureNN and the SequenceNN models.

Original: Then we present our experimental results in which we compare the two models with respect to their accuracy in predicting the preference score, as well as interpretability.
Revised: Then we present our experimental results in which we compare the two models with respect to their accuracy in predicting the preference score, as well as interpretability.

Original: Lastly, we discuss the results and possible extensions to the work.
Revised: Lastly, we discuss the results and possible extensions to the work.

Original: Problem Deﬁnition
Revised: Problem Deﬁnition

Original: Given a Domain D , with a set of features F , and a planner
Revised: Given a Domain D , with a set of features F , and a planner

Original: P , the problem is to learn the preference function F p () that captures the user’s preference model U p () and scores traces accordingly.
Revised: P , the problem is to learn the preference function F p () that captures the user’s preference model U p () and scores traces accordingly.

Original: The types of preferences we learn in this work are a function of the feature set F of the domain and not hidden variables or action costs.
Revised: The types of preferences we learn in this work are a function of the feature set F of the domain and not hidden variables or action costs.

Original: The user U is available to rate a plan trace on its preference, and annotate what features contributed positively or negatively to the rating.
Revised: The user U is available to rate a plan trace on its preference, and annotate what features contributed positively or negatively to the rating.

Original: Features can be categorical or cardinal(count), and involve sequences.
Revised: Features can be categorical or cardinal(count), and involve sequences.

Original: Plans are rated between [0 , 1] with higher values indicating a greater preference.
Revised: Plans are rated between [0 , 1] with higher values indicating a greater preference.

Original: If there are no features in the plan that either contributed positively or negatively to the preference, then the preference score is 0 .
Revised: If there are no features in the plan that either contributed positively or negatively to the preference, then the preference score is 0 .

Original: An equivalent problem formulation assumes that instead of a domain D and planner P, we are given as input a large enough set of plan traces B (backlog of traces) over a relevant set of initial and goal states.
Revised: An equivalent problem formulation assumes that instead of a domain D and planner P, we are given as input a large enough set of plan traces B (backlog of traces) over a relevant set of initial and goal states.

Original: We assume that this set of plans covers the space of possible preferences that the user might have.
Revised: We assume that this set of plans covers the space of possible preferences that the user might have.

Original: Methodology
Revised: Methodology

Original: For our experiments, we chose to use gridworld with features that any human can relate to.
Revised: For our experiments, we chose to use gridworld with features that any human can relate to.

Original: We chose gridworld as it is easy to quickly generate many diverse plans that cover the range of features.
Revised: We chose gridworld as it is easy to quickly generate many diverse plans that cover the range of features.

Original: Given the domain and a task, we go through r rounds of active learning.
Revised: Given the domain and a task, we go through r rounds of active learning.

Original: Each round comprises of r t traces.
Revised: Each round comprises of r t traces.

Original: Both r and r t are hyperparameters.
Revised: Both r and r t are hyperparameters.

Original: For our experiments, we set the number of rounds at r = 3 .
Revised: For our experiments, we set the number of rounds at r = 3 .

Original: After acquiring the data, we train the NN model and test it on a hidden set of traces.
Revised: After acquiring the data, we train the NN model and test it on a hidden set of traces.

Original: We now go over different parts of our methodology in detail.
Revised: We now go over different parts of our methodology in detail.

Original: Domain
Revised: Domain

Original: The objective in our gridworld domain, which we call
Revised: The objective in our gridworld domain, which we call

Original: Journey − World , is to travel from home to the campsite which is shown in the grid in Figure 1(a).
Revised: Journey − World , is to travel from home to the campsite which is shown in the grid in Figure 1(a).

Original: Each step of a plan corresponds to a cell of the grid.
Revised: Each step of a plan corresponds to a cell of the grid.

Original: While some cells are empty, there a lot of cells that have features.
Revised: While some cells are empty, there a lot of cells that have features.

Original: These features can be eateries(like a coffee shop, restaurant), landmarks(natural history museum) or activities(visiting the library, watching a movie).
Revised: These features can be eateries(like a coffee shop, restaurant), landmarks(natural history museum) or activities(visiting the library, watching a movie).

Original: The user can move through any of these states before reaching the campsite.
Revised: The user can move through any of these states before reaching the campsite.

Original: A majority of the cells also contain landscapes like mountains, lake, sea or industries.
Revised: A majority of the cells also contain landscapes like mountains, lake, sea or industries.

Original: The user is not allowed to move through Landscape cells.
Revised: The user is not allowed to move through Landscape cells.

Original: Moving through cells adjacent to Landscape cells corresponds to seeing the landscape along the journey.
Revised: Moving through cells adjacent to Landscape cells corresponds to seeing the landscape along the journey.

Original: For example, if a step in the plan goes through a cell which is adjacent to a lake, this corresponds to the plan going through a state where the user passed by a lake.
Revised: For example, if a step in the plan goes through a cell which is adjacent to a lake, this corresponds to the plan going through a state where the user passed by a lake.

Original: All non-landscape features (like coffee, donut) are binary features in a plan trace i.e. the user has either visited one or not.
Revised: All non-landscape features (like coffee, donut) are binary features in a plan trace i.e. the user has either visited one or not.

Original: On the other hand, the landscape features are cardinal, i.e. we count the number of such landscape features in the plan trace.
Revised: On the other hand, the landscape features are cardinal, i.e. we count the number of such landscape features in the plan trace.

Original: We assume that the count of cardinal features can make a difference in the preference score.
Revised: We assume that the count of cardinal features can make a difference in the preference score.

Original: In total, there are 13 features in Journey − World .
Revised: In total, there are 13 features in Journey − World .

Original: We had designed Journey − World with simple and commonly understood features to make it easier for subsequent human-studies.
Revised: We had designed Journey − World with simple and commonly understood features to make it easier for subsequent human-studies.

Original: We assume people will have preferences over these features.
Revised: We assume people will have preferences over these features.

Original: User Preference Model
Revised: User Preference Model

Original: For our current experiments, we chose to use a completely deﬁned user preference model to represent the user.
Revised: For our current experiments, we chose to use a completely deﬁned user preference model to represent the user.

Original: This made it easier for us to test and debug our methodology.
Revised: This made it easier for us to test and debug our methodology.

Original: In future extensions of this work, we will include evaluations with human trials.
Revised: In future extensions of this work, we will include evaluations with human trials.

Original: The user’s preference model is deﬁned as follows
Revised: The user’s preference model is deﬁned as follows

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: if ( C and D ) where n ( x ) = min ( x, 2) , C ∈ { 0 , 1 } , D ∈ { 0 , 1 } and L, I ∈ N
Revised: if ( C and D ) where n ( x ) = min ( x, 2) , C ∈ { 0 , 1 } , D ∈ { 0 , 1 } and L, I ∈ N

Original: C is a binary variable that is 1 when the plan trace has a coffee.
Revised: C is a binary variable that is 1 when the plan trace has a coffee.

Original: D is also a binary variable and represents a Donut.
Revised: D is also a binary variable and represents a Donut.

Original: CD represents a binary variable set to true when the plan trace a coffee and a donut.
Revised: CD represents a binary variable set to true when the plan trace a coffee and a donut.

Original: When CD is true, C and D are false and this dependency affects the preference score computation as shown in the preceding equation.
Revised: When CD is true, C and D are false and this dependency affects the preference score computation as shown in the preceding equation.

Original: L and I represent
Revised: L and I represent

Original: [Figure]
Revised: [Figure]

Original: the number of lake and industry regions respectively.
Revised: the number of lake and industry regions respectively.

Original: These are cardinal features, and the preference of the plan increases based on their count, up to 2, and then stops increasing.
Revised: These are cardinal features, and the preference of the plan increases based on their count, up to 2, and then stops increasing.

Original: The function itself, while simple to understand, is non-linear and hidden underneath a large hypothesis space of functions that could be learned in the domain, over a larger set of features (13 in total for the Journey − World domain).
Revised: The function itself, while simple to understand, is non-linear and hidden underneath a large hypothesis space of functions that could be learned in the domain, over a larger set of features (13 in total for the Journey − World domain).

Original: In our experiments, we programmed a separate module to rate and annotate plan traces based on an input preference function like the one described previously.
Revised: In our experiments, we programmed a separate module to rate and annotate plan traces based on an input preference function like the one described previously.

Original: This synthetic human is what rates and annotates in the active learning process that we will describe shortly.
Revised: This synthetic human is what rates and annotates in the active learning process that we will describe shortly.

Original: Using a synthetic human helped speed up the testing and debugging process, and gives us a baseline noiseless scenario to test against.
Revised: Using a synthetic human helped speed up the testing and debugging process, and gives us a baseline noiseless scenario to test against.

Original: User Interface
Revised: User Interface

Original: The current user interface(Figure 1(a)) for Journey −
Revised: The current user interface(Figure 1(a)) for Journey −

Original: World displays the entire grid.
Revised: World displays the entire grid.

Original: Icons are used to show the features present in cells.
Revised: Icons are used to show the features present in cells.

Original: The plans for each round of active learning are then shown one at a time.
Revised: The plans for each round of active learning are then shown one at a time.

Original: The plan steps are visualized as a line going from the home to the campsite.
Revised: The plan steps are visualized as a line going from the home to the campsite.

Original: The user has to input a rating to indicate their preference for the plan based on the features that are visited.
Revised: The user has to input a rating to indicate their preference for the plan based on the features that are visited.

Original: They can also annotate features of the plan that they like(green) or dislike(red) as shown in Figure 1(b).
Revised: They can also annotate features of the plan that they like(green) or dislike(red) as shown in Figure 1(b).

Original: The user can click on the
Revised: The user can click on the

Original: Next Plan button to then move on to the next plan.
Revised: Next Plan button to then move on to the next plan.

Original: The interface automatically switches to the next round of Active
Revised: The interface automatically switches to the next round of Active

Original: Learning when the current round’s plans have all been rated.
Revised: Learning when the current round’s plans have all been rated.

Original: Feature-Directed Active Learning
Revised: Feature-Directed Active Learning

Original: In our active learning process, we go through multiple ( r ) rounds of feedback.
Revised: In our active learning process, we go through multiple ( r ) rounds of feedback.

Original: Each successive round utilizes the knowledge from previous rounds to select the most informative queries.
Revised: Each successive round utilizes the knowledge from previous rounds to select the most informative queries.

Original: In the ﬁrst round, the user is shown the most diverse set of plan trajectories that were generated for the domain.
Revised: In the ﬁrst round, the user is shown the most diverse set of plan trajectories that were generated for the domain.

Original: We choose diverse plans because in the ﬁrst round we do not have any knowledge of what features might affect the user’s preference and hence, we want to cover the feature space as much as possible in r t = 30 traces.
Revised: We choose diverse plans because in the ﬁrst round we do not have any knowledge of what features might affect the user’s preference and hence, we want to cover the feature space as much as possible in r t = 30 traces.

Original: In order to get the required diverse plan set, we ﬁrst generate a large number of plan traces( 10000 plans) over a user speciﬁed set of initial and goal states that we refer to as the backlog of plans, B .
Revised: In order to get the required diverse plan set, we ﬁrst generate a large number of plan traces( 10000 plans) over a user speciﬁed set of initial and goal states that we refer to as the backlog of plans, B .

Original: In our current experiments we only have one initial and one goal state.
Revised: In our current experiments we only have one initial and one goal state.

Original: We are easily able to generate such a large backlog of plans because it is a type of gridworld domain.
Revised: We are easily able to generate such a large backlog of plans because it is a type of gridworld domain.

Original: We did not want the computational cost of diverse plan generation to hamper the work.
Revised: We did not want the computational cost of diverse plan generation to hamper the work.

Original: This is a computational cost that needs to be considered when working with other domains.
Revised: This is a computational cost that needs to be considered when working with other domains.

Original: The plans generated cover the entire feature space of the Journey − World domain.
Revised: The plans generated cover the entire feature space of the Journey − World domain.

Original: We then select themost diverse plans within the set of backlog plans for the ﬁrst round.
Revised: We then select themost diverse plans within the set of backlog plans for the ﬁrst round.

Original: We will now discuss how this is determined.
Revised: We will now discuss how this is determined.

Original: The diversity score between any two plans p a and p b is denoted by d ( p a , p b ) .
Revised: The diversity score between any two plans p a and p b is denoted by d ( p a , p b ) .

Original: The diversity is based on the sum of feature count differences for features f ∈ F that are present p a and p b .
Revised: The diversity is based on the sum of feature count differences for features f ∈ F that are present p a and p b .

Original: For a particular feature f , we compute the feature count difference f ∆ .
Revised: For a particular feature f , we compute the feature count difference f ∆ .

Original: Rather than use the difference in count per feature, we use a geometric series sum as computed in
Revised: Rather than use the difference in count per feature, we use a geometric series sum as computed in

Original: Equation 1.
Revised: Equation 1.

Original: The ﬁrst count in the difference contributes 1, the second count contributes a 0 .
Revised: The ﬁrst count in the difference contributes 1, the second count contributes a 0 .

Original: 5 , the third contributes 0.
Revised: 5 , the third contributes 0.

Original: and so forth.
Revised: and so forth.

Original: So the count difference for a single feature contributes to at most 2 to the diversity computation.
Revised: So the count difference for a single feature contributes to at most 2 to the diversity computation.

Original: This avoids any single feature from dominating the diversity computation.
Revised: This avoids any single feature from dominating the diversity computation.

Original: The diversity between two traces is computed as the average f E ∆ (Equation 2) over all the features in the domain.
Revised: The diversity between two traces is computed as the average f E ∆ (Equation 2) over all the features in the domain.

Original: Finally, we calculate the backlog-diversity d B for a plan p using equation 3.
Revised: Finally, we calculate the backlog-diversity d B for a plan p using equation 3.

Original: The backlog-diversity is the average pairwise diversity over every other plan in the backlog.
Revised: The backlog-diversity is the average pairwise diversity over every other plan in the backlog.

Original: Using this diversity score, we select the top r t plans ( r t = 30 in our experiments) for the user feedback.
Revised: Using this diversity score, we select the top r t plans ( r t = 30 in our experiments) for the user feedback.

Original: [Equation]
Revised: [Equation]

Original: After the ﬁrst round of diverse plans, we then make use of the ratings and annotations provided by the user in the ﬁrst round to generate the most informative plan traces for the subsequent rounds.
Revised: After the ﬁrst round of diverse plans, we then make use of the ratings and annotations provided by the user in the ﬁrst round to generate the most informative plan traces for the subsequent rounds.

Original: Given our acquired knowledge of relevant features from the previous round, our objective now is to ﬁgure out the effects and dependencies between these features.
Revised: Given our acquired knowledge of relevant features from the previous round, our objective now is to ﬁgure out the effects and dependencies between these features.

Original: We also want to select traces for the next round that are more likely to be rated either signiﬁcantly higher or lower.
Revised: We also want to select traces for the next round that are more likely to be rated either signiﬁcantly higher or lower.

Original: This region of data is typically harder to get as we expect most data to be closer to the average.
Revised: This region of data is typically harder to get as we expect most data to be closer to the average.

Original: In order to estimate which plan traces would be either most preferred or least preferred, we use a fast weak predictor that predicts the rating of any arbitrary plan p given prior knowledge.
Revised: In order to estimate which plan traces would be either most preferred or least preferred, we use a fast weak predictor that predicts the rating of any arbitrary plan p given prior knowledge.

Original: We need the predictor to be fast as we have to give traces or queries for the next round in a short amount of time.
Revised: We need the predictor to be fast as we have to give traces or queries for the next round in a short amount of time.

Original: The weak predictor estimates a value for each feature based on the prior annotated data.
Revised: The weak predictor estimates a value for each feature based on the prior annotated data.

Original: It can then estimate the score of an unrated plan as just the sum of the features present in it.
Revised: It can then estimate the score of an unrated plan as just the sum of the features present in it.

Original: The value of each feature is scored using a quick and simple method.
Revised: The value of each feature is scored using a quick and simple method.

Original: First, for each scored plan trace p with rating r p , the feature f is given a score f pscore for that plan by equation 4.
Revised: First, for each scored plan trace p with rating r p , the feature f is given a score f pscore for that plan by equation 4.

Original: Then the feature’s score, f score , is computed as the average f p score over all plans that the feature appears in.
Revised: Then the feature’s score, f score , is computed as the average f p score over all plans that the feature appears in.

Original: Then to predict the score for an unrated plan, the weak predictor assigns a score predict ( p ) which is the sum of f score for all features present in the plan.
Revised: Then to predict the score for an unrated plan, the weak predictor assigns a score predict ( p ) which is the sum of f score for all features present in the plan.

Original: [Equation]
Revised: [Equation]

Original: In addition wanting plans that are likely to be rated much higher or much lower, we also want the next round traces to have two more properties.
Revised: In addition wanting plans that are likely to be rated much higher or much lower, we also want the next round traces to have two more properties.

Original: We still want to include some diversity in the plan traces with respect to the overall backlog of traces to uncover features that we might have missed in the ﬁrst round.
Revised: We still want to include some diversity in the plan traces with respect to the overall backlog of traces to uncover features that we might have missed in the ﬁrst round.

Original: Additionally, we want to maintain some similarity in traces between the rounds.
Revised: Additionally, we want to maintain some similarity in traces between the rounds.

Original: We think that the similarity between plans reduces the cognitive load on the user as they need not parse wholly different traces.
Revised: We think that the similarity between plans reduces the cognitive load on the user as they need not parse wholly different traces.

Original: Given a plan p , we denote its similarity to the already scored traces as S ( p ) .
Revised: Given a plan p , we denote its similarity to the already scored traces as S ( p ) .

Original: Finally, we assign a combined weighted score of p c to all the plans in the backlog given by equation 5.
Revised: Finally, we assign a combined weighted score of p c to all the plans in the backlog given by equation 5.

Original: The top r t =plans are then picked for the next round, and in this way the active learning proceeds for r rounds.
Revised: The top r t =plans are then picked for the next round, and in this way the active learning proceeds for r rounds.

Original: For our experiments r is 3 rounds.
Revised: For our experiments r is 3 rounds.

Original: [Equation]
Revised: [Equation]

Original: Preference Learning using Neural Networks
Revised: Preference Learning using Neural Networks

Original: For learning the preference function we used two models,
Revised: For learning the preference function we used two models,

Original: SequenceNN model and FeatureNN model.
Revised: SequenceNN model and FeatureNN model.

Original: The SequenceNN model uses an LSTM module in it.
Revised: The SequenceNN model uses an LSTM module in it.

Original: We considered an LSTM based model as they are well suited to learning patterns overs sequential data.
Revised: We considered an LSTM based model as they are well suited to learning patterns overs sequential data.

Original: The input plan trace was encoded such that each step was an encoding over the features of the cell visited at that step.
Revised: The input plan trace was encoded such that each step was an encoding over the features of the cell visited at that step.

Original: There are 13 features in total, and so each step is a 13-dimensional vector.
Revised: There are 13 features in total, and so each step is a 13-dimensional vector.

Original: We do not provide or restrict the input to only the features that the user annotated during active learning for the SequenceNN model.
Revised: We do not provide or restrict the input to only the features that the user annotated during active learning for the SequenceNN model.

Original: We wanted to test how easily the model could still ﬁgure out the relevant features and learn the preference function well.
Revised: We wanted to test how easily the model could still ﬁgure out the relevant features and learn the preference function well.

Original: The training data for the SequenceNN model was a set of rated plan traces.
Revised: The training data for the SequenceNN model was a set of rated plan traces.

Original: We varied the number of traces given from as small as 30 to 12,000 in increasingly larger step sizes.
Revised: We varied the number of traces given from as small as 30 to 12,000 in increasingly larger step sizes.

Original: A plan trace would be a N × 13 array where N is the plan length.
Revised: A plan trace would be a N × 13 array where N is the plan length.

Original: We trained the model for 10 epochs with a batch size of 8, a learning rate of 0.01 and using stochastic gradient descent.
Revised: We trained the model for 10 epochs with a batch size of 8, a learning rate of 0.01 and using stochastic gradient descent.

Original: The SequenceNN module in our model has 16 memory cells.
Revised: The SequenceNN module in our model has 16 memory cells.

Original: After processing the plan trace through the LSTM module, we concatenate the output vector and memory nodes of the LSTM module and pass it through a single fully connected hidden layer, followed by the output layer which outputs the preference score between [0,1].
Revised: After processing the plan trace through the LSTM module, we concatenate the output vector and memory nodes of the LSTM module and pass it through a single fully connected hidden layer, followed by the output layer which outputs the preference score between [0,1].

Original: The model summary is in Figure 2.
Revised: The model summary is in Figure 2.

Original: The idea is that the LSTM module output and memory, at the end of processing the sequence, will have the necessary information related to the sequence for predicting the score.
Revised: The idea is that the LSTM module output and memory, at the end of processing the sequence, will have the necessary information related to the sequence for predicting the score.

Original: To test if any variant of the SequenceNN model could learn the relevant information better, we also tried varying the size of the model (number of parameters) to make it more powerful.
Revised: To test if any variant of the SequenceNN model could learn the relevant information better, we also tried varying the size of the model (number of parameters) to make it more powerful.

Original: We varied the model from 16 to 64 weights in the memory layer.
Revised: We varied the model from 16 to 64 weights in the memory layer.

Original: Results are in the Evaluation and Analysis section.
Revised: Results are in the Evaluation and Analysis section.

Original: Forthe FeatureNN model, the input was an encoding of the plan trace that only comprised of the features the user annotated as relevant during active learning.
Revised: Forthe FeatureNN model, the input was an encoding of the plan trace that only comprised of the features the user annotated as relevant during active learning.

Original: The entire plan trace was summarized into one encoded vector.
Revised: The entire plan trace was summarized into one encoded vector.

Original: For example, in the user preference model in our experiments, onlyfeatures matter to the user.
Revised: For example, in the user preference model in our experiments, onlyfeatures matter to the user.

Original: We determine what these features are through active learning, and then deﬁned our FeatureNN model accordingly to take a 5-dimensional vector as input.
Revised: We determine what these features are through active learning, and then deﬁned our FeatureNN model accordingly to take a 5-dimensional vector as input.

Original: For example, if a plan trace had a step with coffee and steps that passed by 3 lakes, then the values at the corresponding indices are set to 1, and 3 respectively.
Revised: For example, if a plan trace had a step with coffee and steps that passed by 3 lakes, then the values at the corresponding indices are set to 1, and 3 respectively.

Original: Note that since coffee is a binary feature, even if two coffee steps were in the plan, it’s value in the encoding is only either 1 or 0.
Revised: Note that since coffee is a binary feature, even if two coffee steps were in the plan, it’s value in the encoding is only either 1 or 0.

Original: As for the model description of FeatureNN , it was a simple fully connected neural net with one hidden layer of 4 dimensions and one output layer.
Revised: As for the model description of FeatureNN , it was a simple fully connected neural net with one hidden layer of 4 dimensions and one output layer.

Original: The model summary is in Figure 3.
Revised: The model summary is in Figure 3.

Original: Note that 4 dimensions or nodes for the hidden layer is not a magic number, and would need to be larger if there were more features.
Revised: Note that 4 dimensions or nodes for the hidden layer is not a magic number, and would need to be larger if there were more features.

Original: We reduced the number of dimensions for the hidden layer until the results were measurably worse.
Revised: We reduced the number of dimensions for the hidden layer until the results were measurably worse.

Original: To train the FeatureNN model we vary the number of traces per round r t from 5 to 50 traces for r = 3 rounds.
Revised: To train the FeatureNN model we vary the number of traces per round r t from 5 to 50 traces for r = 3 rounds.

Original: Since the dataset size is very small (smallest is 15 traces), we create 200 duplicates of the data points uniformly and train forepochs.
Revised: Since the dataset size is very small (smallest is 15 traces), we create 200 duplicates of the data points uniformly and train forepochs.

Original: We also shufﬂe the data and train with a batch size of 8, a learning rate of 0.01, and using stochastic gradient descent.
Revised: We also shufﬂe the data and train with a batch size of 8, a learning rate of 0.01, and using stochastic gradient descent.

Original: Both models were tested on an unused test set oftraces for accuracy and interpretability.
Revised: Both models were tested on an unused test set oftraces for accuracy and interpretability.

Original: Interpretability of the Preference Model
Revised: Interpretability of the Preference Model

Original: A user can interpret a Neural Network’s behavior through analyzing what features are salient to its decision, and by how much.
Revised: A user can interpret a Neural Network’s behavior through analyzing what features are salient to its decision, and by how much.

Original: This can be analyzed by adding or removing features and seeing the resultant effect on the predicted score.
Revised: This can be analyzed by adding or removing features and seeing the resultant effect on the predicted score.

Original: When done over a set of different traces, the user can intuit what mattered and how much.
Revised: When done over a set of different traces, the user can intuit what mattered and how much.

Original: With this in mind, we compute a measure of interpretability we call the Attribution Error AE .
Revised: With this in mind, we compute a measure of interpretability we call the Attribution Error AE .

Original: The AE for a feature f of a Plan p is computed as follows:
Revised: The AE for a feature f of a Plan p is computed as follows:

Original: [Equation]
Revised: [Equation]

Original: where U p () is the preference function of the user (true model of preferences) and F p () is the learned preference function.
Revised: where U p () is the preference function of the user (true model of preferences) and F p () is the learned preference function.

Original: AE is simply the difference in the effect of the feature on the preference scores.
Revised: AE is simply the difference in the effect of the feature on the preference scores.

Original: The overall AE for each test plan p ,
Revised: The overall AE for each test plan p ,

Original: AE ( p ) , is the average of AE ( p, f ) for all f present in p .
Revised: AE ( p ) , is the average of AE ( p, f ) for all f present in p .

Original: We compute the AE score for the test set as the average over only the top 10% of AE(p) errors.
Revised: We compute the AE score for the test set as the average over only the top 10% of AE(p) errors.

Original: We do this because neural networks can sometimes have enough capacity to memorize many cases and increase accuracy.
Revised: We do this because neural networks can sometimes have enough capacity to memorize many cases and increase accuracy.

Original: So it can predict the correct preference score of both of the original trace and modiﬁed trace (with dropped feature) by the memory of very similar traces.
Revised: So it can predict the correct preference score of both of the original trace and modiﬁed trace (with dropped feature) by the memory of very similar traces.

Original: It would then seem like it’s preference function predicts the same way as the ground-truth preference function, but it maybe using unrelated features.
Revised: It would then seem like it’s preference function predicts the same way as the ground-truth preference function, but it maybe using unrelated features.

Original: Therefore, it is in the failure cases that we get a true measure of its generalization and how faithful it is to the true model of preferences.
Revised: Therefore, it is in the failure cases that we get a true measure of its generalization and how faithful it is to the true model of preferences.

Original: That is why we use the average over the top 10% of AE(p) errors.
Revised: That is why we use the average over the top 10% of AE(p) errors.

Original: These failure cases could correspond to the cases when a rare or unseen pattern of features are input, and thus not memorized.
Revised: These failure cases could correspond to the cases when a rare or unseen pattern of features are input, and thus not memorized.

Original: Evaluation and Analysis Evaluation of LSTM model
Revised: Evaluation and Analysis Evaluation of LSTM model

Original: When varying the number of training input traces given to the SequenceNN model, we observed that the accuracy improved (error decreased) as expected(Figure 4(a)).
Revised: When varying the number of training input traces given to the SequenceNN model, we observed that the accuracy improved (error decreased) as expected(Figure 4(a)).

Original: Surprisingly, even with 30 traces, it was able to predict with an error of 2 . 5% over the test set of unseen 1000 traces.
Revised: Surprisingly, even with 30 traces, it was able to predict with an error of 2 . 5% over the test set of unseen 1000 traces.

Original: We attribute this to the fact that there are enough simple correlations with other features that can predict the score well for the preference function that we tested with.
Revised: We attribute this to the fact that there are enough simple correlations with other features that can predict the score well for the preference function that we tested with.

Original: This is evidenced by the fact that the interpretability measure (Attribution Error) is very low for 30 traces(Figure 5 (a)); The attribution error was greater than 0.3 and the value range of AE is [0,1].
Revised: This is evidenced by the fact that the interpretability measure (Attribution Error) is very low for 30 traces(Figure 5 (a)); The attribution error was greater than 0.3 and the value range of AE is [0,1].

Original: Additionally, we give the most diverse N traces for each training set size to the SequenceNN model.
Revised: Additionally, we give the most diverse N traces for each training set size to the SequenceNN model.

Original: Diverse traces are more likely to contain relevant information.
Revised: Diverse traces are more likely to contain relevant information.

Original: The interpretability of the LSTM model was not impressive.
Revised: The interpretability of the LSTM model was not impressive.

Original: The attribution error did decrease over the range of training set sizes, but only as low as 0 . 09 as shown in Figure 5(a).
Revised: The attribution error did decrease over the range of training set sizes, but only as low as 0 . 09 as shown in Figure 5(a).

Original: Given that the preference scores are between [0 , 1] , this would correspond to a 9 percent error after 7500 rated traces.
Revised: Given that the preference scores are between [0 , 1] , this would correspond to a 9 percent error after 7500 rated traces.

Original: Needless to say, it is unreasonable to expect a single human to rate 7500 traces.
Revised: Needless to say, it is unreasonable to expect a single human to rate 7500 traces.

Original: We also tried varying the size of the SequenceNN model from 16 to 64 dimensions.
Revised: We also tried varying the size of the SequenceNN model from 16 to 64 dimensions.

Original: This improved accuracy by a minuscule amount (order of 1 e − 4 ), and interpretability did not improve.
Revised: This improved accuracy by a minuscule amount (order of 1 e − 4 ), and interpretability did not improve.

Original: Evaluation of Feature-NN model
Revised: Evaluation of Feature-NN model

Original: The performance of the FeatureNN model was signiﬁcantly better both in accuracy(lower error) and interpretability than the SequenceNN model as seen in Figure 5.
Revised: The performance of the FeatureNN model was signiﬁcantly better both in accuracy(lower error) and interpretability than the SequenceNN model as seen in Figure 5.

Original: This should come as no surprise since we restrict the input space based on user feedback (knowledge) on relevant features.
Revised: This should come as no surprise since we restrict the input space based on user feedback (knowledge) on relevant features.

Original: This also restricts the hypothesis space of functions that the simple feed-forward network could search over.
Revised: This also restricts the hypothesis space of functions that the simple feed-forward network could search over.

Original: We think this will make it more likely that the NN will ﬁnd a good and faithful
Revised: We think this will make it more likely that the NN will ﬁnd a good and faithful

Original: [Figure] [Figure]
Revised: [Figure] [Figure]

Original: approximation function to the true preference function.
Revised: approximation function to the true preference function.

Original: This is as opposed to discovering predictive but incorrect correlations.
Revised: This is as opposed to discovering predictive but incorrect correlations.

Original: What is interesting to note is that the interpretability, as measured by the AE error drops to as little as 2.5 % in as little as 60 traces (20 traces per round over 3 rounds) for
Revised: What is interesting to note is that the interpretability, as measured by the AE error drops to as little as 2.5 % in as little as 60 traces (20 traces per round over 3 rounds) for

Original: FeatureNN model as shown in Figure 5b.
Revised: FeatureNN model as shown in Figure 5b.

Original: It drops below% with 150 traces.
Revised: It drops below% with 150 traces.

Original: The FeatureNN model with 90 traces is as accurate as the
Revised: The FeatureNN model with 90 traces is as accurate as the

Original: SequenceNN with 7500 traces in our problem, with 8% less
Revised: SequenceNN with 7500 traces in our problem, with 8% less

Original: Attribution Error (more interpretable).
Revised: Attribution Error (more interpretable).

Original: While we expected
Revised: While we expected

Original: FeatureNN to be better, we did not expect such a large difference in efﬁciency.
Revised: FeatureNN to be better, we did not expect such a large difference in efﬁciency.

Original: Analysis and Discussion
Revised: Analysis and Discussion

Original: Even with as little as 13 features and a relatively uncomplicated preference function, a sufﬁciently powerful SequenceNN model did not ﬁnd the underlying preference function.
Revised: Even with as little as 13 features and a relatively uncomplicated preference function, a sufﬁciently powerful SequenceNN model did not ﬁnd the underlying preference function.

Original: Instead, it found correlations that predicted the preference score to a very high level of accuracy.
Revised: Instead, it found correlations that predicted the preference score to a very high level of accuracy.

Original: This, unfortunately, makes the model suffer in interpretability.
Revised: This, unfortunately, makes the model suffer in interpretability.

Original: As the number of features increases, the hypothesis space of a NN will increase signiﬁcantly.
Revised: As the number of features increases, the hypothesis space of a NN will increase signiﬁcantly.

Original: This makes it much more likely for any NN to ﬁnd spurious correlations, and suffer in interpretability.
Revised: This makes it much more likely for any NN to ﬁnd spurious correlations, and suffer in interpretability.

Original: So active learning and using a simpler
Revised: So active learning and using a simpler

Original: NN becomes very important for learning preferences in plan traces.
Revised: NN becomes very important for learning preferences in plan traces.

Original: As for prior feature knowledge, we assumed knowledge about what features were categorical (binary in our experiments) and what features were cardinal.
Revised: As for prior feature knowledge, we assumed knowledge about what features were categorical (binary in our experiments) and what features were cardinal.

Original: Rather than assume this knowledge, we can get this from the user as well, and reduce the assumptions about the domain features.
Revised: Rather than assume this knowledge, we can get this from the user as well, and reduce the assumptions about the domain features.

Original: Alternatively, we could have just encoded all features as cardinal features, and let the neural network determine what features were categorical.
Revised: Alternatively, we could have just encoded all features as cardinal features, and let the neural network determine what features were categorical.

Original: While this is certainly possible, we think it better to get this knowledge from the user and encode the plan trace based on this knowledge.
Revised: While this is certainly possible, we think it better to get this knowledge from the user and encode the plan trace based on this knowledge.

Original: This makes the job of the neural network easier, and less likely to learn spurious correlations.
Revised: This makes the job of the neural network easier, and less likely to learn spurious correlations.

Original: In our current encoding of features in FeatureNN model and our experiments, we have not included a preference dependency that considers the number of steps between fea- tures.
Revised: In our current encoding of features in FeatureNN model and our experiments, we have not included a preference dependency that considers the number of steps between fea- tures.

Original: For example, I would like to have a donut withinplan steps after having a coffee .
Revised: For example, I would like to have a donut withinplan steps after having a coffee .

Original: This omission was not intentional.
Revised: This omission was not intentional.

Original: One can easily encode such a sequential feature as a variable as well.
Revised: One can easily encode such a sequential feature as a variable as well.

Original: The number of steps between the two (state) features becomes a cardinal variable to represents this sequential feature.
Revised: The number of steps between the two (state) features becomes a cardinal variable to represents this sequential feature.

Original: Related Work
Revised: Related Work

Original: Two well known paradigms for learning, representing and reasoning over preferences are CP-nets and Generalized additive independence (and their variants).
Revised: Two well known paradigms for learning, representing and reasoning over preferences are CP-nets and Generalized additive independence (and their variants).

Original: Both of them were intended for preferences over outcomes.
Revised: Both of them were intended for preferences over outcomes.

Original: Each outcome can be comprised of many parts(decisions).
Revised: Each outcome can be comprised of many parts(decisions).

Original: One can think of each decision as choosing a value for a variable.
Revised: One can think of each decision as choosing a value for a variable.

Original: The user would have preferences over the possible outcomes, or there maybe a utility (value) associated to each outcome.
Revised: The user would have preferences over the possible outcomes, or there maybe a utility (value) associated to each outcome.

Original: In CPnets (Boutilier et al.
Revised: In CPnets (Boutilier et al.

Original: 2004), the decisions or variables are represented in a graph, and there exists dependencies over variables.
Revised: 2004), the decisions or variables are represented in a graph, and there exists dependencies over variables.

Original: The preferences of a variable’s values are affected by the value of the parent variables.
Revised: The preferences of a variable’s values are affected by the value of the parent variables.

Original: The CP in CP-nets stands for Ceteris Paribus or ”all else being equal” .
Revised: The CP in CP-nets stands for Ceteris Paribus or ”all else being equal” .

Original: Here, the all else refers to the parents of the node, and when they are equal, then a particular set of preference orderings for the child variable’s values hold.
Revised: Here, the all else refers to the parents of the node, and when they are equal, then a particular set of preference orderings for the child variable’s values hold.

Original: The knowledge of the dependence graph is either known apriori, or can be queried from the user.
Revised: The knowledge of the dependence graph is either known apriori, or can be queried from the user.

Original: Once the hierarchy of dependence is known, the user is then queried about preferences at each node.
Revised: Once the hierarchy of dependence is known, the user is then queried about preferences at each node.

Original: For CPnets to be used in plan trace preferences, we would have to ask the user what the dependencies are.
Revised: For CPnets to be used in plan trace preferences, we would have to ask the user what the dependencies are.

Original: Then we would have to ask the user for their relative preferences over features, given ﬁxed parent feature values.
Revised: Then we would have to ask the user for their relative preferences over features, given ﬁxed parent feature values.

Original: Note that the variables for plan preferences may also have to incorporate information about order.
Revised: Note that the variables for plan preferences may also have to incorporate information about order.

Original: So there are signiﬁcantly more variables (features) to consider in sequential data versus unordered data.
Revised: So there are signiﬁcantly more variables (features) to consider in sequential data versus unordered data.

Original: We think querying for such knowledge is very demanding, and not natural for preferences over plan traces.
Revised: We think querying for such knowledge is very demanding, and not natural for preferences over plan traces.

Original: We believe it is more natural for the user to specify the relevant conditional dependencies over features while annotating a plan trace.
Revised: We believe it is more natural for the user to specify the relevant conditional dependencies over features while annotating a plan trace.

Original: Additionally, we think it easier to give a preference value for the plan trace rather than relative preference orderings over the features in the domain.
Revised: Additionally, we think it easier to give a preference value for the plan trace rather than relative preference orderings over the features in the domain.

Original: The features could include sequential dependencies or positiondependent features.
Revised: The features could include sequential dependencies or positiondependent features.

Original: We think it would be hard for the user to be able to describe sequential features and the relative ordering over them.
Revised: We think it would be hard for the user to be able to describe sequential features and the relative ordering over them.

Original: Lastly, CP-nets do not compute utility values and some outcomes can be incomparable for a particular network of dependencies.
Revised: Lastly, CP-nets do not compute utility values and some outcomes can be incomparable for a particular network of dependencies.

Original: In our problem, we would like a total order over the plans, to select the most preferred plan.
Revised: In our problem, we would like a total order over the plans, to select the most preferred plan.

Original: So it helps to have a utility/preference value for every plan trace.
Revised: So it helps to have a utility/preference value for every plan trace.

Original: On the other hand, GAI (Braziunas and Boutilier 2006) models do provide a single utility value for a set of features.
Revised: On the other hand, GAI (Braziunas and Boutilier 2006) models do provide a single utility value for a set of features.

Original: As stated in their work, they provide an additive decomposition of a utility function (into sub-utility functions) in situations where single attributes are not additively independent, but (possibly overlapping) subsets of attributes are.
Revised: As stated in their work, they provide an additive decomposition of a utility function (into sub-utility functions) in situations where single attributes are not additively independent, but (possibly overlapping) subsets of attributes are.

Original: Since the subsets of attributes for the different sub-utility functions can overlap, one must query either with only global queries or a combination of local queries (over the subsets of features) with global queries to calibrate (as was done in the
Revised: Since the subsets of attributes for the different sub-utility functions can overlap, one must query either with only global queries or a combination of local queries (over the subsets of features) with global queries to calibrate (as was done in the

Original: GAI work) (Braziunas and Boutilier 2006)(p. 3).
Revised: GAI work) (Braziunas and Boutilier 2006)(p. 3).

Original: To learn a
Revised: To learn a

Original: GAI with active learning from a single user, there are one of two methods.
Revised: GAI with active learning from a single user, there are one of two methods.

Original: We could make assumptions about what subsets of variables are part of each sub-utility function, and what those functions are, or the user would need to know and give us this information.
Revised: We could make assumptions about what subsets of variables are part of each sub-utility function, and what those functions are, or the user would need to know and give us this information.

Original: We think this is a very difﬁcult task for the user.
Revised: We think this is a very difﬁcult task for the user.

Original: In our approach, we only do full trace queries and ask for annotations and preference ratings.
Revised: In our approach, we only do full trace queries and ask for annotations and preference ratings.

Original: We think it is more natural to ask the user for their overall rating of a plan, rather than how much each subset of features affected the rating.
Revised: We think it is more natural to ask the user for their overall rating of a plan, rather than how much each subset of features affected the rating.

Original: The neural network then handles the job of learning the preference function over the user-speciﬁed features (and any dependencies).
Revised: The neural network then handles the job of learning the preference function over the user-speciﬁed features (and any dependencies).

Original: Theotherformalism for specifying preferences are LTL rules (Huth and Ryan 2004) (p.175), which allow the user to specify sequential patterns.
Revised: Theotherformalism for specifying preferences are LTL rules (Huth and Ryan 2004) (p.175), which allow the user to specify sequential patterns.

Original: Expecting the user to be able to specify LTL rules might be unreasonable.
Revised: Expecting the user to be able to specify LTL rules might be unreasonable.

Original: The user would also have to give utilities or preference orderings over the speciﬁed LTL rules.
Revised: The user would also have to give utilities or preference orderings over the speciﬁed LTL rules.

Original: One can interpret our interface as extracting a subset of simple LTL rules (through annotations) which are present in a plan trace.
Revised: One can interpret our interface as extracting a subset of simple LTL rules (through annotations) which are present in a plan trace.

Original: The user gives a rating to the trace, as well as what features (LTL rules) were good or bad.
Revised: The user gives a rating to the trace, as well as what features (LTL rules) were good or bad.

Original: Extending the LTL analogy, our encoding of a plan trace can be seen as a vector of the relevant LTL rules.
Revised: Extending the LTL analogy, our encoding of a plan trace can be seen as a vector of the relevant LTL rules.

Original: The index corresponding to an LTL rule is set to 1 if the rule is satisﬁed in the plan trace.
Revised: The index corresponding to an LTL rule is set to 1 if the rule is satisﬁed in the plan trace.

Original: However, recall that we also allow cardinal features (counts) in our encoding, and not just binary variables.
Revised: However, recall that we also allow cardinal features (counts) in our encoding, and not just binary variables.

Original: Our interface and learning framework does not handle the entire gamut of possible LTL rules.
Revised: Our interface and learning framework does not handle the entire gamut of possible LTL rules.

Original: We are working on extending the types of sequential preferences supported, while keeping the interface intuitive and expressive.
Revised: We are working on extending the types of sequential preferences supported, while keeping the interface intuitive and expressive.

Original: Conclusion and Future Work
Revised: Conclusion and Future Work

Original: In our approach, we use feature-directed Active Learning complemented with an intuitive and expressive user interface to learn the user’s preference function efﬁciently.
Revised: In our approach, we use feature-directed Active Learning complemented with an intuitive and expressive user interface to learn the user’s preference function efﬁciently.

Original: The traces obtained during active learning are rated and annotated by the user.
Revised: The traces obtained during active learning are rated and annotated by the user.

Original: These traces are encoded as a vector over the features that the user indicated as relevant to their preferences.
Revised: These traces are encoded as a vector over the features that the user indicated as relevant to their preferences.

Original: The feature vectors are used to train a simple feedforward Neural Network to learn the preference function.
Revised: The feature vectors are used to train a simple feedforward Neural Network to learn the preference function.

Original: We show that the SimpleNN neural network is more accurate and interpretable with fewer, more informative plan traces as compared to the LSTM based SequenceNN model.
Revised: We show that the SimpleNN neural network is more accurate and interpretable with fewer, more informative plan traces as compared to the LSTM based SequenceNN model.

Original: The latter was trained with a larger dataset of rated plan traces without active learning.
Revised: The latter was trained with a larger dataset of rated plan traces without active learning.

Original: Our current experiments use a user preference function over only a few variables.
Revised: Our current experiments use a user preference function over only a few variables.

Original: It is important to see how efﬁciently our framework learns a more complex preference function.
Revised: It is important to see how efﬁciently our framework learns a more complex preference function.

Original: Moreover, the current preference function is completely deterministic as it provides consistent annotation and rating to the plan trace.
Revised: Moreover, the current preference function is completely deterministic as it provides consistent annotation and rating to the plan trace.

Original: A human, however, might not behave in a consistent manner.
Revised: A human, however, might not behave in a consistent manner.

Original: We will test with a noisy or probabilistic preference model in future work.
Revised: We will test with a noisy or probabilistic preference model in future work.

Original: The user interface itself can be extended to include more complex annotations.
Revised: The user interface itself can be extended to include more complex annotations.

Original: For example, the user can also provide annotations for some features to be added/dropped from the plan.
Revised: For example, the user can also provide annotations for some features to be added/dropped from the plan.

Original: This is especially useful for cardinal feature as the modiﬁed feature count represents what is ideal to the user.
Revised: This is especially useful for cardinal feature as the modiﬁed feature count represents what is ideal to the user.

Original: For example, if the user’s preference doesn’t increase after visiting more than 2 lakes .
Revised: For example, if the user’s preference doesn’t increase after visiting more than 2 lakes .

Original: Then this can be communicated by removing extra lake features from a plan trace.
Revised: Then this can be communicated by removing extra lake features from a plan trace.

Original: We have mentioned categorical and cardinal features but our framework is also intended to support real-valued features.
Revised: We have mentioned categorical and cardinal features, but our framework is also intended to support real-valued features.

Original: We would need to adapt our active learning process to elicit feedback as to what the minimum, optimum and maximum values of such features are.
Revised: We would need to adapt our active learning process to elicit feedback as to what the minimum, optimum and maximum values of such features are.

Original: These would be the minimum essential points to sample for approximating the underlying utility function.
Revised: These would be the minimum essential points to sample for approximating the underlying utility function.

Original: Lastly we would like to simplify the function by which we choose plan traces in successive rounds of active learning.
Revised: Lastly, we would like to simplify the function by which we choose plan traces in successive rounds of active learning.

Original: We think that the similarity with traces from previous rounds is unnecessary, and might not appreciably reduce the cognitive load on the user.
Revised: We think that the similarity with traces from previous rounds is unnecessary, and might not appreciably reduce the cognitive load on the user.

Original: We think that just diversity and selecting traces that are much more preferred(closer to 1.0) or much less preferred(closer to 0.0) would be sufﬁcient. 
Revised: We think that just diversity and selecting traces that are much more preferred(closer to 1.0) or much less preferred(closer to 0.0) would be sufﬁcient. Acknowledgments

Revised: This research is supported in part by the ONR grants

Revised: N00014-16-1-2892,

Revised: N00014-18-1-2442,

Revised: N00014-18-12840, the AFOSR grant FA9550-18-1-0067, NASA grant

Revised: NNX17AD06G and JP Morgan faculty research grant.