U SING L OGICAL S PECIFICATIONS OF O BJECTIVES IN M ULTI -O BJECTIVE R EINFORCEMENT L EARNING
Anonymous authors
Paper under double-blind review
A BSTRACT
In the multi-objective reinforcement learning (MORL) paradigm, the relative importance of each environment objective is often unknown prior to training, so agents must learn to specialize their behavior to optimize different combinations of environment objectives that are speciﬁed post-training.
These are typically linear combinations, so the agent is effectively parameterized by a weight vector that describes how to balance competing environment objectives.
However, many real world behaviors require non-linear combinations of objectives.
Additionally, the conversion between desired behavior and weightings is often unclear.
In this work, we explore the use of a language based on propositional logic with quantitative semantics–in place of weight vectors–for specifying non-linear behaviors in an interpretable way.
We use a recurrent encoder to encode logical combinations of objectives, and train a MORL agent to generalize over these encodings.
We test our agent in several grid worlds with various objectives and show that our agent can generalize to many never-before-seen speciﬁcations with performance comparable to single policy baseline agents.
We also demonstrate our agent’s ability to generate meaningful policies when presented with novel speciﬁcations and quickly specialize to novel speciﬁcations.
I NTRODUCTION
Reinforcement Learning (RL) is a method for learning behavior policies by maximizing expected reward through interactions with an environment.
RL has grown in popularity as RL agents have excelled at increasingly complex tasks, including board games (Silver et al., 2016), video games (Mnih et al., 2015), robotic control (Haarnoja et al., 2018), and other high dimensional, complex tasks.
RL continues to be a valued area of research as algorithms become more generalizable and sample efﬁcient, making them more feasible for deployment in real world scenarios.
Many RL tasks can be imagined in which multiple possibly conﬂicting objectives exist.
The relative importance of each objective may not be known by the system designer prior to training, and–when it comes to real world deployment–it may be difﬁcult or impossible to retrain agents as the priorities of objectives change over time.
Rather than retrain agents for each prioritization, multiple objective
RL (MORL) (Roijers et al., 2013) seeks to learn a set of potential policies so that importance of objectives can be speciﬁed after training, thus creating more ﬂexible, adaptable agents.
For example, a cleaning agent in a house environment may have several objectives such as dusting, sweeping, and vacuuming.
The agent can learn to complete each of these objectives, but certain objectives may take priority over others.
A user may specify that dusting is twice as important as sweeping and that vacuuming is not important at all.
These priorities may change, and MORL allows agents to learn policies that can satisfy any prioritization of objectives.
As part of MORL, a scalarization function is chosen to convert a multiple objective reward vector into a scalar.
The most common scalarization function for MORL is a linear combination in the form of a weight vector.
However, many real world scalarization functions are non-linear.
Additionally, weight vectors are not ideal for specifying desired behavior.
A user may need to experiment in order to ﬁnd the weights that result in a desired behavior.
Thus scalarization functions that are interpretable and allow for non-linearities are preferred.
Returning to the cleaning agent example, a user may specify that the cleaning agent should either sweep or vacuum, but it is not necessary to do both.
Perhaps the decision should be determined by whichever is more likely to be done well in consideration of other prioritized objectives.
Or, it may be useful to specify that the house should always be kept 75% dusted and that the rest of the agent’s time should be spent vacuuming.
These speciﬁcations become difﬁcult or impossible to encode with simple linear weightings.
Multi-task RL (MTRL) (Caruana, 1997) is a generalization of MORL in which an agent learns to complete multiple tasks simultaneously, often with the ultimate goal of completing some more complex task.
Recently, Universal Value Function Approximators (UVFA) (Schaul et al., 2015) were developed for MTRL to learn state or q-values over multiple goals.
UVFAs require a goal parameterization as input usually deﬁned as an element of the state (Andrychowicz et al., 2017).
However, the environment state is not always expressive enough to parameterize all of the goals we may want to learn.
Multiple objectives may provide a better way to deﬁne additional goals that are not part of the environment state.
The contributions of this work can be deﬁned in two parts.
First, we propose a simple language based on propositional logic to specify logical combinations of multiple objectives.
This language is equipped with quantitative semantics that are used to deﬁne scalarization functions for MORL.
The resulting scalarization functions can express non-linear combinations of objectives and are more interpretable than traditional weight vectors.
The language also acts as a way of specifying goals for multi-task learning that will be parameterized to use UVFAs.
Second, we develop a MORL agent for use with this language.
The agent prepends a recurrent encoder onto a UVFA architecture to parameterize goals speciﬁed in our language.
The agent also follows a learning curriculum over speciﬁcations to improve training speed and performance.
We demonstrate that our agent generalizes to never-before-seen speciﬁcations deﬁned in our language by providing a test set of novel speciﬁcations; this can be seen as a form of zero-shot RL.
Performance on test sets is compared to baseline agents trained on single speciﬁcations, and we show that our agent performs comparably to baselines despite having never been trained on the test speciﬁcations.
We demonstrate this over multiple grid worlds with various objectives.
Finally, we demonstrate our agent’s ability to quickly specialize to novel speciﬁcations post-training.
B ACKGROUND 2. R EINFORCEMENT L EARNING
Traditional RL problems are often modeled by a Markov Decision Process (MDP) deﬁned by the tuple ( S , S 0 , A , p ) where S is the set of states, S 0 is the set of initial states, A is the set of actions, and p ( r, s (cid:48) | s, a ) deﬁnes transition probabilities for the environment.
Here s (cid:48) ∈ S is the subsequent state and r ∈ R is the reward.
An agent’s objective is to maximize expected return:
[Equation]
where R π is the return under a policy π : S
→ A mapping states to actions, and γ is a discount factor.
Q-learning is an approach to solving reinforcement learning problems.
The Deep Q-Learning (DQN) algorithm (Mnih et al., 2015) utilizes a neural network Q to estimate the Q-value of a state-action pair.
The return under a policy R π can be expressed by the Q-value when the policy is deﬁned by the network Q :
[Equation]
We utilize the DQN algorithm in this paper to build our MORL agent, although our language and the way we construct our agent are not limited to the DQN algorithm.
2. M ULTI -O BJECTIVE R EINFORCEMENT L EARNING
We alter the traditional RL formulation as an MDP by using a vector of rewards rather than a scalar reward, resulting in a Multi-Objective Markov Decision Process (MOMDP) (Roijers et al., 2013).
We represent a MOMDP as a tuple ( S , S 0 , A , p ) where p ( r , s (cid:48) | s, a ) maps to the reward vector r ∈ R n rather than a scalar reward.
Each of the n elements in r represents the reward for a particular objective.
Next, we modify the DQN algorithm for MORL.
Equation 2 requires a scalar reward to sum with the discounted future rewards.
Since the new transition function p provides a vector reward r we need a scalarization function.
Typically, the scalarization function is deﬁned as a linear combination of objectives in the form of a weight vector (Abels et al., 2018).
In this work, we explore the use of non-linear scalarizations functions deﬁned by logical combinations of objectives.
In ofﬂine MORL, the scalarization function is unknown prior to training, so an agent’s goal is to learn a coverage set (Roijers et al., 2013) of policies in which there is at least one optimal policy for any scalarization function.
Previously this has been done by ﬁnding a convex coverage set of policies (Mossalam et al., 2016).
However, this method does not work for non-linear scalarization functions.
Abels et al.
(2018) explored the use of UVFAs to train MORL agents in an online setting.
In an online MORL setting, an agent must learn on the ﬂy and is not given the opportunity to learn a coverage set prior to testing.
Instead the agent must adapt quickly to new scalarization functions without forgetting policies for previously seen scalarization functions.
Online MORL is outside of the scope of this paper, but we also utilize UVFAs to instead learn a dense set of policies, trained ofﬂine, to be used post training with arbitrary behavior speciﬁcations.
2. U NIVERSAL V ALUE F UNCTION A PPROXIMATORS
Universal Value Function Approximators (UVFA) (Schaul et al., 2015) learn value functions over multiple goals, taking as input to a neural network a goal parameterization along with environment state or a state action pair.
This dense set of value functions induces an equally dense set of policies which can be applied to their respective goals.
Empirically, UFVAs are shown to be able to generalize to novel goals.
We utilize this method to parameterize goals speciﬁed by logical expressions in our language and show that these statements can be generalized across as well.
Many similar techniques are applied in multi-task learning scenarios.
Although, the purpose of multi-task learning is often to improve the learning speed of an agent rather than improve its ability to generalize to new tasks.
The latter is our objective.
M ULTI -O BJECTIVE RL WITH L OGICAL C OMBINATIONS OF O BJECTIVES
Rather than learn a set of individual policies to approximate a coverage set over scalarization functions, we train a single agent to generalize over encoded representations of scalarization functions.
We deﬁne scalarization functions in the semantics of a custom language.
Behavior speciﬁcations are represented as logic strings, passed through an encoder, and then given to a Q-network for learning.
In this section we lay out our language’s syntax and semantics, the agent architecture, the encoding method, and the learning curriculum for our agent.
3. L ANGUAGE S YNTAX AND S EMANTICS
We deﬁne a language based on propositional logic to specify combinations of objectives.
The logic operates over objective rewards between [0 , 1] inclusive, and is deﬁned as follows.
3.1. S YNTAX
We deﬁne the grammar of our speciﬁcation logic as follows:
[Equation]
Here, o n is the value of the nth objective, or the nth element in a reward vector, and c is a constant.
For the purposes of training, we use the current environment’s reward function to deﬁne the possible values of c for each objective.
Also notice that the ∧ and ∨ operations can be performed non-terminal elements of the grammar.
3.1. S EMANTICS
The quantitative semantics of our speciﬁcation language are deﬁned below:
[Equation]
Notice that any objective may be either minimized or maximized according to a soft ( o n , ¬ o n ) or hard ( o n ≥ c , o n ≤ c ) constraint.
The soft constraints return the value of the objective reward while the hard constraints return a value of one or zero.
With these semantics, we are able to specify complex desired behaviors.
Consider once again a cleaning agent tasked with several cleaning objectives: 1) dust, 2) sweep, and 3) vacuum.
In this scenario, the reward vector returned by the environment represents what percentage of the house is cleaned sufﬁciently for each of the objectives.
That means our language allows us to instruct an agent to sweep as much as possible, maintain a certain percentage swept, or even make as big of a mess as possible.
Additionally, if we instruct an agent to dust or sweep, we can expect it to complete one objective and not the other.
We can also expect that the agent will choose to complete the objective that can be satisﬁed more quickly because it will have a higher expected return.
With the above deﬁned predicate logic we can express desired behaviors such as:
o 1 ≥ .
5 ∧ oDust 50% of the house and sweep as much as possible.
o 1 ≥ .
8 ∧ o 2 ≥ .
8 ∧ o 3 ≥ .Keep
the house 80% dusted, swept, and vacuumed.
( o 1 ∧ o 2) ∨ ( o 2 ∧ o 3)
Either dust and sweep or sweep and vacuum.
o
2 ∧ ( o 1 ∨ o 3)
Sweep and either dust or vacuum.
o 1 ∧ ¬ oDust while increasing the amount of sweeping to be done.
3. M ULTI -O BJECTIVE DQN
For our MORL agent we implement a vanilla DQN as described by Mnih et al.
(2015), although our method is easily applicable to most RL algorithms.
Recent advances in MTRL, such as the use of UVFAs for generalizing across goals, has become more common in multiple objective settings (Friedman & Fontaine, 2018; Abels et al., 2018).
We also utilize UVFAs to generalize across encoded behavior speciﬁcations and output Q-values for each environment action.
The system architecture for the MORL agent is diagrammed in Figure 1.
Output from the encoder is concatenated with a state representation and passed through a four-layer neural network of hidden size 128 that outputs Q-values for each action in the ﬁnite action space.
As the agent gains experience, tuples of state, action, next state, terminal, and reward vector ( s, a, s (cid:48) , t, r ) are stored in a replay buffer for future training.
In our implementation, every 5 steps a batch of size 32 is sampled from the replay buffer, and every sampled tuple is augmented withdifferent behavior speciﬁcations ψ .
Experience tuples are augmented using the reward vector from the tuple and the semantics of a sampled speciﬁcation to generate a scalar reward.
The loss is then calculated over this batch using the formula:
[Equation]
Here f : r × ψ → R are the language semantics that map reward vectors and speciﬁcations to scalar rewards.
Also, ˆ Q is the target Q-network, part of the typical DQN algorithm, a bootstrapped estimate of the true Q-value updated regularly with the parameters of Q .
3. L ANGUAGE E NCODER
We utilize Gated Recurrent Units (GRU) (Cho et al., 2014) in order to encode behavior speciﬁcations in the previously deﬁned language.
We implement this encoder with three bidirectional layers of hidden size 64 as shown in Figure 1.
The input to our encoder is a sequence of one-hot encodings generated from a tokenized speciﬁcation.
The hidden states of the last layer in each direction are used as the speciﬁcation encoding.
The output of this encoder then serves as input to our MORL agent.
We train this language encoder end-to-end along with the rest of our agent.
During testing, we found that allowing gradients to ﬂow from the DQN back through the encoder gave us better results than other attempts to train the encoder in a separate supervised setting.
In the supervised training scenario, we trained using the output of the encoder along with a sampled reward vector to predict the scalarized reward produced by the speciﬁcation’s semantics (the idea being that the encoded speciﬁcation should retain sufﬁcient information to predict the scalarized reward).
However, after several experiments with supervised pretrained and co-trained encoders, we determined that end-toend training provided the DQN agent with better speciﬁcation encodings.
3. L EARNING C URRICULUM
We started by training our agent with a new, randomly sampled behavior speciﬁcation for each training episode.
Additionally, we randomly sampled behavior speciﬁcations to augment training batches.
However, we wondered if it would struggle to generalize to long, complex speciﬁcations.
We therefore experimented with using curricula to slowly increase the complexity of sampled speciﬁcations.
Under this curriculum, we only use a subset of all behavior speciﬁcations when sampling for environment episodes and batch augmentation.
This subset of behaviors is deﬁned by the maximum length of speciﬁcation strings in the subset.
In our ﬁnal agent, we increment the maximum length of speciﬁcations in the subset every 5000 timesteps.
The curriculum increments the maximum length a total of 20 times, starting at a base length of 25 characters up to the maximum speciﬁcation length for the entire set of speciﬁcations.
We compare learners trained with and without a learning curriculum in the experiments in Section 4.
E XPERIMENTAL R ESULTS
We present the following experiments and results to demonstrate our agent’s ability to encode the quantitative semantics of behavior speciﬁcations and behave appropriately according to neverbefore-seen speciﬁcations.
The gridworld shown in Figure 2 is a diagram of the environment used to test our MORL agent’s ability to generalize to new behavior speciﬁcations in a multi-objective setting.
We refer to this as the navigation environment.
It has three objectives: 1) stay on the road, 2) avoid hazards, and 3) move right.
The ﬁrst is marked by increasingly dark cells indicating increasing reward.
Reward for the second objective decreases starting inside the red outline until it reaches zero at the location of the darkest cell.
Finally, the third objective reward increases closer to the right side of the grid.
Environment state is ﬁnite and can be represented by the location of an agent in the grid.
The environment’s action space is also ﬁnite and composed of up, down, left, and right movements.
All transitions have a 10% chance of being random.
The agent’s initial state is indicated in Figure 2, one square down and to right left
4. Z ERO -S HOT G ENERALIZATION R ESULTS
Our initial experiments compare our agent to baseline agents trained on a single policy.
For these experiments, we use the navigation environment deﬁned previously with three objectives: stay on the road, avoid hazards, and move right.
Note that the opposite of each of these objectives are also included in possible behavior speciﬁcations due to the semantics of our language that enable minimization.
We deﬁne a set of 50,000 speciﬁcations to sample from during training, with which the agent learns to generalize to novel speciﬁcations.
These speciﬁcations are randomly generated according to number of atomic statements, logical connectives, hard vs. soft constraints, and value of constraints.
We randomly sample test speciﬁcations from these generated speciﬁcations.
The left plot in Figure 3 shows the average episodic reward for 100 never-before-seen test speciﬁcations throughout training for our agent with and without curriculum learning.
We compare the results of these two agents with 100 baseline DQN agents trained on each of these 100 behaviors.
The error bars in Figure 3 show one standard deviation in average reward for multiple agent initializations.
Our results indicate that learning curricula with respect to speciﬁcation length improve the speed and performance of learning and decrease the variance of learned policies.
Our agents are able to learn near baseline policies for never-before-seen speciﬁcations with the same amount of experience and network updates that it takes baseline DQN agents to learn individual policies.
Figure 4 visualizes resulting reward functions and policies learned by our agent for three neverbefore-seen behavior speciﬁcations.
The policies generated by our agent demonstrate an understanding of the speciﬁed behavior.
For example, the speciﬁcation shown in the left grid in Figure 4, o 2 ≥ 1 ∧ o 3 , can be interpreted as: avoid being within 4 squares of the marked hazards while maximizing the proximity to the rightmost column.
The derived policy, focuses on arriving at the rightmost column of the grid where it will receive the greatest reward while moving away from the hazards when appropriate.
The middle grid’s speciﬁcation, o 3 ≥ .
9 ∨ o 3 ≤ 0 , instructs the agent to move to the right two or the left two columns.
Note that the policy correctly sends the agent to the appropriate columns near the ends of the grid, but the center of the grid contains a few loops and other artifacts of conﬂicting policies.
During execution, the agent eventually reaches one of the speciﬁed goals due to the stochasticity of the environment.
The right grid shows the actual reward function and the generated policy for the speciﬁcation o 1 ≥ .
8 ∧ o 3 ≤ .
This behavior requires the agent to stay near the road while avoiding the four rightmost grid columns.
Notice that the agent’s policy focuses on remaining on the road when to the left of the 4 rightmost columns, otherwise the agent is only concerned with avoiding those columns.
4. W ARM -S TART T RAINING R ESULTS
While the results in Figure 3 show impressive generalization to never-before-seen tasks, performance is not quite optimal.
This begs the question: can the parameters learned when training on a variety of tasks be ﬁne-tuned on a single task?
Here, we test this by training a single policy agent initialized with parameters taken from one of our curriculum agent at 100,000 timesteps.
We compare to the baseline DQN agent (which is always trained on a single task).
The middle plot in Figure 3 shows this warm start agent compared to the baseline agents.
The graph demonstrates that our agent trained to generalize over speciﬁcations has the ability to specialize to individual speciﬁcations much faster than a baseline agent with traditional, randomly initialized parameters.
4. L INEAR A GENT C OMPARISON
Our agent learns a the non-linear scalarization function deﬁned by the semantics of the propositional logic deﬁned in Section 3.1.
Most of the speciﬁcations that can be expressed in this language are not easily represented by linear weights.
However, speciﬁcations that only use soft maximization constraints and logical and connectives can be expressed as weight vectors that contain only ones and zeros.
For example, o 1 → (1 0 0) and o 2 ∧ o 3 → (0 1 1) .
We use this method to compare our agent trained on logical speciﬁcations to agents from recent related work trained on linear weights.
We compare the performance of each of these agents throughout training on the seven possible combinations of objectives according to the method just described.
The linear agent uses the same parameters, architecture, and training algorithm as the logical agent but replaces sampled logical speciﬁcations with sampled weight vectors.
We sample weight vectors from a Dirichlet distribution ( α = 1) following the method of Abels et al.
(2018) who also utilize UVFAs for training their multiobjective agent.
The right plot in Figure 3 shows the results of this experiment.
Our linear agent was able to learn to satisfy the linear objectives with a speed and level of performance that mirrors the linear agent.
These results indicate that our agent does not lose performance on traditional linearly weighted objectives while learning to satisfy more complex non-linear combinations of objectives.
4. S PECIFICATION E NCODING V ISUALIZATIONS
Although our agent demonstrates impressive performance across a large number of speciﬁcations we would like to be assured that the agent is actually learning the semantics of logical speciﬁcations and correctly estimating Q-values for speciﬁcations across states.
The results of Figure 4 begin to demonstrate this by visualizing the learned policy for several logical speciﬁcations.
We further demonstrate our agent’s ability to learn semantics and implement policies across states and speciﬁcations through a number of experiments, the results of which are found in Figure 5.
The top sequence of images in the ﬁgure contains heat maps of our agent’s predicted values over environment states.
{+The heat maps interpolate between the speciﬁcations − o 
+}
and o 3 from left to right by moving between the encodings of each in speciﬁcation encoding space.
Value clearly shifts from left to right as expected.
Interestingly, traces of the ﬁrst objective can be seen during the transition when the agent in not preferring left nor right.
The bottom right graph in the ﬁgure plots a T-SNE visualization of 1,600 encodings of logical speciﬁcations.
These speciﬁcations are organized into 8 buckets of 200 semantically equivalent speciﬁcations.
For example, the speciﬁcations o 1 ∧ o 2 ∧ o 3 would be found in the same bucket as o 1 ∧ ( o 3 ∧ o 2) or ( o 1 ∧ o 1) ∧ ( o 2 ∧ o 3) .
We generate 200 unique speciﬁcations using the method described in Section 4.1 for each of the speciﬁcations listed in Figure 5’s T-SNE visualization.
The plot indicates that our agent is indeed consistently learning the semantics of various language speciﬁcations.
Finally, the bottom left grid in Figure 5 shows the state value heat maps for four semantically equivalent speciﬁcations − o 3 , ( − o 3) , (( − o 3)) , and (( − o 3 ∨ − o 3)) labeled as A, B, C, and D respectively.
The heat maps show that our agent ﬁnds similar state values (and thus policies) for semantically equivalent logcial speciﬁcations in addition to placing them close together in speciﬁcation encoding space.
4. N AVIGATION E NVIRONMENT V ARIATIONS
To test our agent on a variety of environments with various numbers of objectives we designed modiﬁed navigation environments.
We scaled the environment to 5x5 and 20x20 grids as outlined in
Figure 6 along with the 12x12 version described previously.
We train on these modiﬁed navigation
The medium environment is identical to the one described in the previous section, and the others are variations of it.
The objectives are added in the following order: 1) stay on the road, 2) avoid hazards, 3) move right, 4) move up, 5) move towards the horizontal center, and 6) move towards the vertical center.
environments with two, four, and six objectives. The two objective environment includes staying on the road and avoiding hazards. The four objective environment adds moving to the right and moving up. The six objective environment adds moving towards the center row and moving towards the center column.
With the latter four objectives, behavior speciﬁcations can include moving to arbitrary locations in the grid.
Figure 7 shows plots for the nine resulting combinations of environment size and objective count.
In these experiments we use predeﬁned sets of 40,000, 60,000, and 80,000 speciﬁcations with an 80: training-testing split for the two, four, and six objective versions respectively.
We randomly sampled 20% of the speciﬁcations to form test sets and show that our agent again generalizes to never-beforeseen behavior speciﬁcations over various environment sizes and objective counts.
We found that increased environment size and objective count increased the generalization difﬁculty, as indicated by the increase in disparity between our agent and baseline performance, but our agent still manages to generalize in these more difﬁcult environments.
It is interesting to note that with increased number of objectives, the difference between non-curriculum and curriculum agents becomes more apparent.
C ONCLUSIONS AND F UTURE W ORK
An ideal decision making agent has the ability to adjust behavior according to the needs and preferences of a user.
Preferably, this would be done without retraining.
Rather than learning separate policies for each desired behavior, we have shown that information about state, transitions, and the interactions between objectives can be captured and shared in a single model.
This work can be framed in many ways: as a non-linear generalization of MORL, or as a generalization of UVFA where value functions generalize across complex tasks speciﬁcations, instead of a single goal state.
Either way, our results demonstrate that deep RL agents can successfully learn about, and generalize across, complex task speciﬁcations encoded in complex languages.
We are particularly excited about our zero-shot results: our agents generalize to never-before-seen task speciﬁcations that are complex and nuanced, and are able to perform almost optimally with no taskspeciﬁc training.
If we allow the agent to train on new tasks, our warm-start experiments suggest that the parameters we have found serve as an excellent initialization that enables an agent to rapidly achieve optimal performance on a new task.
We have also shown preliminary evidence that when trying to summarize and utilize complex speciﬁcations, some sort of curriculum-based learning is likely to be necessary.
By increasingly complexifying the task speciﬁcations, we were generally able to achieve better, more stable, and lowervariance behavior.
In this work, our MORL agent demonstrates that it is possible to generalize over language deﬁned speciﬁcations and behave well when given completely novel behavior speciﬁcations.
This is possible in part through learning semantic representations of language expressions and then generalizing over those representations, an idea that can be generalized to other languages.
This naturally begs the question: could we encode task speciﬁcations using natural language, instead of a logical language?
Previous attempts have been made to combine natural and formal language with RL.
Natural language has been used for advice giving (Kuhlmann et al., 2004), reward shaping (Goyal et al., 2019), and deﬁning reward functions (Fu et al., 2019) in RL tasks.
Quantitative semantics of temporal logics have been used to deﬁne reward functions (Aksaray et al., 2016; Li et al., 2017) and ensure safe exploration (Li & Belta, 2019).
More complex languages like these would allow for more complex behaviors.
Finally, in this work, we use a simple language based on propositional logic to specify desired behavior, but in future work, MORL scalarization functions may include temporal and other formal logics.
Temporal logics would allow for complex sequences of commands with respect to multiple objectives, becoming much more applicable to real world scenarios.
Natural languages come with additional difﬁculties due their lack of concrete semantics.
However, improvements in natural language representation may also enable the use of natural language to specify desired behavior in MORL.