Collaborative Decision Making Using Action Suggestions
Anonymous Author(s)
AffiliationAddress email
Abstract
The level of autonomy is increasing in systems spanning multiple domains, butthese systems still experience failures.
One way to mitigate the risk of failures isto integrate human oversight of the autonomous systems and rely on the humanto take control when the autonomy fails.
In this work, we formulate a methodof collaborative decision making through action suggestions that improves actionselection without taking control of the system.
Our approach uses each suggestionefficiently by incorporating the implicit information shared through suggestions tomodify the agent’s belief and achieves better performance with fewer suggestionsthan naively following the suggested actions.
We assume collaborative agentsshare the same objective and communicate through valid actions.
By assumingthe suggested action is dependent only on the state, we can incorporate the suggested action as an independent observation of the environment.
The assumptionof a collaborative environment enables us to use the agent’s policy to estimate thedistribution over action suggestions.
We propose two methods that use suggestedactions and demonstrate the approach through simulated experiments.
The proposed methodology results in increased performance while also being robust tosuboptimal suggestions.
Introduction
Autonomous systems and humans have different strengths.
Moravec [1] claimed in 1988 that “Itis comparatively easy to make computers exhibit adult level performance on intelligence tests orplaying checkers, and difficult or impossible to give them the skills of a one-year-old when it comesto perception and mobility.”
Thirty years later, computers have achieved superhuman level performance on many games like Go and StarCraft II [2], [3].
Despite major strides in perception,algorithms still exhibit brittleness and struggle with consistency in tasks that most would considertrivial for a young adult, like reliably generating a plausible sentence from a list of words [4]–[7].
Humans often make suboptimal decisions; however, they are exceptionally good at finding ways toaccomplish tasks, balance risk, reason through unstructured problems and incorporate new information into decisions without prior experience in a situation.
Systems integrating autonomy oftentake advantage of human strengths by using a human-on-the-loop structure.
This framework allowsfor humans to use their experience to recognize situations where the autonomy might struggle or isfailing and to take control [8], [9].
We hypothesize that instead of requiring the human to assumecontrol, an agent can use occasional control inputs from the human-on-the-loop in the form of actionsuggestions to improve its understanding of the environment.
In this work, we explore a method of collaboration that enables combining benefits of heterogeneoussystems like humans and machines through action suggestions versus direct control.
Our approach was inspired by the way that we see humans incorporate action suggestions in daily life.
A real world example is a pilot flying an aircraft while coordinating with air traffic control (ATC).
In the United
States, there are various services ATC provides to ensure safety.
Many of these services providesuggestions to pilots, but do not mandate an action or remove responsibilities from the pilot (e.g.radar assistance to VFR aircraft) [10]. Consider a situation where a pilot is flying at low altitude,has knowledge of another aircraft to the left, and an ATC controller recommends a turn to the left.
This action suggestion would cause the pilot to consider situations where a left turn towards trafficwould be optimal (e.g. terrain ahead) and reassess their belief about the environment.
In our problem of interest, we are assuming the agent has the same objective as a suggester anduse action suggestions as additional information about the environment to increase the quality ofthe action selection.
We avoid having to share exact belief states between agents by treating thesuggested action as an observation.
Using the assumption that the suggester is collaborative, we usethe agent’s policy to infer a distribution over suggested actions and use this distribution to updatethe agent’s belief.
This approach allows collaboration between different types of systems and avoidsexplicit translation of complex belief spaces.
In this paper, we first review related work in section 2 and discuss how our approach differs fromprevious methods.
Our primary contributions are in section 3.
We first provide an overview ofa partially observable Markov decision process (POMDP) and then outline the problem setting.
Section 3.3 demonstrates how we can view action suggestions as independent observations andmodify the belief update process.
We then provide two methods to estimate the distribution oversuggested actions using the agent’s policy in order to update the belief in section 3.4.
Section 4presents the experiments and discusses key results demonstrating the effectiveness of our approach.
Previous Work
Collaborative sequential decision making between two agents is a form of shared autonomy.
Theexact definition of shared autonomy and control varies in the literature [11].
A common themeamong the approaches is combining inputs between two agents to achieve performance better thanthe agent’s acting alone.
The problems often involve one of two main categories: 1 ) assistance froman autonomous agent to a human or 2 ) assistance from a human to the autonomous system.
Theproblem framework we are considering aligns with the second category.
In the first category, an autonomous agent is seeking to perform actions to assist the other agent(often a human).
Dragan and Srinivasa [12] discuss key aspects of this category and decompose itinto prediction of the user’s intent and arbitration between the user’s inputs and the action chosen bythe autonomous system.
Nguyen et al.
[13] propose a method that decomposes a game into subtasksthat can be modelled as a Markov decision process.
The autonomous agent then infers the subtaskthe other agent is attempting to accomplish and selects actions to assist.
Others model this categoryas a POMDP where the uncertainty is around the goal of the assisted agent [14]–[16].
The second category often involves forms of corrections.
Losey and O’Malley [17] use aspects ofoptimal control to learn from physical human interventions.
Nemec et al.
[18] first learn from humandemonstrations and then refine the policy through incremental learning from kinesthetic guidanceof the autonomous agent.
Other approaches perform the corrections in an iterative cycle during aplanning process.
Reardon et al.
[19] present a method that allows a human to interactively providefeedback during planning.
The feedback is in the form of hints or suggestions and is used to modifythe optimization process.
This approach is similar to the concept of reward shaping in reinforcementlearning.
Cognetti et al.
[20] provide a method that allows for real time modifications of a path while Hagenowet al.
[21] present a method that permits an outside agent to modify key robot state variables.
Thesechanges are then blended with the original control.
The concept of following guidance versus actingindependently and how to blend the two was explored by Evrard and Kheddar [22] where they usetwo controllers and alternate roles of following and leading.
Medina et al.
[23] present a methodthat selects between two strategies depending on the level of disagreement between agents.
Another related area of research is imitation learning.
Within this field, there are methods to learnfrom feedback by querying the expert during training or expert intervention real-time [24], [25].
These methodologies are a form of human-machine collaboration by using the expert knowledge ofthe human to train or correct the system and then letting the system perform the task autonomously
These approaches generally rely on strict assumptions or an explicit model of the other agent sothe autonomous system can interpret inputs in a way to reason how to integrate them.
In the firstcategory, Jeon et al.
[26] propose a model structure to map human inputs to different actions basedon the robot’s confidence in the goal.
Reddy et al.
[27] deviate from the use of model based methodsby using human-in-the-loop deep reinforcement learning to map from observations and user inputsto an agent action.
Our approach builds on the concept of mapping user inputs to an agent action.
However, we differfrom other methods in that we do not build an explicit model of the suggester nor learn a mappingthat might be suggester dependent.
We achieve this mapping by assuming the suggester is collaborative and shares the same objective when providing inputs.
This assumption enables the use ofthe key insight from Spencer et al.
[25], “... any amount of expert feedback ... provides informationabout the quality of the current state, the quality of the action, or both”, and we build an implicitmodel of the suggester using only the agent’s knowledge of the environment and the agent’s policy.
Action Suggestions as Observations
3.1 Background
A partially observable Markov decision process (POMDP) is a mathematical framework to modelsequential decision making problems under uncertainty [28].
A POMDP is represented as a tuple( S , A , O , T, O, R, γ ) , where S is a set of states, A is a set of actions, and O is a set of observations.
At each time step, an agent starts in state s ∈ S and chooses an action, a ∈ A .
The agent transitionsfrom state s to state s ′ based on the transition function T ( s, a, s ′ )
= p ( s ′ | s, a ) , which represents theconditional probability of transitioning to state s ′ from state s after choosing action a .
The agent doesnot directly observe the state, but receives an observation o ∈ O based on the observation function
O ( s ′ , a, o )
= p ( o | s ′ , a ) , which represents the conditional probability of observing observation ogiven the agent chose action a and transitioned to state s ′ .
At each time step the agent receives a reward, R ( s, a ) ∈ R for choosing action a from state s .
Forinfinite horizon POMDPs, the discount factor γ ∈ [0 , 1) is applied to the reward at each time step.
The goal of an agent is to maximize the total expected reward E [ (cid:80) ∞ t =
γ t R ( s t , a t )] , where s t anda t are the state and action at time t .
One method to solve a POMDP is to infer a belief distributionb ∈ B over S and then solve for a policy π that maps the belief to an action where B is the set ofbeliefs over S [29].
Executing with this type of policy requires maintaining b through updates aftereach time step.
Policies can be generated offline or computed online during execution.
In this work, we focus onapplying our method to policies generated offline and leave the application to online solvers forfuture work.
Many approximate offline solvers involve point-based value iteration.
The idea is tosample the belief space and perform backup operations on the sampled points of the belief space,iteratively applying the Bellman equation until the value function converges.
PBVI [30], HSVI2[31], FSVI [32], and SARSOP [33] are examples of such an approach, though they differ in theselection of initial belief points, the generation of points at each iteration, and the choice of whichpoints to backup.
These algorithms represent the policy as a set of alpha vectors.
In this work, weused SARSOP to generate the policies; however, any algorithm that produces a policy where theutility of a belief can be calculated could be implemented with little change to our methods.
3.2 Problem Formulation
For a given problem of sequential decision making under uncertainty, we choose to model the problem as a POMDP and use a point-based solution method.
In this work, we assume discrete state,action, and observation spaces but the methods can be generalized to continuous spaces.
Our problem involves two entities: an autonomous system using the POMDP policy to perform actions andinteract with the environment that we will refer to as the agent, and a suggester providing actionsuggestions to the agent.
The suggester can observe the environment, but not interact or affect thestate except to provide recommended actions to the agent.
The suggester is not required to providesuggestions but can provide a maximum of one action suggestion at each time step.
The agent and suggester are collaborative and share the same objective of maximizing the total expected reward.
The agent and suggester can receive different information and maintain separate beliefs of the environment.
The separation of the actors allows each to process and receive information independently and capitalize on strengths differently.
An example problem is an expert humanreceiving observations not modeled by an autonomous robot and providing intermittent suggestions.
3.3 Incorporating Action Suggestions
There are different ways for the agent to use action suggestions.
One approach would be to modelthe suggester in the original POMDP.
This approach is similar to previous shared autonomy work(section 2).
Modeling the suggester within the POMDP would provide a fundamental way of incorporating suggestions, but would require a priori knowledge of the suggester to develop an explicitmodel to solve for the policy and also require maintaining a belief over the model during execution.
A simple method that does not require a model of the suggester would be for the agent to naivelyfollow each suggestion.
A similar and more robust method would be to follow a suggestion if itmeets some defined criteria, thus potentially disregarding suboptimal suggestions (e.g. only followa suggestion if it is a specific action).
These approaches are simple and can incorporate suggestions;however, they do not benefit from incorporating implied information contained with each suggestedaction.
Applying our inspirations we introduced in the flight example to our problem, each action suggestioncontains information related to the suggester’s belief of the environment.
We propose treating eachaction suggestion as an observation of the state in order to update the agent’s belief.
This idea enablesthe suggester to influence the agent while the agent remains autonomous.
Our problem assumes thesuggester and the agent maintain independent beliefs.
If we further assume the suggested action isnot influenced by the agent’s previous action, the suggested action is only dependent on the state.
The independence of the agent’s previous action and the suggester allows a modification of our beliefupdate process that only involves the probability of receiving the suggested action given the currentstate.
The suggested action is not always independent of the agent’s action, but the assumption thatit only depends on the current state is often reasonable.
Our belief at time t over state s ∈ S with observations o ∈ O and action suggestions o s ∈
A isp ( s t | a 0: t − 1 , o o : t , o so : t ) .
Using Bayes’ theorem, we can rewrite our expression as
[Equation]
where the subscript 0: t refers to all instances of that variable from 0 to t , and t − = t − 1 .
Thisexpression can be simplified using the independence assumption, the law of total probability, andthe Markov property to
[Equation]
Equation (2) is a simple modification to our standard belief update procedure with POMDPs and isan expression of updating a belief with two independent observations.
3.4 Inferring the Distribution over Suggested Actions
We can update the agent’s belief based on p ( o st | s t ) .
However, the agent cannot calculate thisdistribution directly.
We propose using the assumption that the agents are collaborative and estimatep ( o st | s t ) by using the agent’s policy.
In our flight example, when the pilot receives the suggestion toturn left they would use their experience to reason through what scenarios they would have providedthat same suggestion.
Scaled Rational.
One approach is to assume the suggester is perfectly rational and has a policyidentical to the agent.
The suggester would only give actions that would maximize the total expectedreward using the agent’s policy π .
In other words, p ( o st | s t ) ≈ 1 ( o st = π ( s t ))
where 1 ( · ) is theindicator function.
To relax the assumption of a perfectly rational agent and an identical policy, weintroduce a scaling factor, τ ∈ (0 , 1] .
With a scaling factor, we assume the suggester acts rationallyand with the agent’s policy a fraction τ of the time and uses a random policy otherwise.
The scaled rational update can be expressed as
[Equation]
Noisy Rational.
Another approach is to assume the likelihood of the suggested action is relatedto the total expected reward of choosing that action.
Shepard [34] developed a noisy rational modelfrom a psychological perspective, and this model has been widely used in robotics to model suboptimal decision making [35]–[37].
With this model, the suggester is most likely to choose the actionwith the highest expected return and less likely to choose suboptimal actions.
This model requirescalculating the total expected return of each action.
The action value function Q ( s, a ) returns the expected value of performing action a in state s andthen executing optimally thereafter.
We can use the agent’s policy and the reward function andperform a one step look ahead to calculate the Q-function [29].
Using a policy represented withalpha vectors, we can calculate Q ( s, a ) by performing a one step look ahead using a belief that theagent is in state s .
The noisy rational model has the same form as the Boltzmann distribution andthe softmax function.
We can express this model as
[Equation]
where λ ∈ [0 , ∞ ) is a hyperparameter often referred to as the rationality coefficient.
The distribution approaches a uniform distribution as λ approaches 0 .
As λ increases, the model approaches aperfectly rational agent.
Experiments
The proposed methodology to incorporate action suggestions as observations was evaluated on twoclassic POMDP problems, Tag [30] and RockSample [38].
These domains are relatively simplebut provide a way to evaluate the merits of the approach by removing domain-specific variablesthat might influence performance.
The simulations were constructed to first evaluate the effectiveness and efficiency of the proposed approach and then to test the robustness to suboptimal actionsuggestions.
4.1 Environments
The two environments chosen to evaluate our approach have discrete action, state, and observationspaces.
The structure of the problems allow for the visualization of the belief space while alsoproviding a modest scalability problem.
Tag.
The Tag environment was first introduced by Pineau et al.
The layout of the environmentcan be seen in fig.
The agent and an opponent are initialized randomly in the grid.
The goal of theagent is to tag the opponent by performing the tag action while in the same square as the opponent.
The agent can move in the four cardinal directions or perform the tag action.
The movement ofthe agent is deterministic based on its selected action.
A reward of − 1 is imposed for each motionaction and the tag action results in a +10 for a successful tag and − 10 otherwise.
The agent’sposition is fully observable but the opponent’s position is unobserved unless both actors are in thesame cell.
The opponent moves stochastically according to a fixed policy away from the agent.
Theopponent moves away from the agent 80 % of the time and stays in the same cell otherwise.
Ourimplementation of the opponent’s movement policy varies slightly from the original paper allowingmore movement away from the agent, thus making the scenario slightly more challenging. Detailsof the differences are described in more detail in the supplemental material.
RockSample.
The RockSample environment consists of a robot that must explore an environmentand sample rocks of scientific value [38].
Each rock can either be good or bad and the robot receivesrewards accordingly.
The robot also receives a reward for departing the environment by entering anexit region.
The robot knows the positions of every rock and its own location exactly, but does notknow whether each rock is good or bad.
The robot has a noisy sensor to check if a rock is good or bad and the accuracy of the sensor depends on the distance to the rock.
Upon each use of thesensor, the robot receives a negative reward.
In the following sections, a RockSample problem willbe designated as RockSample ( n, k, sr, sp ) where n designates a grid size of n × n , k is the numberof rocks, sr is the sensor range efficiency, and sp is the penalty for using the sensor.
The reward forsampling a good rock and exiting the environment is +10 and the penalty for sampling a bad rockis − 10 .
4.2 Simulation Details
The simulation environment was built using the POMDPs.jl framework [39].
The Tag environmentused the standard parameters and was simulated until the first of the agent tagging the opponent or100 steps.
The RockSample environment was simulated until the agent exited the environment.
The
SARSOP algorithm was used to generate policies for the agent.
All of the agents used the samepolicies for the environment but incorporated the action suggestions differently.
If the suggestedaction was the same as the selected action with the current belief, no modification of the belief wasperformed.
This implementation decision potentially prohibits valuable information to be passedwhen actions align.
However, simulations with the RockSample and Tag environments showeddifferences were negligible and the slight decrease in computation time allowed for more simulationsto be performed.
Performing belief updates with aligned suggestions would likely be critical indifferent scenarios.
In RockSample, the rocks were randomly initialized based on a uniform distribution.
The agent’sbelief was initialized with a uniform distribution over the state space in Tag and a uniform distribution over the rocks in RockSample.
The number of simulations for a given scenario varied andwe provide the 95 % confidence interval for each value reported in the results section. The rewardsreported are the mean value over all simulations for a given scenario.
The number of suggestionsrefers to the number of times the suggested action differed from the action initially selected by theagent before considering the suggestion.
To simulate a suggester that was not always present orcould not communicate actions reliably, the simulation also supported varying the percentage ofsuggestions passed to the agent.
One suggester was used but the quality and consistency of the action suggestions varied.
Differentagents were simulated to evaluate our proposed approach and establish baselines in each scenario.
Details of each agent are provided below.
• Suggester .
The suggester is a combination of an all-knowing agent and a purely randomsuggester.
The rate of randomness is adjusted to scale from purely random to completelyall-knowing.
If the suggester is sending a non-random suggestion, the action is selectedfrom the POMDP policy using the true state of the environment. • Normal Agent .
The normal agent executes in the environment using the policy withoutconsidering any action suggestions.
This agent provides a baseline when action suggestionsare not considered.
• Perfect Agent.
The perfect agent is initialized and executed with perfect knowledge of the state.
At each time step an action is selected from the policy given the true state of the system.
This agent provides and upper bound on the total expected reward executing with the given policy. • Random Agent.
The random agent chooses an action from a uniform distribution over the action space at each time step.
This agent provides a lower bound on the total expectedreward when investigating robustness to suboptimal and random action suggestions. • Naive Agent.
The naive agent executes in the environment using the POMDP policy.
Whenaction suggestions are received, it follows the action suggestions naively, and performs no modifications to its belief state.
The rate at which the naive agent follows the suggestions was adjusted to investigate robustness.
The rate of following the suggestion is depicted byν in the results.
• Scaled Agent.
The scaled agent incorporates the methodology outlined in section 3.4 anduses eq.
(3) to estimate p ( o st | s t ) .
The hyperparameter τ is kept constant for each simulation and the value used is shown with the presented results.
The value for τ was notadjusted to fine-tune performance.
It was broadly changed to show overall effects of thehyperparameter in different situations. [Table]
• Noisy Agent.
The noisy agent also incorporates the ideas from section 3.4.
This agent useseq.
(4) to estimate p ( o st | s t ) .
The hyperparamter λ is kept constant for each simulationand the value is shown with the respective results.
Like the scaled agent, the parameterλ was not fine-tuned for performance.
Coarse adjustments were made to depict how thehyperparameter influenced the algorithm.
4.3 Effectiveness and Efficiency Results
The different agents were compared using an all-knowing suggester with a 100 % message reception rate.
Simulations were ran on Tag, RockSample (7 , 8 , 20 , 0) , and RockSample (8 , 4 , 10 , − 1) .
RockSample (8 , 4 , 10 , − 1) was designed with the rocks near the corners of the environment.
Thislayout emphasized the importance of each movement direction.
The results are summarized in table 1.
We also compared the different agents using suggesters with partial views of the state andincluded those results in the supplemental material.
As expected, incorporating actions suggestions from an all-knowing suggester improves performance across all scenarios.
Despite not directly following the suggested action, the Scaled and
Noisy agents were able to reach a near perfect reward.
A key metric to note with these results isthe number of suggestions.
The naive agent was able to achieve perfect scores by simply followingthe suggestions; however, it did not integrate any information contained from each suggestion andresulted in requiring more suggestions to achieve similar scores.
While the scores for all assistedagents decrease when the hyperparameters change (become less trusting of the suggester) the naiveagent’s score decreases at a faster rate.
Again, this difference is an artifact of the benefit of updatingthe belief of the agent with each suggestion.
A visual depiction of the change in the belief state of an agent in the Tag environment after incorporating an action suggestion is depicted in fig.
Figure 1a shows the belief the agent has of where thetarget is located.
With that belief, the best action according to the agent’s policy would be to movenorth.
However, the agent receives a suggestion from a suggester to move west.
Figure 1b showsthe updated belief after using the noisy rational approach with λ = 1 .
There are multiple states tothe west of the agent where the target might be located.
The update process was able to incorporatethe implied information of the west suggestion by shifting the distribution towards the west side ofthe grid.
Collaborators often are not capable of continuously providing suggestions to an agent.
The percentage of suggestions received was varied to investigate the effectiveness of the agents in the presenceof intermittent action suggestions.
A suggester with perfect state knowledge was used but the mes- [Figure] [Figure] sage reception rate was varied.
The results for the Tag environment are shown in fig.
The changein reward is shown in fig.
2a.
Figure 2b shows how the ratio of the number of suggestions to thenumber of steps required to tag changes.
The increased reward and lower suggestion ratio furtheremphasizes the benefit of our approach.
4.4 Robustness Results
The previous results assumed a perfectly rational suggester, but this is not always realistic.
Weevaluated the robustness of the different agents by adjusting the randomness of the suggester.
Thechance of a random action suggestion (uniformly picked from the action space) was varied from0 to 1 representing a perfect suggester to a completely random suggester.
The results from thesesimulations are shown for both the Tag environment and RockSample (8 , 4 , 10 , − 1) in fig.
As expected, the naive agent that follows all suggestions performs poorly as the randomness of thesuggester increases.
Decreasing ν increases the robustness, but sacrifices performance.
The scaledand noisy approaches perform well even with high trust parameter settings ( τ and ν ).
These resultsdemonstrate the value of not naively following suggestions and the benefit of balancing the agent’sinitial belief state with the information received from the action suggestion.
Conclusion
We developed a new method to increase collaboration between heterogeneous agents through theuse of action suggestions.
Because a suggested action conveys information about the suggester’sbelief, we used the agent’s current policy to transform the suggested action into a distribution that [Figure] we used to update the agent’s belief.
We applied this approach in simulation and demonstrated theincreased efficiency by requiring fewer suggestions and suggestion rate.
We further demonstratedour approach was robust to suboptimal decisions and could still perform better than an unassistedagent with more than 50 % of the suggestions being random.
This methodology does not rely onknowledge of other agents.
The only requirement is a shared objective and communication throughthe agent’s set of actions.
This low threshold of coordination increases collaboration opportunitieswith other systems including humans.
Integrating action suggestions as observations requires a change to the belief update process of theagent.
The independence of the observations allow the update process to occur simultaneously or inany order which allows for asynchronous and unreliable suggestions without added complexity.
Thisapproach also scales linearly with the number of suggesters because the agent must only approximatethe distribution over action suggestions for each suggester.
A critical assumption that enables this process with no knowledge of the suggester is that the suggester is collaborative.
The scaled and noisy rational approaches do factor in suboptimal decisionsbut ultimately assume the suggester is providing actions in the best interest of the agent.
If a suggester was maleficent, the belief would be skewed towards distributions that would explain the suggestions.
The proposed approach is designed to be robust to suboptimal suggestions, but not resilientto bad actors.
There are many opportunities of future work expanding on these ideas.
As previously discussed, thisapproach was formalized and applied to offline methods, but the same concept can be extended toonline solvers.
Online approaches also offer unique opportunities to use a suggested action to helpbalance exploration and exploitation while searching for an action.
The experiment results demonstrated a trade-off of performance and robustness when changing the hyperparameters.
Finding away to learn the quality of the suggester and adjust the parameters real time is a promising idea.
The benefit of modifying the agent’s belief through an action suggestion is not limited to situationswhen the subsequent, updated belief is more accurate.
A modified belief could be beneficial if theresulting actions are optimal regardless of the accuracy of the belief.
This idea opens up possibilities of using action suggestions to increase performance when an agent’s policy does not match theenvironment.
Checklist
For all authors...
(a) Do the main claims made in the abstract and introduction accurately reflect the paper’scontributions and scope?
[Yes] Section 3.3 and section 3.3 provide the key contributions and section 4 documents the experimentation results.(b)
Did you describe the limitations of your work?
[Yes] Key assumptions were documented when applying them and a key limitation of being robust and not resilient wasrevisited in section 5.
(c) Did you discuss any potential negative societal impacts of your work?
[No] This paper
provides a general approach to enable collaboration.
Based on the guidelines and it
being more of a foundational research paper, we did not discuss societal impacts as
the list would include most applications of collaboration using a POMDP framework.
(d) Have you read the ethics review guidelines and ensured that your paper conforms to
them?
[Yes]
If you are including theoretical results...
(a) Did you state the full set of assumptions of all theoretical results?
[Yes] The step to in-
tegrate the action suggestion as an independent observation was shown in section 3.3.
The assumption of dependence only on the state was mentioned in that section as well.
(b) Did you include complete proofs of all theoretical results?
[N/A]
The derivation of an
independent update in a Bayes’ filter is well established, but the key steps were shown
in section 3.3.
If you ran experiments...
(a) Did you include the code, data, and instructions needed to reproduce the main exper-
imental results (either in the supplemental material or as a URL)?
[Yes] Included in
the supplemental material.
All policies for the environments were not provided based
on file size, but the policies can be generated with a provided script.
(b) Did you specify all the training details (e.g., data splits, hyperparameters,
how they
were chosen)?
[Yes] Experiment details are documented in section 4.
(c) Did you report error bars (e.g., with respect to the random seed after running experiments multiple times)?
[Yes] See section 4.
(d) Did you include the total amount of compute and the type of resources used (e.g., type of GPUs, internal cluster, or cloud provider)?
[No] Our approach does not require a lot
of computation.
Computation time and amount was not a concern and not provided.
If you are using existing assets (e g., code, data, models) or curating/releasing new assets...
(a) If your work uses existing assets, did you cite the creators?
[Yes] Experiments were
built on the POMDPs.jl framework and documented in section 4.2.
(b) Did you mention the license of the assets?
[No] POMDPs.jl is available free to use,
copy, modify, merge, publish, distribute, and/or sublicense and is distributed with an
MIT "Expat" License.
(c) Did you include any new assets either in the supplemental material or as a URL?
[N/A]
The Tag environment is not provided by POMDPs.jl, but is supplied to recreate
the experiments in the supplemental material.
(d) Did you discuss whether and how consent was obtained from people whose data
[Table]
you’re using/curating?
[N/A]
[Table]
(e) Did you discuss whether the data you are using/curating contains personally identifi-
[Table]
able information or offensive content?
[N/A]
[Table]
If you used crowdsourcing or conducted research with human subjects...
[Table]
(a) Did you include the full text of instructions given to participants and screenshots, if [Table] applicable?
[N/A]
[Table]
(b) Did you describe any potential participant risks, with links to Institutional Review
[Table]
Board (IRB) approvals, if applicable?
[N/A]
[Table]
(c) Did you include the estimated hourly wage paid to participants and the total amount
[Table]
spent on participant compensation?
[N/A]
[Table]
A Different Quality Suggester Results
This section presents results on RockSample (8 , 4 , 10 , − 1) when the suggester is not always allknowing.
In our approach, we formulated the belief update based on assuming the suggester observed the environment.
These results demonstrate that our approach extends beyond an all-knowingsuggester and can incorporate information from suggestions developed from different beliefs of thestate.
Table 3 contains the mean rewards and table 4 contains the mean number of suggestions consideredby the agent.
The details of the agents are provided in section 4.2.
In these simulations, insteadof having an all-knowing suggester, the suggester maintained a belief on the state of the rocks andprovided suggestions based on that belief.
The suggester’s initial belief varied and was determinedbased on two parameters.
We represent a suggester with different parameters as ( G | B ) .
Theparameters can be thought of as G being the initial belief that good rocks are good and B is theinitial belief that bad rocks are good.
For example, a suggester with parameters of (1 . 0 | 0 .
0)would represent perfect knowledge of the environment while (0 . 5 | 0 .
5) would be an initial beliefof a uniform distribution.
Examples of different initial beliefs based on different parameters areprovided in table 2.
In table 2, the examples are based on a true state of [1 , 1 , 0 , 0] where a onerepresents a good rock and a zero represents a bad rock.
[Table]
[Table]
[Table]
[Table]
[Table]
[Table]
[Table]
[Table]
[Table]
[Table]
[Table]
[Table]
[Table]
[Table]
[Table]
[Table]
[Table]
[Table]
[Table]
[Table]
[Table]
[Table]
[Table]
[Table]
[Table]
[Table]
[Table]
B Environment Differences
B.1 Tag Differences
Our implementation of the Tag environment differs slightly from the original implementation.
Thedifference is in the transition function and results in our scores differing than those presented in otherpapers [30], [33].
The original problem described the opponent’s transition as moving away fromthe agent 80 % of the time and staying in the same cell otherwise [30].
The original implementationconsidered all actions away from the agent as valid and did not consider if an action would resultin hitting a wall.
Additionally, the actions were considered in pairs (i.e. east/west and north/
south)with 40 % allocated to each pair.
This implementation choice often results in the opponent stayingin the same cell greater than 20 % of the time despite having valid moves away from the agent.
[Table]
[Table]
Our implementation only considered actions that would result in a valid move away.
The moveaway probability was distributed equally to each valid action.
This change results in the opponentmoving away from the agent more than the original implementation and results in a slightly morechallenging scenario.
An example of the differences in the opponent transition probabilities in a3 × 3 environment is show in fig. 4.0.
[Figure]
[Figure]
B.2 RockSample Additions
Our implementation of the RockSample environment is the same as previous work except for anadditional penalty for performing the sense action (i.e. we added a penalty when the agent uses thesensor to gather information about a rock).
We added this penalty when formulating a problem toemphasize the importance of each action selection.
Our implementation is identical to other workswhen the penalty is set to zero and can be verified by our RockSample (7 , 8 , 20 , 0) experiments.