Repository: google-deepmind/meltingpot
Issue: Running with Scissors in the Matrix results discrepancy
Root cause: Documentation Deficiency

Hi,

The results of the OPRE paper and MeltingPot paper in the environment Running with the Scissors in the matrix (which is almost the same in both the papers as discussed in #68) do not match.

### OPRE Paper self-play results:
![Screenshot from 2022-09-01 17-26-39](https://user-images.githubusercontent.com/19221028/187908936-ec44c0f0-e0b8-47cd-a67e-25ac96bc1417.png)


### MeltingPot reported scores:
**Note:** All Meltingpot agents are trained using self-play as mentioned in the paper.
![Screenshot from 2022-09-01 17-26-23](https://user-images.githubusercontent.com/19221028/187909022-2cb715ab-f464-440f-905c-e6a66f873c42.png)




The only difference based on my understanding is that the OPRE paper initialized 6 different agents (RWS has only 2 agents in the environment) with different seeds whereas the MeltinPot paper initialized 2 agents (RWS has only 2 agents in the environment)

Is there anything I'm missing here?
Could you please comment on what could be the reason behind this discrepancy?

Thanks
Kinal

Hi! 
Another difference is that in OPRE paper the overall reward scale of the game. The Melting Pot version has rewards scaled down by 0.01 compared to the original (OPRE paper) version. Essentially, to compare one has to multiply Melting Pot results by 100

Oh, I somehow missed that.
Thank you for clarifying.

With this scaling, the scores of meltingpot paper are `~5` by averaging over sc2 (_pure rock_), sc3(_pure paper_), sc4(_pure scissors_).




I think this one is on us, I just looked at the melting pot paper and it's not that easy to get it (you have to pay attention to the matrix of RWS game). We should have been more explicit.


> With this scaling, the scores of meltingpot paper are ~5 by averaging over sc2 (pure rock), sc3(pure paper), sc4(pure scissors).

The corresponding score in the MP paper is against SC 1, which is a pure bot (r/s/p) chosen at random vs the agent. But yes, you could also average the SC 2-4. 

Overall, my explanation of the difference between OPRE paper and Melting Pot results is, as you correctly pointed out, due to the training protocol. Having 6 agents playing each other helps with i) exploration ii) not getting stuck in local minima via diversity (having to respond to 5 different agents at all times instead of just one). 

Thank you so much for this perspective.

Hey @vezhnick,

I had a question regarding the training of OPRE.

It is mentioned in the OPRE paper that **all 6 agents learn simultaneously**.
Considering the RWS environment with 2 agents, I believe the experience collection loop samples 2 agents from the set of 6 and uses those 2 to generate actions/experiences.
In the learner, is it that for both the players from the environment, each agent is updated playing against itself as follows
- `agent 0` vs `agent 0`
- `agent 1` vs `agent 1`
- and so on.

Or is it that 3 random pairs are made for training
- `agent 1` vs `agent 3`
- `agent 2` vs `agent 0`
- `agent 4` vs `agent 5`

Or is there any other strategy to train the entire population on the same data?

Thanks

Hey there!

It is the second version, meaning they are mixed at random. We ran several hundreds of matches between random pairs of agents in parallel, collecting the data and shipping it to learners in distributed fashion.

As you are running experience collection in parallel between random pairs, I believe the process of learning and experience collection is asynchronous. 

- So, in the learner, is the information about which experience is between which pair of agents passed through the replay buffer? Or is the paring in learner independent of how the experience was collected?
  E.g.:
  - Process 1 experience `agent 3` vs `agent 0`
  - Process 2 experience `agent 1` vs `agent 4`
  
  And as the learner is getting samples from both of these matches, how to manage matching in the learner?


I also had more questions regarding the OPRE paper. Hope you don't mind me asking them here.

- In section 5.4 ablations, a variant called PureMix is mentioned, which is used to check the effectiveness of the mixture model. Is the PureMix variant just conditioning the final policy on the latent dimension $z$, whereas the OPRE uses a mixture?

  Based on my understanding, the Value function and $\mu$ and $\pi$ policy for PureMix would be:
  
  $V = c(x_e)$ and $\pi = \eta(a|x_e)$
  
  where $x_e = concatenate([z,x_{\le t}])$ and $z = q(x'_t)$ a categorical distribution predicted using concealed information. And similarly, for $\mu$

Hey! Sorry for a late responce.

> So, in the learner, is the information about which experience is between which pair of agents passed through the replay buffer? Or is the paring in learner independent of how the experience was collected?

We did not use replay at all, it was all done in online fasion. You could think of it though as a replay buffer, where each trajectory is sampled only once and buffer is a FIFO. 

> And as the learner is getting samples from both of these matches, how to manage matching in the learner?

Actually, in your example the experience from the first match would only go to the 3rd and 0th learner, while the second would go to 1st and 4th. I'm not sure I get the question right though.

> In section 5.4 ablations, a variant called PureMix is mentioned...

Pure mix is very simple, it is the same as the basic baseline (advantage actor critic with V-trace), but policy is represented as a mixture. It is a lot closer to baseline than to OPRE. Sorry if that wasn't very clear.
The version you are suggesting would be a good ablation as well.

P.S. feel free to shoot me an email




Thank you for these details. 
I'll email you with more questions and details about my work.

I believe your email-id is: vezhnick [at] deepmind [dot] com
Please correct me if I'm wrong.


Closing as I think this is resolved.

Hey @vezhnick,

I have mailed you a few more queries regarding OPRE on "vezhnick [at] deepmind [dot] com". I hope you have received it.