From: NeurIPS.cc/2022/Workshop/NeurReps/Program_Chairs
title: Paper Decision
decision: Accept (Poster)

From: NeurIPS.cc/2022/Workshop/NeurReps/Paper41/Reviewer_2AB7
title: Impressive work in which the task seems designed to make the idea look good, but it's a good idea anyway.
confidence: 3: The reviewer is fairly confident that the evaluation is correct
submission_track: Proceedings Paper (9 Page)
summary: This paper shows that if one approaches object rearrangement by decomposing objects into identity and state, and learn transitions between states, that one can do a better job on stacking and rearrangement tasks.

Neat idea, to factor each the scene into object identities and states and to learn transitions from state to state.  I love that sparsity emerged.  And I am not surprised that one does well on stacking and rearrangement tasks if one learns to factor objects into object identity and state, and then learns to group transitions between states, since manipulation of state and not identity results in object rearrangement and stacking and trying to manipulate object identity would be pointless.
strengths_and_weaknesses: This is a really neat idea, and I suspect there is some hardcore underlying group theory that makes this all work.  I think leaving an explication of that to future work is a fine idea.

Along those lines, I suspect that a purer group theoretic approach would be a bit more elegant-- I can't help but feel like the kitchen sink was thrown at this problem.  This results in a high quality submission for the NeurIPS workshop, but in the future, there might be a more elegant way of approaching it.  And the tasks are chosen well to showcase the benefits of doing this kind of decomposition.

Clarity: I was not familiar with SLATE or TD and so I had to be confused until I read the Appendix.  You might want to reference the Appendix on dSLATE in the main text.  Also, you might want to say that some of your tasks are MDPs and some POMDPs (I think)... relevant for if you use just the present observation and action or would ideally want to use longer pasts of observations and actions to learn transitions.

Significance: Time will tell, but I think that this kind of decomposition will be useful more generally.  Humans certainly do it.

Minor typo: How well they can reused for solving tasks — > How well they can be reused for solving tasks
questions: Does it matter that only the present $o_t, a_t$ are used if there are POMDPs?

See previous comments.
limitations: I think some comments about whether or not dSLATE would work in less limited environments would be greatly appreciated.  I intuitively have no feel for this and very much want to know whether or not dSLATE would work in more naturalistic environments, which can be far more complicated!
soundness: 3: Good
presentation: 3: Good
contribution: 3: Good
overall_rating: 8: Technically strong paper with novel ideas and high-to-excellent impact, and excellent evaluation and reproducibility.
relevance: 3: Solid fit
recommended_decision: 3: Accept

From: NeurIPS.cc/2022/Workshop/NeurReps/Paper41/Reviewer_4bdq
title: Generalizing across visual features and actions
confidence: 2: The reviewer is willing to defend the evaluation, but it is quite likely that the reviewer did not understand central parts of the paper
submission_track: Proceedings Paper (9 Page)
summary: The paper studies the problem of object rearrangement, or how to take actions to transition from some observation state to a goal state. Since the space of possible transitions is combinatorially large, the authors propose an approach that first groups visual features into sets, and then.
strengths_and_weaknesses: Overall: I'm not sure whether this paper fits with the theme of the workshop, though perhaps the case can be made. I think the paper does a nice job of documenting the results that it has, but I'm not yet convinced at how the method would scale to more difficult transitions/tasks with similar (but perhaps more probabilistic) structure.

Strengths:
- Figures are well-explained
- Learning graph structures and combining this with previous methods seems like a promising avenue

Weaknesses:
- There's little to no discussion of potential limitations/future steps for the model
- It's unclear how well the model would generalize to more challenging tasks
questions: "We implement a version of MPC (Veerapaneni et al., 2020) by replacing our factorized graph search with CEM." 
- These acronyms should be spelled out (in the main paper, rather than the appendix).
limitations: Appendix E is helpful in understanding limitations of the work. I'd like to better understand how the model might do if noise is added to the representations or during dSlate training, to test whether this affects usefulness of certain actions.
It'd be also helpful to understand how the model could be scaled up to more challenging problems of object rearrangement.
soundness: 3: Good
presentation: 3: Good
contribution: 3: Good
overall_rating: 5: Technically solid paper with some limitations e.g., limited analysis or evaluation.
relevance: 2: Limited relevance
recommended_decision: 2: Borderline

From: NeurIPS.cc/2022/Workshop/NeurReps/Paper41/Reviewer_DygD
title: A object controller that learns to generalise from interactions with previous objects
confidence: 2: The reviewer is willing to defend the evaluation, but it is quite likely that the reviewer did not understand central parts of the paper
submission_track: Proceedings Paper (9 Page)
summary: In this work the authors propose Hierarchical Abstraction (HA), an algorithm that learns to control objects compositionally. The basic logic is that actions affect different objects in the same ways, so you should learn the consequence of actions generally, and be able to flexible apply it to any object. To achieve this goal, HA takes as input pairs of images before and after an action has been applied, and performs two tasks. 

First, it forms a latent representation of the image in which each object has a latent, and the latent is made of two parts: an identity (green, round) and a state (position). Identity and state are defined by their permanence: identities never change, whereas between frames actions change some object's states. This is done using slot attention.

Second, a state transition graph is formed from all experiences, abstracted from the identities of the objects. This graph tells you which actions transform states to states.

Finally, a novel rearrangement task can be solved by breaking down the initial and goal image into a series of required transitions, and choosing the goal action associated with these transitions.
strengths_and_weaknesses: Originality

I really liked the algorithm, but I found it hard to assess the originality, I don't know the literature well. It seemed like a rebooting of many sensible ideas that are probably found in the literature with modern ML methods. This combination seemed both clever and to work well.

Quality

The submission seemed sound

Clarity

The writing was mostly clear and organised, though the figures were often ludicrously tiny (no need for that, you are so far from the page limit!) and I found the description overly brief, so that I had to really dig into the ideas to get what was being said.

Significance

This seems an interesting question to answer, and the authors appear to do it well.
questions: I eventually think I understood most things and thought it was a very neat little implementation. But I found it quite tough going, perhaps thanks to my ignorance. I will try and outline some of the things I found confusing so that, if the authors wanted to, they could make their work clearer for people like me. Before I do that I will include a couple of more general comments, and I'll end this section with some typos I found.

I was trying to understand how broadly this approach could be applied. Flexible state abstraction seem great but complex action choices that depend on other states, like in the block stacking example, required janky prerequisite state add-ins. As such, more complex action dependency could be hard, is this fair?

I thought the question was nicely introduced, and the literature review was nice, especially the bit in the appendix.

Why was this submitted to this workshop? It was cool, but didn't seem very related. This confusion might be related to one of my major confusions:

There were regular mentions of symmetry, but I don't think it was ever made clear which symmetry was being talked about. I assumed it was object permutation symmetry, is this correct? If so, when you call it an equivariant network, is that equivariant with respect to object permutation? At some point could this be made clear. Even with this symmetry, it certainly doesn't seem bang on the theme of the workshop, it's a paper about generalising actions no?

For this reason and a couple of others, I found level 1 the most confusing section, some more things:

I was surprised when time popped up, in all the discussion previously there had been only a before and after image, then suddenly there was a stream and I didn't know whether this was the stream of frames in the video starting at the before image, ending with the after; or separate instances of before-after pairs tied together. I eventually concluded it was the latter, if I got that wrong then I misunderstood a lot. Either way could this be made clear?

For a long time I thought z was just an ID, 1, 2, 3, etc. Then you started doing the Hungarian algorithm on ID and I realised it must be some summary of the features of the object. I probably shouldn't have realised that so late, could you make it clearer earlier what is in z? 

Is z_t+1 = z_t for all t? 

What is the nature of the independence the authors are looking for? That each object's latent representation is independent from the other objects? In which case why does sampling h_0 independently achieve this?

In the phrase 'Factorized graph over state transitions', what is it being factorized from, object ID? Could that be said?

The authors phrased their key insight as seeing that the two parts of HA were in fact the same (2.1), partitioning space into discrete groups. That didn't seem the key insight to me, that seemed a smaller one compared to the whole idea about separating ID from state and learning actions in a state not ID dependent way. This insight also arrived unintroduced, the introduction had very much led me to think that compositional action learning was the key insight. Why do the authors think this nicely symmetric two level clustering thing is their key insight, rather than just a convenient way of summarising their algorithm?

Figure 3 I think some figure labels went awry? And it was also, in theme for the paper, a far too tiny figure.

TYPOS

Fourth line of last para in into, in italics, missing 'be'

HA' should surely be written HA's?

First para of page 12, extra 'us'
limitations: The authors could have done more to highlight the limitations of their approach, for example the methodology for generalising to constrained state transition cases seemed janky.
soundness: 4: Excellent
presentation: 3: Good
contribution: 3: Good
overall_rating: 7: Technically solid paper, with moderate-to-high impact, and good-to-excellent evaluation and reproducibility.
relevance: 2: Limited relevance
recommended_decision: 2: Borderline