From: NIPS.cc/2018/Workshop/IRASL
title: Acceptance Decision
decision: Accept (Demo)

From: NIPS.cc/2018/Workshop/IRASL/Paper62/AnonReviewer3
title: Need "proof"
comment: I think the hardest thing about any visualization is demonstrating efficacy. Your anecdotal evidence is a good first step.  But it's only a first step.

From: NIPS.cc/2018/Workshop/IRASL/Paper62/Authors
title: We will provide more explanations on how to read our visualizations, what to pay attention to, and expand on our musical analysis
comment: Thank you for spending the time to review our workshop paper and visualizations.  We will put more focus on explaining the visualizations and how they are musically relevant, for example how the attention pays attention harmonically to chord progressions (along with voice-leading) to generate music with phrasing, melodically to motifs to enable repetition and variation, and to scale-like passages to stay in key.

From: NIPS.cc/2018/Workshop/IRASL/Paper62/Authors
title: Our main contribution is our interactive visualization tool, ready for demo!
comment: We invite readers to view a website of video animations and interactive visualizations at
https://storage.googleapis.com/nips-workshop-visualization/index.html
(also linked to in the workshop paper).

The visualizations are interactive and you can click on any note to see what the model was attending to when it generated that note.  You can also use the left and right arrow key (on the keyboard) to step through the music.

We realize the visualization can be overwhelming at first, and hard to know where to start and what to focus on.  We are working on showing a heatmap-like "map" to highlight where the interesting parts are.  For future work, we think it will be interesting to cluster/segment the music into larger chunks to show higher-level musical structure. 

We imagine this can also be a very useful tool for text, allowing the viewer to "play through" sentences and paragraphs to see the causal structure of how earlier parts of the paragraph influences later generative decisions.

From: NIPS.cc/2018/Workshop/IRASL/Paper62/Authors
title: We will rebalance our paper and provide more details on our examples, and include more references
comment: Thank you for your review and for giving us many help suggestions and references!  We completely agree that we could have summarized the text on describing the prior model and left more space to go through our examples.  We will elaborate on them in the next version.

From: NIPS.cc/2018/Workshop/IRASL/Paper62/Authors
title: Animating the self-attention visualizations with the music playback could make it easier to find correspondences
comment: Thank you for taking the time to review our workshop paper!  This is a short paper, documenting work-in-progress.  Our main contribution is our visualization tool, and we hope the community will find it useful as an exploration tool to form hypotheses on how attention works.  

We implemented an animation feature because it allows the viewer/listener to quickly play through the attention steps and be able to pause whenever they see something interesting.  This also allows the listener to see the visualizations in the context of the music, and hopefully make it easier to find correspondences. 

For the JSB Chorale dataset, we did not know the learned regular attention was practically a local model until we visualized it, especially since the model architecture allows it to attend globally.  From the samples we knew it had less interesting chord progressions but we did not know why.  Our visualization tool allowed us to "debug" and see this very clearly.  The animation allowed us to see how consistent this is throughout the piece, and not just for certain timesteps.

From: NIPS.cc/2018/Workshop/IRASL/Paper62/AnonReviewer1
title: Interesting application, but the provided results are not  convincing
review: The abstract promises a very interesting application of visualizing attention for understanding music. 

I however found the paper little difficult to read, and don't understand anything from the visualizations provided on the website. I am confused regarding the visualizations, and not sure what sort of musical conclusions we should draw out of them. 

I commend the authors for putting in the effort to build the visualization tool. If somehow authors can demonstrate meaningful musical conclusions in future re-iterations, I do think this can turn out to be a useful tool for analyzing music. 
rating: 2: Marginally below acceptance threshold
confidence: 2: The reviewer is fairly confident that the evaluation is correct

From: NIPS.cc/2018/Workshop/IRASL/Paper62/AnonReviewer2
title: Interesting visualizations, still seeking clarity
review: The attention visualizations are quite interesting, but it is unclear what exactly they *mean*, in a musical sense. Taking one or two examples, and showing clearly what the relative connections mean would be beneficial for understanding. As it stands, the visuals are compelling. Seeing the "attention deltas" or the difference in what is attended (perhaps thresholded on magnitude), over time could also be useful, as there seem to be a few examples where the attention is relatively static or bounces between two sets of notes in the far past. Exponentially weighted moving averages of the attention weights themselves, over time might be another interesting way to see if the model has a "favorite" part when composing future steps.

It would also be interesting to see gradient based visualizations, and how they relate to the attention visualizations. Some examples from Alex Graves' talk at the RAM workshop for NIPS 2015 can be seen at http://www.thespermwhale.com/jaseweston/ram/slides/session2/Smooth%20Operators-NIPS2015.pptx

Applying this setup to some music which is decidedly "non-Bach" or "non-classical", to see if the model can find motifs or riffs which are classically inspired would be another useful setting for analysis.

It be nice to beef up the citations a bit, as there have been many other works analyzing attention patterns in other domains (especially text). This would help interested readers find further reading about visualization, analysis and interpretation both inside the transformer framework, attention models at large, and in deep learning generally. I will add a few examples below, which the authors can use or not at their discretion - there may also be secondary references from these that prove useful.

Seq2SeqVis (they also have many relevant citations here): https://arxiv.org/abs/1804.09299
Get To The Point: Summarization with Pointer-Generator Networks (esp. the appendix): https://arxiv.org/abs/1704.04368 
Smooth Operators: the Rise of Differentiable Attention in Deep Learning (linked above): http://www.thespermwhale.com/jaseweston/ram/slides/session2/Smooth%20Operators-NIPS2015.pptx

The focus of the paper (from the abstract) is visualization, but the bulk of the text itself is about the minutiae of self-attention and relative embeddings (which has been fully described in past papers). Maybe there is a way to "rebalance" the focus of the paper, or spend more text space "breaking down" a particular example? This would greatly strengthen the paper.

As it stands the paper alone feels somewhat incomplete - the website is incredibly thorough, but without a paper or section to guide my understanding, the visualizations are more "pretty pictures" than "useful visualizations" for me personally. Just visualizing the internals of transformer doesn't really help me directly understand what parts are musically useful or meaningful in composition. Specifically the use case "... we use it to analyze existing music to see if the resulting self-attention structure corroborates with the musical structure known from music theory." hasn't been shown in great detail for the current iteration of the paper. 

The website is extensive, and clearly took a lot of effort - with some extra care the paper can be similarly impressive, as the tools are already built! I look forward to seeing more work in this vein of interpreting and analyzing powerful generative models, in order to make stronger tools for creativity.

Detail:
I don't like the use of "checkout" in the last line of the abstract, suggested a change here but again leave it to author discretion

"We invite the reader to checkout video animations of music attention and interact with the visualizations" -> "We invite readers to view a website of video animations and interactive visualizations at"
rating: 3: Marginally above acceptance threshold
confidence: 3: The reviewer is absolutely certain that the evaluation is correct and very familiar with the relevant literature

From: NIPS.cc/2018/Workshop/IRASL/Paper62/AnonReviewer3
title: Paper not finished?
review: It appears that the paper is not finished.  

I like the idea of seeing an attention mechanism in action. The paper describes a nice system.

But what are we to learn from this animation? The paper talks about some short-term versus relative connections, but I don't know how to interpret that result.  Does the animation help?
rating: 1: Strong rejection
confidence: 2: The reviewer is fairly confident that the evaluation is correct