From: NeurIPS.cc/2022/Workshop/SVRHM/Paper41/Authors
title: Response to Reviewer ir2Q
comment: We thank the reviewer for their evaluation and feedback.

Response to major points:
- Random projection versus random subselection: This is a great idea, and we've tried it before (though not for this project). Given that the fitted coefficients are relatively uniform across the features from the best performing layers, it seems likely (though not a given) that the majority of features are important. Random subselection would constitute a major speedup for the pipeline.
- Error analysis: Given time constraints, we we might not be able to include the image-by-image analysis in the current manuscript, but we've done something similar (with different image *supercategories* -- faces, landscapes, scenes, et cetera) in a previous work, referenced below [1]. The overarching question -- which images does the model predict well and which does it not? -- is an area of active investigation for us.
- Meaningful baselines and categories: There are a number of meaningful baselines we didn't have the space to discuss, but which will appear in a future version of the manuscript. Baselines we've considered include fully random features (a la Rahimi & Recht), and other image-computable features such as GIST features. However, the baseline the author mentions, that of image category, is one the particular dataset we use (Oasis) allows us somewhat a priori to dispense, since the Oasis images were selected to include a range of valence and arousal *within* individual categories. Dog, for example, includes both an ecstatic golden retriever and a deceased german shephard it's best not to describe in further detail (all images of Oasis are publicly available for further scrutiny). The reviewer is right though directly performing this kind of analysis with category labels could still be illuminating.

Minor: 
 - Supervised probing: The reviewer is right that our primary probing method requires a supervised parametric mapping between deep net features and affect ratings. We are currently experimenting with nonparametric mappings, but these still require labels to some extent? We will try and allude to this in manuscript.

References: 

[1] Conwell, Colin and Graham, D and Vessel, EA (2021) The Perceptual Primacy of Feeling: Affectless machine vision models robustly predict human visual arousal, valence, and aesthetics. Psyarxiv: https://psyarxiv.com/5wg4s/

From: NeurIPS.cc/2022/Workshop/SVRHM/Paper41/Authors
title: Response to Reviewer hMvm
comment: We thank the reviewer for their evaluation and feedback.

Per the point about testing the limits of the neural network decoding approach, well-taken! One of the main candidate limits we're now testing are limits imposed by experiential subjectivity (a self-supervised visual learner trained on natural image statistics will never be able to predict the aesthetic ratings of a woman that loves Rothko because her mother's name is Rothka). Stay tuned!

Per the point about the use of a single lambda parameter, again, well-taken. In other similar projects, we typically use a nested cross-validation procedure that involves first splitting the data in half, then performing a lambda grid search on the training half, before evaluating the test set with leave one-out-cross-validation. Early piloting with different kinds of nested cross-validation suggested the lambda mattered far less than we suspected in comparisons between models, but the reviewer is right that we should revisit this point. A follow-up validation is a good suggestion.

On the minor nits:
- Valence and arousal ratings: We have the results for both of these, but didn't include them for narrative clarity and space!
- Last layer, best layer?: The final layer is best for the majority of models, but not all. (We've now mentioned this in the caption.)
- Subject bootstrapping: The reviewer is absolutely correct that assessing variance over multiple retrainings of the same model is the variance that matters here. With more computational resources, we would do exactly this.
- 1B-C: Fixed!
- The reviewer is correct that we do not fully delineate the relationship between the number of SRPs and $R^2$, and a parameter search over the epsilon distortion (inasmuch as it dictates the number of SRPs) or even just over a number of handselected SRPs is one we should consider. While we did not include this in the writeup due to space constraints, we have in the past replicated a number of analyses on the best layers of each model with and without SRPs, finding the maximum difference to be less than 0.05 $r_{Pearson}$. We should probably rereplicate this, and include it in future versions of this work.

From: NeurIPS.cc/2022/Workshop/SVRHM/Paper41/Authors
title: Response to Reviewer mynZ
comment: We thank the reviewer for the evaluation and feedback!

Responses to specific points below:

Quality:
- We share a similar concern regarding the use of a single lambda parameter for all models. In other similar projects, we typically use a nested cross-validation procedure that involves first splitting the data in half, then performing a lambda grid search on the training half, before evaluating the test set with leave one-out-cross-validation. For the particular datasets we use, however, this leaves us with very little data. In future work, we plan on including multiple forms of cross-validation and demonstrating the effect of a varying lambda.
- We see the ambiguity the reviewer is referring to in the word "mimics" in the sentence on decoding. We'll try and rephrase this for the final manuscript.

Clarity:
- A schematic would definitely facilitate better understanding of the pipeline in Section 2. While beyond the scope of the current submission, we plan on making this kind of schematic for the ultimate product.
- We've now added an explanation to the figure caption that clarifies what these lines are (they are the lines that correspond to the 3 different sizes -- small, base, large -- of the SLIP vision transformers).

Originality:
- The "previous work" we referred to is actually our own, but we elided the reference to preserve anonymity. The reference will be included in the final camera ready version.

Significance:
- The reviewer is correct in pointing out all the ways that fully separating perception from language is difficult, especially since there seem to be a number of covariance structures shared between the two; however, to the extent that these models are trained in the absence of explicit learning signals that directly enforce the learning of these covariances across modalities is the extent to which we can empirically isolate them. We now have models that learn solely from language (at least, the transition probabilities of language) and vision (in visual self-supervised algorithms). While they may shared learned representational structure, they are not directly trained to do so. This is mainly what we mean by disambiguating vision from language.

From: NeurIPS.cc/2022/Workshop/SVRHM/Paper41/Authors
title: Response to Reviewer vVV3 
comment: We thank the reviewer for their evaluation and feedback.

Response to comments and suggestions:

"Subjectivity and individual ratings": The reviewer is absolutely right that the group-averaging in a major sense undermines the subjective aspects of aesthetic evaluation. While we didn't have space to put these results here, in other work, we are actively assessing the fit of vision and language models to individual subject ratings. The long and short of the results is that we can predict some individual ratings extremely well, and others not at all. We're currently testing a few hypotheses about why this might be so, but one of our hypotheses we're preliminarily calling "automated hipster detection". Stay tuned for more!

"Task performance versus affect predictivity": Like in work using deep nets to predict brains, we often see a strong initial spike in predictions of affect as a function of task performance (e.g. ImageNet classification), but this trend very quickly plateaus with the newest models (which tend to cohere pretty strongly in their task performance). In short, because we have somewhat saturated many standard computer vision and language benchmarks, it's somewhat unclear how certain kinds of task performance relate to affective prediction.

"Using human caption with GPT2": Human-generated captions (and human evaluation of the aesthetic value of images associated with a given caption) is definitely one the more straightforward of the ways we can evaluate the ineffability of aesthetic preference. We hope to do this in future work!

From: NeurIPS.cc/2022/Workshop/SVRHM/Paper41/Authors
title: Response to Reviewer 6z4w
comment: We thank the review for their evaluation and feedback!

Per your comment on lines 122-123: It is true in some sense that the caption+prefix embedding is effectively superfluous since all the information necessary to make the caption is available in the prefix. This was, on the other hand, something that might only be obvious in hindsight. We suspected that the caption+prefix embedding to potentially be more predictive in the same way that recurrence can sometimes disambiguate initially noisy signals. We've also seen instances in which concatenating representations from earlier and later layers of a deep net is sometimes more predictive than either of the individual layers alone. We thought it might be the case that there is information in the early to mid layers that is lost by the time the full caption is produced. The results bore out neither of our intuitions, but we included them as part of the process if others had similar intuitions.

Figure 2 caption error: Fixed in the final manuscript!

From: NeurIPS.cc/2022/Workshop/SVRHM/Paper41/Reviewer_6z4w
title: Novel way to disambiguate the role of visual and language features in aesthetic judgments
review: Summary:

The paper tries to show that visual aesthetic experience is a multi-modal phenomenon, shaped by both visual perception and language. For this, the authors use computational modelling to disentangle the role of visual and linguistic features in explaining human aesthetic ratings of natural images. The primary findings are:

1.Visual features alone can explain around 75 % of the variance in average human aesthetic ratings but adding language features can explain up to 87% of the variance in human ratings.
2.To further understand how visual and language features interact to improve the prediction power of aesthetic ratings, the authors used a technique where they use the embedding learned by a CLIP model to disentangle the contributions of visual and language features. For this, they extract a prefix embedding that captures the features from both vision and language. Further, they use this embedding to generate captions of images using GPT2. Now they compare the predictive power of layer activations in GPT 2 model, when it is fed (a) the prefix embedding along with the caption, (b) the prefix embedding without the caption, and (c) the caption alone. They found that conditions (a) and (b) resulted in equal predictive power as CLIP (~85%). However, (c) only had a predictive power of 38.6%. The main take away is that language features can capture a non-trivial portion of variance in aesthetic ratings.

Pros:

1.This paper provides a novel way to approach a very complex topic and demonstrates how one can harness the power of deep learning models to infer mechanisms of how humans perceive aesthetics. As such the topic is very relevant to the theme of the workshop and is an important line of work. 
2.The paper is well-written, and the results are easy to understand.
3.The design, implementation and analysis were expertly conducted.  

Comment:

1.Lines 122-123: This finding appears to be an artifact of how the analysis was set up. The prefix embedding, which comes from CLIP, presumably contains information that correlates with both visual and language content in the images (since that is how CLIP is trained). The fact that the predictive power of “prefix embedding only” matches that of CLIP suggests that the CLIP-Cap method can harness the full expressive power of CLIP in terms of explaining variation in aesthetic ratings.  Now, when GPT 2 is fed the “prefix embedding only”, it will generate a caption that is based on this information. Thus, the information captured by the caption needs to be equal to or a subset of the information present in the prefix embedding. The “caption only” condition only accounted for 38.6% of the variance in aesthetics ratings. This result shows that converting the preferred embedding to a caption by GPT 2 results in partial loss of information reducing the predictive power from 84.8% to 38.6%. Since the caption only condition captures a subset of the variance explained by the “prefix embedding” condition, the “caption+prefix embedding” condition as well as the “prefix embedding only” conditions are expected to have the same predictive power (as was shown in the paper). Based on this argument, one might draw the same conclusions as in the paper without having to use the “caption+prefix embedding” condition.  While this does not change any findings of the paper, it may require re-writing the section from line 122-126. 

Minor error:

1. Figure 2 caption: The facet references in the text seem to be reversed. The left facet consists of the red trace, while the right facet consists of the green, blue and purple traces. 

rating: 7: Good paper, accept
confidence: 3: The reviewer is fairly confident that the evaluation is correct

From: NeurIPS.cc/2022/Workshop/SVRHM/Paper41/Reviewer_vVV3
title: Fascinating approach, many questions remain
review: This paper aims to predict human ratings of image aesthetics, using features extracted from several different deep neural network models pre-trained with different objectives, to dissociate the contributions of vision and language in aesthetic judgment. The paper is extremely well written and the model comparisons have been very thoughtfully chosen to control for as many factors as possible.  

Below, I offer a few suggestions / comments, which I hope will be helpful for strengthening this very interesting paper.

  - The authors allude to the "inherent subjectivity of affective self-report" but avoid this subjectivity by relying on group-averaged aesthetic judgments. While it is certainly fascinating and unexpected that these DNN models can explain so much of the variance in these group averaged aesthetic judgments, this begs the question of how much interesting variance is being excluded by the group-averaging process. That is, would these vision-trained or vision&language-trained models do as well at predicting individual subjects' aesthetic judgments? 
  - I appreciate the authors' using the SLIP family of models to fairly compare training objectives, while holding constant the effect of dataset size and architecture. 
  - I would be curious how the relative performance of these models at the aesthetic prediction relates to the relative performance of these models at a more standard computer vision task (e.g. Imagenet top-1 performance). This might help to determine whether the aesthetic judgment behavioral task is actually capturing anything more than a standard visual discrimination/categorization task.
  - The experiment using the CLIP-GPT2-caption only embedding is quite interesting and helps to make the argument about the "ineffability" of beauty judgments, though I'm not entirely certain it is a fair comparison given the number of additional steps required to get that representation. Perhaps asking humans to generate a verbal caption and then using that as input to a GPT2 embedding might more directly capture the effect of language alone. It would also be interesting to see how humans rate the aesthetics of an image caption, without being able to see the image it is associated with, and how consistent these ratings would be with human ratings of the image itself.
rating: 8: Top 50% of accepted papers, clear accept
confidence: 4: The reviewer is confident but not absolutely certain that the evaluation is correct

From: NeurIPS.cc/2022/Workshop/SVRHM/Paper41/Reviewer_mynZ
title: An attempt at disentangling vision and language with differences in training
review: This paper takes a dataset of human ratings of aesthetics and tries to partial out the difference between visual and linguistic representations by using neural networks trained with self-supervision and those trained with language alignment. 

Quality: 
* One methodological concern is the use of only a single $\lambda$ parameter across all of the models/layers, chosen based on another model. Depending on the distribution of model activations this choice could dramatically change the predictions, and I was left wondering if that alone could explain the results in Figure 2 and Figure 3.   
* The statement in lines 50-51 seems unfounded -- it seems like an open question of whether this type of decoding "mimics" how information is decoded from neurophysiological signals (and even if true, it is likely a simplification)?

Clarity: 
* The initial results on Unimodel and Multimodel visual models are fairly clear (and the Schematic in Figure 1S helps) however the section on “Language Models via Captions” is very hard to follow. In fact, the authors refer to their own approach as “somewhat oblique” in the appendix (which still did not help the clarity or motivation of this section). Including a schematic which has the inputs/outputs of the different models that are chained together would improve the paper. 
* Why are there multiple lines for each model on Figure 1B Left? This is not explained in the caption. 

Originality:
* Using different training of these models to capture aspects of language vs. perception and relate this to aesthetics seems original.  
* On line 69 the authors state that their unimodal vision model is “recapitulating previous work” but do not provide a citation. What previous work are they referring to and how does their work extend on it? 

Significance: 
* Building image-computable models of human judgements of complex features such as “aesthetics” which are hard to directly define is an interesting direction, and using the features of neural networks for this purpose seems promising. 
* Fully separating language from vision is in these models seems tricky to nail down, because it has been shown that the readouts of the final layers of visual trained models are able to support tasks such as classification, and in fact the models that are released publicly are chosen to have this feature. Because of this, the statement that their results show “a general methodology for disambiguating perceptual and linguistic abstractions” seems too strong. 
rating: 5: Marginally below acceptance threshold
confidence: 4: The reviewer is confident but not absolutely certain that the evaluation is correct

From: NeurIPS.cc/2022/Workshop/SVRHM/Paper41/Reviewer_hMvm
title: Review
review: This paper investigates the extent to which neural network representations can be used to predict human aesthetic ratings for images. It finds that a SEER-pretrained RegNet64 model can explain 75% of the explainable variance in this dataset, but OpenAI's CLIP models perform even better, explaining 87% of the variance. A set of experiments with models trained on the same dataset using either SimCLR, CLIP, or SLIP training shows that language pretraining generally provides a small but significant boost in accuracy. However, using CLIP-Cap to try to generate new captions from the images does not lead to any improvement.

Overall, this is an interesting study and the experimental setup is appropriate. I appreciate that the authors have controlled for the effect of training data by investigating SLIP models. A longer version of this work might attempt to provide more interpretable insights into what aspects of aesthetics can and cannot be captured by neural network representations or through text, but I recognize that there are space constraints given the venue.

My only major concern is that, from the appendix, it seems like, when fitting ridge regression models, the authors consider only a single value of the regularization hyperparameter, which was selected using a different network from the networks studied here. However, representations from different networks could require different regularization strengths to achieve optimal predictive power. It would be better to use some sort of nested cross-validation to pick the optimal regularization strength for each representation. It would also suffice to validate that the selected regularization strength is close to optimal for all models, if that is indeed the case.

Minor nits:

- According to L44, the dataset investigated has valence, arousal, and aesthetic ratings. Why not repeat this analysis for the valence and arousal ratings as well?
- In Figure 1B (left), it seems like the final layer is best for many models, but I can't tell if it's best for all models. Since which layer is optimal seems like the most important piece of information in the plot, it would be helpful to clarify it, either visually or in the text.
- The significance testing on L100 resamples the human subject pool but doesn't account for variance due to initialization used for pretraining, so it's not useful if the goal is to compare SimCLR pretraining to SLIP pretraining (vs. comparing these specific SimCLR models to these specific SLIP models). However, eyeballing Figure 1B, the difference between SimCLR and SLIP is probably significant for each of the model sizes individually, which probably means it is robust to initialization.
- L102 refers to "Figure 1B-C" but there is no Figure 1C.
- I am not entirely convinced that the sparse random projection doesn't affect the results. Clearly, with enough dimensions, $R^2$ converges to the correct value, but the current analysis bounds only the distortion of the embeddings and not of $R^2$ as a function of the number of dimensions, and some additional analysis is necessary to bound the error in $R^2$. It would be good to derive a bound for $R^2$, to empirically validate that the number of features used for the sparse projection has no effect on the results, or simply to solve the problem exactly. For $X \in \mathbb{R}^{n \times D}$ with $n = 900, D = 24\times 24\times 1024$ (the size of a CLIP L/14@336 representation), it should be tractable to compute $XX^\top$ and then either use that as a kernel matrix for kernel ridge regression or take its (matrix) square root and use that as a $900 \times 900$ feature matrix.
rating: 7: Good paper, accept
confidence: 4: The reviewer is confident but not absolutely certain that the evaluation is correct

From: NeurIPS.cc/2022/Workshop/SVRHM/Paper41/Reviewer_ir2Q
title: Interesting, sound, standard parts & sections possibly missing
review: 
**Major**:

- The statistical testing and idea behind the pipeline (feature extraction, dimension reduction, training) seems sound. For the dimensionality reduction, the authors used sparse random projects. This seems well-motivated theoretically in the appendix. However, for some tasks, it has been shown [1-3] that just selecting a random subset is also a viable strategy which can outperform the sparse and gaussian random projections. Would it be possible for the authors to try it or do they consider this to not be the performance bottleneck of the pipeline?
- The error analysis for more intuitive understanding of the problem seems to be missing. Would it be possible for authors to include the following images (and associated discussion) in the appendix?
  1. Cases where the prediction is aligned with the human judgement?
  2. Cases where the prediction differs from the human judgement?
- Baseline or any other meaningful comparison seems to be missing. Although it is not the goal of the paper to show the possibly high quality of the probes and rather compare them among the models of different modalities, having a baseline to compare would be very beneficial to a reader. E.g. 60% explainable variance explained may be both very low (if close to the baseline) or surprisingly high. How high would the baseline of just the image class go? I imagine that the class of _cat_ has on average much higher ranking than _house_.
- Explanation of the showed results seems to be missing. Do the probes work because they essentially detect classes (see previous point)? Why do the probes get better the deeper you go to the model?

**Minor**:
- The general approach appears to be supervised probing, which should be more explicitly acknowledged.
- The large figure captions are hard to parse but this may be justified given the page requirements.
- I enjoyed the writing which seems to be without any typos and seems clear and well motivated.
- The research question of multimodality seems interesting.

**References (for dimensionality reduction)**:
- [1] Padmaja, D. Lakshmi, and B. Vishnuvardhan. "Comparative study of feature subset selection methods for dimensionality reduction on scientific data." 2016 IEEE 6th International Conference on Advanced Computing (IACC). IEEE, 2016. [link](https://www.researchgate.net/profile/Lakshmi-Padmaja-Dhyaram/publication/306304009_Comparative_Study_of_Feature_Subset_Selection_Methods_for_Dimensionality_Reduction_on_Scientific_Data/links/61e5a2468d338833e37690af/Comparative-Study-of-Feature-Subset-Selection-Methods-for-Dimensionality-Reduction-on-Scientific-Data.pdf)
- [2] Huertas-García, Álvaro, et al. "Exploring Dimensionality Reduction Techniques in Multilingual Transformers." arXiv preprint arXiv:2204.08415 (2022). [link](https://arxiv.org/pdf/2204.08415.pdf) 
- [3] Zouhar, Vilém, et al. "Knowledge Base Index Compression via Dimensionality and Precision Reduction." Proceedings of the 1st Workshop on Semiparametric Methods in NLP: Decoupling Logic from Knowledge. 2022. [link](https://arxiv.org/pdf/2204.02906.pdf)
rating: 6: Marginally above acceptance threshold
confidence: 4: The reviewer is confident but not absolutely certain that the evaluation is correct