Paper:
Mapping cognitive ontologies to and from the brain
Imaging neuroscience links brain activation maps to behavior and cognition via correlational studies. Due to the nature of the individual experiments, based on eliciting neural response from a small number of stimuli, this link is incomplete, and unidirectional from the causal point of view. To come to conclusions on the function implied by the activation of brain regions, it is necessary to combine a wide exploration of the various brain functions and some inversion of the statistical inference. Here we introduce a methodology for accumulating knowledge towards a bidirectional link between observed brain activity and the corresponding function. We rely on a large corpus of imaging studies and a predictive engine. Technically, the challenges are to find commonality between the studies without denaturing the richness of the corpus. The key elements that we contribute are labeling the tasks performed with a cognitive ontology, and modeling the long tail of rare paradigms in the corpus. To our knowledge, our approach is the first demonstration of predicting the cognitive content of completely new brain images.
To that end, we propose a method that predicts the experimental paradigms across different studies.

Review:
The authors provide a methodology for analyzing fMRI 
      data from a database containing 19 studies, which allows making reverse 
      inferences of brain regions specifically related to a given term, as 
      opposed to forward inferences that are usually used in fMRI studies but 
      that do not provide specificity. The authors apply this method to predict 
      cognitive ontology terms for new brain images (not used in training). 
      Differently from ref.[28] (Yarkoni et al., 2011), the authors do not use 
      peak coordinates text-mined from journal articles, but analyze actual fMRI 
      contrasts annotated with specific terms. Multivoxel pattern 
      recognition studies and automated meta-analyses over numerous neuroimaging 
      experiments are becoming increasingly popular and it's only a question of 
      time when they will become a common practice in the neuroimaging field, 
      which still suffers from problems of low statistical power, simplistic 
      assumptions (of candidate regions) and limited comparability. The current 
      paper provides an interesting methodology for obtaining reasonably 
      accurate reverse inferences using a relatively small number of studies 
      with few subjects. Although a similar database has been used for 
      classification of mental states in ref.[18] (Poldrack et al., 2009), the 
      current paper focuses on several difficult issues not resolved in [18], 
      such as predictability across different studies, mitigating various 
      confounding effects, and using cognitive ontology terms instead of classes 
      based on task/study. Although the paper is good and appropriate 
      for NIPS, it could be improved in several ways: - the last sentence of 
      the abstract is too general and a more specific claim of novelty (such as 
      predicting ontology terms across different studies) should be used; - 
      to my knowledge, ref.[28] does not only use Talairach coordinates, as 
      stated in the beginning of para.2 of the introduction; - assertion in 
      section 3.1 that images from many different teams are used in the database 
      does not seem to be very accurate, as the vast majority of studies seem to 
      have only 2 different senior authors (one of which is sometimes the first 
      author). More specific information regarding the number of different 
      scanners and truly independent study designs should be reported. It should 
      also be discussed whether prediction across different studies is 
      successful also for these truly independent studies (i.e. with no related 
      study being in the training set) or only for similar studies (when a 
      training set contains at least one study with shared authors, scanners or 
      designs). Currently only mean performances are reported, which are not 
      very informative in this regard. - It would be useful to discuss why 
      "buttons" and not "saccades" are inversely correlated with "none" (section 
      4.1); - measures of "precision" and "recall" should be explicitly 
      defined in the text or footnotes even if they are standard in certain 
      fields; - what are the chance levels for predictions in fig.2? - 
      without them it's very difficult (or impossible) to evaluate the quality 
      of predictions. - it is not clear what are the criteria of 
      classification into 3 "operating regimes" in fig.2 and what exactly do 
      these regimes mean (if nothing particular, such classification should be 
      avoided); - what is the rationale behind choices of coordinates for 
      brain slices in fig.3 maps? In addition, the following 
      typos/errors should be corrected: - in the abstract it should be 
      "nature" instead of "natural" (line 2) and "bidirectional" better than 
      "bilateral" (line 8); - in the 3rd sentence of section 4.1 (lines 3-4) 
      I suppose it should be "visual" instead of "not visual"; - in the 5th 
      line from the end of page 7, it should be "albeit" instead of "ableit"; 
      - on page 8, there should be "fewer" instead of "less" in line 5 of 
      para.1 and the last line of para.2, "many" instead of "much" in the last 
      line of para.1 and "whereas" instead of "will" in line 9 of para.2. 
      Finally, it's more conventional for references to be numbered in 
      the order of their presentation, not alphabetically. --- 
      The authors addressed most of my major concerns regarding 
      weaknesses of the study, therefore, assuming their responses and 
      additional data are incorporated into the final version and that minor 
      points are corrected as well, this is a solid paper that should be useful 
      for the NIPS community. A few more points regarding author responses: 
      - The analysis of training on Dehaene's data / testing on Poldrack's 
      and vice versa is interesting and should be mentioned in the final version 
      as well. It is not clear which term is "read" (no such term appeared in 
      the paper) More detailed explanations regarding the substantial 
      deterioration of "auditory" scores would be helpful. - Chance levels 
      should also be provided for the Poldrack vs. Dehaene testing. It is also a 
      bit strange how the "saccades" term with both precision and recall close 
      to 0 (fig. 2) can be above chance. - I agree with Reviewer 3 that fig. 
      3 should be presented/explained in a more informative way, indicating 
      statistical significances. As the involvement of cerebellum in cognition 
      is not so well known as in motor response, this result is particularly 
      interesting and it would be useful to discuss it in the text. Regarding 
      the prediction of auditory stimuli by voxels in the visual cortex, it's 
      not clear based on the shown data if the result is genuine, but if it is, 
      it may be related to evidence of multisensory integration between primary 
      sensory areas (see the work of Micah M. 
      Murray)