In this task, you are given a paper review. Based on the review, your job is to identify language and generate "en" if the review is in English or generate "es" if the review is in Spanish. Note that URLs in the text have been replaced with [Link].

Example Input: In this paper, the authors propose a new method to learn hierarchical representations of sentences, based on reinforcement learning. They propose to learn a neural shift-reduce parser, such that the induced tree structures lead to good performance on a downstream task. They use reinforcement learning (more specifically, the policy gradient method REINFORCE) to learn their model. The reward of the algorithm is the evaluation metric of the downstream task. The authors compare two settings, (1) no structure information is given (hence, the only supervision comes from the downstream task) and (2) actions from an external parser is used as supervision to train the policy network, in addition to the supervision from the downstream task. The proposed approach is evaluated on four tasks: sentiment analysis, semantic relatedness, textual entailment and sentence generation. I like the idea of learning tree representations of text which are useful for a downstream task. The paper is clear and well written. However, I am not convinced by the experimental results presented in the paper. Indeed, on most tasks, the proposed model is far from state-of-the-art models: - sentiment analysis, 86.5 v.s. 89.7 (accuracy); - semantic relatedness, 0.32 v.s. 0.25 (MSE); - textual entailment, 80.5 v.s. 84.6 (accuracy). From the results presented in the paper, it is hard to know if these results are due to the model, or because of the reinforcement learning algorithm. PROS: - interesting idea: learning structures of sentences adapted for a downstream task. - well written paper. CONS: - weak experimental results (do not really support the claim of the authors). Minor comments: In the second paragraph of the introduction, one might argue that bag-of-words is also a predominant approach to represent sentences. Paragraph titles (e.g. in section 3.2) should have a period at the end. ---------------------------------------------------------------------------------------------------------------------- UPDATE I am still not convinced by the results presented in the paper, and in particular by the fact that one must combine the words in a different way than left-to-right to obtain state of the art results. However, I do agree that this is an interesting research direction, and that the results presented in the paper are promising. I am thus updating my score from 5 to 6.
Example Output: en

Example Input: - Strengths: -- A well-motivated approach, with a clear description and solid results. - Weaknesses: -- Nothing substantial other than the comments below. - General Discussion: The paper describes a new method called attention-over-attention for reading comprehension. First layers of the network compute a vector for each query word and document word, resulting in a |Q|xK matrix for the query and a |D|xK for the document. Since the answer is a document word, an attention mechanism is used for assigning weights to each word, depending on their interaction with query words. In this work, the authors deepen a traditional attention mechanism by computing a weight for each query word through a separate attention and then using that to weight the main attention over document words. Evaluation is properly conducted on benchmark datasets, and various insights are presented through an analysis of the results as well as a comparison to prior work. I think this is a solid piece of work on an important problem, and the method is well-motivated and clearly described, so that researchers can easily reproduce results and apply the same techniques to other similar tasks. - Other remarks: -- p4, Equation 12: I am assuming i is iterating over training set and p(w) is referring to P(w|D,Q) in the previous equation? Please clarify to avoid confusion. -- I am wondering whether you explored/discussed initializing word embeddings with existing vectors such as Google News or Glove? Is there a reason to believe the general-purpose word semantics would not be useful in this task? -- p6 L589-592: It is not clear what the authors are referring to when they say 'letting the model explicitly learn weights between individual attentions'? Is this referring to their own architecture, more specifically the GRU output indirectly affecting how much attention will be applied to each query and document word? Clarifying that would be useful. Also, I think the improvement on validation is not 4.1, rather 4.0 (72.2-68.2). -- p7 Table 5: why do you think the weight for local LM is relatively higher for the CN task while the benefit of adding it is less? Since you included the table, I think it'll be nice to provide some insights to the reader. -- I would have liked to see the software released as part of this submission. -- Typo p2 L162 right column: "is not that effective than expected" --> "is not as effective as expected"? -- Typo p7 L689 right column: "appear much frequent" --> "appears more frequently"? -- Typo p8 L719-721 left column: "the model is hard to" --> "it is hard for the model to"? & "hard to made" --> "hard to make"?
Example Output: en

Example Input: *** Paper Summary *** This paper simplify matching network by considering only a single prototype per class which is obtained as the average of the embedding of the training class samples. Empirical comparisons with matching networks are reported. *** Review *** The paper reads well and clearly motivate the work. This work of learning metric learning propose to simplify an earlier work (matching network) which is a great objective. However, I am not sure it achieve better results than matching networks. The space of learning embeddings to optimize nearest neighbor classification has been explored before, but the idea of averaging the propotypes is interesting (as a non-linear extension of Mensink et al 2013). I would suggest to improve the discussion of related work and to consolidate the results section to help distinguish between the methods you outperform and the one you do not. The related work section can be extended to include work on learning distance metric to optimize a nearest neighbor classification, see Weinberger et al, 2005 and subsequent work. Extensions to perform the same task with neural networks can be found in Min et al, 09 that purse a goal very close to yours. Regarding approaches pursuing similar goals with a different learning objective, you cite siamese network with pairwise supervision. The learning to rank (for websearch) litterature with triplet supervision or global ranking losses is also highly relevant, ie. one example "the query" defines the class and the embedding space need to be such that positive/relevant document are closer to the query than the others. I would suggest to start with Chris Burges 2010 tutorial. One learning class I am not sure the reported results correctly reflect the state of the art for all tasks. The results are positive on Omniglot but I feel that you should also report the better results of matching networks on miniImageNet with fine tuning and full contextual embeddings. It can be considered misleading not to report it. On Cub 200, I thought that the state-of-the-art was 50.1%, when using features from GoogLeNet (Akata et al 2015), could you comment on this? Overall, paper could greatly be improved, both in the discussion of related work and with a less partial reporting of prior empirical results. *** References *** Large Margin Nearest Neighbors. Weinberger et al, 2005 From RankNet to LambdaRank to LambdaMART: An Overview, Chris J.C. Burges, June 23, 2010 A Deep Non-linear Feature Mapping for Large-Margin kNN Classification, Min et al, 09
Example Output:
en