In this task, you are given a paper review. Based on the review, your job is to identify language and generate "en" if the review is in English or generate "es" if the review is in Spanish. Note that URLs in the text have been replaced with [Link].

Example Input: This paper discusses Hyperband, an extension of successive halving by Jamieson & Talwalkar (AISTATS 2016). Successive halving is a very nice algorithm that starts evaluating many configurations and repeatedly cuts off the current worst half to explore many configuration for a limited budget. Having read the paper for the question period and just rereading it again, I am now not entirely sure what its contribution is meant to be: the only improvement of Hyperband vs. successive halving is in the theoretical worst case bounds (not more than 5x worse than random search), but you can (a) trivially obtain that bound by using a fifth of your time for running random configurations to completion and (b) the theoretical analysis to show this is said to be beyond the scope of the paper. That makes me wonder whether the theoretical results are the contribution of this paper, or whether they are the subject of a different paper and the current paper is mostly an empirical study of the method? I hope to get a response by the authors and see this made clearer in an updated version of the paper. In terms of experiments, the paper fails to show a case where Hyperband actually performs better than the authors' previous algorithm successive halving with its most agressive setting of bracket b=4. Literally, in every figure, bracket b=4 is at least as good (and sometimes substantially better) than Hyperband. That makes me think that in practice I would prefer successive halving with b=4 over Hyperband. (And if I really want Hyperband's guarantee of not being more than 5x worse than random search I can run random search on a fifth of my machines.) The experiments also compare to some Bayesian optimization methods, but not to the most relevant very closely related Multi-Task Bayesian Optimization methods that have been dominating effective methods for deep learning in that area in the last 3 years: "Multi-Task Bayesian Optimization" by Swersky, Snoek, and Adams (2013) already showed 5x speedups for deep learning by starting with smaller datasets, and there have been several follow-up papers showing even larger speedups. Given that this prominent work on multitask Bayesian optimization exists, I also think the introduction, which sells Hyperband as a very new approach to hyperparameter optimization is misleading. I would've much preferred a more down-to-earth pitch that says "configuration evaluation" has been becoming a very important feature in hyperparameter optimization, including Bayesian optimization, that sometimes yields very large speedups (this can be quantified by examples from existing papers) and this paper adds some much-needed theoretical understanding to this and demonstrates how important configuration evaluation is even in the simplest case of being used with random search. I think this could be done easily and locally by adding a paragraph to the intro. As another point regarding novelty, I think the authors should make clear that approaches for adaptively deciding how many resources to use for which evaluation have been studied for (at least) 23 years in the ML community -- see Maron & Moore, NIPS 1993: "Hoeffding Races: Accelerating Model Selection Search for Classification and Function Approximation" (
Example Output: en

Example Input: 26: An End-to-End Model for Question Answering over Knowledge Base with Cross-Attention Combining Global Knowledge This paper presents an approach for factoid question answering over a knowledge graph (Freebase), by using a neural model that attempts to learn a semantic correlation/correspondence between various "aspects" of the candidate answer (e.g., answer type, relation to question entity, answer semantic, etc.) and a subset of words of the question. A separate correspondence component is learned for each "aspect" of the candidate answers. The two key contributions of this work are: (1) the creation of separate components to capture different aspects of the candidate answer, rather than relying on a single semantic representation, and (2) incorporating global context (from the KB) of the candidate answers. The most interesting aspect of this work, in my opinion, is the separation of candidate answer representation into distinct aspects, which gives us (the neural model developer) a little more control over guiding the NN models towards information that would be more beneficial in its decision making. It sort of harkens to the more traditional algorithms that rely on feature engineering. But in this case the "feature engineering" (i.e., aspects) is more subtle, and less onerous. I encourage the authors to continue refining this system along these lines. While the high-level idea is fairly clear to a reasonably informed reader, the devil in the details would make it hard for some audience to immediately grasp key insights from this work. Some parts of the paper could benefit from more explanation... Specifically: (1) Context aspect of candidate answers (e_c) is not clearly explained in the paper. Therefore, the last two sentences of Section 3.2.2 seem unclear. (2) Mention of OOV in the abstract and introduction need more explanation. As such, I think the current exposition in the paper assumes a deep understanding of prior work by the reader. (3) The experiments conducted in this paper restrict comparison to IR-based system -- and the reasoning behind this decision is reasonable. But it is not clear then why the work of Yang et al. (2014) -- which is described to be SP-based -- is part of the comparison. While, I am all for including more systems in the comparison, there seem to be some inconsistencies in what should and should not be compared. Additionally, I see not harm in also mentioning the comparable performance numbers for the best SP-based systems. I observe in the paper that the embeddings are learned entirely from the training data. I wonder how much impact the random initialization of these embeddings has on the end performance. It would be interesting to determine (and list) the variance if any. Additionally, if we were to start with pre-trained embeddings (e.g., from word2vec) instead of the randomly initialized ones, would that have any impact? As I read the paper, one possible direction of future work that occurred to me was to possibly include structured queries (from SP-based methods) as part of the cross-attention mechanism. In other words, in addition to using the various aspects of the candidate answers as features, one could include structured queries that generate the produce that candidate answer as an additional aspect of the candidate answer. An attention mechanism could then also focus on various parts of the structured query, and its (semantic) matches to the input question as an additional signal for the NN model. Just a thought. Some notes regarding the positioning of the paper: I hesitate to call the model proposed here "attention" models, because (per my admittedly limited understanding) attention mechanisms apply to "encoder-decoder" situations, where semantics expressed in one structured form (e.g., image, sentence in one language, natural language question, etc.) are encoded into an abstract representation, and then generated into another structured form (e.g., caption, sentence in another language, structured query, etc.). The attention mechanism allows the "encoder" to jump around and attend to different parts of the input (instead of sequentially) as the output is being generated by the decoder. This paper does not appear to fit this notion, and may be confusing to a broader audience. ------ Thank you for clarifications in the author response.
Example Output: en

Example Input: NO ME ENCUENTRO CAPACITADO PARA EFECTUAR ESTA REVISION. ESTA MUY LEJOS DE MI ESPECIALIZACION.
Example Output:
es