Definition: In this task, you are given a paper review. Based on the review, your job is to identify language and generate "en" if the review is in English or generate "es" if the review is in Spanish. Note that URLs in the text have been replaced with [Link].
Input: The paper describes a deep-learning-based model for parsing the creole Singaporean English to Universal Dependencies. They implement a parser based on the model by Dozat and Manning (2016) and add neural stacking (Chen et al., 2016) to it. They train an English model and then use some of the hidden representations of the English model as input to their Singlish parser. This allows them to make use of the much larger English training set along with a small Singlish treebank, which they annotate. They show that their approach (LAS 76.57) works better than just using an English parser (LAS 65.6) or training a parser on their small Singlish data set (LAS 64.01). They also analyze for which common constructions, their approach improves parsing quality. They also describe and evaluate a stacked POS model based on Chen et al. (2016), they discuss how common constructions should be analyzed in the UD framework, and they provide an annotated treebank of 1,200 sentences. 100 of them were annotated by two people and their inter-annotator agreement was 85.3 UAS and 75.7 LAS. - Strengths: - They obtain good results and their experimental setup appears to be solid. - They perform many careful analyses and explore the influence on many parameters of their model. - They provide a small Singlish treebank annotated according to the Universal Dependencies v1.4 guidelines. - They propose very sound guidelines on how to analyze common Singlish constructions in UD. - Their method is linguistically informed and they nicely exploit similarity between standard English and the creole Singaporean English. - The paper presents methods for a low-resource language. - They are not just applying an existing English method to another language but instead present a method that can be potentially used for other closely related language pairs. - They use a well-motivated method for selecting the sentences to include in their treebank. - The paper is very well written and easy to read. - Weaknesses: - The annotation quality seems to be rather poor. They performed double annotation of 100 sentences and their inter-annotator agreement is just 75.72% in terms of LAS. This makes it hard to assess how reliable the estimate of the LAS of their model is, and the LAS of their model is in fact slightly higher than the inter-annotator agreement. UPDATE: Their rebuttal convincingly argued that the second annotator who just annotated the 100 examples to compute the IAA didn't follow the annotation guidelines for several common constructions. Once the second annotator fixed these issues, the IAA was reasonable, so I no longer consider this a real issue. - General Discussion: I am a bit concerned about the apparently rather poor annotation quality of the data and how this might influence the results, but overall, I liked the paper a lot and I think this would be a good contribution to the conference. - Questions for the authors: - Who annotated the sentences? You just mention that 100 sentences were annotated by one of the authors to compute inter=annotator agreement but you don't mention who annotated all the sentences. - Why was the inter-annotator agreement so low? In which cases was there disagreement? Did you subsequently discuss and fix the sentences for which there was disagreement? - Table A2: There seem to be a lot of discourse relations (almost as many as dobj relations) in your treebank. Is this just an artifact of the colloquial language or did you use "discourse" for things that are not considered "discourse" in other languages in UD? - Table A3: Are all of these discourse particles or discourse + imported vocab? If the latter, perhaps put them in separate tables, and glosses would be helpful. - Low-level comments: - It would have been interesting if you had compared your approach to the one by Martinez et al. (2017, [Link] Perhaps you should mention this paper in the reference section. - You use the word "grammar" in a slightly strange way. I think replacing "grammar" with syntactic constructions would make it clearer what you try to convey. (e.g., line 90) - Line 291: I don't think this can be regarded as a variant of it-extraposition. But I agree with the analysis in Figure 2, so perhaps just get rid of this sentence. - Line 152: I think the model by Dozat and Manning (2016) is no longer state-of-the art, so perhaps just replace it with "very high performing model" or something like that. - It would be helpful if you provided glosses in Figure 2.
Output:
en