Teacher:In this task, you are given a paper review. Based on the review, your job is to identify language and generate "en" if the review is in English or generate "es" if the review is in Spanish. Note that URLs in the text have been replaced with [Link].
Teacher: Now, understand the problem? Solve this instance: - Strengths: *- Task *- Simple model, yet the best results on SQuAD (single model0 *- Evaluation and comparison - Weaknesses: *- Analysis of errors/results (See detailed comments below) - General Discussion: In this paper the authors present a method for directly querying Wikipedia to answer open domain questions. The system consist of two components - a module to query/fetch wikipedia articles and a module to answer the question given the fetched set of wikipedia articles. The document retrieval system is a traditional IR system relying on term frequency models and ngram counts. The answering system uses a feature representation for paragraphs that consists of word embeddings, indicator features to determine whether a paragraph word occurs in a question, token-level features including POS, NER etc and a soft feature for capturing similarity between question and paragraph tokens in embedding space. A combined feature representation is used as an input to a bi-direction LSTM RNN for encoding. For questions an RNN that works on the word embeddings is used. These are then used to train an overall classifier independently for start and end spans of sentences within a paragraph to answer questions. The system has been trained using different Open Domain QA datasets such as SQuAD and WebQuestions by modifying the training data to include articles fetched by the IR engine instead of just the actual correct document/passage. Overall, an easy to follow interesting paper but I had a few questions: 1) The IR system has a Accuracy@5 of over 75 %, and individually the document reader performs well and can beat the best single models on SquAD. What explains the significant drop in Table 6. The authors mention that instead of the fetched results, if they test using the best paragraph the accuracy reaches just 0.49 (from 0.26) but that is still significantly below the 0.78-79 in the SQuAD task. So, presumably the error is this large because the neural network for matching isnt doing as good a job in learning the answers when using the modified training set (which includes fetched articles) instead of the case when training and testing is done for the document understanding task. Some analysis of whats going on here should be provided. What was the training accuracy in the both cases? What can be done to improve it? To be fair, the authors to allude to this in the conclusion but I think it still needs to be part of the paper to provide some meaningful insights. 2) I understand the authors were interested in treating this as a pure machine comprehension task and therefore did not want to rely on external sources such as Freebase which could have helped with entity typing but that would have been interesting to use. Tying back to my first question -- if the error is due to highly relevant topical sentences as the authors mention, could entity typing have helped? The authors should also refer to QuASE (Sun et. al 2015 at WWW2015) and similar systems in their related work. QuASE is also an Open domain QA system that answers using fetched passages - but it relies on the web instead of just Wikipedia.
Student:
en