In this task, you will be presented with a context from an academic paper and a question based on the context. You have to classify the questions into "Extractive", "Abstractive", or "Yes-no" questions. Extractive questions can be answered by concatenating extracts taken from a context into a summary while answering abstractive questions involves paraphrasing the context using novel sentences. Yes-no question is a question whose expected answer is one of two choices, one that affirms the question and one that denies the question. Typically, the choices are either yes or no.

However, all previous IE benchmarks BIBREF18 are too small to train neural network models typically used in QA, and thus we need to build a large benchmark. Therefore, we build a large scale benchmark named QA4IE benchmark which consists of 293K Wikipedia articles and 2M golden relation triples with 636 different relation types. We manually find 148 relations which can be projected to a WikiData relation out of 2064 DBpedia relations.
Question: Was this benchmark automatically created from an existing dataset?
Yes-no

A quantitative measure is required to reliably evaluate the achieved improvement. One of the methods proposed to measure the interpretability is the word intrusion test BIBREF41 . But, this method is expensive to apply since it requires evaluations from multiple human evaluators for each embedding dimension. In this study, we use a semantic category-based approach based on the method and category dataset (SEMCAT) introduced in BIBREF27 to quantify interpretability. Specifically, we apply a modified version of the approach presented in BIBREF40 in order to consider possible sub-groupings within the categories in SEMCAT.
Question: What experiments do they use to quantify the extent of interpretability?
Abstractive

As a first baseline we use Bag-of-Words, a well-known and robust text representations used in various domains BIBREF21 , combined with a standard shallow classifier, namely, a Support Vector Machine with linear kernel. We used LIBSVM implementation of SVM.
Question: Which shallow approaches did they experiment with?
Extractive