TASK DEFINITION: In this task, you will be presented with a context from an academic paper and a question based on the context. You have to classify the questions into "Extractive", "Abstractive", or "Yes-no" questions. Extractive questions can be answered by concatenating extracts taken from a context into a summary while answering abstractive questions involves paraphrasing the context using novel sentences. Yes-no question is a question whose expected answer is one of two choices, one that affirms the question and one that denies the question. Typically, the choices are either yes or no.
PROBLEM: Validated transcripts were sent to professional translators. In order to control the quality of the professional translations, we applied various sanity checks to the translations BIBREF11. We also sanity check the overlaps of train, development and test sets in terms of transcripts and voice clips (via MD5 file hashing), and confirm they are totally disjoint.
Question: How is the quality of the data empirically evaluated?

SOLUTION: Extractive

PROBLEM: (proposed) Bi-LSTM/CRF + Bi-CharLSTM with modality attention (W+C): uses the modality attention to merge word and character embeddings.

(proposed) Bi-LSTM/CRF + Bi-CharLSTM + Inception (W+C+V): takes as input visual contexts extracted from InceptionNet as well, concatenated with word and char vectors.

(proposed) Bi-LSTM/CRF + Bi-CharLSTM + Inception with modality attention (W+C+V): uses the modality attention to merge word, character, and visual embeddings as input to entity LSTM.
Question: Does their NER model learn NER from both text and images?

SOLUTION: Yes-no

PROBLEM: Before designing this NLP toolkit, we conducted a survey among engineers and identified a spectrum of three typical personas.
Question: How do the authors evidence the claim that many engineers find it a big overhead to choose from multiple frameworks, models and optimization techniques?

SOLUTION:
Abstractive