In this task, you will be presented with a context from an academic paper and a question separated with a
. You have to answer the question based on the context.

The base sentence embedding model is a BiLSTM over all words in the respective sequences with randomly initialised word embeddings, following BIBREF30 . We opt for this strong baseline sentence encoding model, as opposed to engineering sentence embeddings that work particularly well for this dataset, to showcase the dataset. We would expect pre-trained contextual encoding models, e.g. ELMO BIBREF33 , ULMFit BIBREF34 , BERT BIBREF35 , to offer complementary performance gains, as has been shown for a few recent papers BIBREF36 , BIBREF37 .
Question: What were the baselines?
a BiLSTM over all words in the respective sequences with randomly initialised word embeddings, following BIBREF30

More than 2,100 texts were paired with 15 questions each, resulting in a total number of approx. 32,000 annotated questions. For 13% of the questions, the workers did not agree on one of the 4 categories with a 3 out of 5 majority, so we did not include these questions in our dataset.

The distribution of category labels on the remaining 87% is shown in Table TABREF10 . 14,074 (52%) questions could be answered. Out of the answerable questions, 10,160 could be answered from the text directly (text-based) and 3,914 questions required the use of commonsense knowledge (script-based). After removing 135 questions during the validation, the final dataset comprises 13,939 questions, 3,827 of which require commonsense knowledge (i.e. 27.4%). This ratio was manually verified based on a random sample of questions.
Question: what dataset statistics are provided?
More than 2,100 texts were paired with 15 questions each, resulting in a total number of approx. 32,000 annotated questions. 13% of the questions are not answerable. Out of the answerable questions, 10,160 could be answered from the text directly (text-based) and 3,914 questions required the use of commonsense knowledge (script-based). The final dataset comprises 13,939 questions, 3,827 of which require commonsense knowledge (i.e. 27.4%).

The second series of tests consisted of using all the documents of the subcorpus "specialist" $E$, because the documents of the subcorpus of Annodis are not identical. As far as system evaluations are concerned, we use the 78 $E$ documents as reference. We calculate the precision $P$, the recall $R$ and the $F$-score on the text corpus used in our tests, as follow:
Question: How is segmentation quality evaluated?
Segmentation quality is evaluated by calculating the precision, recall, and F-score of the automatic segmentations in comparison to the segmentations made by expert annotators from the ANNODIS subcorpus.