instruction:
In this task, you will be presented with a context from an academic paper and a question separated with a
. You have to answer the question based on the context.
question:
The test-set accuracies obtained by different learning methods, including the current state-of-the-art results, are presented in Table TABREF11 .
Question: what models did they compare to?
answer:
High-order CNN, Tree-LSTM, DRNN, DCNN, CNN-MC, NBoW and SVM

question:
A recently published dataset called SCAN BIBREF2 (Simplified version of the CommAI Navigation tasks), tests compositional generalization in a sequence-to-sequence (seq2seq) setting by systematically holding out of the training set all inputs containing a basic primitive verb ("jump"), and testing on sequences containing that verb.
Question: How does the SCAN dataset evaluate compositional generalization?
answer:
it systematically holds out inputs in the training set containing basic primitive verb, "jump", and tests on sequences containing that verb.

question:
We evaluate our model on two benchmark datasets BIBREF9 . The homographic dataset contains 2,250 contexts, 1,607 of which contain a pun. The heterographic dataset consists of 1,780 contexts with 1,271 containing a pun. We notice there is no standard splitting information provided for both datasets. Thus we apply 10-fold cross validation. To make direct comparisons with prior studies, following BIBREF4 , we accumulated the predictions for all ten folds and calculate the scores in the end.
Question: What datasets are used in evaluation?
answer:
The homographic dataset contains 2,250 contexts, 1,607 of which contain a pun. The heterographic dataset consists of 1,780 contexts with 1,271 containing a pun.