In this task, you will be presented with a context from an academic paper and a question based on the context. You have to classify the questions into "Extractive", "Abstractive", or "Yes-no" questions. Extractive questions can be answered by concatenating extracts taken from a context into a summary while answering abstractive questions involves paraphrasing the context using novel sentences. Yes-no question is a question whose expected answer is one of two choices, one that affirms the question and one that denies the question. Typically, the choices are either yes or no.

We compare the SBERT sentence embeddings to other sentence embeddings methods on the following seven SentEval transfer tasks:

MR: Sentiment prediction for movie reviews snippets on a five start scale BIBREF25.

CR: Sentiment prediction of customer product reviews BIBREF26.

SUBJ: Subjectivity prediction of sentences from movie reviews and plot summaries BIBREF27.

MPQA: Phrase level opinion polarity classification from newswire BIBREF28.

SST: Stanford Sentiment Treebank with binary labels BIBREF29.

TREC: Fine grained question-type classification from TREC BIBREF30.

MRPC: Microsoft Research Paraphrase Corpus from parallel news sources BIBREF31.
Question: What transfer learning tasks are evaluated?
Extractive

Human Judgments
Following BIBREF11 , BIBREF12 and the vast amount of previous work on semantic similarity, we ask nine undergraduate subjects to assess the similarity of 360 pairs of relations from a subset of Wikidata BIBREF8 that are chosen to cover from high to low levels of similarity. In our experiment, subjects were asked to rate an integer similarity score from 0 (no similarity) to 4 (perfectly the same) for each pair.
Question: How do they gather human judgements for similarity between relations?
Abstractive

We used the following affective resources relying on different emotion models.

Emolex: it contains 14,182 words associated with eight primary emotion based on the Plutchik model BIBREF10 , BIBREF11 .

EmoSenticNet(EmoSN): it is an enriched version of SenticNet BIBREF12 including 13,189 words labeled by six Ekman's basic emotion BIBREF13 , BIBREF14 .

Dictionary of Affect in Language (DAL): includes 8,742 English words labeled by three scores representing three dimensions: Pleasantness, Activation and Imagery BIBREF15 .

Affective Norms for English Words (ANEW): consists of 1,034 English words BIBREF16 rated with ratings based on the Valence-Arousal-Dominance (VAD) model BIBREF17 .

Linguistic Inquiry and Word Count (LIWC): this psycholinguistic resource BIBREF18 includes 4,500 words distributed into 64 emotional categories including positive (PosEMO) and negative (NegEMO).
Question: What affective-based features are used?
Abstractive