TASK DEFINITION: In this task, you will be presented with a context from an academic paper and you have to write an answerable question based on the context. Your questions can be extractive, abstractive, or yes-no questions.
PROBLEM: We use the annotations from experts for an abstract if it exists otherwise use crowd annotations.

SOLUTION: How do they match annotators to instances?

PROBLEM: Results on all sets show an important difference between precision and recall, precision being significantly higher than recall. There is also a significant difference between the results obtained on development and test set (F1 = 89.60, F1 = 87.82) and the results on the supplemental test set (F1 = 71.49). The time difference between the supplemental test set and the development and test set (the headlines from the the supplemental test set being from a different time period to the training set) can probably explain these differences.

SOLUTION: What is the performance of the CRF model on the task described?

PROBLEM: Therefore, we implement reinforcement learning and elaborately design two rewards to describe the irony accuracy and sentiment preservation, respectively.

SOLUTION:
What is the combination of rewards for reinforcement learning?