TASK DEFINITION: In this task, you will be presented with a context from an academic paper and a question separated with a
. You have to answer the question based on the context.
PROBLEM: Given a statement and articles, workers are asked to judge whether the statement can be derived from the articles at three grades: True, Likely (i.e. Answerable), or Unsure (i.e. Unanswerable). If a worker selects Unsure, we ask workers to tell us why they are unsure from two choices ("Not stated in the article" or "Other"). If a worker selects True or Likely in the judgement task, we first ask which sentences in the given articles are justification explanations for a given statement, similarly to HotpotQA BIBREF2. The "summary" text boxes (i.e. NLDs) are then initialized with these selected sentences.
Question: How was the dataset annotated?

SOLUTION: True, Likely (i.e. Answerable), or Unsure (i.e. Unanswerable) why they are unsure from two choices ("Not stated in the article" or "Other") The "summary" text boxes

PROBLEM: A few sample advice rules in English (these are converted to first-order logic format and given as input to our algorithm) are presented in Table TABREF11 . We modified the work of Odom et al. odomAIME15,odomAAAI15 to learn RDNs in the presence of advice. The key idea is to explicitly represent advice in calculating gradients.
Question: How do they incorporate human advice?

SOLUTION: by converting human advice to first-order logic format and use as an input to calculate gradient

PROBLEM: Therefore, we decided to compile our own corpora based on English documents, which we crawled from different publicly accessible sources.
Question: Do they report results only on English data?

SOLUTION:
Yes