TASK DEFINITION: In this task, you will be presented with a context from an academic paper and a question separated with a
. You have to answer the question based on the context.
PROBLEM: Given a statement and articles, workers are asked to judge whether the statement can be derived from the articles at three grades: True, Likely (i.e. Answerable), or Unsure (i.e. Unanswerable). If a worker selects Unsure, we ask workers to tell us why they are unsure from two choices ("Not stated in the article" or "Other"). If a worker selects True or Likely in the judgement task, we first ask which sentences in the given articles are justification explanations for a given statement, similarly to HotpotQA BIBREF2. The "summary" text boxes (i.e. NLDs) are then initialized with these selected sentences.
Question: How was the dataset annotated?

SOLUTION: True, Likely (i.e. Answerable), or Unsure (i.e. Unanswerable) why they are unsure from two choices ("Not stated in the article" or "Other") The "summary" text boxes

PROBLEM: Since our methodology does not use a seed list of offensive words, it is not biased by topic, target, or dialect. Using our methodology, we tagged 10,000 Arabic tweet dataset for offensiveness, where offensive tweets account for roughly 19% of the tweets. Further, we labeled tweets as vulgar or hate speech. To date, this is the largest available dataset, which we plan to make publicly available along with annotation guidelines. We use this dataset to characterize Arabic offensive language to ascertain the topics, dialects, and users' gender that are most associated with the use of offensive language.
Question: In what way is the offensive dataset not biased by topic, dialect or target?

SOLUTION: It does not use a seed list to gather tweets so the dataset does not skew to specific topics, dialect, targets.

PROBLEM: Given a set of evaluation documents, each having a known correct label from a closed set of labels (often referred to as the "gold-standard"), and a predicted label for each document from the same set, the document-level accuracy is the proportion of documents that are correctly labeled over the entire evaluation collection. This is the most frequently reported metric and conveys the same information as the error rate, which is simply the proportion of documents that are incorrectly labeled (i.e. INLINEFORM0 ). There are two distinct ways in which results are generally summarized per-language: (1) precision, in which documents are grouped according to their predicted language; and (2) recall, in which documents are grouped according to what language they are actually written in. It is also common practice to report an F-score INLINEFORM0 , which is the harmonic mean of precision and recall.
Question: what evaluation methods are discussed?

SOLUTION:
document-level accuracy precision recall F-score