TASK DEFINITION: In this task, you will be presented with a context from an academic paper and a question separated with a
. You have to answer the question based on the context.
PROBLEM: Since our methodology does not use a seed list of offensive words, it is not biased by topic, target, or dialect. Using our methodology, we tagged 10,000 Arabic tweet dataset for offensiveness, where offensive tweets account for roughly 19% of the tweets. Further, we labeled tweets as vulgar or hate speech. To date, this is the largest available dataset, which we plan to make publicly available along with annotation guidelines. We use this dataset to characterize Arabic offensive language to ascertain the topics, dialects, and users' gender that are most associated with the use of offensive language.
Question: In what way is the offensive dataset not biased by topic, dialect or target?

SOLUTION: It does not use a seed list to gather tweets so the dataset does not skew to specific topics, dialect, targets.

PROBLEM: Metrics. We use tolerance accuracy BIBREF16, which measures how far away the predicted span is from the gold standard span, as a metric. The rationale behind the metric is that, in practice, it suffices to recommend a rough span which contains the answer a difference of a few seconds would not matter much to the user. Metrics. We used accuracy and MRR (Mean Reciprocal Ranking) as metrics. Metrics. To evaluate our pipeline approach we use overall accuracy after filtering and accuracy given that the segment is in the top 10 videos.
Question: What evaluation metrics were used in the experiment?

SOLUTION: For sentence-level prediction they used tolerance accuracy, for segment retrieval accuracy and MRR and for the pipeline approach they used overall accuracy

PROBLEM: We evaluated our models on 3 different datasets:

CSAT dataset for CSAT prediction, consisting of spoken transcripts (automatic via ASR).

20 newsgroups for topic identification task, consisting of written text;

Fisher Phase 1 corpus for topic identification task, consisting of spoken transcripts (manual);
Question: What datasets did they use for evaluation?

SOLUTION:
CSAT dataset 20 newsgroups Fisher Phase 1 corpus