TASK DEFINITION: In this task, you will be presented with a context from an academic paper and a question separated with a
. You have to answer the question based on the context.
PROBLEM: The datasets used for training, validation and testing contain sentences extracted from the Europarl corpus BIBREF1 and SoNaR corpus BIBREF2. The Europarl corpus is an open-source parallel corpus containing proceedings of the European Parliament. The Dutch section consists of 2,333,816 sentences and 53,487,257 words. The SoNaR corpus comprises two corpora: SONAR500 and SONAR1. The SONAR500 corpus consists of more than 500 million words obtained from different domains.
Question: What are the sizes of both datasets?

SOLUTION: The Dutch section consists of 2,333,816 sentences and 53,487,257 words. The SONAR500 corpus consists of more than 500 million words obtained from different domains.

PROBLEM: We introduce a surrogate training objective that avoids these problems and as a result is fully continuous. Specifically, we form a continuous function softLB that seeks to approximate the result of running our decoder on input INLINEFORM0 and then evaluating the result against INLINEFORM1 using INLINEFORM2 .
Question: Do they provide a framework for building a sub-differentiable for any final loss metric?

SOLUTION: Yes

PROBLEM: A meta-research study found only 1 in 9 qualitative papers in Human-Computer Interaction reported inter-rater reliability metrics BIBREF31. Another related area are meta-research and methods papers focused on identifying or preventing low-effort responses from crowdworkers sometimes called "spam" or "random" responses, or alternatively "fraudsters" or "cheaters." Rates of "self-agreement" are often used, determining if the same person labels the same item differently at a later stage.
Question: What are the key issues around whether the gold standard data produced in such an annotation is reliable?

SOLUTION:
only 1 in 9 qualitative papers in Human-Computer Interaction reported inter-rater reliability metrics low-effort responses from crowdworkers