Title: Add ASNQ dataset
Author: mkserge (CONTRIBUTOR)
State: closed
Created: 2020-10-29T23:31:56Z

This pull request adds the ASNQ dataset. It is a dataset for answer sentence selection derived from Google Natural Questions (NQ) dataset (Kwiatkowski et al. 2019). The dataset details can be found in the paper at https://arxiv.org/abs/1911.04118

The dataset is authored by Siddhant Garg, Thuy Vu and Alessandro Moschitti. 

_Please note that I have no affiliation with the authors._

Repo: https://github.com/alexa/wqa_tanda

Comments:
Very nice !
What do the `sentence1` and `sentence2` correspond to exactly ?
Also maybe you could use the `ClassLabel` feature type for the `label` field (see [snli](https://github.com/huggingface/datasets/blob/master/datasets/snli/snli.py) for example)

> What do the `sentence1` and `sentence2` correspond to exactly ?

`sentence1` is a question, and `sentence2` is a candidate answer sentence. The labels are [1, 2, 3, 4] defining a relation between the answer sentence and the question. For example, label 4 means that the answer sentence is inside the _long_answer_ passage AND that the _short_answer_ is within the answer sentence. All the other labels are the negatives with different characteristics. (the short_answer, long_answer terminology is borrowed from Google's NQ dataset)

Should I label them simply as `question` and `answer`? I was going more with what I saw in the examples/run_glue.py script, but I realize now there is no restriction around this.

> Also maybe you could use the `ClassLabel` feature type for the `label` field (see [snli](https://github.com/huggingface/datasets/blob/master/datasets/snli/snli.py) for example)

I am finding it difficult to assign names to each class, but perhaps it's possible. Here's the description of each class from the paper.

1. Sentences from the document that are in the long answer but do not contain the annotated short answers. It is possible that these sentences might contain the short answer.
2. Sentences from the document that are not in the long answer but contain the short answer string, that is, such occurrence is purely accidental.
3. Sentences from the document that are neither in the long answer nor contain the short answer.
4. Sentences from the document that are in the long answer and do contain the annotated short answers.

Any ideas?



Yes it's better to have explicit feature names. Maybe go with question/answer or question/sentence.
I read in the paper that 1,2 and 3 are considered negative and 4 positive.
We could have a binary classification label `label` (either positive of negative) and then two boolean fields `short_answser_in_sentence` and `sentence_in_long_answer`. What do you think ?

> Yes it's better to have explicit feature names. Maybe go with question/answer or question/sentence.
> I read in the paper that 1,2 and 3 are considered negative and 4 positive.
> We could have a binary classification label `label` (either positive of negative) and then two boolean fields `short_answser_in_sentence` and `sentence_in_long_answer`. What do you think ?

Ok, sounds good. I went with `sentence` to keep it consistent with `short_answer_in_sentence` and `sentence_in_long_answer`. 

I changed it to a ClassLabel with pos and neg classes and added the two above as features. Let me know if this is not what you had in mind.