added metrics for CUAD
> For now I've added F1, AUPR, Precision at 80% recall, and Precision at 90%. Last 3 metrics were reported in the [paper](https://arxiv.org/pdf/2103.06268.pdf). Please let me know if we require `exact_match` metric too here

@bhavitvyamalik  I guess the mentioned metrics are enough but it would be better if exact match is also added since the standard SQUAD dataset also has it. I would like to quote it from the website that I am following to learn
these things.
Exact Match:
This metric is as simple as it sounds. For each question+answer pair, if
the characters of the model's prediction exactly match the characters of
*(one
of) the True Answer(s)*, EM = 1, otherwise EM = 0. This is a strict
all-or-nothing metric; being off by a single character results in a score
of 0. When assessing against a negative example, if the model predicts any
text at all, it automatically receives a 0 for that example.

So, I guess you need to ensure at least 1 predicted answer matches for EM
to be 1.
Source:
https://qa.fastforwardlabs.com/no%20answer/null%20threshold/bert/distilbert/exact%20match/f1/robust%20predictions/2020/06/09/Evaluating_BERT_on_SQuAD.html

You can go to their homepage and read the other links. They have detailed
explanations on evaluation metrics. You can also have a look at the
squad_v2 metric file for further clarification.

Regards,
Mohammed Rakib

On Sun, 25 Apr 2021 at 15:20, Bhavitvya Malik ***@***.***>
wrote:

> I'm a little confused when it comes to 2 ground truths which can be a
> possible answer. Like here for eg.
>
> predictions = [{'prediction_text': ['The seller:', 'The buyer/End-User:
> Shenzhen LOHAS Supply Chain Management Co., Ltd.'], 'id':
> 'LohaCompanyltd_20191209_F-1_EX-10.16_11917878_EX-10.16_Supply
> Agreement__Parties'}]
>
> references = [{'answers': {'answer_start': [143, 49], 'text': ['The
> seller:', 'The buyer/End-User: Shenzhen LOHAS Supply Chain Management Co.,
> Ltd.']}, 'id':
> 'LohaCompanyltd_20191209_F-1_EX-10.16_11917878_EX-10.16_Supply
> Agreement__Parties'}]
>
> Should I ensure at least 1 predicted answer matches or both predicted
> answers should match (like in this case) for EM to be 1?
>
> —
> You are receiving this because you commented.
> Reply to this email directly, view it on GitHub
> <https://github.com/huggingface/datasets/pull/2257#issuecomment-826289753>,
> or unsubscribe
> <https://github.com/notifications/unsubscribe-auth/AHMYZAZSAEZNFWEMVAPK6M3TKPNHLANCNFSM43QFZVPQ>
> .
>
 Updated the same @MohammedRakib! Even if a single answer matches I'm returning 1 in that case for EM (not traversing all predictions once we have one `exact_match` from prediction)
For now I've added F1, AUPR, Precision at 80% recall, and Precision at 90%. Last 3 metrics were reported in the [paper](https://arxiv.org/pdf/2103.06268.pdf). Please let me know if we require `exact_match` metric too here