|
--- |
|
license: cc-by-4.0 |
|
--- |
|
This is model based on mT5-XXL that predicts a binary label for a given article and summary for Q1 (comprehensibility), as defined in the [SEAHORSE paper](https://arxiv.org/abs/2305.13194) (Clark et al., 2023). |
|
|
|
It is trained similarly to the [TRUE paper (Honovich et al, 2022)](https://arxiv.org/pdf/2204.04991.pdf) on human ratings from the SEAHORSE dataset in 6 languages: |
|
- German |
|
- English |
|
- Spanish |
|
- Russian |
|
- Turkish |
|
- Vietnamese |
|
|
|
The input format for the model is: "premise: ARTICLE hypothesis: SUMMARY", where ARTICLE is the document being summarized and SUMMARY is the candidate summary. |
|
|
|
There is also a smaller (mT5-L) version of this model, as well as metrics trained for each of the other 5 dimensions described in the original paper. |
|
|
|
The full citation for the SEAHORSE paper is: |
|
``` |
|
@misc{clark2023seahorse, |
|
title={SEAHORSE: A Multilingual, Multifaceted Dataset for Summarization Evaluation}, |
|
author={Elizabeth Clark and Shruti Rijhwani and Sebastian Gehrmann and Joshua Maynez and Roee Aharoni and Vitaly Nikolaev and Thibault Sellam and Aditya Siddhant and Dipanjan Das and Ankur P. Parikh}, |
|
year={2023}, |
|
eprint={2305.13194}, |
|
archivePrefix={arXiv}, |
|
primaryClass={cs.CL} |
|
} |
|
``` |
|
|
|
Contact: seahorse-authors@google.com |
