Hugging Face's logo

sapienzanlp
/
sentinel-ref-da

Translation
Transformers
multilingual
SENTINEL-REF-DA
Inference Endpoints
Model card Files Community
sentinel-ref-da
File size: 3,409 Bytes 
811695e
 
 
 
 
 
d448c08
 
 
 
 
 
 
 
 
 
 
 
550a433
 
d448c08
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
edf9cfe
d448c08
 
 
 
edf9cfe
 
 
 
 
 
 
 
d448c08
 
 
 
 
 
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
 
---
pipeline_tag: translation
language: multilingual
library_name: transformers
license: cc-by-nc-sa-4.0
---

<div align="center">

<h1 style="font-family: 'Arial', sans-serif; font-size: 28px; font-weight: bold; color: black;">
    🛡️ Guardians of the Machine Translation Meta-Evaluation:<br>
    Sentinel Metrics Fall In!
</h1>

</div>

<div style="display:flex; justify-content: center; align-items: center; flex-direction: row;">
    <a href="https://2024.aclweb.org/"><img src="http://img.shields.io/badge/ACL-2024-4b44ce.svg"></a> &nbsp; &nbsp; 
    <a href="https://aclanthology.org/2024.acl-long.856/"><img src="http://img.shields.io/badge/paper-ACL--anthology-B31B1B.svg"></a> &nbsp; &nbsp;
    <a href="https://arxiv.org/abs/2408.13831"><img src="https://img.shields.io/badge/arXiv-2408.13831-b31b1b.svg"></a> &nbsp; &nbsp;
    <a href="https://creativecommons.org/licenses/by-nc-sa/4.0/"><img src="https://img.shields.io/badge/License-CC%20BY--NC--SA%204.0-lightgrey.svg"></a>
</div>
<div style="display:flex; justify-content: center; align-items: center; flex-direction: row;">
    <a href="https://huggingface.co/collections/sapienzanlp/mt-sentinel-metrics-66ab643b32aab06f3157e5c1"><img src="https://img.shields.io/badge/%F0%9F%A4%97%20Hugging%20Face-Collection-FCD21D"></a> &nbsp; &nbsp;
    <a href="https://github.com/SapienzaNLP/guardians-mt-eval"><img src="https://img.shields.io/badge/GitHub-Repo-121013?logo=github&logoColor=white"></a> &nbsp; &nbsp;
</div>

This repository contains the **SENTINEL<sub>REF</sub>** metric model trained on Direct Assessments (DA) annotations. For details on how to use our sentinel metric models, check our [GitHub repository](https://github.com/SapienzaNLP/guardians-mt-eval).

## Usage

After having installed our repository package, you can use this model within Python in the following way:

```python
from sentinel_metric import download_model, load_from_checkpoint

model_path = download_model("sapienzanlp/sentinel-ref-da")
model = load_from_checkpoint(model_path)

data = [
    {"ref": "There's no place like home."},
    {"ref": "Toto, I've a feeling we're not in Kansas anymore."}
]

output = model.predict(data, batch_size=8, gpus=1)
```

Output:
```python
# Segment scores
>>> output.scores
[0.26086926460266113, 0.17275342345237732]

# System score
>>> output.system_score
0.21681134402751923
```

## Cite this work
This work has been published at [ACL 2024 (Main Conference)](https://aclanthology.org/2024.acl-long.856/). If you use any part, please consider citing our paper as follows:

```bibtex
@inproceedings{perrella-etal-2024-guardians,
    title = "Guardians of the Machine Translation Meta-Evaluation: Sentinel Metrics Fall In!",
    author = "Perrella, Stefano  and Proietti, Lorenzo  and Scir{\`e}, Alessandro  and Barba, Edoardo  and Navigli, Roberto",
    editor = "Ku, Lun-Wei  and Martins, Andre  and Srikumar, Vivek",
    booktitle = "Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)",
    month = aug,
    year = "2024",
    address = "Bangkok, Thailand", publisher = "Association for Computational Linguistics",
    url = "https://aclanthology.org/2024.acl-long.856",
    pages = "16216--16244",
}
```

## License
This work is licensed under [Creative Commons Attribution-ShareAlike-NonCommercial 4.0](https://creativecommons.org/licenses/by-nc-sa/4.0/).