sequence-evaluate 0.0.3
A python package to evaluate sequence-to-sequence models
Requires Python: >=3.6

# SeqEval: A python package for evaluating Seq2Seq models

Sequence Evaluator (SeqEval) is a python package that computes metrics useful for evaluating Seq2Seq models for many tasks such as: machine translation, dialogue response generation, and text summarization. There already exists many packages to compute those metrics, but Seq2Eval puts them all in one place!

## Installation

```python
pip install sequence-evaluate
```

## Usage

You can easily compute all metrics in just a few lines of code. \
You first import the class ```SeqEval``` and create an instance of it:

```python
from seq_eval import SeqEval
evaluator = SeqEval()
```

The evaluator expects two python lists containing ```candidates``` (outputs generated by the model) and ```references``` (ground-truth data). \
For example:
```python
candidates = ["he began by starting a five person war cabinet and included chamberlain as lord president of the council",
             "the siege lasted from 250 to 241 bc, the romans laid siege to lilybaeum",
             "the original ocean water was found in aquaculture"]

references = ["he began his premiership by forming a five-man war cabinet which included chamberlain as lord president of the council",
             "the siege of lilybaeum lasted from 250 to 241 bc, as the roman army laid siege to the carthaginian-held sicilian city of lilybaeum",
             "the original mission was for research into the uses of deep ocean water in ocean thermal energy conversion (otec) renewable energy production and in aquaculture"]
```

You can now compute all metrics by using the ```evaluate``` function: (Setting ```verbose=True``` prints out the results)
```python
scores = evaluator.evaluate(candidates, references, verbose=True)
```
The function returns a dictionary containing all computed metric values:

```
{'bleu_1': 0.4428272792647754,
 'bleu_2': 0.35920252706356015,
 'bleu_3': 0.29702864345243746,
 'bleu_4': 0.2527668976020239,
 'inter_dist1': 0.1294642799346348,
 'inter_dist2': 0.5837103808275891,
 'intra_dist1': 0.31033264382268116,
 'intra_dist2': 0.7908440001400115,
 'rouge_1_f1': 0.6512670259900423,
 'rouge_1_precision': 0.8539562289562289,
 'rouge_1_recall': 0.5528035775713794,
 'rouge_2_f1': 0.3928074411537155,
 'rouge_2_precision': 0.5244559362206421,
 'rouge_2_recall': 0.3353174603174603,
 'rouge_l_f1': 0.6282785202429159,
 'rouge_l_precision': 0.8122895622895623,
 'rouge_l_recall': 0.5369305616983636,
 'semantic_textual_similarity': 0.8229544957478842}
```
 
## Dependencies
Make sure that you have the following libraries installed:

```
transformers 4.16.2
sentence-transformers 2.2.0
ntlk 3.2.5
torch 1.10.0+cu111
rouge 1.0.1
numpy 1.21.5
```

## Contact
**Tarek Naous**: [Scholar](https://scholar.google.com/citations?user=ImyLv44AAAAJ&hl=en) | [Github](https://github.com/tareknaous?tab=repositories) |
[Linkedin](https://www.linkedin.com/in/tareknaous/) |  [Research Gate](https://www.researchgate.net/profile/Tarek_Naous?ev=hdr_xprf) | [Personal Wesbite](https://www.sites.google.com/view/tareknaous)
| tareknaous@gmail.com
```