Update README.md
Browse files
README.md
CHANGED
@@ -35,8 +35,48 @@ model-index:
|
|
35 |
---
|
36 |
## Model objective
|
37 |
|
38 |
-
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
39 |
|
40 |
## Metrics
|
41 |
|
42 |
-
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
35 |
---
|
36 |
## Model objective
|
37 |
|
38 |
+
TBF
|
39 |
+
|
40 |
+
## Model specs
|
41 |
+
|
42 |
+
This model is a fine-tuned version of [spanish-t5-small](https://huggingface.co/flax-community/spanish-t5-small) on the data described below.
|
43 |
+
It achieves the following results on the evaluation set:
|
44 |
+
- 'eval_bleu': 93.8347,
|
45 |
+
- 'eval_f1': 0.9904,
|
46 |
+
|
47 |
+
## Training procedure
|
48 |
+
### Training hyperparameters
|
49 |
+
The following hyperparameters were used during training:
|
50 |
+
- learning_rate: 1e-04
|
51 |
+
- train_batch_size: 32
|
52 |
+
- seed: 42
|
53 |
+
- num_epochs: 10
|
54 |
+
- weight_decay: 0,01
|
55 |
+
|
56 |
+
## Training and evaluation data
|
57 |
+
|
58 |
+
TBF
|
59 |
|
60 |
## Metrics
|
61 |
|
62 |
+
For training, we used both Blue (sacrebleu implementation in HF) and BertScore. The first one, a standard in Machine Translation processes, has been added for ensuring robustness of the newly generated data, while the second one is kept for keeping the expected semantic similarity.
|
63 |
+
|
64 |
+
However, given the actual use case, we expect generated segments to be very close to input segments and to label segments in training. As an example, we can take the following:
|
65 |
+
|
66 |
+
inputSegment = 'De acuerdo con las informaciones anteriores , las alumnas se han quejado de la actitud de los profesores en los exámenes finales. Los representantes estudiantiles son los alumnos Juanju y Javi.'
|
67 |
+
expectedOutput (label) = 'De acuerdo con las informaciones anteriores, el alumnado se ha quejado de la actitud del profesorado en los exámenes finales. Los representantes estudiantiles son los alumnos Juanju y Javi.'
|
68 |
+
actualOutput = 'De acuerdo con las informaciones anteriores, el alumnado se ha quejado de la actitud del profesorado en los exámenes finales. Los representantes estudiantiles son el alumnado Juanju y Javi.'
|
69 |
+
|
70 |
+
As you can see, segments are pretty similar. So, instead of measuring Bleu or BertScore here, we propose an alternate metric that would be DiffBleu:
|
71 |
+
|
72 |
+
$$DiffBleu = BLEU(actualOutput - inputSegment, labels - inputSegment)$$
|
73 |
+
|
74 |
+
Where the minuses as in set notation. This way, we also evaluate DiffBleu after the model has been trained.
|
75 |
+
|
76 |
+
|
77 |
+
## Usage example
|
78 |
+
|
79 |
+
|
80 |
+
|
81 |
+
|
82 |
+
Enjoy!
|