End of training

README.md

@@ -0,0 +1,69 @@
+---
+license: apache-2.0
+base_model: sentence-transformers/paraphrase-MiniLM-L3-v2
+tags:
+- generated_from_trainer
+datasets:
+- nyt_ingredients
+model-index:
+- name: nyt_ingredients-crf-tagger-paraphrase-MiniLM-L3-v2
+  results: []
+---
+
+<!-- This model card has been generated automatically according to the information the Trainer had access to. You
+should probably proofread and complete it, then remove this comment. -->
+
+# nyt_ingredients-crf-tagger-paraphrase-MiniLM-L3-v2
+
+This model is a fine-tuned version of [sentence-transformers/paraphrase-MiniLM-L3-v2](https://huggingface.co/sentence-transformers/paraphrase-MiniLM-L3-v2) on the nyt_ingredients dataset.
+It achieves the following results on the evaluation set:
+- Loss: 10.2590
+- Comment: {'precision': 0.03657262277951933, 'recall': 0.0264750378214826, 'f1': 0.030715225976305396, 'number': 1322}
+- Name: {'precision': 0.5238095238095238, 'recall': 0.01245753114382786, 'f1': 0.024336283185840708, 'number': 1766}
+- Qty: {'precision': 0.0234375, 'recall': 0.0020920502092050207, 'f1': 0.003841229193341869, 'number': 1434}
+- Range End: {'precision': 0.0, 'recall': 0.0, 'f1': 0.0, 'number': 17}
+- Unit: {'precision': 0.0, 'recall': 0.0, 'f1': 0.0, 'number': 1166}
+- Overall Precision: 0.0419
+- Overall Recall: 0.0105
+- Overall F1: 0.0168
+- Overall Accuracy: 0.1284
+
+## Model description
+
+More information needed
+
+## Intended uses & limitations
+
+More information needed
+
+## Training and evaluation data
+
+More information needed
+
+## Training procedure
+
+### Training hyperparameters
+
+The following hyperparameters were used during training:
+- learning_rate: 5e-05
+- train_batch_size: 32
+- eval_batch_size: 32
+- seed: 42
+- optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-08
+- lr_scheduler_type: linear
+- num_epochs: 2
+
+### Training results
+
+| Training Loss | Epoch | Step | Validation Loss | Comment                                                                                                      | Name                                                                                                          | Qty                                                                                                   | Range End                                                  | Unit                                                         | Overall Precision | Overall Recall | Overall F1 | Overall Accuracy |
+|:-------------:|:-----:|:----:|:---------------:|:------------------------------------------------------------------------------------------------------------:|:-------------------------------------------------------------------------------------------------------------:|:-----------------------------------------------------------------------------------------------------:|:----------------------------------------------------------:|:------------------------------------------------------------:|:-----------------:|:--------------:|:----------:|:----------------:|
+| No log        | 1.0   | 54   | 11.5992         | {'precision': 0.03826530612244898, 'recall': 0.0340393343419062, 'f1': 0.036028823058446756, 'number': 1322} | {'precision': 0.9047619047619048, 'recall': 0.010758776896942242, 'f1': 0.021264689423614997, 'number': 1766} | {'precision': 0.0, 'recall': 0.0, 'f1': 0.0, 'number': 1434}                                          | {'precision': 0.0, 'recall': 0.0, 'f1': 0.0, 'number': 17} | {'precision': 0.0, 'recall': 0.0, 'f1': 0.0, 'number': 1166} | 0.0526            | 0.0112         | 0.0185     | 0.1319           |
+| No log        | 2.0   | 108  | 10.2590         | {'precision': 0.03657262277951933, 'recall': 0.0264750378214826, 'f1': 0.030715225976305396, 'number': 1322} | {'precision': 0.5238095238095238, 'recall': 0.01245753114382786, 'f1': 0.024336283185840708, 'number': 1766}  | {'precision': 0.0234375, 'recall': 0.0020920502092050207, 'f1': 0.003841229193341869, 'number': 1434} | {'precision': 0.0, 'recall': 0.0, 'f1': 0.0, 'number': 17} | {'precision': 0.0, 'recall': 0.0, 'f1': 0.0, 'number': 1166} | 0.0419            | 0.0105         | 0.0168     | 0.1284           |
+
+
+### Framework versions
+
+- Transformers 4.34.0
+- Pytorch 2.0.1+cu118
+- Datasets 2.14.5
+- Tokenizers 0.14.0

added_tokens.json

@@ -0,0 +1,7 @@
+{
+  "[CLS]": 101,
+  "[MASK]": 103,
+  "[PAD]": 0,
+  "[SEP]": 102,
+  "[UNK]": 100
+}

all_results.json

@@ -0,0 +1,41 @@
+{
+    "epoch": 2.0,
+    "eval_COMMENT": {
+        "f1": 0.030715225976305396,
+        "number": 1322,
+        "precision": 0.03657262277951933,
+        "recall": 0.0264750378214826
+    },
+    "eval_NAME": {
+        "f1": 0.024336283185840708,
+        "number": 1766,
+        "precision": 0.5238095238095238,
+        "recall": 0.01245753114382786
+    },
+    "eval_QTY": {
+        "f1": 0.003841229193341869,
+        "number": 1434,
+        "precision": 0.0234375,
+        "recall": 0.0020920502092050207
+    },
+    "eval_RANGE_END": {
+        "f1": 0.0,
+        "number": 17,
+        "precision": 0.0,
+        "recall": 0.0
+    },
+    "eval_UNIT": {
+        "f1": 0.0,
+        "number": 1166,
+        "precision": 0.0,
+        "recall": 0.0
+    },
+    "eval_loss": 10.259025573730469,
+    "eval_overall_accuracy": 0.12838815472171314,
+    "eval_overall_f1": 0.016813787305590584,
+    "eval_overall_precision": 0.04189944134078212,
+    "eval_overall_recall": 0.010517090271691499,
+    "eval_runtime": 15.7061,
+    "eval_samples_per_second": 108.365,
+    "eval_steps_per_second": 3.438
+}

config.json

@@ -0,0 +1,55 @@
+{
+  "_name_or_path": "sentence-transformers/paraphrase-MiniLM-L3-v2",
+  "architectures": [
+    "PretrainedCRFModel"
+  ],
+  "attention_probs_dropout_prob": 0.1,
+  "auto_map": {
+    "AutoModel": "crf_model.PretrainedCRFModel"
+  },
+  "classifier_dropout": null,
+  "gradient_checkpointing": false,
+  "hidden_act": "gelu",
+  "hidden_dropout_prob": 0.1,
+  "hidden_size": 384,
+  "id2label": {
+    "0": "O",
+    "1": "B-COMMENT",
+    "2": "I-COMMENT",
+    "3": "B-NAME",
+    "4": "I-NAME",
+    "5": "B-RANGE_END",
+    "6": "I-RANGE_END",
+    "7": "B-QTY",
+    "8": "I-QTY",
+    "9": "B-UNIT",
+    "10": "I-UNIT"
+  },
+  "initializer_range": 0.02,
+  "intermediate_size": 1536,
+  "label2id": {
+    "B-COMMENT": 1,
+    "B-NAME": 3,
+    "B-QTY": 7,
+    "B-RANGE_END": 5,
+    "B-UNIT": 9,
+    "I-COMMENT": 2,
+    "I-NAME": 4,
+    "I-QTY": 8,
+    "I-RANGE_END": 6,
+    "I-UNIT": 10,
+    "O": 0
+  },
+  "layer_norm_eps": 1e-12,
+  "max_position_embeddings": 512,
+  "model_type": "bert",
+  "num_attention_heads": 12,
+  "num_hidden_layers": 3,
+  "pad_token_id": 0,
+  "position_embedding_type": "absolute",
+  "torch_dtype": "float32",
+  "transformers_version": "4.34.0",
+  "type_vocab_size": 2,
+  "use_cache": true,
+  "vocab_size": 30522
+}

crf.py

@@ -0,0 +1,243 @@
+import unittest
+from dataclasses import dataclass
+from typing import Optional, Tuple
+
+import torch
+from torch import nn
+from torch.autograd import Variable
+
+
+@dataclass
+class CRFOutput:
+    loss: Optional[torch.tensor]
+    real_path_score: Optional[torch.tensor]
+    total_score: torch.tensor
+    best_path_score: torch.tensor
+    best_path: torch.tensor
+
+
+class MaskedCRFLoss(nn.Module):
+    __constants__ = ["num_tags", "mask_id"]
+
+    num_tags: int
+    mask_id: int
+
+    def __init__(self, num_tags: int, mask_id: int = 0):
+        super().__init__()
+        self.num_tags = num_tags
+        self.mask_id = mask_id
+        self.transitions = nn.Parameter(torch.randn(num_tags, num_tags))
+        self.start_transitions = nn.Parameter(torch.randn(num_tags))
+        self.stop_transitions = nn.Parameter(torch.randn(num_tags))
+
+    def extra_repr(self) -> str:
+        s = "num_tags={num_tags}, mask_id={mask_id}"
+        return s.format(**self.__dict__)
+
+    def forward(self, emissions, tags, mask, return_best_path=False):
+        # emissions: (seq_length, batch_size, num_tags)
+        # tags: (seq_length, batch_size)
+        # mask: (seq_length, batch_size)
+
+        seq_length, batch_size = tags.shape
+        mask = mask.float()
+        # set return_best_path as True always during eval.
+        # During training it slows things down as best path is not needed
+        if not self.training:
+            return_best_path = True
+        # Compute the total likelihood
+        total_score, best_path_score, best_path = self.compute_log_partition_function(
+            emissions, mask, return_best_path=return_best_path
+        )
+
+        if tags is None:
+            return CRFOutput(None, None, total_score, best_path_score, best_path)
+
+        # Compute the likelihood of the real path
+        real_path_score = torch.zeros(batch_size).to(tags.device)
+        real_path_score += self.start_transitions[tags[0]]  # batch_size
+        for i in range(1, seq_length):
+            current_tag = tags[i]
+            real_path_score += self.transitions[tags[i - 1], current_tag] * mask[i]
+            real_path_score += emissions[i, range(batch_size), current_tag] * mask[i]
+        # Transition to STOP_TAG
+        real_path_score += self.stop_transitions[tags[-1]]
+
+        # Return the negative log likelihood
+        loss = torch.mean(total_score - real_path_score)
+        return CRFOutput(loss, real_path_score, total_score, best_path_score, best_path)
+
+    def compute_log_partition_function(self, emissions, mask, return_best_path=False):
+        init_alphas = self.start_transitions + emissions[0]  # (batch_size, num_tags)
+        forward_var = init_alphas
+        forward_viterbi_var = init_alphas
+        # backpointers holds the best tag id at each time step, we accumulate these in reverse order
+        backpointers = []
+
+        for i, emission in enumerate(emissions[1:, :, :], 1):
+            broadcast_emission = emission.unsqueeze(2)  # (batch_size, num_tags, 1)
+            broadcast_transmissions = self.transitions.unsqueeze(
+                0
+            )  # (1, num_tags, num_tags)
+
+            # Compute next
+            next_tag_var = (
+                forward_var.unsqueeze(1) + broadcast_emission + broadcast_transmissions
+            )
+            next_tag_viterbi_var = (
+                forward_viterbi_var.unsqueeze(1)
+                + broadcast_emission
+                + broadcast_transmissions
+            )
+
+            next_unmasked_forward_var = torch.logsumexp(next_tag_var, dim=2)
+            viterbi_scores, best_next_tags = torch.max(next_tag_viterbi_var, dim=2)
+            # If mask == 1 use the next_unmasked_forward_var else copy the forward_var
+            # Update forward_var
+            forward_var = (
+                mask[i].unsqueeze(-1) * next_unmasked_forward_var
+                + (1 - mask[i]).unsqueeze(-1) * forward_var
+            )
+            # Update viterbi with mask
+            forward_viterbi_var = (
+                mask[i].unsqueeze(-1) * viterbi_scores
+                + (1 - mask[i]).unsqueeze(-1) * forward_viterbi_var
+            )
+            backpointers.append(best_next_tags)
+
+        # Transition to STOP_TAG
+        terminal_var = forward_var + self.stop_transitions
+        terminal_viterbi_var = forward_viterbi_var + self.stop_transitions
+
+        alpha = torch.logsumexp(terminal_var, dim=1)
+        best_path_score, best_final_tags = torch.max(terminal_viterbi_var, dim=1)
+
+        best_path = None
+        if return_best_path:
+            # backtrace
+            best_path = [best_final_tags]
+            for bptrs, mask_data in zip(reversed(backpointers), torch.flip(mask, [0])):
+                best_tag_id = torch.gather(
+                    bptrs, 1, best_final_tags.unsqueeze(1)
+                ).squeeze(1)
+                best_final_tags.masked_scatter_(
+                    mask_data.to(dtype=torch.bool),
+                    best_tag_id.masked_select(mask_data.to(dtype=torch.bool)),
+                )
+                best_path.append(best_final_tags)
+            # Reverse the order because we were appending in reverse
+            best_path = torch.stack(best_path[::-1])
+            best_path = best_path.where(mask == 1, -100)
+
+        return alpha, best_path_score, best_path
+
+    def viterbi_decode(self, emissions, mask):
+        seq_len, batch_size, num_tags = emissions.shape
+
+        # backpointers holds the best tag id at each time step, we accumulate these in reverse order
+        backpointers = []
+
+        # Initialize the viterbi variables in log space
+        init_vvars = self.start_transitions + emissions[0]  # (batch_size, num_tags)
+        forward_var = init_vvars
+
+        for i, emission in enumerate(emissions[1:, :, :], 1):
+            broadcast_emission = emission.unsqueeze(2)
+            broadcast_transmissions = self.transitions.unsqueeze(0)
+            next_tag_var = (
+                forward_var.unsqueeze(1) + broadcast_emission + broadcast_transmissions
+            )
+
+            viterbi_scores, best_next_tags = torch.max(next_tag_var, 2)
+            # If mask == 1 use the next_unmasked_forward_var else copy the forward_var
+            forward_var = (
+                mask[i].unsqueeze(-1) * viterbi_scores
+                + (1 - mask[i]).unsqueeze(-1) * forward_var
+            )
+            backpointers.append(best_next_tags)
+
+        # Transition to STOP_TAG
+        terminal_var = forward_var + self.stop_transitions
+        best_path_score, best_final_tags = torch.max(terminal_var, dim=1)
+
+        # backtrace
+        best_path = [best_final_tags]
+        for bptrs, mask_data in zip(reversed(backpointers), torch.flip(mask, [0])):
+            best_tag_id = torch.gather(bptrs, 1, best_final_tags.unsqueeze(1)).squeeze(
+                1
+            )
+            best_final_tags.masked_scatter_(
+                mask_data.to(dtype=torch.bool),
+                best_tag_id.masked_select(mask_data.to(dtype=torch.bool)),
+            )
+            best_path.append(best_final_tags)
+
+        # Reverse the order because we were appending in reverse
+        best_path = torch.stack(best_path[::-1])
+        best_path = best_path.where(mask == 1, -100)
+
+        return best_path, best_path_score
+
+
+class MaskedCRFLossTest(unittest.TestCase):
+    def setUp(self):
+        self.num_tags = 5
+        self.mask_id = 0
+
+        self.crf_model = MaskedCRFLoss(self.num_tags, self.mask_id)
+
+        self.seq_length, self.batch_size = 11, 5
+        # Making up some inputs
+        # emissions = Variable(torch.randn(seq_length, batch_size, num_tags))
+        # tags = Variable(torch.randint(num_tags, (seq_length, batch_size)))
+        # mask = Variable(torch.ones(seq_length, batch_size))
+        self.emissions = torch.randn(self.seq_length, self.batch_size, self.num_tags)
+        self.tags = torch.randint(self.num_tags, (self.seq_length, self.batch_size))
+        # mask = torch.ones(seq_length, batch_size)
+        self.mask = torch.randint(2, (self.seq_length, self.batch_size))
+
+    def test_forward(self):
+        # Checking if forward runs successfully
+        try:
+            output = self.crf_model(self.emissions, self.tags, self.mask)
+            print("Forward function runs successfully!")
+        except Exception as e:
+            print("Forward function couldn't run successfully:", e)
+
+    def test_viterbi_decode(self):
+        # Checking if viterbi_decode runs successfully
+        try:
+            path, best_path_score = self.crf_model.viterbi_decode(
+                self.emissions, self.mask
+            )
+            print(path.T)
+            print("Viterbi decoding function runs successfully!")
+        except Exception as e:
+            print("Viterbi decoding function couldn't run successfully:", e)
+
+    def test_forward_output(self):
+        # Simple check if losses are non-negative
+        output = self.crf_model(self.emissions, self.tags, self.mask)
+        loss = output.loss
+        self.assertTrue((loss > 0).all())
+
+    def test_compute_log_partition_function_output(self):
+        # Simply checking if the output is non-negative
+        (
+            partition,
+            best_path_score,
+            best_path,
+        ) = self.crf_model.compute_log_partition_function(self.emissions, self.mask)
+        self.assertTrue((partition > 0).all())
+
+    def test_viterbi_decode_output(self):
+        print(self.mask.T)
+        # Check whether the output shape is correct and lies within valid tag range
+        path, best_path_score = self.crf_model.viterbi_decode(self.emissions, self.mask)
+        print(path.T)
+        self.assertEqual(
+            path.shape, (self.seq_length, self.batch_size)
+        )  # checking dimensions
+        self.assertTrue(
+            ((0 <= path) | (path == -100)).all() and (path < self.num_tags).all()
+        )  # checking tag validity

crf_model.py

@@ -0,0 +1,78 @@
+from dataclasses import dataclass
+from typing import Optional, Tuple
+
+import datasets
+import torch
+from datasets import load_dataset
+from torch import nn
+from transformers import AutoConfig, AutoModelForTokenClassification, AutoTokenizer
+from transformers.modeling_outputs import TokenClassifierOutput
+from transformers.modeling_utils import PreTrainedModel
+
+from .crf import MaskedCRFLoss
+
+
+@dataclass
+class TokenClassifierCRFOutput(TokenClassifierOutput):
+    loss: Optional[torch.FloatTensor] = None
+    real_path_score: Optional[torch.FloatTensor] = None
+    total_score: torch.FloatTensor = None
+    best_path_score: torch.FloatTensor = None
+    best_path: Optional[torch.LongTensor] = None
+    hidden_states: Optional[Tuple[torch.FloatTensor]] = None
+    attentions: Optional[Tuple[torch.FloatTensor]] = None
+
+
+class PretrainedCRFModel(PreTrainedModel):
+    config_class = AutoConfig
+
+    def __init__(self, config):
+        super().__init__(config)
+        self.encoder = AutoModelForTokenClassification.from_pretrained(
+            config._name_or_path, config=config
+        )
+        self.crf_model = MaskedCRFLoss(self.config.num_labels)
+        self.post_init()
+
+    def forward(
+        self,
+        input_ids=None,
+        token_type_ids=None,
+        attention_mask=None,
+        labels=None,
+        return_best_path=False,
+        **kwargs
+    ):
+        encoder_output = self.encoder(
+            input_ids=input_ids,
+            token_type_ids=token_type_ids,
+            attention_mask=attention_mask,
+            **kwargs
+        )
+
+        # Convert output to seq length as first dim
+
+        emissions = encoder_output.logits.transpose(1, 0)
+        tags = labels.transpose(1, 0)
+        mask = tags != -100
+        tags = tags.where(mask, 0)  # CRF cant support -100 id
+
+        crf_output = self.crf_model(
+            emissions, tags, mask, return_best_path=return_best_path
+        )
+
+        # Convert best_path to batch first
+        best_path = crf_output.best_path
+        if best_path is not None:
+            best_path = best_path.transpose(1, 0)
+
+        output = TokenClassifierCRFOutput(
+            loss=crf_output.loss,
+            real_path_score=crf_output.real_path_score,
+            total_score=crf_output.total_score,
+            best_path_score=crf_output.best_path_score,
+            best_path=best_path,
+            hidden_states=encoder_output.hidden_states,
+            attentions=encoder_output.attentions,
+        )
+        return output

pytorch_model.bin

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:957a43de6de74f7ecd4ba71f7c05807847718572ae4a3af8cd70c7f799baf978
+size 69004255

special_tokens_map.json

@@ -0,0 +1,7 @@
+{
+  "cls_token": "[CLS]",
+  "mask_token": "[MASK]",
+  "pad_token": "[PAD]",
+  "sep_token": "[SEP]",
+  "unk_token": "[UNK]"
+}

tokenizer_config.json

@@ -0,0 +1,65 @@
+{
+  "added_tokens_decoder": {
+    "0": {
+      "content": "[PAD]",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "100": {
+      "content": "[UNK]",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "101": {
+      "content": "[CLS]",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "102": {
+      "content": "[SEP]",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "103": {
+      "content": "[MASK]",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    }
+  },
+  "additional_special_tokens": [],
+  "clean_up_tokenization_spaces": true,
+  "cls_token": "[CLS]",
+  "do_basic_tokenize": true,
+  "do_lower_case": true,
+  "mask_token": "[MASK]",
+  "max_length": 128,
+  "model_max_length": 512,
+  "never_split": null,
+  "pad_to_multiple_of": null,
+  "pad_token": "[PAD]",
+  "pad_token_type_id": 0,
+  "padding_side": "right",
+  "sep_token": "[SEP]",
+  "stride": 0,
+  "strip_accents": null,
+  "tokenize_chinese_chars": true,
+  "tokenizer_class": "BertTokenizer",
+  "truncation_side": "right",
+  "truncation_strategy": "longest_first",
+  "unk_token": "[UNK]"
+}

trainer_state.json

@@ -0,0 +1,112 @@
+{
+  "best_metric": null,
+  "best_model_checkpoint": null,
+  "epoch": 2.0,
+  "eval_steps": 500,
+  "global_step": 108,
+  "is_hyper_param_search": false,
+  "is_local_process_zero": true,
+  "is_world_process_zero": true,
+  "log_history": [
+    {
+      "epoch": 1.0,
+      "eval_COMMENT": {
+        "f1": 0.036028823058446756,
+        "number": 1322,
+        "precision": 0.03826530612244898,
+        "recall": 0.0340393343419062
+      },
+      "eval_NAME": {
+        "f1": 0.021264689423614997,
+        "number": 1766,
+        "precision": 0.9047619047619048,
+        "recall": 0.010758776896942242
+      },
+      "eval_QTY": {
+        "f1": 0.0,
+        "number": 1434,
+        "precision": 0.0,
+        "recall": 0.0
+      },
+      "eval_RANGE_END": {
+        "f1": 0.0,
+        "number": 17,
+        "precision": 0.0,
+        "recall": 0.0
+      },
+      "eval_UNIT": {
+        "f1": 0.0,
+        "number": 1166,
+        "precision": 0.0,
+        "recall": 0.0
+      },
+      "eval_loss": 11.59915828704834,
+      "eval_overall_accuracy": 0.13186071187421627,
+      "eval_overall_f1": 0.018494437220054907,
+      "eval_overall_precision": 0.05263157894736842,
+      "eval_overall_recall": 0.011218229623137599,
+      "eval_runtime": 14.249,
+      "eval_samples_per_second": 119.447,
+      "eval_steps_per_second": 3.79,
+      "step": 54
+    },
+    {
+      "epoch": 2.0,
+      "eval_COMMENT": {
+        "f1": 0.030715225976305396,
+        "number": 1322,
+        "precision": 0.03657262277951933,
+        "recall": 0.0264750378214826
+      },
+      "eval_NAME": {
+        "f1": 0.024336283185840708,
+        "number": 1766,
+        "precision": 0.5238095238095238,
+        "recall": 0.01245753114382786
+      },
+      "eval_QTY": {
+        "f1": 0.003841229193341869,
+        "number": 1434,
+        "precision": 0.0234375,
+        "recall": 0.0020920502092050207
+      },
+      "eval_RANGE_END": {
+        "f1": 0.0,
+        "number": 17,
+        "precision": 0.0,
+        "recall": 0.0
+      },
+      "eval_UNIT": {
+        "f1": 0.0,
+        "number": 1166,
+        "precision": 0.0,
+        "recall": 0.0
+      },
+      "eval_loss": 10.259025573730469,
+      "eval_overall_accuracy": 0.12838815472171314,
+      "eval_overall_f1": 0.016813787305590584,
+      "eval_overall_precision": 0.04189944134078212,
+      "eval_overall_recall": 0.010517090271691499,
+      "eval_runtime": 15.3099,
+      "eval_samples_per_second": 111.17,
+      "eval_steps_per_second": 3.527,
+      "step": 108
+    },
+    {
+      "epoch": 2.0,
+      "step": 108,
+      "total_flos": 3334407253032.0,
+      "train_loss": 12.8824462890625,
+      "train_runtime": 151.3626,
+      "train_samples_per_second": 22.489,
+      "train_steps_per_second": 0.714
+    }
+  ],
+  "logging_steps": 500,
+  "max_steps": 108,
+  "num_train_epochs": 2,
+  "save_steps": 500,
+  "total_flos": 3334407253032.0,
+  "trial_name": null,
+  "trial_params": null
+}

training_args.bin

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:2c5254dbe381bcf40ffb21d73c66be21031100bd1243da923a4f4347812cc60c
+size 4155

validation_results.json

@@ -0,0 +1,41 @@
+{
+    "epoch": 2.0,
+    "eval_COMMENT": {
+        "f1": 0.030715225976305396,
+        "number": 1322,
+        "precision": 0.03657262277951933,
+        "recall": 0.0264750378214826
+    },
+    "eval_NAME": {
+        "f1": 0.024336283185840708,
+        "number": 1766,
+        "precision": 0.5238095238095238,
+        "recall": 0.01245753114382786
+    },
+    "eval_QTY": {
+        "f1": 0.003841229193341869,
+        "number": 1434,
+        "precision": 0.0234375,
+        "recall": 0.0020920502092050207
+    },
+    "eval_RANGE_END": {
+        "f1": 0.0,
+        "number": 17,
+        "precision": 0.0,
+        "recall": 0.0
+    },
+    "eval_UNIT": {
+        "f1": 0.0,
+        "number": 1166,
+        "precision": 0.0,
+        "recall": 0.0
+    },
+    "eval_loss": 10.259025573730469,
+    "eval_overall_accuracy": 0.12838815472171314,
+    "eval_overall_f1": 0.016813787305590584,
+    "eval_overall_precision": 0.04189944134078212,
+    "eval_overall_recall": 0.010517090271691499,
+    "eval_runtime": 15.7061,
+    "eval_samples_per_second": 108.365,
+    "eval_steps_per_second": 3.438
+}