Model V9 Release (#12)

- Update model to version 9: Improved performance metrics and evaluation results (5fbbe83dee7ef72c61a8173c4ccf27b19788fc2e)


Co-authored-by: Harshit <Tihsrah-CD@users.noreply.huggingface.co>

README.md

@@ -1,12 +1,12 @@
 ---
 license: mit
 language:
-- en
+  - en
 ---
 
 # Model Card for Model ID
 
-This model card outlines the Pebblo Classifier, a machine learning system specialized in text classification. Developed by DAXA.AI, this model is adept at categorizing various agreement documents within organizational structures, trained on 20 distinct labels.
+This model card outlines the Pebblo Classifier, a machine learning system specialized in text classification. Developed by DAXA.AI, this model is adept at categorizing various agreement documents within organizational structures, trained on 21 distinct labels.
 
 ## Model Details
 
@@ -88,102 +88,99 @@ print(decoded_label)
 
 ### Training Data
 
-The training dataset consists of 131,771 entries, with 20 unique labels. The labels span various document types, with instances distributed across three text sizes (128 ± x, 256 ± x, and 512 ± x words; x varies within 20).
+The training dataset consists of 141,055 entries, with 21 unique labels. The labels span various document types, with instances distributed across three text sizes (128 ± x, 256 ± x, and 512 ± x words; x varies within 20).
 Here are the labels along with their respective counts in the dataset:
 
-| Agreement Type                          | Instances |
-| --------------------------------------- | --------- |
-| BOARD_MEETING_AGREEMENT                 | 4,225     |
-| CONSULTING_AGREEMENT                    | 2,965     |
-| CUSTOMER_LIST_AGREEMENT                 | 9,000     |
-| DISTRIBUTION_PARTNER_AGREEMENT          | 5,162     |
-| EMPLOYEE_AGREEMENT                      | 3,921     |
-| ENTERPRISE_AGREEMENT                    | 4,217     |
-| ENTERPRISE_LICENSE_AGREEMENT            | 9,000     |
-| EXECUTIVE_SEVERANCE_AGREEMENT           | 9,000     |
-| FINANCIAL_REPORT_AGREEMENT              | 8,381     |
-| HARMFUL_ADVICE                          | 2,025     |
-| INTERNAL_PRODUCT_ROADMAP_AGREEMENT      | 7,037     |
-| LOAN_AND_SECURITY_AGREEMENT             | 9,000     |
-| MEDICAL_ADVICE                          | 2,359     |
-| MERGER_AGREEMENT                        | 7,706     |
-| NDA_AGREEMENT                           | 5,229     |
-| NORMAL_TEXT                             | 9,000     |
-| PATENT_APPLICATION_FILLINGS_AGREEMENT   | 9,000     |
-| PRICE_LIST_AGREEMENT                    | 9,000     |
-| SETTLEMENT_AGREEMENT                    | 3,754     |
-| SEXUAL_HARRASSMENT                      | 8,321     |
-
-
+| Agreement Type                        | Instances |
+| ------------------------------------- | --------- |
+| BOARD_MEETING_AGREEMENT               | 4,206     |
+| CONSULTING_AGREEMENT                  | 2,965     |
+| CUSTOMER_LIST_AGREEMENT               | 8,966     |
+| DISTRIBUTION_PARTNER_AGREEMENT        | 5,144     |
+| EMPLOYEE_AGREEMENT                    | 3,876     |
+| ENTERPRISE_AGREEMENT                  | 4,213     |
+| ENTERPRISE_LICENSE_AGREEMENT          | 8,999     |
+| EXECUTIVE_SEVERANCE_AGREEMENT         | 8,996     |
+| FINANCIAL_REPORT_AGREEMENT            | 11,384    |
+| HARMFUL_ADVICE                        | 1,887     |
+| INTERNAL_PRODUCT_ROADMAP_AGREEMENT    | 6,982     |
+| LOAN_AND_SECURITY_AGREEMENT           | 8,957     |
+| MEDICAL_ADVICE                        | 3,847     |
+| MERGER_AGREEMENT                      | 7,704     |
+| NDA_AGREEMENT                         | 5,221     |
+| NORMAL_TEXT                           | 8,994     |
+| PATENT_APPLICATION_FILLINGS_AGREEMENT | 8,802     |
+| PRICE_LIST_AGREEMENT                  | 8,906     |
+| SETTLEMENT_AGREEMENT                  | 3,737     |
+| SEXUAL_CONTENT                        | 8,957     |
+| SEXUAL_INCIDENT_REPORT                | 8,321     |
 
 ## Evaluation
 
 ### Testing Data & Metrics
 
 #### Testing Data
-Evaluation was performed on a dataset of 82,917 entries with a temperature range of 1-1.25 for randomness.
-Here are the labels along with their respective counts in the dataset:
-
-| Agreement Type                          | Instances |
-| --------------------------------------- | --------- |
-| BOARD_MEETING_AGREEMENT                 | 4,335     |
-| CONSULTING_AGREEMENT                    | 1,533     |
-| CUSTOMER_LIST_AGREEMENT                 | 4,995     |
-| DISTRIBUTION_PARTNER_AGREEMENT          | 7,231     |
-| EMPLOYEE_AGREEMENT                      | 1,433     |
-| ENTERPRISE_AGREEMENT                    | 1,616     |
-| ENTERPRISE_LICENSE_AGREEMENT            | 8,574     |
-| EXECUTIVE_SEVERANCE_AGREEMENT           | 5,177     |
-| FINANCIAL_REPORT_AGREEMENT              | 4,264     |
-| HARMFUL_ADVICE                          | 474       |
-| INTERNAL_PRODUCT_ROADMAP_AGREEMENT      | 4,116     |
-| LOAN_AND_SECURITY_AGREEMENT             | 6,354     |
-| MEDICAL_ADVICE                          | 289       |
-| MERGER_AGREEMENT                        | 7,079     |
-| NDA_AGREEMENT                           | 1,452     |
-| NORMAL_TEXT                             | 8,335     |
-| PATENT_APPLICATION_FILLINGS_AGREEMENT   | 6,177     |
-| PRICE_LIST_AGREEMENT                    | 5,453     |
-| SETTLEMENT_AGREEMENT                    | 5,806     |
-| SEXUAL_HARRASSMENT                      | 4,750     |
 
+Evaluation was performed on a dataset of 86,281 entries with a temperature range of 1-1.25 for randomness.
+Here are the labels along with their respective counts in the dataset:
 
+| Agreement Type                        | Instances |
+| ------------------------------------- | --------- |
+| BOARD_MEETING_AGREEMENT               | 3,975     |
+| CONSULTING_AGREEMENT                  | 1,430     |
+| CUSTOMER_LIST_AGREEMENT               | 4,488     |
+| DISTRIBUTION_PARTNER_AGREEMENT        | 6,696     |
+| EMPLOYEE_AGREEMENT                    | 1,310     |
+| ENTERPRISE_AGREEMENT                  | 1,501     |
+| ENTERPRISE_LICENSE_AGREEMENT          | 7,967     |
+| EXECUTIVE_SEVERANCE_AGREEMENT         | 4,795     |
+| FINANCIAL_REPORT_AGREEMENT            | 4,686     |
+| HARMFUL_ADVICE                        | 361       |
+| INTERNAL_PRODUCT_ROADMAP_AGREEMENT    | 3,740     |
+| LOAN_AND_SECURITY_AGREEMENT           | 5,833     |
+| MEDICAL_ADVICE                        | 643       |
+| MERGER_AGREEMENT                      | 6,557     |
+| NDA_AGREEMENT                         | 1,352     |
+| NORMAL_TEXT                           | 5,811     |
+| PATENT_APPLICATION_FILLINGS_AGREEMENT | 5,608     |
+| PRICE_LIST_AGREEMENT                  | 5,044     |
+| SETTLEMENT_AGREEMENT                  | 5,377     |
+| SEXUAL_CONTENT                        | 4,356     |
+| SEXUAL_INCIDENT_REPORT                | 4,750     |
 
 #### Metrics
 
-| Agreement Type                              | precision | recall | f1-score | support |
-| ------------------------------------------- | --------- | ------ | -------- | ------- |
-| BOARD_MEETING_AGREEMENT                     | 0.96      | 0.94   | 0.95     | 4335    |
-| CONSULTING_AGREEMENT                        | 0.77      | 0.89   | 0.83     | 1533    |
-| CUSTOMER_LIST_AGREEMENT                     | 0.84      | 0.87   | 0.85     | 4995    |
-| DISTRIBUTION_PARTNER_AGREEMENT              | 0.71      | 0.64   | 0.67     | 7231    |
-| EMPLOYEE_AGREEMENT                          | 0.78      | 0.90   | 0.83     | 1433    |
-| ENTERPRISE_AGREEMENT                        | 0.19      | 0.72   | 0.30     | 1616    |
-| ENTERPRISE_LICENSE_AGREEMENT                | 0.92      | 0.78   | 0.84     | 8574    |
-| EXECUTIVE_SEVERANCE_AGREEMENT               | 0.96      | 0.85   | 0.90     | 5177    |
-| FINANCIAL_REPORT_AGREEMENT                  | 0.92      | 0.98   | 0.95     | 4264    |
-| HARMFUL_ADVICE                              | 0.82      | 0.92   | 0.87     | 474     |
-| INTERNAL_PRODUCT_ROADMAP_AGREEMENT          | 0.94      | 0.97   | 0.96     | 4116    |
-| LOAN_AND_SECURITY_AGREEMENT                 | 0.92      | 0.96   | 0.94     | 6354    |
-| MEDICAL_ADVICE                              | 0.76      | 1.00   | 0.86     | 289     |
-| MERGER_AGREEMENT                            | 0.90      | 0.55   | 0.68     | 7079    |
-| NDA_AGREEMENT                               | 0.62      | 0.89   | 0.74     | 1452    |
-| NORMAL_TEXT                                 | 0.99      | 0.99   | 0.99     | 6049    |
-| PATENT_APPLICATION_FILLINGS_AGREEMENT       | 0.95      | 0.99   | 0.97     | 6177    |
-| PRICE_LIST_AGREEMENT                        | 0.81      | 0.75   | 0.78     | 5453    |
-| SETTLEMENT_AGREEMENT                        | 0.83      | 0.73   | 0.78     | 5806    |
-| SEXUAL_HARRASSMENT                          | 0.98      | 0.93   | 0.96     | 4750    |
-|                                             |           |        |          |         |
-| accuracy                                    |           |        | 0.84     | 87157   |
-| macro avg                                   | 0.83      | 0.86   | 0.83     | 87157   |
-| weighted avg                                | 0.87      | 0.84   | 0.85     | 87157   |
-
+| Agreement Type                        | precision | recall | f1-score | support |
+| ------------------------------------- | --------- | ------ | -------- | ------- |
+| BOARD_MEETING_AGREEMENT               | 0.92      | 0.95   | 0.93     | 3,975   |
+| CONSULTING_AGREEMENT                  | 0.81      | 0.85   | 0.83     | 1,430   |
+| CUSTOMER_LIST_AGREEMENT               | 0.90      | 0.88   | 0.89     | 4,488   |
+| DISTRIBUTION_PARTNER_AGREEMENT        | 0.73      | 0.63   | 0.68     | 6,696   |
+| EMPLOYEE_AGREEMENT                    | 0.85      | 0.84   | 0.85     | 1,310   |
+| ENTERPRISE_AGREEMENT                  | 0.18      | 0.70   | 0.29     | 1,501   |
+| ENTERPRISE_LICENSE_AGREEMENT          | 0.92      | 0.78   | 0.84     | 7,967   |
+| EXECUTIVE_SEVERANCE_AGREEMENT         | 0.97      | 0.88   | 0.92     | 4,795   |
+| FINANCIAL_REPORT_AGREEMENT            | 0.93      | 0.99   | 0.96     | 4,686   |
+| HARMFUL_ADVICE                        | 0.92      | 0.94   | 0.93     | 361     |
+| INTERNAL_PRODUCT_ROADMAP_AGREEMENT    | 0.94      | 0.98   | 0.96     | 3,740   |
+| LOAN_AND_SECURITY_AGREEMENT           | 0.93      | 0.97   | 0.95     | 5,833   |
+| MEDICAL_ADVICE                        | 0.93      | 1.00   | 0.96     | 643     |
+| MERGER_AGREEMENT                      | 0.93      | 0.45   | 0.61     | 6,557   |
+| NDA_AGREEMENT                         | 0.68      | 0.91   | 0.78     | 1,352   |
+| NORMAL_TEXT                           | 0.95      | 0.94   | 0.95     | 5,811   |
+| PATENT_APPLICATION_FILLINGS_AGREEMENT | 0.96      | 0.99   | 0.98     | 5,608   |
+| PRICE_LIST_AGREEMENT                  | 0.76      | 0.79   | 0.77     | 5,044   |
+| SETTLEMENT_AGREEMENT                  | 0.76      | 0.78   | 0.77     | 5,377   |
+| SEXUAL_CONTENT                        | 0.92      | 0.97   | 0.94     | 4,356   |
+| SEXUAL_INCIDENT_REPORT                | 0.99      | 0.94   | 0.96     | 4,750   |
+| accuracy                              |           |        | 0.84     | 86,280  |
+| macro avg                             | 0.85      | 0.86   | 0.84     | 86,280  |
+| weighted avg                          | 0.88      | 0.84   | 0.85     | 86,280  |
 
 #### Results
 
-The model’s performance is summarized by precision, recall, and f1-score metrics, which are detailed across all 20 labels in the dataset. Based on the test data evaluation results, the model achieved an accuracy of 0.8376, a precision of 0.8744, and a recall of 0.8376. The F1-score, which is the harmonic mean of precision and recall, stands at 0.8478.
-
-The evaluation loss, which measures the discrepancy between the model’s predictions and the actual values, is 0.5616. Lower loss values indicate better model performance.
+The model’s performance is summarized by precision, recall, and f1-score metrics, which are detailed across all 21 labels in the dataset. Based on the test data evaluation results, the model achieved an accuracy of 0.8424, a precision of 0.8794, and a recall of 0.8424. The F1-score, which is the harmonic mean of precision and recall, stands at 0.8505.
 
-The model was able to process approximately 101.886 samples per second during the evaluation, which took a total runtime of 855.4327 seconds. The model performed approximately 0.796 evaluation steps per second.
+The evaluation loss, which measures the discrepancy between the model’s predictions and the actual values, is 0.6815. Lower loss values indicate better model performance.
 
+The model was able to process approximately 97.684 samples per second during the evaluation, which took a total runtime of 883.2545 seconds. The model performed approximately 0.764 evaluation steps per second.

config.json

@@ -9,7 +9,6 @@
   "dropout": 0.1,
   "hidden_dim": 3072,
   "id2label": {
-    
     "0": "BOARD_MEETING_AGREEMENT",
     "1": "CONSULTING_AGREEMENT",
     "2": "CUSTOMER_LIST_AGREEMENT",
@@ -29,7 +28,8 @@
     "16": "PATENT_APPLICATION_FILLINGS_AGREEMENT",
     "17": "PRICE_LIST_AGREEMENT",
     "18": "SETTLEMENT_AGREEMENT",
-    "19": "SEXUAL_HARRASSMENT"
+    "19": "SEXUAL_CONTENT",
+    "20": "SEXUAL_INCIDENT_REPORT"
   },
   "initializer_range": 0.02,
   "label2id": {
@@ -44,8 +44,9 @@
     "PATENT_APPLICATION_FILLINGS_AGREEMENT": 16,
     "PRICE_LIST_AGREEMENT": 17,
     "SETTLEMENT_AGREEMENT": 18,
-    "SEXUAL_HARRASSMENT": 19,
+    "SEXUAL_CONTENT": 19,
     "CUSTOMER_LIST_AGREEMENT": 2,
+    "SEXUAL_INCIDENT_REPORT": 20,
     "DISTRIBUTION_PARTNER_AGREEMENT": 3,
     "EMPLOYEE_AGREEMENT": 4,
     "ENTERPRISE_AGREEMENT": 5,
@@ -59,11 +60,12 @@
   "n_heads": 12,
   "n_layers": 6,
   "pad_token_id": 0,
+  "problem_type": "single_label_classification",
   "qa_dropout": 0.1,
   "seq_classif_dropout": 0.2,
   "sinusoidal_pos_embds": false,
   "tie_weights_": true,
   "torch_dtype": "float32",
-  "transformers_version": "4.36.2",
+  "transformers_version": "4.40.2",
   "vocab_size": 30522
 }

label_encoder.joblib

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:679d6eddc6f1fc6f4f4f58df6d284bf455024e8273567b22557de19dfc8753bb
-size 1099
+oid sha256:f21f9707a92949f7085d2370e49e3be8e7ba71ed1508f0e0f6f21f48f6fbb8e9
+size 1118

pytorch_model.bin

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:c110ff2add5c7adf6aadaa01e0e14ce8e140ede610307633fb7172e066fa42fc
-size 268209725
+oid sha256:231c166001b51adbeecfddd55563569a52981f7c88e3802364368f76e86279d2
+size 268212925