Update README.md

README.md

@@ -176,23 +176,122 @@ Looking forward to your direction on this matter.
 
 
 
+
+
+
 # Example: Process anonymized version with GPT4 and change enteties back
 ```python
 import torch
+import json
+import re
 from transformers import AutoTokenizer, AutoModelForCausalLM
+
 tokenizer = AutoTokenizer.from_pretrained("metricspace/EntityAnonymization-3B-V0.9")
 model = AutoModelForCausalLM.from_pretrained("metricspace/EntityAnonymization-3B-V0.9", torch_dtype=torch.bfloat16)
+model.to("cuda:0")
 
-import re
 
-def extract_last_assistant_response(input_text):
-    # Find the occurrence of "ASSISTANT:" in the input text
-    match = re.search(r'ASSISTANT:', input_text)
+# Anonymized input
+anonymized_text = '''Subject: HR Incident Report: Speculation of Drug Misuse by Mr. Edward Martin
 
-    # Get the index where the last "ASSISTANT:" ends
-    start_index = match.end()
-    response = input_text[start_index:].strip()
-    return response
+Dear Mrs. Charlotte Johnson,
+
+I trust you're well. I wish to bring to your attention a concerning matter involving one of our esteemed employees, Mr. Edward Martin.
+
+Employee Details:
+
+Name: Edward Martin
+Position: Senior Marketing Creative
+Department: Marketing
+Date of Joining: January 15, 2020
+Reporting Manager: Mrs. Jane Anderson
+
+Incident Details:
+Date: October 25, 2023
+Location: Restroom, topmost floor
+Time: midday
+
+Description of Incident:
+On the date specified, a few colleagues reported unusual behavior exhibited by Mr. Martin, which raised concerns about potential drug misuse. Witnesses mentioned that Edward appeared disoriented and was found in the restroom for an extended period. Some employees also discovered unidentified pills in close proximity to his chair.
+
+Witness Accounts:
+Ms. Marie Foster: "Edward seemed distracted and not his usual self today. He's been taking frequent breaks and appears a bit disoriented."
+Mr. Benjamin Adams: "I found some pills near his chair on the floor. It's concerning, and I felt it necessary to report."
+
+Immediate Actions Taken:
+Mr. Edward Martin was approached by People Management for a preliminary conversation to understand the situation.
+Mrs. Jane Anderson, his reporting manager, was made aware of the concerns.
+
+Recommendations:
+It's crucial to have a private and supportive conversation with Mr. Martin to understand if there's an underlying issue.
+Consider referring Edward to our Personal Assistance Program (PAP) for counseling or support.
+It may be beneficial to organize a session on drug awareness and workplace safety for all employees.
+It's of utmost importance to handle this situation with sensitivity and discretion, ensuring the wellbeing of Mr. Martin and maintaining the integrity of our workplace environment. This email serves as a formal documentation of the incident. We'll determine the subsequent course of action based on your guidance and the recommendations provided.
+
+Looking forward to your direction on this matter.
+'''
+
+
+# Entities map
+
+entities_map = '''
+{
+    "Mr. Benjamin Mitchell": "Mr. Edward Martin",
+    "Mrs. Alice Williams": "Mrs. Charlotte Johnson",
+    "January 15, 2020": "January 15, 2020",
+    "Mrs. Jane Fitzgerald": "Mrs. Jane Anderson",
+    "October 25, 2023": "October 25, 2023",
+    "4th Floor": "topmost floor",
+    "11:45 AM": "midday",
+    "Emily Clark": "Marie Foster",
+    "Employee Assistance Program (EAP)": "Personal Assistance Program (PAP)",
+    "Robert Taylor": "Benjamin Adams",
+}
+'''
+
+
+
+# Step 1: Processing anonymized text with GPT-4
+
+import openai
+openai.api_key = f"<API_KEY>"
+
+completion = openai.ChatCompletion.create(
+  model="gpt-4",
+  messages=[
+    {"role": "user", "content": f"Write an official warning letter to the employee, that we do not tolerate it and with the next incident he gets fired in the name of Mrs. Charlotte Johnson Human Resources Manager. Here is the report with information. {anonymized_info}"}
+  ]
+)
+
+print(completion.choices[0].message.content)
+#output
+'''
+Subject: Official Warning – Substance Misuse Policy Violation
+
+Dear Mr. Edward Martin,
+
+We hope this letter finds you well. This letter serves as an official warning in regards to an incident that occurred on October 25, 2023, in which signs of drug misuse were reported. This alleged conduct is highly alarming and contrary to our company’s policies and guidelines.
+
+The incident detailed allegations of unusual behavior indicative of possible substance abuse. Colleagues reported that you appeared disoriented and were found in the restroom for an extended period. Additionally, unidentified pills were discovered near your workspace.
+
+Our company is committed to providing a safe and non-detrimental work environment for all its workforce. This commitment is compromised when any type of drug misuse occurs. We draw your attention to our Employee Handbook, specifically 'Section 5: Substance Misuse', that states any illegal drug use, substance misuse or distribution thereof is strictly prohibited and could be subject to severe disciplinary action, including termination.
+
+This is an official warning that such behavior misaligns with our workplace norms and cannot be tolerated. Another incident like this or similar breach of company guidelines may lead to further disciplinary action, up to and including termination of employment.
+
+Please note that this is not an assumption of your guilt but an assertion to remain vigilant against any practices that could harm you or the workplace environment. We encourage you to utilize our Personal Assistance Program (PAP) as a tool for counseling and support, if needed.
+
+We believe in your potential to rectify this situation and to maintain the high standards we are all accustomed to in our organization.
+
+Should you need assistance or if you wish to discuss this matter further, please feel free to reach out to me. We appreciate your immediate attention to this important issue.
+
+Yours sincerely,
+
+Mrs. Charlotte Johnson
+Human Resources Manager
+'''
+
+
+# Step 2: Replace the entities back in processed by GPT-4 text.
 
 import ast
 
@@ -208,59 +307,57 @@ def swap_keys_and_values_in_string(input_str):
 
     return swapped_str
 
-# sample text for entitity extraction and resampling
 
-original_text = '''Our organization, XYZ Biotech, operates at the forefront of groundbreaking pharmaceutical research, renowned for our pioneering drug development and breakthrough treatments. In light of the ever-evolving regulatory landscape and the need to safeguard our research endeavors, we've recognized the critical importance of enhancing our compliance and data security protocols. To this end, we are on the lookout for a top-notch regulatory affairs specialist to spearhead this transformation, ensuring the rigorous adherence to industry standards and the protection of our confidential research data.
+gpt_response = completion.choices[0].message.content
+entities_map = swap_keys_and_values_in_string(entities_map)
+prompt = f"USER: Rephrase with {entities_map}: {gpt_response}\n\nASSISTANT:"
+inputs = tokenizer(prompt, return_tensors='pt').to('cuda:0')
+outputs = model.generate(inputs.input_ids, max_new_tokens=2048)
+output_text = tokenizer.decode(outputs[0], skip_special_tokens=True)
 
-This comprehensive initiative encompasses not only ensuring regulatory compliance but also the implementation of three vital security measures. We will be utilizing CipherGuard's state-of-the-art encryption technology to secure our research data, deploying BioShield's advanced security protocols for laboratory access, and integrating SecureLabs' real-time data monitoring and threat detection systems.
+#output:
+'''
+Subject: Official Warning – Substance Misuse Policy Violation
 
-The enhancement of our regulatory affairs and data security measures is a critical component in safeguarding our proprietary research, reinforcing our commitment to drug development excellence. While we prioritize compliance and data protection, the user experience for our research teams and partners will remain user-friendly and efficient, whether they are using our proprietary research software, "BioDiscover," or our mobile applications.
+Dear Mr. Benjamin Mitchell,
 
-We are actively in search of a regulatory affairs specialist who can comprehend the importance of maintaining compliance and data security in our industry and who can deliver a comprehensive, airtight solution that not only ensures our adherence to regulations but also safeguards the confidential nature of our research at XYZ Biotech.''',
-'''
+We hope this letter finds you well. This letter serves as an official warning in regards to an incident that occurred on January 15,
 
+2020, in which signs of drug misuse were reported. This alleged conduct is highly alarming and contrary to our company’s policies and guidelines.
 
-# another different anonymized text with replaced entitie
+The incident detailed allegations of unusual behavior indicative of possible substance abuse. Colleagues reported that you appeared disoriented and
 
-anonymized_text = '''ABC Pharmaceuticals, a renowned player in the pharmaceutical industry, is dedicated to pioneering drug development and breakthrough treatments. In response to the ever-evolving regulatory landscape and the need to protect our research initiatives, we have identified the paramount importance of enhancing our compliance and data security protocols. As a part of this strategic shift, we are actively searching for a top-tier regulatory affairs specialist to lead this transformation, ensuring unwavering adherence to industry standards and the safeguarding of our confidential research data.
+were found in the restroom for an extended period. Additionally, unidentified pills were discovered near your workspace.
 
-This comprehensive initiative goes beyond regulatory compliance and entails the implementation of three crucial security measures. We will be leveraging the cutting-edge encryption technology provided by CodeGuard to secure our research data, implementing BioProtect's advanced security protocols for laboratory access, and integrating the real-time data monitoring and threat detection systems offered by SecureTech.
+Our company is committed to providing a safe and non-detrimental work environment for all its workforce. This commitment is compromised when any
 
-The enhancement of our regulatory affairs and data security measures is a pivotal component in safeguarding our proprietary research, reinforcing our commitment to excellence in drug development. While we prioritize compliance and data protection, the user experience for our research teams and partners will remain user-friendly and efficient, whether they are using our proprietary research software, "BioDiscover," or our mobile applications.
+type of drug misuse occurs. We draw your attention to our Employee Handbook, specifically 'Section 5: Substance Misuse', that states any illegal
 
-We are actively seeking a regulatory affairs specialist who comprehends the critical importance of upholding compliance and data security in our industry and possesses the expertise to deliver a comprehensive and impervious solution that ensures not only our adherence to regulations but also preserves the confidentiality of our research data at ABC Pharmaceuticals.'''
+drug use, substance misuse or distribution thereof is strictly prohibited and could be subject to severe disciplinary action, including termination.
 
+This is an official warning that such behavior misaligns with our workplace norms and cannot be tolerated. Another incident like this or similar breach
 
-prompt = f'USER: Resample the entities: {original_text}\n\nASSISTANT:'
-inputs = tokenizer(prompt, return_tensors='pt').to('cuda')
-outputs = model.generate(inputs.input_ids, max_new_tokens=250, do_sample=False, top_k=50, top_p=0.98, num_beams=1)
-output_text_1 = tokenizer.decode(outputs[0], skip_special_tokens=True)
+of company guidelines may lead to further disciplinary action, up to and including termination of employment.
 
+Please note that this is not an assumption of your guilt but an assertion to remain vigilant against any practices that could harm you or the workplace
 
-generated_part = extract_assistant_response(output_text_1)
+environment. We encourage you to utilize our Employee Assistance Program (EAP) as a tool for counseling and support, if needed.
 
-# inverting the entity map
-# {'XYZ Biotech': 'ABC Pharmaceuticals', 'CipherGuard': 'CodeGuard', 'BioShield': 'BioProtect', 'SecureLabs': 'SecureTech', 'BioDiscover': 'BioDiscover'}
-# inverted to this:
-# {'ABC Pharmaceuticals': 'XYZ Biotech', 'CodeGuard': 'CipherGuard', 'BioProtect': 'BioShield', 'SecureTech': 'SecureLabs', 'BioDiscover': 'BioDiscover'}
+We believe in your potential to rectify this situation and to maintain the high standards we are all accustomed to in our organization.
 
-inverted_entities = swap_keys_and_values_in_string(generated_part)
+Should you need assistance or if you wish to discuss this matter further, please feel free to reach out to me. We appreciate your immediate attention
 
+to this important issue.
 
-prompt_2 = f"USER: Rephrase with {inverted_entities}: {anonymized_text}\n\nASSISTANT:"
-inputs = tokenizer(prompt_2, return_tensors='pt').to('cuda')
-outputs = model.generate(inputs.input_ids, max_new_tokens=500, do_sample=False, top_k=50, top_p=0.98)
-output_text_2 = tokenizer.decode(outputs[0], skip_special_tokens=True)
+Yours sincerely,
 
-print(output_text_2)
+Mrs. Alice Williams,
 
+Human Resources Manager.
 '''
-XYZ Biotech, a renowned player in the biotech industry, is dedicated to pioneering drug development and breakthrough treatments. In response to the ever-evolving regulatory landscape and the need to protect our research initiatives, we have identified the paramount importance of enhancing our compliance and data security protocols. As a part of this strategic shift, we are actively searching for a top-tier regulatory affairs specialist to lead this transformation, ensuring unwavering adherence to industry standards and the safeguarding of our confidential research data.
-This comprehensive initiative goes beyond regulatory compliance and entails the implementation of three crucial security measures. We will be leveraging the cutting-edge encryption technology provided by CipherGuard to secure our research data, implementing BioShield's advanced security protocols for laboratory access, and integrating the real-time data monitoring and threat detection systems offered by SecureLabs.
-The enhancement of our regulatory affairs and data security measures is a pivotal component in safeguarding our proprietary research, reinforcing our commitment to excellence in drug development. While we prioritize compliance and data protection, the user experience for our research teams and partners will remain user-friendly and efficient, whether they are using our proprietary research software, "BioDiscover," or our mobile applications.
-We are actively seeking a regulatory affairs specialist who comprehends the critical importance of upholding compliance and data security in our industry and possesses the expertise to deliver a comprehensive and impervious solution that ensures not only our adherence to regulations but also preserves the confidentiality of our research data at XYZ Biotech.
-'''
+
 ```
+
 …
 # Dataset and Training Documentation for Audit
 If you require the original dataset used for training this model, or further documentation related to its training and architecture for audit purposes, you can request this information by contacting us.