commit

README.md

@@ -1,3 +1,68 @@
 ---
 license: apache-2.0
+datasets:
+- metricspace/AnonymeData
+pipeline_tag: text2text-generation
 ---
+
+# EntityAnonymization-3B-V0.9
+
+# License
+This Natural Language Processing (NLP) model is made available under the Apache License, Version 2.0. You are free to use, modify, and distribute this software according to the terms and conditions of the Apache 2.0 License. For the full license text, please refer to the Apache 2.0 License.
+# Usage and Specific Capabilities
+## Text Length Limitation
+The model is optimized to analyze texts containing up to 2048 tokens. If your text exceeds this limit, we recommend splitting it into smaller chunks, each containing no more than 512 tokens. Each chunk can then be processed separately.
+## Supported Languages
+Bulgarian, Chinese, Czech, Dutch, English, Estonian, Finnish, French, German, Greek, Indonesian, Italian, Japanese, Korean, Lithuanian, Norwegian, Polish, Portuguese, Romanian, Russian, Slovak, Spanish, Swedish, Turkish
+
+
+# Use Cases
+## Entity Resampling and Anonymization
+Introducing a cutting-edge model tailored to the task of extracting entities from sensitive text and anonymizing it. This model specializes in identifying and safeguarding confidential information, ensuring organizations' compliance with stringent data privacy regulations and minimizing the potential for inadvertent disclosure of classified data and trade secrets.
+# Example Usage
+```python
+import torch
+from transformers import AutoTokenizer, AutoModelForCausalLM
+tokenizer = AutoTokenizer.from_pretrained("metricspace/EntityAnonymization-3B-V0.9")
+model = AutoModelForCausalLM.from_pretrained("metricspace/EntityAnonymization-3B-V0.9", torch_dtype=torch.bfloat16)
+
+def extract_assistant_response(input_text):
+    # Find all occurrences of "ASSISTANT:" in the input text
+    matches = re.finditer(r'ASSISTANT:', input_text)
+
+    # Extract text after each occurrence of "ASSISTANT:"
+    assistant_responses = []
+    for match in matches:
+        start_index = match.end()  # Get the index where "ASSISTANT:" ends
+        response = input_text[start_index:].strip()
+        assistant_responses.append(response)
+
+    return assistant_responses
+
+
+text_to_anonymize = "Sophia had always been enchanted by Venice, a historic city nestled in the heart of the Venetian lagoon. She had explored Venice on numerous occasions, each visit revealing hidden treasures in the enchanting city. On her latest trip, Sophia met Marco, a local historian, who shared captivating stories about the history of Venice.""
+
+
+prompt = f'USER: Resample the entities: {text_to_anonymize}\n\nASSISTANT:'
+inputs = tokenizer(prompt, return_tensors='pt').to('cuda')
+output_entities = model.generate(inputs.input_ids, max_new_tokens=250, do_sample=False, top_k=50, top_p=0.98, num_beams=1)
+output_entities_text = tokenizer.decode(output_entities[0], skip_special_tokens=True)
+
+# extracting entities text from assistant response
+generated_part = extract_assistant_response(output_text_1)[0]
+
+prompt_2 = f"USER: Rephrase with {generated_part}: {text_to_anonymize}\n\nASSISTANT:"
+inputs = tokenizer(prompt_2, return_tensors='pt').to('cuda')
+output_resampled = model.generate(inputs.input_ids, max_new_tokens=500, do_sample=False, top_k=50, top_p=0.98)
+output_resampled_text = tokenizer.decode(output_resampled[0], skip_special_tokens=True)
+
+
+print(output_resampled_text)
+```
+…
+# Dataset and Training Documentation for Audit
+If you require the original dataset used for training this model, or further documentation related to its training and architecture for audit purposes, you can request this information by contacting us.
+Further Tuning Services for Custom Use Cases
+For specialized needs or custom use cases, we offer further tuning services to adapt the model to your specific requirements. To inquire about these services, please reach out to us at:
+📧 Email: info@metric-space.ai
+Please note that the availability of the dataset, additional documentation, and tuning services may be subject to certain conditions and limitations.