Adding Custom Entity into model architecture!
is it possible to add a custom entities into the model architecture ?
You would have to finetune the base model to include that
Hello there, what is the process ?
Let's say I want to create a new category called "avenger", and I got the list of name of all marvel superhero (hulk, ironman, etc...) ? Can someone give an overview of how to complete the base model so it can label "avenger" the name "ironman" when found in a sentence ?
Hello there, what is the process ?
Let's say I want to create a new category called "avenger", and I got the list of name of all marvel superhero (hulk, ironman, etc...) ? Can someone give an overview of how to complete the base model so it can label "avenger" the name "ironman" when found in a sentence ?
Anyone has a proposed method for this?
Hello there, what is the process ?
Let's say I want to create a new category called "avenger", and I got the list of name of all marvel superhero (hulk, ironman, etc...) ? Can someone give an overview of how to complete the base model so it can label "avenger" the name "ironman" when found in a sentence ?Anyone has a proposed method for this?
Here's a detailed guide:
- Data collection and preparation
Data collection: Collect data with texts containing personally identifiable information (PII) that you want to discover. This data should be tagged, meaning each entity should be labeled with the appropriate tag (e.g. AVENGER for all identifiable avenger names).
Data Format: Data should generally be in a format compatible with the library you are using (e.g., CoNLL). Make sure the data is labeled correctly and contains all necessary tags.
Model Setup
Model selection: Use a model that supports the desired language.
Data preparation: Use transformers for tokenization and data preparation. Make sure the data is split into training and test samples.Model pre-training
Hyperparameter tuning: Define hyperparameters for training such as batch size, number of epochs, learning rate, etc.
Training process: Use PyTorch or other suitable library to pre-train the model on your data. Note the use of GPUs to speed up the process.
Model evaluation: After training, evaluate the model on a test sample to ensure its quality. Use metrics such as accuracy, completeness, and F1-score.Save model you fine-tuned and test it