Token Classification

Token classification is the task of classifying each token in a sequence. This can be used for Named Entity Recognition (NER), Part-of-Speech (POS) tagging, and more. Get your data ready in proper format and then with just a few clicks, your state-of-the-art model will be ready to be used in production.

Data Format

The data should be in the following CSV format:

tokens,tags
"['I', 'love', 'Paris']", "['O', 'O', 'B-LOC']"
"['I', 'live', 'in', 'New', 'York']", "['O', 'O', 'O', 'B-LOC', 'I-LOC']"
.
.
.

or you can also use JSONL format:

{"tokens": ["I", "love", "Paris"], "tags": ["O", "O", "B-LOC"]}
{"tokens": ["I", "live", "in", "New", "York"], "tags": ["O", "O", "O", "B-LOC", "I-LOC"]}
.
.
.

As you can see, we have two columns in the CSV file. One column is the tokens and the other is the tags. Both the columns are stringified lists! The tokens column contains the tokens of the sentence and the tags column contains the tags for each token.

If your CSV is huge, you can divide it into multiple CSV files and upload them separately. Please make sure that the column names are the same in all CSV files.

One way to divide the CSV file using pandas is as follows:

import pandas as pd

# Set the chunk size
chunk_size = 1000
i = 1

# Open the CSV file and read it in chunks
for chunk in pd.read_csv('example.csv', chunksize=chunk_size):
    # Save each chunk to a new file
    chunk.to_csv(f'chunk_{i}.csv', index=False)
    i += 1

Columns

Your CSV/JSONL dataset must have two columns: tokens and tags.

Parameters

❯ autotrain token-classification --help
usage: autotrain <command> [<args>] token-classification [-h] [--train] [--deploy] [--inference] [--username USERNAME]
                                                         [--backend {local-cli,spaces-a10gl,spaces-a10gs,spaces-a100,spaces-t4m,spaces-t4s,spaces-cpu,spaces-cpuf}]
                                                         [--token TOKEN] [--push-to-hub] --model MODEL --project-name PROJECT_NAME
                                                         [--data-path DATA_PATH] [--train-split TRAIN_SPLIT] [--valid-split VALID_SPLIT]
                                                         [--batch-size BATCH_SIZE] [--seed SEED] [--epochs EPOCHS]
                                                         [--gradient_accumulation GRADIENT_ACCUMULATION] [--disable_gradient_checkpointing]
                                                         [--lr LR] [--log {none,wandb,tensorboard}] [--tokens-column TOKENS_COLUMN]
                                                         [--tags-column TAGS_COLUMN] [--max-seq-length MAX_SEQ_LENGTH] [--warmup-ratio WARMUP_RATIO]
                                                         [--optimizer OPTIMIZER] [--scheduler SCHEDULER] [--weight-decay WEIGHT_DECAY]
                                                         [--max-grad-norm MAX_GRAD_NORM] [--logging-steps LOGGING_STEPS]
                                                         [--evaluation-strategy {steps,epoch,no}] [--save-total-limit SAVE_TOTAL_LIMIT]
                                                         [--auto-find-batch-size]
                                                         [--mixed-precision {fp16,bf16,None}]

✨ Run AutoTrain Token Classification

options:
  -h, --help            show this help message and exit
  --train               Command to train the model
  --deploy              Command to deploy the model (limited availability)
  --inference           Command to run inference (limited availability)
  --username USERNAME   Hugging Face Hub Username
  --backend {local-cli,spaces-a10gl,spaces-a10gs,spaces-a100,spaces-t4m,spaces-t4s,spaces-cpu,spaces-cpuf}
                        Backend to use: default or spaces. Spaces backend requires push_to_hub & username. Advanced users only.
  --token TOKEN         Your Hugging Face API token. Token must have write access to the model hub.
  --push-to-hub         Push to hub after training will push the trained model to the Hugging Face model hub.
  --model MODEL         Base model to use for training
  --project-name PROJECT_NAME
                        Output directory / repo id for trained model (must be unique on hub)
  --data-path DATA_PATH
                        Train dataset to use. When using cli, this should be a directory path containing training and validation data in appropriate
                        formats
  --train-split TRAIN_SPLIT
                        Train dataset split to use
  --valid-split VALID_SPLIT
                        Validation dataset split to use
  --batch-size BATCH_SIZE
                        Training batch size to use
  --seed SEED           Random seed for reproducibility
  --epochs EPOCHS       Number of training epochs
  --gradient_accumulation GRADIENT_ACCUMULATION
                        Gradient accumulation steps
  --disable_gradient_checkpointing
                        Disable gradient checkpointing
  --lr LR               Learning rate
  --log {none,wandb,tensorboard}
                        Use experiment tracking
  --tokens-column TOKENS_COLUMN
                        Tokens column to use. Must be a stringified list of tokens if using a CSV file. Default is 'tokens'.
  --tags-column TAGS_COLUMN
                        Tags column to use. Must be a stringified list of tags if using a CSV file. Default is 'tags'.
  --max-seq-length MAX_SEQ_LENGTH
                        Set the maximum sequence length (number of tokens) that the model should handle in a single input. Longer sequences are
                        truncated. Affects both memory usage and computational requirements. Default is 128 tokens.
  --warmup-ratio WARMUP_RATIO
                        Define the proportion of training to be dedicated to a linear warmup where learning rate gradually increases. This can help
                        in stabilizing the training process early on. Default ratio is 0.1.
  --optimizer OPTIMIZER
                        Choose the optimizer algorithm for training the model. Different optimizers can affect the training speed and model
                        performance. 'adamw_torch' is used by default.
  --scheduler SCHEDULER
                        Select the learning rate scheduler to adjust the learning rate based on the number of epochs. 'linear' decreases the
                        learning rate linearly from the initial lr set. Default is 'linear'. Try 'cosine' for a cosine annealing schedule.
  --weight-decay WEIGHT_DECAY
                        Set the weight decay rate to apply for regularization. Helps in preventing the model from overfitting by penalizing large
                        weights. Default is 0.0, meaning no weight decay is applied.
  --max-grad-norm MAX_GRAD_NORM
                        Specify the maximum norm of the gradients for gradient clipping. Gradient clipping is used to prevent the exploding gradient
                        problem in deep neural networks. Default is 1.0.
  --logging-steps LOGGING_STEPS
                        Determine how often to log training progress. Set this to the number of steps between each log output. -1 determines logging
                        steps automatically. Default is -1.
  --evaluation-strategy {steps,epoch,no}
                        Specify how often to evaluate the model performance. Options include 'no', 'steps', 'epoch'. 'epoch' evaluates at the end of
                        each training epoch by default.
  --save-total-limit SAVE_TOTAL_LIMIT
                        Limit the total number of model checkpoints to save. Helps manage disk space by retaining only the most recent checkpoints.
                        Default is to save only the latest one.
  --auto-find-batch-size
                        Enable automatic batch size determination based on your hardware capabilities. When set, it tries to find the largest batch
                        size that fits in memory.
  --mixed-precision {fp16,bf16,None}
                        Choose the precision mode for training to optimize performance and memory usage. Options are 'fp16', 'bf16', or None for
                        default precision. Default is None.

< > Update on GitHub