# Examples This folder contains actively maintained examples of use of 🤗 Transformers organized along NLP tasks. If you are looking for an example that used to be in this folder, it may have moved to our [research projects](https://github.com/huggingface/transformers/tree/master/examples/research_projects) subfolder (which contains frozen snapshots of research projects). ## Important note **Important** To make sure you can successfully run the latest versions of the example scripts, you have to **install the library from source** and install some example-specific requirements. To do this, execute the following steps in a new virtual environment: ```bash git clone https://github.com/huggingface/transformers cd transformers pip install . ``` Then cd in the example folder of your choice and run ```bash pip install -r requirements.txt ``` To browse the examples corresponding to released versions of 🤗 Transformers, click on the line below and then on your desired version of the library:
Examples for older versions of 🤗 Transformers - [v4.3.3](https://github.com/huggingface/transformers/tree/v4.3.3/examples) - [v4.2.2](https://github.com/huggingface/transformers/tree/v4.2.2/examples) - [v4.1.1](https://github.com/huggingface/transformers/tree/v4.1.1/examples) - [v4.0.1](https://github.com/huggingface/transformers/tree/v4.0.1/examples) - [v3.5.1](https://github.com/huggingface/transformers/tree/v3.5.1/examples) - [v3.4.0](https://github.com/huggingface/transformers/tree/v3.4.0/examples) - [v3.3.1](https://github.com/huggingface/transformers/tree/v3.3.1/examples) - [v3.2.0](https://github.com/huggingface/transformers/tree/v3.2.0/examples) - [v3.1.0](https://github.com/huggingface/transformers/tree/v3.1.0/examples) - [v3.0.2](https://github.com/huggingface/transformers/tree/v3.0.2/examples) - [v2.11.0](https://github.com/huggingface/transformers/tree/v2.11.0/examples) - [v2.10.0](https://github.com/huggingface/transformers/tree/v2.10.0/examples) - [v2.9.1](https://github.com/huggingface/transformers/tree/v2.9.1/examples) - [v2.8.0](https://github.com/huggingface/transformers/tree/v2.8.0/examples) - [v2.7.0](https://github.com/huggingface/transformers/tree/v2.7.0/examples) - [v2.6.0](https://github.com/huggingface/transformers/tree/v2.6.0/examples) - [v2.5.1](https://github.com/huggingface/transformers/tree/v2.5.1/examples) - [v2.4.0](https://github.com/huggingface/transformers/tree/v2.4.0/examples) - [v2.3.0](https://github.com/huggingface/transformers/tree/v2.3.0/examples) - [v2.2.0](https://github.com/huggingface/transformers/tree/v2.2.0/examples) - [v2.1.1](https://github.com/huggingface/transformers/tree/v2.1.0/examples) - [v2.0.0](https://github.com/huggingface/transformers/tree/v2.0.0/examples) - [v1.2.0](https://github.com/huggingface/transformers/tree/v1.2.0/examples) - [v1.1.0](https://github.com/huggingface/transformers/tree/v1.1.0/examples) - [v1.0.0](https://github.com/huggingface/transformers/tree/v1.0.0/examples)
Alternatively, you can find switch your cloned 🤗 Transformers to a specific version (for instance with v3.5.1) with ```bash git checkout tags/v3.5.1 ``` and run the example command as usual afterward. ## The Big Table of Tasks Here is the list of all our examples: - with information on whether they are **built on top of `Trainer`/`TFTrainer`** (if not, they still work, they might just lack some features), - whether or not they leverage the [🤗 Datasets](https://github.com/huggingface/datasets) library. - links to **Colab notebooks** to walk through the scripts and run them easily, | Task | Example datasets | Trainer support | TFTrainer support | 🤗 Datasets | Colab |---|---|:---:|:---:|:---:|:---:| | [**`language-modeling`**](https://github.com/huggingface/transformers/tree/master/examples/language-modeling) | Raw text | ✅ | - | ✅ | [![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/huggingface/blog/blob/master/notebooks/01_how_to_train.ipynb) | [**`multiple-choice`**](https://github.com/huggingface/transformers/tree/master/examples/multiple-choice) | SWAG, RACE, ARC | ✅ | ✅ | ✅ | [![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/ViktorAlm/notebooks/blob/master/MPC_GPU_Demo_for_TF_and_PT.ipynb) | [**`question-answering`**](https://github.com/huggingface/transformers/tree/master/examples/question-answering) | SQuAD | ✅ | ✅ | ✅ | [![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/huggingface/notebooks/blob/master/examples/question_answering.ipynb) | [**`summarization`**](https://github.com/huggingface/transformers/tree/master/examples/seq2seq) | CNN/Daily Mail | ✅ | - | - | - | [**`text-classification`**](https://github.com/huggingface/transformers/tree/master/examples/text-classification) | GLUE, XNLI | ✅ | ✅ | ✅ | [![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/huggingface/notebooks/blob/master/examples/text_classification.ipynb) | [**`text-generation`**](https://github.com/huggingface/transformers/tree/master/examples/text-generation) | - | n/a | n/a | - | [![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/huggingface/blog/blob/master/notebooks/02_how_to_generate.ipynb) | [**`token-classification`**](https://github.com/huggingface/transformers/tree/master/examples/token-classification) | CoNLL NER | ✅ | ✅ | ✅ | [![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/huggingface/notebooks/blob/master/examples/token_classification.ipynb) | [**`translation`**](https://github.com/huggingface/transformers/tree/master/examples/seq2seq) | WMT | ✅ | - | - | - ## Distributed training and mixed precision All the PyTorch scripts mentioned above work out of the box with distributed training and mixed precision, thanks to the [Trainer API](https://huggingface.co/transformers/main_classes/trainer.html). To launch one of them on _n_ GPUS, use the following command: ```bash python -m torch.distributed.launch \ --nproc_per_node number_of_gpu_you_have path_to_script.py \ --all_arguments_of_the_script ``` As an example, here is how you would fine-tune the BERT large model (with whole word masking) on the text classification MNLI task using the `run_glue` script, with 8 GPUs: ```bash python -m torch.distributed.launch \ --nproc_per_node 8 text-classification/run_glue.py \ --model_name_or_path bert-large-uncased-whole-word-masking \ --task_name mnli \ --do_train \ --do_eval \ --max_seq_length 128 \ --per_device_train_batch_size 8 \ --learning_rate 2e-5 \ --num_train_epochs 3.0 \ --output_dir /tmp/mnli_output/ ``` If you have a GPU with mixed precision capabilities (architecture Pascal or more recent), you can use mixed precision training with PyTorch 1.6.0 or latest, or by installing the [Apex](https://github.com/NVIDIA/apex) library for previous versions. Just add the flag `--fp16` to your command launching one of the scripts mentioned above! Using mixed precision training usually results in 2x-speedup for training with the same final results (as shown in [this table](https://github.com/huggingface/transformers/tree/master/examples/text-classification#mixed-precision-training) for text classification). ## Running on TPUs When using Tensorflow, TPUs are supported out of the box as a `tf.distribute.Strategy`. When using PyTorch, we support TPUs thanks to `pytorch/xla`. For more context and information on how to setup your TPU environment refer to Google's documentation and to the very detailed [pytorch/xla README](https://github.com/pytorch/xla/blob/master/README.md). In this repo, we provide a very simple launcher script named [xla_spawn.py](https://github.com/huggingface/transformers/tree/master/examples/xla_spawn.py) that lets you run our example scripts on multiple TPU cores without any boilerplate. Just pass a `--num_cores` flag to this script, then your regular training script with its arguments (this is similar to the `torch.distributed.launch` helper for `torch.distributed`): ```bash python xla_spawn.py --num_cores num_tpu_you_have \ path_to_script.py \ --all_arguments_of_the_script ``` As an example, here is how you would fine-tune the BERT large model (with whole word masking) on the text classification MNLI task using the `run_glue` script, with 8 TPUs: ```bash python xla_spawn.py --num_cores 8 \ text-classification/run_glue.py \ --model_name_or_path bert-large-uncased-whole-word-masking \ --task_name mnli \ --do_train \ --do_eval \ --max_seq_length 128 \ --per_device_train_batch_size 8 \ --learning_rate 2e-5 \ --num_train_epochs 3.0 \ --output_dir /tmp/mnli_output/ ``` ## Logging & Experiment tracking You can easily log and monitor your runs code. The following are currently supported: * [TensorBoard](https://www.tensorflow.org/tensorboard) * [Weights & Biases](https://docs.wandb.ai/integrations/huggingface) * [Comet ML](https://www.comet.ml/docs/python-sdk/huggingface/) ### Weights & Biases To use Weights & Biases, install the wandb package with: ```bash pip install wandb ``` Then log in the command line: ```bash wandb login ``` If you are in Jupyter or Colab, you should login with: ```python import wandb wandb.login() ``` To enable logging to W&B, include `"wandb"` in the `report_to` of your `TrainingArguments` or script. Or just pass along `--report_to all` if you have `wandb` installed. Whenever you use `Trainer` or `TFTrainer` classes, your losses, evaluation metrics, model topology and gradients (for `Trainer` only) will automatically be logged. Advanced configuration is possible by setting environment variables:
Environment Variables Options
WANDB_LOG_MODEL Log the model as artifact at the end of training (false by default)
WANDB_WATCH
  • gradients (default): Log histograms of the gradients
  • all: Log histograms of gradients and parameters
  • false: No gradient or parameter logging
WANDB_PROJECT Organize runs by project
Set run names with `run_name` argument present in scripts or as part of `TrainingArguments`. Additional configuration options are available through generic [wandb environment variables](https://docs.wandb.com/library/environment-variables). Refer to related [documentation & examples](https://docs.wandb.ai/integrations/huggingface). ### Comet.ml To use `comet_ml`, install the Python package with: ```bash pip install comet_ml ``` or if in a Conda environment: ```bash conda install -c comet_ml -c anaconda -c conda-forge comet_ml ```