Processors
Processors can mean two different things in the Transformers library:
- the objects that pre-process inputs for multi-modal models such as Wav2Vec2 (speech and text) or CLIP (text and vision)
- deprecated objects that were used in older versions of the library to preprocess data for GLUE or SQUAD.
Multi-modal processors
Any multi-modal model will require an object to encode or decode the data that groups several modalities (among text, vision and audio). This is handled by objects called processors, which group together two or more processing objects such as tokenizers (for the text modality), image processors (for vision) and feature extractors (for audio).
Those processors inherit from the following base class that implements the saving and loading functionality:
This is a mixin used to provide saving/loading functionality for all processor classes.
apply_chat_template
< source >( conversation: List chat_template: Optional = None tokenize: bool = False **kwargs )
Parameters
- conversation (
List[Dict, str, str]
) — The conversation to format. - chat_template (
Optional[str]
, optional) — The Jinja template to use for formatting the conversation. If not provided, the tokenizer’s chat template is used. - tokenize (
bool
, optional, defaults toFalse
) — Whether to tokenize the output or not. **kwargs — Additional keyword arguments
Similar to the apply_chat_template
method on tokenizers, this method applies a Jinja template to input
conversations to turn them into a single tokenizable string.
from_args_and_dict
< source >( args processor_dict: Dict **kwargs ) → ~processing_utils.ProcessingMixin
Parameters
- processor_dict (
Dict[str, Any]
) — Dictionary that will be used to instantiate the processor object. Such a dictionary can be retrieved from a pretrained checkpoint by leveraging the~processing_utils.ProcessingMixin.to_dict
method. - kwargs (
Dict[str, Any]
) — Additional parameters from which to initialize the processor object.
Returns
~processing_utils.ProcessingMixin
The processor object instantiated from those parameters.
Instantiates a type of ~processing_utils.ProcessingMixin
from a Python dictionary of parameters.
from_pretrained
< source >( pretrained_model_name_or_path: Union cache_dir: Union = None force_download: bool = False local_files_only: bool = False token: Union = None revision: str = 'main' **kwargs )
Parameters
- pretrained_model_name_or_path (
str
oros.PathLike
) — This can be either:- a string, the model id of a pretrained feature_extractor hosted inside a model repo on huggingface.co.
- a path to a directory containing a feature extractor file saved using the
save_pretrained() method, e.g.,
./my_model_directory/
. - a path or url to a saved feature extractor JSON file, e.g.,
./my_model_directory/preprocessor_config.json
. **kwargs — Additional keyword arguments passed along to both from_pretrained() and~tokenization_utils_base.PreTrainedTokenizer.from_pretrained
.
Instantiate a processor associated with a pretrained model.
This class method is simply calling the feature extractor
from_pretrained(), image processor
ImageProcessingMixin and the tokenizer
~tokenization_utils_base.PreTrainedTokenizer.from_pretrained
methods. Please refer to the docstrings of the
methods above for more information.
get_processor_dict
< source >( pretrained_model_name_or_path: Union **kwargs ) → Tuple[Dict, Dict]
Parameters
- pretrained_model_name_or_path (
str
oros.PathLike
) — The identifier of the pre-trained checkpoint from which we want the dictionary of parameters. - subfolder (
str
, optional, defaults to""
) — In case the relevant files are located inside a subfolder of the model repo on huggingface.co, you can specify the folder name here.
Returns
Tuple[Dict, Dict]
The dictionary(ies) that will be used to instantiate the processor object.
From a pretrained_model_name_or_path
, resolve to a dictionary of parameters, to be used for instantiating a
processor of type ~processing_utils.ProcessingMixin
using from_args_and_dict
.
push_to_hub
< source >( repo_id: str use_temp_dir: Optional = None commit_message: Optional = None private: Optional = None token: Union = None max_shard_size: Union = '5GB' create_pr: bool = False safe_serialization: bool = True revision: str = None commit_description: str = None tags: Optional = None **deprecated_kwargs )
Parameters
- repo_id (
str
) — The name of the repository you want to push your processor to. It should contain your organization name when pushing to a given organization. - use_temp_dir (
bool
, optional) — Whether or not to use a temporary directory to store the files saved before they are pushed to the Hub. Will default toTrue
if there is no directory named likerepo_id
,False
otherwise. - commit_message (
str
, optional) — Message to commit while pushing. Will default to"Upload processor"
. - private (
bool
, optional) — Whether or not the repository created should be private. - token (
bool
orstr
, optional) — The token to use as HTTP bearer authorization for remote files. IfTrue
, will use the token generated when runninghuggingface-cli login
(stored in~/.huggingface
). Will default toTrue
ifrepo_url
is not specified. - max_shard_size (
int
orstr
, optional, defaults to"5GB"
) — Only applicable for models. The maximum size for a checkpoint before being sharded. Checkpoints shard will then be each of size lower than this size. If expressed as a string, needs to be digits followed by a unit (like"5MB"
). We default it to"5GB"
so that users can easily load models on free-tier Google Colab instances without any CPU OOM issues. - create_pr (
bool
, optional, defaults toFalse
) — Whether or not to create a PR with the uploaded files or directly commit. - safe_serialization (
bool
, optional, defaults toTrue
) — Whether or not to convert the model weights in safetensors format for safer serialization. - revision (
str
, optional) — Branch to push the uploaded files to. - commit_description (
str
, optional) — The description of the commit that will be created - tags (
List[str]
, optional) — List of tags to push on the Hub.
Upload the processor files to the 🤗 Model Hub.
Examples:
from transformers import AutoProcessor
processor = AutoProcessor.from_pretrained("google-bert/bert-base-cased")
# Push the processor to your namespace with the name "my-finetuned-bert".
processor.push_to_hub("my-finetuned-bert")
# Push the processor to an organization with the name "my-finetuned-bert".
processor.push_to_hub("huggingface/my-finetuned-bert")
register_for_auto_class
< source >( auto_class = 'AutoProcessor' )
Register this class with a given auto class. This should only be used for custom feature extractors as the ones
in the library are already mapped with AutoProcessor
.
This API is experimental and may have some slight breaking changes in the next releases.
save_pretrained
< source >( save_directory push_to_hub: bool = False **kwargs )
Parameters
- save_directory (
str
oros.PathLike
) — Directory where the feature extractor JSON file and the tokenizer files will be saved (directory will be created if it does not exist). - push_to_hub (
bool
, optional, defaults toFalse
) — Whether or not to push your model to the Hugging Face model hub after saving it. You can specify the repository you want to push to withrepo_id
(will default to the name ofsave_directory
in your namespace). - kwargs (
Dict[str, Any]
, optional) — Additional key word arguments passed along to the push_to_hub() method.
Saves the attributes of this processor (feature extractor, tokenizer…) in the specified directory so that it can be reloaded using the from_pretrained() method.
This class method is simply calling save_pretrained() and save_pretrained(). Please refer to the docstrings of the methods above for more information.
to_dict
< source >( ) → Dict[str, Any]
Returns
Dict[str, Any]
Dictionary of all the attributes that make up this processor instance.
Serializes this instance to a Python dictionary.
to_json_file
< source >( json_file_path: Union )
Save this instance to a JSON file.
to_json_string
< source >( ) → str
Returns
str
String containing all the attributes that make up this feature_extractor instance in JSON format.
Serializes this instance to a JSON string.
Deprecated processors
All processors follow the same architecture which is that of the DataProcessor. The processor returns a list of InputExample. These InputExample can be converted to InputFeatures in order to be fed to the model.
Base class for data converters for sequence classification data sets.
Gets a collection of InputExample for the dev set.
Gets an example from a dict with tensorflow tensors.
Gets the list of labels for this data set.
Gets a collection of InputExample for the test set.
Gets a collection of InputExample for the train set.
Some tensorflow_datasets datasets are not formatted the same way the GLUE datasets are. This method converts examples to the correct format.
class transformers.InputExample
< source >( guid: str text_a: str text_b: Optional = None label: Optional = None )
A single training/test example for simple sequence classification.
Serializes this instance to a JSON string.
class transformers.InputFeatures
< source >( input_ids: List attention_mask: Optional = None token_type_ids: Optional = None label: Union = None )
A single set of features of data. Property names are the same names as the corresponding inputs to a model.
Serializes this instance to a JSON string.
GLUE
General Language Understanding Evaluation (GLUE) is a benchmark that evaluates the performance of models across a diverse set of existing NLU tasks. It was released together with the paper GLUE: A multi-task benchmark and analysis platform for natural language understanding
This library hosts a total of 10 processors for the following tasks: MRPC, MNLI, MNLI (mismatched), CoLA, SST2, STSB, QQP, QNLI, RTE and WNLI.
Those processors are:
~data.processors.utils.MrpcProcessor
~data.processors.utils.MnliProcessor
~data.processors.utils.MnliMismatchedProcessor
~data.processors.utils.Sst2Processor
~data.processors.utils.StsbProcessor
~data.processors.utils.QqpProcessor
~data.processors.utils.QnliProcessor
~data.processors.utils.RteProcessor
~data.processors.utils.WnliProcessor
Additionally, the following method can be used to load values from a data file and convert them to a list of InputExample.
transformers.glue_convert_examples_to_features
< source >( examples: Union tokenizer: PreTrainedTokenizer max_length: Optional = None task = None label_list = None output_mode = None )
Loads a data file into a list of InputFeatures
XNLI
The Cross-Lingual NLI Corpus (XNLI) is a benchmark that evaluates the quality of cross-lingual text representations. XNLI is crowd-sourced dataset based on MultiNLI: pairs of text are labeled with textual entailment annotations for 15 different languages (including both high-resource language such as English and low-resource languages such as Swahili).
It was released together with the paper XNLI: Evaluating Cross-lingual Sentence Representations
This library hosts the processor to load the XNLI data:
~data.processors.utils.XnliProcessor
Please note that since the gold labels are available on the test set, evaluation is performed on the test set.
An example using these processors is given in the run_xnli.py script.
SQuAD
The Stanford Question Answering Dataset (SQuAD) is a benchmark that evaluates the performance of models on question answering. Two versions are available, v1.1 and v2.0. The first version (v1.1) was released together with the paper SQuAD: 100,000+ Questions for Machine Comprehension of Text. The second version (v2.0) was released alongside the paper Know What You Don’t Know: Unanswerable Questions for SQuAD.
This library hosts a processor for each of the two versions:
Processors
Those processors are:
~data.processors.utils.SquadV1Processor
~data.processors.utils.SquadV2Processor
They both inherit from the abstract class ~data.processors.utils.SquadProcessor
Processor for the SQuAD data set. overridden by SquadV1Processor and SquadV2Processor, used by the version 1.1 and version 2.0 of SQuAD, respectively.
Returns the evaluation example from the data directory.
Creates a list of SquadExample
using a TFDS dataset.
Returns the training examples from the data directory.
Additionally, the following method can be used to convert SQuAD examples into
~data.processors.utils.SquadFeatures
that can be used as model inputs.
transformers.squad_convert_examples_to_features
< source >( examples tokenizer max_seq_length doc_stride max_query_length is_training padding_strategy = 'max_length' return_dataset = False threads = 1 tqdm_enabled = True )
Converts a list of examples into a list of features that can be directly given as input to a model. It is model-dependant and takes advantage of many of the tokenizer’s features to create the model’s inputs.
Example:
processor = SquadV2Processor()
examples = processor.get_dev_examples(data_dir)
features = squad_convert_examples_to_features(
examples=examples,
tokenizer=tokenizer,
max_seq_length=args.max_seq_length,
doc_stride=args.doc_stride,
max_query_length=args.max_query_length,
is_training=not evaluate,
)
These processors as well as the aforementioned method can be used with files containing the data as well as with the tensorflow_datasets package. Examples are given below.
Example usage
Here is an example using the processors as well as the conversion method using data files:
# Loading a V2 processor
processor = SquadV2Processor()
examples = processor.get_dev_examples(squad_v2_data_dir)
# Loading a V1 processor
processor = SquadV1Processor()
examples = processor.get_dev_examples(squad_v1_data_dir)
features = squad_convert_examples_to_features(
examples=examples,
tokenizer=tokenizer,
max_seq_length=max_seq_length,
doc_stride=args.doc_stride,
max_query_length=max_query_length,
is_training=not evaluate,
)
Using tensorflow_datasets is as easy as using a data file:
# tensorflow_datasets only handle Squad V1.
tfds_examples = tfds.load("squad")
examples = SquadV1Processor().get_examples_from_dataset(tfds_examples, evaluate=evaluate)
features = squad_convert_examples_to_features(
examples=examples,
tokenizer=tokenizer,
max_seq_length=max_seq_length,
doc_stride=args.doc_stride,
max_query_length=max_query_length,
is_training=not evaluate,
)
Another example using these processors is given in the run_squad.py script.
< > Update on GitHub