Processors

在 Transformers 库中，processors可以有两种不同的含义：

为多模态模型，例如Wav2Vec2（语音和文本）或CLIP（文本和视觉）预处理输入的对象
在库的旧版本中用于预处理GLUE或SQUAD数据的已弃用对象。

多模态processors

任何多模态模型都需要一个对象来编码或解码将多个模态（包括文本、视觉和音频）组合在一起的数据。这由称为processors的对象处理，这些processors将两个或多个处理对象组合在一起，例如tokenizers（用于文本模态），image processors（用于视觉）和feature extractors（用于音频）。

这些processors继承自以下实现保存和加载功能的基类：

class transformers.ProcessorMixin

< source >

( *args **kwargs )

This is a mixin used to provide saving/loading functionality for all processor classes.

from_args_and_dict

< source >

( args processor_dict: Dict **kwargs ) → ~processing_utils.ProcessingMixin

Parameters

processor_dict (Dict[str, Any]) — Dictionary that will be used to instantiate the processor object. Such a dictionary can be retrieved from a pretrained checkpoint by leveraging the ~processing_utils.ProcessingMixin.to_dict method.
kwargs (Dict[str, Any]) — Additional parameters from which to initialize the processor object.

Returns

~processing_utils.ProcessingMixin

The processor object instantiated from those parameters.

Instantiates a type of ~processing_utils.ProcessingMixin from a Python dictionary of parameters.

from_pretrained

< source >

( pretrained_model_name_or_path: Union cache_dir: Union = None force_download: bool = False local_files_only: bool = False token: Union = None revision: str = 'main' **kwargs )

Parameters

pretrained_model_name_or_path (str or os.PathLike) — This can be either:
- a string, the model id of a pretrained feature_extractor hosted inside a model repo on huggingface.co.
- a path to a directory containing a feature extractor file saved using the save_pretrained() method, e.g., ./my_model_directory/.
- a path or url to a saved feature extractor JSON file, e.g., ./my_model_directory/preprocessor_config.json. **kwargs — Additional keyword arguments passed along to both from_pretrained() and ~tokenization_utils_base.PreTrainedTokenizer.from_pretrained.

Instantiate a processor associated with a pretrained model.

This class method is simply calling the feature extractor from_pretrained(), image processor ImageProcessingMixin and the tokenizer ~tokenization_utils_base.PreTrainedTokenizer.from_pretrained methods. Please refer to the docstrings of the methods above for more information.

get_processor_dict

< source >

( pretrained_model_name_or_path: Union **kwargs ) → Tuple[Dict, Dict]

Parameters

pretrained_model_name_or_path (str or os.PathLike) — The identifier of the pre-trained checkpoint from which we want the dictionary of parameters.
subfolder (str, optional, defaults to "") — In case the relevant files are located inside a subfolder of the model repo on huggingface.co, you can specify the folder name here.

Returns

Tuple[Dict, Dict]

The dictionary(ies) that will be used to instantiate the processor object.

From a pretrained_model_name_or_path, resolve to a dictionary of parameters, to be used for instantiating a processor of type ~processing_utils.ProcessingMixin using from_args_and_dict.

push_to_hub

< source >

( repo_id: str use_temp_dir: Optional = None commit_message: Optional = None private: Optional = None token: Union = None max_shard_size: Union = '5GB' create_pr: bool = False safe_serialization: bool = True revision: str = None commit_description: str = None tags: Optional = None **deprecated_kwargs )

Parameters

repo_id (str) — The name of the repository you want to push your processor to. It should contain your organization name when pushing to a given organization.
use_temp_dir (bool, optional) — Whether or not to use a temporary directory to store the files saved before they are pushed to the Hub. Will default to True if there is no directory named like repo_id, False otherwise.
commit_message (str, optional) — Message to commit while pushing. Will default to "Upload processor".
private (bool, optional) — Whether or not the repository created should be private.
token (bool or str, optional) — The token to use as HTTP bearer authorization for remote files. If True, will use the token generated when running huggingface-cli login (stored in ~/.huggingface). Will default to True if repo_url is not specified.
max_shard_size (int or str, optional, defaults to "5GB") — Only applicable for models. The maximum size for a checkpoint before being sharded. Checkpoints shard will then be each of size lower than this size. If expressed as a string, needs to be digits followed by a unit (like "5MB"). We default it to "5GB" so that users can easily load models on free-tier Google Colab instances without any CPU OOM issues.
create_pr (bool, optional, defaults to False) — Whether or not to create a PR with the uploaded files or directly commit.
safe_serialization (bool, optional, defaults to True) — Whether or not to convert the model weights in safetensors format for safer serialization.
revision (str, optional) — Branch to push the uploaded files to.
commit_description (str, optional) — The description of the commit that will be created
tags (List[str], optional) — List of tags to push on the Hub.

Upload the processor files to the 🤗 Model Hub.

Examples:

from transformers import AutoProcessor

processor = AutoProcessor.from_pretrained("google-bert/bert-base-cased")

# Push the processor to your namespace with the name "my-finetuned-bert".
processor.push_to_hub("my-finetuned-bert")

# Push the processor to an organization with the name "my-finetuned-bert".
processor.push_to_hub("huggingface/my-finetuned-bert")

register_for_auto_class

< source >

( auto_class = 'AutoProcessor' )

Parameters

auto_class (str or type, optional, defaults to "AutoProcessor") — The auto class to register this new feature extractor with.

Register this class with a given auto class. This should only be used for custom feature extractors as the ones in the library are already mapped with AutoProcessor.

This API is experimental and may have some slight breaking changes in the next releases.

save_pretrained

< source >

( save_directory push_to_hub: bool = False **kwargs )

Parameters

save_directory (str or os.PathLike) — Directory where the feature extractor JSON file and the tokenizer files will be saved (directory will be created if it does not exist).
push_to_hub (bool, optional, defaults to False) — Whether or not to push your model to the Hugging Face model hub after saving it. You can specify the repository you want to push to with repo_id (will default to the name of save_directory in your namespace).
kwargs (Dict[str, Any], optional) — Additional key word arguments passed along to the push_to_hub() method.

Saves the attributes of this processor (feature extractor, tokenizer…) in the specified directory so that it can be reloaded using the from_pretrained() method.

This class method is simply calling save_pretrained() and save_pretrained(). Please refer to the docstrings of the methods above for more information.

to_dict

< source >

( ) → Dict[str, Any]

Returns

Dict[str, Any]

Dictionary of all the attributes that make up this processor instance.

Serializes this instance to a Python dictionary.

to_json_file

< source >

( json_file_path: Union )

Parameters

json_file_path (str or os.PathLike) — Path to the JSON file in which this processor instance’s parameters will be saved.

Save this instance to a JSON file.

to_json_string

< source >

( ) → str

Returns

str

String containing all the attributes that make up this feature_extractor instance in JSON format.

Serializes this instance to a JSON string.

已弃用的processors

所有processor都遵循与 DataProcessor 相同的架构。processor返回一个 InputExample 列表。这些 InputExample 可以转换为 InputFeatures 以供输送到模型。

class transformers.DataProcessor

< source >

( )

Base class for data converters for sequence classification data sets.

get_dev_examples

< source >

( data_dir )

Gets a collection of InputExample for the dev set.

get_example_from_tensor_dict

< source >

( tensor_dict )

Gets an example from a dict with tensorflow tensors.

get_labels

< source >

( )

Gets the list of labels for this data set.

get_test_examples

< source >

( data_dir )

Gets a collection of InputExample for the test set.

get_train_examples

< source >

( data_dir )

Gets a collection of InputExample for the train set.

tfds_map

< source >

( example )

Some tensorflow_datasets datasets are not formatted the same way the GLUE datasets are. This method converts examples to the correct format.

class transformers.InputExample

< source >

( guid: str text_a: str text_b: Optional = None label: Optional = None )

A single training/test example for simple sequence classification.

to_json_string

< source >

( )

Serializes this instance to a JSON string.

class transformers.InputFeatures

< source >

( input_ids: List attention_mask: Optional = None token_type_ids: Optional = None label: Union = None )

A single set of features of data. Property names are the same names as the corresponding inputs to a model.

to_json_string

< source >

( )

Serializes this instance to a JSON string.

GLUE

General Language Understanding Evaluation (GLUE) 是一个基准测试，评估模型在各种现有的自然语言理解任务上的性能。它与论文 GLUE: A multi-task benchmark and analysis platform for natural language understanding 一同发布。

该库为以下任务提供了总共10个processor：MRPC、MNLI、MNLI（mismatched）、CoLA、SST2、STSB、QQP、QNLI、RTE 和 WNLI。

这些processor是：

~data.processors.utils.MrpcProcessor
~data.processors.utils.MnliProcessor
~data.processors.utils.MnliMismatchedProcessor
~data.processors.utils.Sst2Processor
~data.processors.utils.StsbProcessor
~data.processors.utils.QqpProcessor
~data.processors.utils.QnliProcessor
~data.processors.utils.RteProcessor
~data.processors.utils.WnliProcessor

此外，还可以使用以下方法从数据文件加载值并将其转换为 InputExample 列表。

transformers.glue_convert_examples_to_features

< source >

( examples: Union tokenizer: PreTrainedTokenizer max_length: Optional = None task = None label_list = None output_mode = None )

Loads a data file into a list of InputFeatures

XNLI

跨语言NLI语料库（XNLI）是一个评估跨语言文本表示质量的基准测试。XNLI是一个基于MultiNLI的众包数据集：”文本对“被标记为包含15种不同语言（包括英语等高资源语言和斯瓦希里语等低资源语言）的文本蕴涵注释。

它与论文 XNLI: Evaluating Cross-lingual Sentence Representations 一同发布。

该库提供了加载XNLI数据的processor：

~data.processors.utils.XnliProcessor

请注意，由于测试集上有“gold”标签，因此评估是在测试集上进行的。

使用这些processor的示例在 run_xnli.py 脚本中提供。

SQuAD

斯坦福问答数据集（SQuAD）是一个评估模型在问答上性能的基准测试。有两个版本，v1.1 和 v2.0。第一个版本（v1.1）与论文 SQuAD: 100,000+ Questions for Machine Comprehension of Text 一同发布。第二个版本（v2.0）与论文 Know What You Don’t Know: Unanswerable Questions for SQuAD 一同发布。

该库为两个版本各自提供了一个processor：

Processors

这两个processor是：

~data.processors.utils.SquadV1Processor
~data.processors.utils.SquadV2Processor

它们都继承自抽象类 ~data.processors.utils.SquadProcessor。

class transformers.data.processors.squad.SquadProcessor

< source >

( )

Processor for the SQuAD data set. overridden by SquadV1Processor and SquadV2Processor, used by the version 1.1 and version 2.0 of SQuAD, respectively.

get_dev_examples

< source >

( data_dir filename = None )

Returns the evaluation example from the data directory.

get_examples_from_dataset

< source >

( dataset evaluate = False )

Creates a list of SquadExample using a TFDS dataset.

Examples:

>>> import tensorflow_datasets as tfds

>>> dataset = tfds.load("squad")

>>> training_examples = get_examples_from_dataset(dataset, evaluate=False)
>>> evaluation_examples = get_examples_from_dataset(dataset, evaluate=True)

get_train_examples

< source >

( data_dir filename = None )

Returns the training examples from the data directory.

此外，可以使用以下方法将 SQuAD 示例转换为可用作模型输入的 ~data.processors.utils.SquadFeatures。

transformers.squad_convert_examples_to_features

< source >

( examples tokenizer max_seq_length doc_stride max_query_length is_training padding_strategy = 'max_length' return_dataset = False threads = 1 tqdm_enabled = True )

Converts a list of examples into a list of features that can be directly given as input to a model. It is model-dependant and takes advantage of many of the tokenizer’s features to create the model’s inputs.

Example:

processor = SquadV2Processor()
examples = processor.get_dev_examples(data_dir)

features = squad_convert_examples_to_features(
    examples=examples,
    tokenizer=tokenizer,
    max_seq_length=args.max_seq_length,
    doc_stride=args.doc_stride,
    max_query_length=args.max_query_length,
    is_training=not evaluate,
)

这些processor以及前面提到的方法可以与包含数据的文件以及tensorflow_datasets包一起使用。下面给出了示例。

Example使用

以下是使用processor以及使用数据文件的转换方法的示例：

# Loading a V2 processor
processor = SquadV2Processor()
examples = processor.get_dev_examples(squad_v2_data_dir)

# Loading a V1 processor
processor = SquadV1Processor()
examples = processor.get_dev_examples(squad_v1_data_dir)

features = squad_convert_examples_to_features(
    examples=examples,
    tokenizer=tokenizer,
    max_seq_length=max_seq_length,
    doc_stride=args.doc_stride,
    max_query_length=max_query_length,
    is_training=not evaluate,
)

使用 tensorflow_datasets 就像使用数据文件一样简单：

# tensorflow_datasets only handle Squad V1.
tfds_examples = tfds.load("squad")
examples = SquadV1Processor().get_examples_from_dataset(tfds_examples, evaluate=evaluate)

features = squad_convert_examples_to_features(
    examples=examples,
    tokenizer=tokenizer,
    max_seq_length=max_seq_length,
    doc_stride=args.doc_stride,
    max_query_length=max_query_length,
    is_training=not evaluate,
)

另一个使用这些processor的示例在 run_squad.py 脚本中提供。