Models
The base classes PreTrainedModel, TFPreTrainedModel, and FlaxPreTrainedModel implement the common methods for loading/saving a model either from a local file or directory, or from a pretrained model configuration provided by the library (downloaded from HuggingFace’s AWS S3 repository).
PreTrainedModel and TFPreTrainedModel also implement a few methods which are common among all the models to:
- resize the input token embeddings when new tokens are added to the vocabulary
- prune the attention heads of the model.
The other methods that are common to each model are defined in ModuleUtilsMixin
(for the PyTorch models) and TFModuleUtilsMixin
(for the TensorFlow models) or
for text generation, GenerationMixin (for the PyTorch models),
TFGenerationMixin (for the TensorFlow models) and
FlaxGenerationMixin (for the Flax/JAX models).
PreTrainedModel
Base class for all models.
PreTrainedModel takes care of storing the configuration of the models and handles methods for loading, downloading and saving models as well as a few methods common to all models to:
- resize the input embeddings,
- prune heads in the self-attention heads.
Class attributes (overridden by derived classes):
config_class (PretrainedConfig) — A subclass of PretrainedConfig to use as configuration class for this model architecture.
load_tf_weights (
Callable
) — A python method for loading a TensorFlow checkpoint in a PyTorch model, taking as arguments:- model (PreTrainedModel) — An instance of the model on which to load the TensorFlow checkpoint.
- config (
PreTrainedConfig
) — An instance of the configuration associated to the model. - path (
str
) — A path to the TensorFlow checkpoint.
base_model_prefix (
str
) — A string indicating the attribute associated to the base model in derived classes of the same architecture adding modules on top of the base model.is_parallelizable (
bool
) — A flag indicating whether this model supports model parallelization.main_input_name (
str
) — The name of the principal input to the model (ofteninput_ids
for NLP models,pixel_values
for vision models andinput_values
for speech models).
push_to_hub
< source >( repo_id: str use_temp_dir: typing.Optional[bool] = None commit_message: typing.Optional[str] = None private: typing.Optional[bool] = None use_auth_token: typing.Union[bool, str, NoneType] = None max_shard_size: typing.Union[int, str, NoneType] = '10GB' create_pr: bool = False **deprecated_kwargs )
Parameters
-
repo_id (
str
) — The name of the repository you want to push your model to. It should contain your organization name when pushing to a given organization. -
use_temp_dir (
bool
, optional) — Whether or not to use a temporary directory to store the files saved before they are pushed to the Hub. Will default toTrue
if there is no directory named likerepo_id
,False
otherwise. -
commit_message (
str
, optional) — Message to commit while pushing. Will default to"Upload model"
. -
private (
bool
, optional) — Whether or not the repository created should be private (requires a paying subscription). -
use_auth_token (
bool
orstr
, optional) — The token to use as HTTP bearer authorization for remote files. IfTrue
, will use the token generated when runninghuggingface-cli login
(stored in~/.huggingface
). Will default toTrue
ifrepo_url
is not specified. -
max_shard_size (
int
orstr
, optional, defaults to"10GB"
) — Only applicable for models. The maximum size for a checkpoint before being sharded. Checkpoints shard will then be each of size lower than this size. If expressed as a string, needs to be digits followed by a unit (like"5MB"
). -
create_pr (
bool
, optional, defaults toFalse
) — Whether or not to create a PR with the uploaded files or directly commit.
Upload the model file to the 🤗 Model Hub while synchronizing a local clone of the repo in
repo_path_or_name
.
Examples:
from transformers import AutoModel
model = AutoModel.from_pretrained("bert-base-cased")
# Push the model to your namespace with the name "my-finetuned-bert".
model.push_to_hub("my-finetuned-bert")
# Push the model to an organization with the name "my-finetuned-bert".
model.push_to_hub("huggingface/my-finetuned-bert")
from_pretrained
< source >( pretrained_model_name_or_path: typing.Union[str, os.PathLike, NoneType] *model_args **kwargs )
Parameters
-
pretrained_model_name_or_path (
str
oros.PathLike
, optional) — Can be either:- A string, the model id of a pretrained model hosted inside a model repo on huggingface.co.
Valid model ids can be located at the root-level, like
bert-base-uncased
, or namespaced under a user or organization name, likedbmdz/bert-base-german-cased
. - A path to a directory containing model weights saved using
save_pretrained(), e.g.,
./my_model_directory/
. - A path or url to a tensorflow index checkpoint file (e.g,
./tf_model/model.ckpt.index
). In this case,from_tf
should be set toTrue
and a configuration object should be provided asconfig
argument. This loading path is slower than converting the TensorFlow checkpoint in a PyTorch model using the provided conversion scripts and loading the PyTorch model afterwards. - A path or url to a model folder containing a flax checkpoint file in .msgpack format (e.g,
./flax_model/
containingflax_model.msgpack
). In this case,from_flax
should be set toTrue
. None
if you are both providing the configuration and state dictionary (resp. with keyword argumentsconfig
andstate_dict
).
- A string, the model id of a pretrained model hosted inside a model repo on huggingface.co.
Valid model ids can be located at the root-level, like
-
model_args (sequence of positional arguments, optional) —
All remaining positional arguments will be passed to the underlying model’s
__init__
method. -
config (
Union[PretrainedConfig, str, os.PathLike]
, optional) — Can be either:- an instance of a class derived from PretrainedConfig,
- a string or path valid as input to from_pretrained().
Configuration for the model to use instead of an automatically loaded configuration. Configuration can be automatically loaded when:
- The model is a model provided by the library (loaded with the model id string of a pretrained model).
- The model was saved using save_pretrained() and is reloaded by supplying the save directory.
- The model is loaded by supplying a local directory as
pretrained_model_name_or_path
and a configuration JSON file named config.json is found in the directory.
-
state_dict (
Dict[str, torch.Tensor]
, optional) — A state dictionary to use instead of a state dictionary loaded from saved weights file.This option can be used if you want to create a model from a pretrained configuration but load your own weights. In this case though, you should check if using save_pretrained() and from_pretrained() is not a simpler option.
-
cache_dir (
Union[str, os.PathLike]
, optional) — Path to a directory in which a downloaded pretrained model configuration should be cached if the standard cache should not be used. -
from_tf (
bool
, optional, defaults toFalse
) — Load the model weights from a TensorFlow checkpoint save file (see docstring ofpretrained_model_name_or_path
argument). -
from_flax (
bool
, optional, defaults toFalse
) — Load the model weights from a Flax checkpoint save file (see docstring ofpretrained_model_name_or_path
argument). -
ignore_mismatched_sizes (
bool
, optional, defaults toFalse
) — Whether or not to raise an error if some of the weights from the checkpoint do not have the same size as the weights of the model (if for instance, you are instantiating a model with 10 labels from a checkpoint with 3 labels). -
force_download (
bool
, optional, defaults toFalse
) — Whether or not to force the (re-)download of the model weights and configuration files, overriding the cached versions if they exist. -
resume_download (
bool
, optional, defaults toFalse
) — Whether or not to delete incompletely received files. Will attempt to resume the download if such a file exists. -
proxies (
Dict[str, str]
, optional) — A dictionary of proxy servers to use by protocol or endpoint, e.g.,{'http': 'foo.bar:3128', 'http://hostname': 'foo.bar:4012'}
. The proxies are used on each request. -
output_loading_info(
bool
, optional, defaults toFalse
) — Whether ot not to also return a dictionary containing missing keys, unexpected keys and error messages. -
local_files_only(
bool
, optional, defaults toFalse
) — Whether or not to only look at local files (i.e., do not try to download the model). -
use_auth_token (
str
or bool, optional) — The token to use as HTTP bearer authorization for remote files. IfTrue
, will use the token generated when runninghuggingface-cli login
(stored in~/.huggingface
). -
revision (
str
, optional, defaults to"main"
) — The specific model version to use. It can be a branch name, a tag name, or a commit id, since we use a git-based system for storing models and other artifacts on huggingface.co, sorevision
can be any identifier allowed by git. -
mirror (
str
, optional) — Mirror source to accelerate downloads in China. If you are from China and have an accessibility problem, you can set this option to resolve it. Note that we do not guarantee the timeliness or safety. Please refer to the mirror site for more information. -
_fast_init(
bool
, optional, defaults toTrue
) — Whether or not to disable fast initialization.One should only disable _fast_init to ensure backwards compatibility with
transformers.__version__ < 4.6.0
for seeded model initialization. This argument will be removed at the next major version. See pull request 11471 for more information.
Parameters for big model inference
-
low_cpu_mem_usage(
bool
, optional) — Tries to not use more than 1x model size in CPU memory (including peak memory) while loading the model. This is an experimental feature and a subject to change at any moment. -
torch_dtype (
str
ortorch.dtype
, optional) — Override the defaulttorch.dtype
and load the model under this dtype. If"auto"
is passed the dtype will be automatically derived from the model’s weights. -
device_map (
str
orDict[str, Union[int, str, torch.device]]
, optional) — A map that specifies where each submodule should go. It doesn’t need to be refined to each parameter/buffer name, once a given module name is inside, every submodule of it will be sent to the same device.To have Accelerate compute the most optimized
device_map
automatically, setdevice_map="auto"
. For more information about each option see designing a device map. -
max_memory (
Dict
, optional) — A dictionary device identifier to maximum memory. Will default to the maximum memory available for each GPU and the available CPU RAM if unset. -
offload_folder (
str
oros.PathLike
, optional) — If thedevice_map
contains any value"disk"
, the folder where we will offload weights. -
offload_state_dict (
bool
, optional) — IfTrue
, will temporarily offload the CPU state dict to the hard drive to avoid getting out of CPU RAM if the weight of the CPU state dict + the biggest shard of the checkpoint does not fit. Defaults toTrue
when there is some disk offload. -
load_in_8bit (
bool
, optional, defaults toFalse
) — IfTrue
, will convert the loaded model into mixed-8bit quantized model. To use this feature please installbitsandbytes
compiled with your CUDA version by runningpip install -i https://test.pypi.org/simple/ bitsandbytes-cudaXXX
where XXX is your CUDA version (e.g. 11.6 = 116). Make also sure that you have enough GPU RAM to store half of the model size since the 8bit modules are not compiled and adapted for CPUs. -
int8_threshold (
float
, optional, defaults to 6) — Works together withload_in_8bit
. This corresponds to the outlier threshold for outlier detection as described inGPT3.int8() : 8-bit Matrix Multiplication for Transformers at Scale
paper. Any hidden states value that is above this threshold will be considered an outlier and the operation on those values will be done in fp16. Values are usually normally distributed, that is, most values are in the range [-3.5, 3.5], but there are some exceptional systematic outliers that are very differently distributed for large models. These outliers are often in the interval [-60, -6] or [6, 60]. Int8 quantization works well for values of magnitude ~5, but beyond that, there is a significant performance penalty. A good default threshold is 6, but a lower threshold might be needed for more unstable models (small models, fine-tuning). -
subfolder (
str
, optional, defaults to""
) — In case the relevant files are located inside a subfolder of the model repo on huggingface.co, you can specify the folder name here. -
kwargs (remaining dictionary of keyword arguments, optional) —
Can be used to update the configuration object (after it being loaded) and initiate the model (e.g.,
output_attentions=True
). Behaves differently depending on whether aconfig
is provided or automatically loaded:- If a configuration is provided with
config
,**kwargs
will be directly passed to the underlying model’s__init__
method (we assume all relevant updates to the configuration have already been done) - If a configuration is not provided,
kwargs
will be first passed to the configuration class initialization function (from_pretrained()). Each key ofkwargs
that corresponds to a configuration attribute will be used to override said attribute with the suppliedkwargs
value. Remaining keys that do not correspond to any configuration attribute will be passed to the underlying model’s__init__
function.
- If a configuration is provided with
Instantiate a pretrained pytorch model from a pre-trained model configuration.
The model is set in evaluation mode by default using model.eval()
(Dropout modules are deactivated). To train
the model, you should first set it back in training mode with model.train()
.
The warning Weights from XXX not initialized from pretrained model means that the weights of XXX do not come pretrained with the rest of the model. It is up to you to train those weights with a downstream fine-tuning task.
The warning Weights from XXX not used in YYY means that the layer XXX is not used by YYY, therefore those weights are discarded.
Passing `use_auth_token=True“ is required when you want to use a private model.
Activate the special “offline-mode” to use this method in a firewalled environment.
Examples:
>>> from transformers import BertConfig, BertModel
>>> # Download model and configuration from huggingface.co and cache.
>>> model = BertModel.from_pretrained("bert-base-uncased")
>>> # Model was saved using *save_pretrained('./test/saved_model/')* (for example purposes, not runnable).
>>> model = BertModel.from_pretrained("./test/saved_model/")
>>> # Update configuration during loading.
>>> model = BertModel.from_pretrained("bert-base-uncased", output_attentions=True)
>>> assert model.config.output_attentions == True
>>> # Loading from a TF checkpoint file instead of a PyTorch model (slower, for example purposes, not runnable).
>>> config = BertConfig.from_json_file("./tf_model/my_tf_model_config.json")
>>> model = BertModel.from_pretrained("./tf_model/my_tf_checkpoint.ckpt.index", from_tf=True, config=config)
>>> # Loading from a Flax checkpoint file instead of a PyTorch model (slower)
>>> model = BertModel.from_pretrained("bert-base-uncased", from_flax=True)
low_cpu_mem_usage
algorithm:
This is an experimental function that loads the model using ~1x model size CPU memory
Here is how it works:
- save which state_dict keys we have
- drop state_dict before the model is created, since the latter takes 1x model size CPU memory
- after the model has been instantiated switch to the meta device all params/buffers that are going to be replaced from the loaded state_dict
- load state_dict 2nd time
- replace the params/buffers from the state_dict
Currently, it can’t handle deepspeed ZeRO stage 3 and ignores loading errors
get_input_embeddings
< source >(
)
→
nn.Module
Returns
nn.Module
A torch module mapping vocabulary to hidden states.
Returns the model’s input embeddings.
get_memory_footprint
< source >( return_buffers = True )
Parameters
-
return_buffers (
bool
, optional, defaults toTrue
) — Whether to return the size of the buffer tensors in the computation of the memory footprint. Buffers are tensors that do not require gradients and not registered as parameters. E.g. mean and std in batch norm layers. Please see: https://discuss.pytorch.org/t/what-pytorch-means-by-buffers/120266/2
Get the memory footprint of a model. This will return the memory footprint of the current model in bytes. Useful to benchmark the memory footprint of the current model and design some tests. Solution inspired from the PyTorch discussions: https://discuss.pytorch.org/t/gpu-memory-that-model-uses/56822/2
get_output_embeddings
< source >(
)
→
nn.Module
Returns
nn.Module
A torch module mapping hidden states to vocabulary.
Returns the model’s output embeddings.
Deactivates gradient checkpointing for the current model.
Note that in other frameworks this feature can be referred to as “activation checkpointing” or “checkpoint activations”.
Activates gradient checkpointing for the current model.
Note that in other frameworks this feature can be referred to as “activation checkpointing” or “checkpoint activations”.
If needed prunes and maybe initializes weights.
A method executed at the end of each Transformer model initialization, to execute code that needs the model’s modules properly initialized (such as weight initialization).
prune_heads
< source >( heads_to_prune: typing.Dict[int, typing.List[int]] )
Prunes heads of the base model.
register_for_auto_class
< source >( auto_class = 'AutoModel' )
Register this class with a given auto class. This should only be used for custom models as the ones in the library are already mapped with an auto class.
This API is experimental and may have some slight breaking changes in the next releases.
resize_token_embeddings
< source >(
new_num_tokens: typing.Optional[int] = None
)
→
torch.nn.Embedding
Parameters
-
new_num_tokens (
int
, optional) — The number of new tokens in the embedding matrix. Increasing the size will add newly initialized vectors at the end. Reducing the size will remove vectors from the end. If not provided orNone
, just returns a pointer to the input tokenstorch.nn.Embedding
module of the model without doing anything.
Returns
torch.nn.Embedding
Pointer to the input tokens Embeddings Module of the model.
Resizes input token embeddings matrix of the model if new_num_tokens != config.vocab_size
.
Takes care of tying weights embeddings afterwards if the model class has a tie_weights()
method.
save_pretrained
< source >( save_directory: typing.Union[str, os.PathLike] is_main_process: bool = True state_dict: typing.Optional[dict] = None save_function: typing.Callable = <function save at 0x7fb4a5816e50> push_to_hub: bool = False max_shard_size: typing.Union[int, str] = '10GB' **kwargs )
Parameters
-
save_directory (
str
oros.PathLike
) — Directory to which to save. Will be created if it doesn’t exist. -
is_main_process (
bool
, optional, defaults toTrue
) — Whether the process calling this is the main process or not. Useful when in distributed training like TPUs and need to call this function on all processes. In this case, setis_main_process=True
only on the main process to avoid race conditions. -
state_dict (nested dictionary of
torch.Tensor
) — The state dictionary of the model to save. Will default toself.state_dict()
, but can be used to only save parts of the model or if special precautions need to be taken when recovering the state dictionary of a model (like when using model parallelism). -
save_function (
Callable
) — The function to use to save the state dictionary. Useful on distributed training like TPUs when one need to replacetorch.save
by another method. -
push_to_hub (
bool
, optional, defaults toFalse
) — Whether or not to push your model to the Hugging Face model hub after saving it. You can specify the repository you want to push to withrepo_id
(will default to the name ofsave_directory
in your namespace). -
max_shard_size (
int
orstr
, optional, defaults to"10GB"
) — The maximum size for a checkpoint before being sharded. Checkpoints shard will then be each of size lower than this size. If expressed as a string, needs to be digits followed by a unit (like"5MB"
).If a single weight of the model is bigger than
max_shard_size
, it will be in its own checkpoint shard which will be bigger thanmax_shard_size
.kwargs — Additional key word arguments passed along to the push_to_hub() method.
Save a model and its configuration file to a directory, so that it can be re-loaded using the
[from_pretrained()](/docs/transformers/v4.22.2/en/main_classes/model#transformers.PreTrainedModel.from_pretrained)
class method.
set_input_embeddings
< source >( value: Module )
Set model’s input embeddings.
Tie the weights between the input embeddings and the output embeddings.
If the torchscript
flag is set in the configuration, can’t handle parameter sharing so we are cloning the
weights instead.
Large model loading
In Transformers 4.20.0, the from_pretrained() method has been reworked to accommodate large models using Accelerate. This requires Accelerate >= 0.9.0 and PyTorch >= 1.9.0. Instead of creating the full model, then loading the pretrained weights inside it (which takes twice the size of the model in RAM, one for the randomly initialized model, one for the weights), there is an option to create the model as an empty shell, then only materialize its parameters when the pretrained weights are loaded.
This option can be activated with low_cpu_mem_usage=True
. The model is first created on the Meta device (with empty weights) and the state dict is then loaded inside it (shard by shard in the case of a sharded checkpoint). This way the maximum RAM used is the full size of the model only.
from transformers import AutoModelForSeq2SeqLM
t0pp = AutoModelForSeq2SeqLM.from_pretrained("bigscience/T0pp", low_cpu_mem_usage=True)
Moreover, you can directly place the model on different devices if it doesn’t fully fit in RAM (only works for inference for now). With device_map="auto"
, Accelerate will determine where to put each layer to maximize the use of your fastest devices (GPUs) and offload the rest on the CPU, or even the hard drive if you don’t have enough GPU RAM (or CPU RAM). Even if the model is split across several devices, it will run as you would normally expect.
When passing a device_map
, low_cpu_mem_usage
is automatically set to True
, so you don’t need to specify it:
from transformers import AutoModelForSeq2SeqLM
t0pp = AutoModelForSeq2SeqLM.from_pretrained("bigscience/T0pp", device_map="auto")
You can inspect how the model was split across devices by looking at its hf_device_map
attribute:
t0pp.hf_device_map
{'shared': 0,
'decoder.embed_tokens': 0,
'encoder': 0,
'decoder.block.0': 0,
'decoder.block.1': 1,
'decoder.block.2': 1,
'decoder.block.3': 1,
'decoder.block.4': 1,
'decoder.block.5': 1,
'decoder.block.6': 1,
'decoder.block.7': 1,
'decoder.block.8': 1,
'decoder.block.9': 1,
'decoder.block.10': 1,
'decoder.block.11': 1,
'decoder.block.12': 1,
'decoder.block.13': 1,
'decoder.block.14': 1,
'decoder.block.15': 1,
'decoder.block.16': 1,
'decoder.block.17': 1,
'decoder.block.18': 1,
'decoder.block.19': 1,
'decoder.block.20': 1,
'decoder.block.21': 1,
'decoder.block.22': 'cpu',
'decoder.block.23': 'cpu',
'decoder.final_layer_norm': 'cpu',
'decoder.dropout': 'cpu',
'lm_head': 'cpu'}
You can also write your own device map following the same format (a dictionary layer name to device). It should map all parameters of the model to a given device, but you don’t have to detail where all the submosules of one layer go if that layer is entirely on the same device. For instance, the following device map would work properly for T0pp (as long as you have the GPU memory):
device_map = {"shared": 0, "encoder": 0, "decoder": 1, "lm_head": 1}
Another way to minimize the memory impact of your model is to instantiate it at a lower precision dtype (like torch.float16
) or use direct quantization techniques as described below.
Model Instantiation dtype
Under Pytorch a model normally gets instantiated with torch.float32
format. This can be an issue if one tries to
load a model whose weights are in fp16, since it’d require twice as much memory. To overcome this limitation, you can
either explicitly pass the desired dtype
using torch_dtype
argument:
model = T5ForConditionalGeneration.from_pretrained("t5", torch_dtype=torch.float16)
or, if you want the model to always load in the most optimal memory pattern, you can use the special value "auto"
,
and then dtype
will be automatically derived from the model’s weights:
model = T5ForConditionalGeneration.from_pretrained("t5", torch_dtype="auto")
Models instantiated from scratch can also be told which dtype
to use with:
config = T5Config.from_pretrained("t5")
model = AutoModel.from_config(config)
Due to Pytorch design, this functionality is only available for floating dtypes.
ModuleUtilsMixin
A few utilities for torch.nn.Modules
, to be used as a mixin.
Add a memory hook before and after each sub-module forward pass to record increase in memory consumption.
Increase in memory consumption is stored in a mem_rss_diff
attribute for each module and can be reset to zero
with model.reset_memory_hooks_state()
.
estimate_tokens
< source >(
input_dict: typing.Dict[str, typing.Union[torch.Tensor, typing.Any]]
)
→
int
Helper function to estimate the total number of tokens from the model inputs.
floating_point_ops
< source >(
input_dict: typing.Dict[str, typing.Union[torch.Tensor, typing.Any]]
exclude_embeddings: bool = True
)
→
int
Parameters
-
batch_size (
int
) — The batch size for the forward pass. -
sequence_length (
int
) — The number of tokens in each line of the batch. -
exclude_embeddings (
bool
, optional, defaults toTrue
) — Whether or not to count embedding and softmax operations.
Returns
int
The number of floating-point operations.
Get number of (optionally, non-embeddings) floating-point operations for the forward and backward passes of a
batch with this transformer model. Default approximation neglects the quadratic dependency on the number of
tokens (valid if 12 * d_model << sequence_length
) as laid out in this
paper section 2.1. Should be overridden for transformers with parameter
re-use e.g. Albert or Universal Transformers, or if doing long-range modeling with very high sequence lengths.
get_extended_attention_mask
< source >( attention_mask: Tensor input_shape: typing.Tuple[int] device: <property object at 0x7fb4422ae040> = None dtype: torch.float32 = None )
Makes broadcastable attention and causal masks so that future and masked tokens are ignored.
get_head_mask
< source >( head_mask: typing.Optional[torch.Tensor] num_hidden_layers: int is_attention_chunked: bool = False )
Parameters
-
head_mask (
torch.Tensor
with shape[num_heads]
or[num_hidden_layers x num_heads]
, optional) — The mask indicating if we should keep the heads or not (1.0 for keep, 0.0 for discard). - num_hidden_layers (
int
) — The number of hidden layers in the model. is_attention_chunked — (bool
, optional, defaults toFalse
): Whether or not the attentions scores are computed by chunks or not.
Prepare the head mask if needed.
invert_attention_mask
< source >(
encoder_attention_mask: Tensor
)
→
torch.Tensor
Invert an attention mask (e.g., switches 0. and 1.).
num_parameters
< source >(
only_trainable: bool = False
exclude_embeddings: bool = False
)
→
int
Get number of (optionally, trainable or non-embeddings) parameters in the module.
Reset the mem_rss_diff
attribute of each module (see add_memory_hooks()).
TFPreTrainedModel
Base class for all TF models.
TFPreTrainedModel takes care of storing the configuration of the models and handles methods for loading, downloading and saving models as well as a few methods common to all models to:
- resize the input embeddings,
- prune heads in the self-attention heads.
Class attributes (overridden by derived classes):
- config_class (PretrainedConfig) — A subclass of PretrainedConfig to use as configuration class for this model architecture.
- base_model_prefix (
str
) — A string indicating the attribute associated to the base model in derived classes of the same architecture adding modules on top of the base model. - main_input_name (
str
) — The name of the principal input to the model (ofteninput_ids
for NLP models,pixel_values
for vision models andinput_values
for speech models).
push_to_hub
< source >( repo_id: str use_temp_dir: typing.Optional[bool] = None commit_message: typing.Optional[str] = None private: typing.Optional[bool] = None use_auth_token: typing.Union[bool, str, NoneType] = None max_shard_size: typing.Union[int, str, NoneType] = '10GB' **model_card_kwargs )
Parameters
-
repo_id (
str
) — The name of the repository you want to push your model to. It should contain your organization name when pushing to a given organization. -
use_temp_dir (
bool
, optional) — Whether or not to use a temporary directory to store the files saved before they are pushed to the Hub. Will default toTrue
if there is no directory named likerepo_id
,False
otherwise. -
commit_message (
str
, optional) — Message to commit while pushing. Will default to"Upload model"
. -
private (
bool
, optional) — Whether or not the repository created should be private (requires a paying subscription). -
use_auth_token (
bool
orstr
, optional) — The token to use as HTTP bearer authorization for remote files. IfTrue
, will use the token generated when runninghuggingface-cli login
(stored in~/.huggingface
). Will default toTrue
ifrepo_url
is not specified. -
max_shard_size (
int
orstr
, optional, defaults to"10GB"
) — Only applicable for models. The maximum size for a checkpoint before being sharded. Checkpoints shard will then be each of size lower than this size. If expressed as a string, needs to be digits followed by a unit (like"5MB"
). model_card_kwargs — Additional keyword arguments passed along to thecreate_model_card()
method.
Upload the model files to the 🤗 Model Hub while synchronizing a local clone of the repo in repo_path_or_name
.
Examples:
from transformers import TFAutoModel
model = TFAutoModel.from_pretrained("bert-base-cased")
# Push the model to your namespace with the name "my-finetuned-bert".
model.push_to_hub("my-finetuned-bert")
# Push the model to an organization with the name "my-finetuned-bert".
model.push_to_hub("huggingface/my-finetuned-bert")
compile
< source >( optimizer = 'rmsprop' loss = 'passthrough' metrics = None loss_weights = None weighted_metrics = None run_eagerly = None steps_per_execution = None **kwargs )
This is a thin wrapper that sets the model’s loss output head as the loss if the user does not specify a loss function themselves.
from_pretrained
< source >( pretrained_model_name_or_path *model_args **kwargs )
Parameters
-
pretrained_model_name_or_path (
str
, optional) — Can be either:- A string, the model id of a pretrained model hosted inside a model repo on huggingface.co.
Valid model ids can be located at the root-level, like
bert-base-uncased
, or namespaced under a user or organization name, likedbmdz/bert-base-german-cased
. - A path to a directory containing model weights saved using
save_pretrained(), e.g.,
./my_model_directory/
. - A path or url to a PyTorch state_dict save file (e.g,
./pt_model/pytorch_model.bin
). In this case,from_pt
should be set toTrue
and a configuration object should be provided asconfig
argument. This loading path is slower than converting the PyTorch model in a TensorFlow model using the provided conversion scripts and loading the TensorFlow model afterwards. None
if you are both providing the configuration and state dictionary (resp. with keyword argumentsconfig
andstate_dict
).
- A string, the model id of a pretrained model hosted inside a model repo on huggingface.co.
Valid model ids can be located at the root-level, like
-
model_args (sequence of positional arguments, optional) —
All remaining positional arguments will be passed to the underlying model’s
__init__
method. -
config (
Union[PretrainedConfig, str]
, optional) — Can be either:- an instance of a class derived from PretrainedConfig,
- a string valid as input to from_pretrained().
Configuration for the model to use instead of an automatically loaded configuration. Configuration can be automatically loaded when:
- The model is a model provided by the library (loaded with the model id string of a pretrained model).
- The model was saved using save_pretrained() and is reloaded by supplying the save directory.
- The model is loaded by supplying a local directory as
pretrained_model_name_or_path
and a configuration JSON file named config.json is found in the directory. from_pt — (bool
, optional, defaults toFalse
): Load the model weights from a PyTorch state_dict save file (see docstring ofpretrained_model_name_or_path
argument).
-
ignore_mismatched_sizes (
bool
, optional, defaults toFalse
) — Whether or not to raise an error if some of the weights from the checkpoint do not have the same size as the weights of the model (if for instance, you are instantiating a model with 10 labels from a checkpoint with 3 labels). -
cache_dir (
str
, optional) — Path to a directory in which a downloaded pretrained model configuration should be cached if the standard cache should not be used. -
force_download (
bool
, optional, defaults toFalse
) — Whether or not to force the (re-)download of the model weights and configuration files, overriding the cached versions if they exist. -
resume_download (
bool
, optional, defaults toFalse
) — Whether or not to delete incompletely received files. Will attempt to resume the download if such a file exists. proxies — (Dict[str, str],
optional): A dictionary of proxy servers to use by protocol or endpoint, e.g.,
{‘http’: ‘foo.bar:3128’, ‘http://hostname’: ‘foo.bar:4012’}. The proxies are used on each request. output_loading_info(
bool, *optional*, defaults to
False`): Whether ot not to also return a dictionary containing missing keys, unexpected keys and error messages. -
local_files_only(
bool
, optional, defaults toFalse
) — Whether or not to only look at local files (e.g., not try doanloading the model). -
use_auth_token (
str
or bool, optional) — The token to use as HTTP bearer authorization for remote files. IfTrue
, will use the token generated when runninghuggingface-cli login
(stored in~/.huggingface
). -
revision (
str
, optional, defaults to"main"
) — The specific model version to use. It can be a branch name, a tag name, or a commit id, since we use a git-based system for storing models and other artifacts on huggingface.co, sorevision
can be any identifier allowed by git. -
mirror (
str
, optional) — Mirror source to accelerate downloads in China. If you are from China and have an accessibility problem, you can set this option to resolve it. Note that we do not guarantee the timeliness or safety. Please refer to the mirror site for more information. -
subfolder (
str
, optional, defaults to""
) — In case the relevant files are located inside a subfolder of the model repo on huggingface.co, you can specify the folder name here. -
kwargs (remaining dictionary of keyword arguments, optional) —
Can be used to update the configuration object (after it being loaded) and initiate the model (e.g.,
output_attentions=True
). Behaves differently depending on whether aconfig
is provided or automatically loaded:- If a configuration is provided with
config
,**kwargs
will be directly passed to the underlying model’s__init__
method (we assume all relevant updates to the configuration have already been done) - If a configuration is not provided,
kwargs
will be first passed to the configuration class initialization function (from_pretrained()). Each key ofkwargs
that corresponds to a configuration attribute will be used to override said attribute with the suppliedkwargs
value. Remaining keys that do not correspond to any configuration attribute will be passed to the underlying model’s__init__
function.
- If a configuration is provided with
Instantiate a pretrained TF 2.0 model from a pre-trained model configuration.
The warning Weights from XXX not initialized from pretrained model means that the weights of XXX do not come pretrained with the rest of the model. It is up to you to train those weights with a downstream fine-tuning task.
The warning Weights from XXX not used in YYY means that the layer XXX is not used by YYY, therefore those weights are discarded.
Passing use_auth_token=True
is required when you want to use a private model.
Examples:
>>> from transformers import BertConfig, TFBertModel
>>> # Download model and configuration from huggingface.co and cache.
>>> model = TFBertModel.from_pretrained("bert-base-uncased")
>>> # Model was saved using *save_pretrained('./test/saved_model/')* (for example purposes, not runnable).
>>> model = TFBertModel.from_pretrained("./test/saved_model/")
>>> # Update configuration during loading.
>>> model = TFBertModel.from_pretrained("bert-base-uncased", output_attentions=True)
>>> assert model.config.output_attentions == True
>>> # Loading from a Pytorch model file instead of a TensorFlow checkpoint (slower, for example purposes, not runnable).
>>> config = BertConfig.from_json_file("./pt_model/my_pt_model_config.json")
>>> model = TFBertModel.from_pretrained("./pt_model/my_pytorch_model.bin", from_pt=True, config=config)
get_bias
< source >(
)
→
tf.Variable
Returns
tf.Variable
The weights representing the bias, None if not an LM model.
Dict of bias attached to an LM head. The key represents the name of the bias attribute.
get_input_embeddings
< source >(
)
→
tf.Variable
Returns
tf.Variable
The embeddings layer mapping vocabulary to hidden states.
Returns the model’s input embeddings layer.
get_lm_head
< source >(
)
→
tf.keras.layers.Layer
Returns
tf.keras.layers.Layer
The LM head layer if the model has one, None if not.
The LM Head layer. This method must be overwritten by all the models that have a lm head.
get_output_embeddings
< source >(
)
→
tf.Variable
Returns
tf.Variable
The new weights mapping vocabulary to hidden states.
Returns the model’s output embeddings
get_output_layer_with_bias
< source >(
)
→
tf.keras.layers.Layer
Returns
tf.keras.layers.Layer
The layer that handles the bias, None if not an LM model.
Get the layer that handles a bias attribute in case the model has an LM head with weights tied to the embeddings
Get the concatenated _prefix name of the bias from the model name to the parent layer
load_repo_checkpoint
< source >(
repo_path_or_name
)
→
dict
Loads a saved checkpoint (model weights and optimizer state) from a repo. Returns the current epoch count when the checkpoint was made.
prepare_tf_dataset
< source >(
dataset: datasets.Dataset
batch_size: int = 8
shuffle: bool = True
tokenizer: typing.Optional[ForwardRef('PreTrainedTokenizerBase')] = None
collate_fn: typing.Optional[typing.Callable] = None
collate_fn_args: typing.Union[typing.Dict[str, typing.Any], NoneType] = None
drop_remainder: typing.Optional[bool] = None
prefetch: bool = True
)
→
Dataset
Parameters
-
dataset (
Any
) — A [~datasets.Dataset
] to be wrapped as atf.data.Dataset
. -
batch_size (
int
, defaults to 8) — The size of batches to return. -
shuffle (
bool
, defaults toTrue
) — Whether to return samples from the dataset in random order. UsuallyTrue
for training datasets andFalse
for validation/test datasets. -
tokenizer (PreTrainedTokenizerBase, optional) —
A
PreTrainedTokenizer
that will be used to pad samples to create batches. Has no effect if a specificcollate_fn
is passed instead. -
collate_fn (
Callable
, optional) — A function that collates samples from the dataset into a single batch. Defaults toDefaultDataCollator
if notokenizer
is supplied orDataCollatorWithPadding
if atokenizer
is passed. -
collate_fn_args (
Dict[str, Any]
, optional) — A dict of arguments to pass to thecollate_fn
alongside the list of samples. -
drop_remainder (
bool
, optional) — Whether to drop the final batch, if the batch_size does not evenly divide the dataset length. Defaults to the same setting asshuffle
. -
prefetch (
bool
, defaults toTrue
) — Whether to add prefetching to the end of thetf.data
pipeline. This is almost always beneficial for performance, but can be disabled in edge cases.
Returns
Dataset
A tf.data.Dataset
which is ready to pass to the Keras API.
Wraps a HuggingFace Dataset as a tf.data.Dataset
with collation and batching. This method is
designed to create a “ready-to-use” dataset that can be passed directly to Keras methods like fit()
without
further modification. The method will drop columns from the dataset if they don’t match input names for the
model. If you want to specify the column names to return rather than using the names that match this model, we
recommend using Dataset.to_tf_dataset()
instead.
prune_heads
< source >( heads_to_prune )
Prunes heads of the base model.
register_for_auto_class
< source >( auto_class = 'TFAutoModel' )
Register this class with a given auto class. This should only be used for custom models as the ones in the library are already mapped with an auto class.
This API is experimental and may have some slight breaking changes in the next releases.
resize_token_embeddings
< source >(
new_num_tokens: typing.Optional[int] = None
)
→
tf.Variable
or tf.keras.layers.Embedding
Parameters
-
new_num_tokens (
int
, optional) — The number of new tokens in the embedding matrix. Increasing the size will add newly initialized vectors at the end. Reducing the size will remove vectors from the end. If not provided orNone
, just returns a pointer to the input tokens without doing anything.
Returns
tf.Variable
or tf.keras.layers.Embedding
Pointer to the input tokens of the model.
Resizes input token embeddings matrix of the model if new_num_tokens != config.vocab_size
.
Takes care of tying weights embeddings afterwards if the model class has a tie_weights()
method.
save_pretrained
< source >( save_directory saved_model = False version = 1 push_to_hub = False max_shard_size: typing.Union[int, str] = '10GB' create_pr: bool = False **kwargs )
Parameters
-
save_directory (
str
) — Directory to which to save. Will be created if it doesn’t exist. -
saved_model (
bool
, optional, defaults toFalse
) — If the model has to be saved in saved model format as well or not. -
version (
int
, optional, defaults to 1) — The version of the saved model. A saved model needs to be versioned in order to be properly loaded by TensorFlow Serving as detailed in the official documentation https://www.tensorflow.org/tfx/serving/serving_basic -
push_to_hub (
bool
, optional, defaults toFalse
) — Whether or not to push your model to the Hugging Face model hub after saving it. You can specify the repository you want to push to withrepo_id
(will default to the name ofsave_directory
in your namespace). -
max_shard_size (
int
orstr
, optional, defaults to"10GB"
) — The maximum size for a checkpoint before being sharded. Checkpoints shard will then be each of size lower than this size. If expressed as a string, needs to be digits followed by a unit (like"5MB"
).If a single weight of the model is bigger than
max_shard_size
, it will be in its own checkpoint shard which will be bigger thanmax_shard_size
. -
create_pr (
bool
, optional, defaults toFalse
) — Whether or not to create a PR with the uploaded files or directly commit.kwargs — Additional key word arguments passed along to the push_to_hub() method.
Save a model and its configuration file to a directory, so that it can be re-loaded using the from_pretrained() class method.
serving_output
< source >( output )
Prepare the output of the saved model. Each model must implement this function.
set_bias
< source >( value )
Set all the bias in the LM head.
set_input_embeddings
< source >( value )
Set model’s input embeddings
set_output_embeddings
< source >( value )
Set model’s output embeddings
A modification of Keras’s default train_step
that correctly handles matching outputs to labels for our models
and supports directly training on the loss output head. In addition, it ensures input keys are copied to the
labels where appropriate. It will also copy label keys into the input dict when using the dummy loss, to ensure
that they are available to the model during the forward pass.
A modification of Keras’s default train_step
that correctly handles matching outputs to labels for our models
and supports directly training on the loss output head. In addition, it ensures input keys are copied to the
labels where appropriate. It will also copy label keys into the input dict when using the dummy loss, to ensure
that they are available to the model during the forward pass.
TFModelUtilsMixin
A few utilities for tf.keras.Model
, to be used as a mixin.
num_parameters
< source >(
only_trainable: bool = False
)
→
int
Get the number of (optionally, trainable) parameters in the model.
FlaxPreTrainedModel
class transformers.FlaxPreTrainedModel
< source >( config: PretrainedConfig module: Module input_shape: typing.Tuple = (1, 1) seed: int = 0 dtype: dtype = <class 'jax.numpy.float32'> _do_init: bool = True )
Base class for all models.
FlaxPreTrainedModel takes care of storing the configuration of the models and handles methods for loading, downloading and saving models.
Class attributes (overridden by derived classes):
- config_class (PretrainedConfig) — A subclass of PretrainedConfig to use as configuration class for this model architecture.
- base_model_prefix (
str
) — A string indicating the attribute associated to the base model in derived classes of the same architecture adding modules on top of the base model. - main_input_name (
str
) — The name of the principal input to the model (ofteninput_ids
for NLP models,pixel_values
for vision models andinput_values
for speech models).
push_to_hub
< source >( repo_id: str use_temp_dir: typing.Optional[bool] = None commit_message: typing.Optional[str] = None private: typing.Optional[bool] = None use_auth_token: typing.Union[bool, str, NoneType] = None max_shard_size: typing.Union[int, str, NoneType] = '10GB' create_pr: bool = False **deprecated_kwargs )
Parameters
-
repo_id (
str
) — The name of the repository you want to push your model to. It should contain your organization name when pushing to a given organization. -
use_temp_dir (
bool
, optional) — Whether or not to use a temporary directory to store the files saved before they are pushed to the Hub. Will default toTrue
if there is no directory named likerepo_id
,False
otherwise. -
commit_message (
str
, optional) — Message to commit while pushing. Will default to"Upload model"
. -
private (
bool
, optional) — Whether or not the repository created should be private (requires a paying subscription). -
use_auth_token (
bool
orstr
, optional) — The token to use as HTTP bearer authorization for remote files. IfTrue
, will use the token generated when runninghuggingface-cli login
(stored in~/.huggingface
). Will default toTrue
ifrepo_url
is not specified. -
max_shard_size (
int
orstr
, optional, defaults to"10GB"
) — Only applicable for models. The maximum size for a checkpoint before being sharded. Checkpoints shard will then be each of size lower than this size. If expressed as a string, needs to be digits followed by a unit (like"5MB"
). -
create_pr (
bool
, optional, defaults toFalse
) — Whether or not to create a PR with the uploaded files or directly commit.
Upload the model checkpoint to the 🤗 Model Hub while synchronizing a local clone of the repo in
repo_path_or_name
.
Examples:
from transformers import FlaxAutoModel
model = FlaxAutoModel.from_pretrained("bert-base-cased")
# Push the model to your namespace with the name "my-finetuned-bert".
model.push_to_hub("my-finetuned-bert")
# Push the model to an organization with the name "my-finetuned-bert".
model.push_to_hub("huggingface/my-finetuned-bert")
from_pretrained
< source >( pretrained_model_name_or_path: typing.Union[str, os.PathLike] dtype: dtype = <class 'jax.numpy.float32'> *model_args **kwargs )
Parameters
-
pretrained_model_name_or_path (
str
oros.PathLike
) — Can be either:- A string, the model id of a pretrained model hosted inside a model repo on huggingface.co.
Valid model ids can be located at the root-level, like
bert-base-uncased
, or namespaced under a user or organization name, likedbmdz/bert-base-german-cased
. - A path to a directory containing model weights saved using
save_pretrained(), e.g.,
./my_model_directory/
. - A path or url to a pt index checkpoint file (e.g,
./tf_model/model.ckpt.index
). In this case,from_pt
should be set toTrue
.
- A string, the model id of a pretrained model hosted inside a model repo on huggingface.co.
Valid model ids can be located at the root-level, like
-
dtype (
jax.numpy.dtype
, optional, defaults tojax.numpy.float32
) — The data type of the computation. Can be one ofjax.numpy.float32
,jax.numpy.float16
(on GPUs) andjax.numpy.bfloat16
(on TPUs).This can be used to enable mixed-precision training or half-precision inference on GPUs or TPUs. If specified all the computation will be performed with the given
dtype
.Note that this only specifies the dtype of the computation and does not influence the dtype of model parameters.
If you wish to change the dtype of the model parameters, see to_fp16() and to_bf16().
-
model_args (sequence of positional arguments, optional) —
All remaining positional arguments will be passed to the underlying model’s
__init__
method. -
config (
Union[PretrainedConfig, str, os.PathLike]
, optional) — Can be either:- an instance of a class derived from PretrainedConfig,
- a string or path valid as input to from_pretrained().
Configuration for the model to use instead of an automatically loaded configuration. Configuration can be automatically loaded when:
- The model is a model provided by the library (loaded with the model id string of a pretrained model).
- The model was saved using save_pretrained() and is reloaded by supplying the save directory.
- The model is loaded by supplying a local directory as
pretrained_model_name_or_path
and a configuration JSON file named config.json is found in the directory.
-
cache_dir (
Union[str, os.PathLike]
, optional) — Path to a directory in which a downloaded pretrained model configuration should be cached if the standard cache should not be used. -
from_pt (
bool
, optional, defaults toFalse
) — Load the model weights from a PyTorch checkpoint save file (see docstring ofpretrained_model_name_or_path
argument). -
ignore_mismatched_sizes (
bool
, optional, defaults toFalse
) — Whether or not to raise an error if some of the weights from the checkpoint do not have the same size as the weights of the model (if for instance, you are instantiating a model with 10 labels from a checkpoint with 3 labels). -
force_download (
bool
, optional, defaults toFalse
) — Whether or not to force the (re-)download of the model weights and configuration files, overriding the cached versions if they exist. -
resume_download (
bool
, optional, defaults toFalse
) — Whether or not to delete incompletely received files. Will attempt to resume the download if such a file exists. -
proxies (
Dict[str, str]
, optional) — A dictionary of proxy servers to use by protocol or endpoint, e.g.,{'http': 'foo.bar:3128', 'http://hostname': 'foo.bar:4012'}
. The proxies are used on each request. -
local_files_only(
bool
, optional, defaults toFalse
) — Whether or not to only look at local files (i.e., do not try to download the model). -
revision (
str
, optional, defaults to"main"
) — The specific model version to use. It can be a branch name, a tag name, or a commit id, since we use a git-based system for storing models and other artifacts on huggingface.co, sorevision
can be any identifier allowed by git. -
subfolder (
str
, optional, defaults to""
) — In case the relevant files are located inside a subfolder of the model repo on huggingface.co, you can specify the folder name here. -
kwargs (remaining dictionary of keyword arguments, optional) —
Can be used to update the configuration object (after it being loaded) and initiate the model (e.g.,
output_attentions=True
). Behaves differently depending on whether aconfig
is provided or automatically loaded:- If a configuration is provided with
config
,**kwargs
will be directly passed to the underlying model’s__init__
method (we assume all relevant updates to the configuration have already been done) - If a configuration is not provided,
kwargs
will be first passed to the configuration class initialization function (from_pretrained()). Each key ofkwargs
that corresponds to a configuration attribute will be used to override said attribute with the suppliedkwargs
value. Remaining keys that do not correspond to any configuration attribute will be passed to the underlying model’s__init__
function.
- If a configuration is provided with
Instantiate a pretrained flax model from a pre-trained model configuration.
The warning Weights from XXX not initialized from pretrained model means that the weights of XXX do not come pretrained with the rest of the model. It is up to you to train those weights with a downstream fine-tuning task.
The warning Weights from XXX not used in YYY means that the layer XXX is not used by YYY, therefore those weights are discarded.
Examples:
>>> from transformers import BertConfig, FlaxBertModel
>>> # Download model and configuration from huggingface.co and cache.
>>> model = FlaxBertModel.from_pretrained("bert-base-cased")
>>> # Model was saved using *save_pretrained('./test/saved_model/')* (for example purposes, not runnable).
>>> model = FlaxBertModel.from_pretrained("./test/saved_model/")
>>> # Loading from a PyTorch checkpoint file instead of a PyTorch model (slower, for example purposes, not runnable).
>>> config = BertConfig.from_json_file("./pt_model/config.json")
>>> model = FlaxBertModel.from_pretrained("./pt_model/pytorch_model.bin", from_pt=True, config=config)
load_flax_sharded_weights
< source >(
shard_files
)
→
Dict
This is the same as flax.serialization.from_bytes
(https:lax.readthedocs.io/en/latest/_modules/flax/serialization.html#from_bytes) but for a sharded checkpoint.
This load is performed efficiently: each checkpoint shard is loaded one by one in RAM and deleted after being loaded in the model.
register_for_auto_class
< source >( auto_class = 'FlaxAutoModel' )
Register this class with a given auto class. This should only be used for custom models as the ones in the library are already mapped with an auto class.
This API is experimental and may have some slight breaking changes in the next releases.
save_pretrained
< source >( save_directory: typing.Union[str, os.PathLike] params = None push_to_hub = False max_shard_size = '10GB' **kwargs )
Parameters
-
save_directory (
str
oros.PathLike
) — Directory to which to save. Will be created if it doesn’t exist. -
push_to_hub (
bool
, optional, defaults toFalse
) — Whether or not to push your model to the Hugging Face model hub after saving it. You can specify the repository you want to push to withrepo_id
(will default to the name ofsave_directory
in your namespace). -
max_shard_size (
int
orstr
, optional, defaults to"10GB"
) — The maximum size for a checkpoint before being sharded. Checkpoints shard will then be each of size lower than this size. If expressed as a string, needs to be digits followed by a unit (like"5MB"
).If a single weight of the model is bigger than
max_shard_size
, it will be in its own checkpoint shard which will be bigger thanmax_shard_size
.kwargs — Additional key word arguments passed along to the push_to_hub() method.
Save a model and its configuration file to a directory, so that it can be re-loaded using the
[from_pretrained()](/docs/transformers/v4.22.2/en/main_classes/model#transformers.FlaxPreTrainedModel.from_pretrained)
class method
to_bf16
< source >( params: typing.Union[typing.Dict, flax.core.frozen_dict.FrozenDict] mask: typing.Any = None )
Cast the floating-point params
to jax.numpy.bfloat16
. This returns a new params
tree and does not cast
the params
in place.
This method can be used on TPU to explicitly convert the model parameters to bfloat16 precision to do full half-precision training or to save weights in bfloat16 for inference in order to save memory and improve speed.
Examples:
>>> from transformers import FlaxBertModel
>>> # load model
>>> model = FlaxBertModel.from_pretrained("bert-base-cased")
>>> # By default, the model parameters will be in fp32 precision, to cast these to bfloat16 precision
>>> model.params = model.to_bf16(model.params)
>>> # If you want don't want to cast certain parameters (for example layer norm bias and scale)
>>> # then pass the mask as follows
>>> from flax import traverse_util
>>> model = FlaxBertModel.from_pretrained("bert-base-cased")
>>> flat_params = traverse_util.flatten_dict(model.params)
>>> mask = {
... path: (path[-2] != ("LayerNorm", "bias") and path[-2:] != ("LayerNorm", "scale"))
... for path in flat_params
... }
>>> mask = traverse_util.unflatten_dict(mask)
>>> model.params = model.to_bf16(model.params, mask)
to_fp16
< source >( params: typing.Union[typing.Dict, flax.core.frozen_dict.FrozenDict] mask: typing.Any = None )
Cast the floating-point parmas
to jax.numpy.float16
. This returns a new params
tree and does not cast the
params
in place.
This method can be used on GPU to explicitly convert the model parameters to float16 precision to do full half-precision training or to save weights in float16 for inference in order to save memory and improve speed.
Examples:
>>> from transformers import FlaxBertModel
>>> # load model
>>> model = FlaxBertModel.from_pretrained("bert-base-cased")
>>> # By default, the model params will be in fp32, to cast these to float16
>>> model.params = model.to_fp16(model.params)
>>> # If you want don't want to cast certain parameters (for example layer norm bias and scale)
>>> # then pass the mask as follows
>>> from flax import traverse_util
>>> model = FlaxBertModel.from_pretrained("bert-base-cased")
>>> flat_params = traverse_util.flatten_dict(model.params)
>>> mask = {
... path: (path[-2] != ("LayerNorm", "bias") and path[-2:] != ("LayerNorm", "scale"))
... for path in flat_params
... }
>>> mask = traverse_util.unflatten_dict(mask)
>>> model.params = model.to_fp16(model.params, mask)
to_fp32
< source >( params: typing.Union[typing.Dict, flax.core.frozen_dict.FrozenDict] mask: typing.Any = None )
Cast the floating-point parmas
to jax.numpy.float32
. This method can be used to explicitly convert the
model parameters to fp32 precision. This returns a new params
tree and does not cast the params
in place.
Examples:
>>> from transformers import FlaxBertModel
>>> # Download model and configuration from huggingface.co
>>> model = FlaxBertModel.from_pretrained("bert-base-cased")
>>> # By default, the model params will be in fp32, to illustrate the use of this method,
>>> # we'll first cast to fp16 and back to fp32
>>> model.params = model.to_f16(model.params)
>>> # now cast back to fp32
>>> model.params = model.to_fp32(model.params)
Pushing to the Hub
A Mixin containing the functionality to push a model or tokenizer to the hub.
push_to_hub
< source >( repo_id: str use_temp_dir: typing.Optional[bool] = None commit_message: typing.Optional[str] = None private: typing.Optional[bool] = None use_auth_token: typing.Union[bool, str, NoneType] = None max_shard_size: typing.Union[int, str, NoneType] = '10GB' create_pr: bool = False **deprecated_kwargs )
Parameters
-
repo_id (
str
) — The name of the repository you want to push your {object} to. It should contain your organization name when pushing to a given organization. -
use_temp_dir (
bool
, optional) — Whether or not to use a temporary directory to store the files saved before they are pushed to the Hub. Will default toTrue
if there is no directory named likerepo_id
,False
otherwise. -
commit_message (
str
, optional) — Message to commit while pushing. Will default to"Upload {object}"
. -
private (
bool
, optional) — Whether or not the repository created should be private (requires a paying subscription). -
use_auth_token (
bool
orstr
, optional) — The token to use as HTTP bearer authorization for remote files. IfTrue
, will use the token generated when runninghuggingface-cli login
(stored in~/.huggingface
). Will default toTrue
ifrepo_url
is not specified. -
max_shard_size (
int
orstr
, optional, defaults to"10GB"
) — Only applicable for models. The maximum size for a checkpoint before being sharded. Checkpoints shard will then be each of size lower than this size. If expressed as a string, needs to be digits followed by a unit (like"5MB"
). -
create_pr (
bool
, optional, defaults toFalse
) — Whether or not to create a PR with the uploaded files or directly commit.
Upload the {object_files} to the 🤗 Model Hub while synchronizing a local clone of the repo in
repo_path_or_name
.
Examples:
from transformers import {object_class}
{object} = {object_class}.from_pretrained("bert-base-cased")
# Push the {object} to your namespace with the name "my-finetuned-bert".
{object}.push_to_hub("my-finetuned-bert")
# Push the {object} to an organization with the name "my-finetuned-bert".
{object}.push_to_hub("huggingface/my-finetuned-bert")
Sharded checkpoints
transformers.modeling_utils.load_sharded_checkpoint
< source >(
model
folder
strict = True
)
→
NamedTuple
Parameters
-
model (
torch.nn.Module
) — The model in which to load the checkpoint. -
folder (
str
oros.PathLike
) — A path to a folder containing the sharded checkpoint. -
strict (
bool
, *optional, defaults to
True`) — Whether to strictly enforce that the keys in the model state dict match the keys in the sharded checkpoint.
Returns
NamedTuple
A named tuple with missing_keys
and unexpected_keys
fields
missing_keys
is a list of str containing the missing keysunexpected_keys
is a list of str containing the unexpected keys
This is the same as
torch.nn.Module.load_state_dict
but for a sharded checkpoint.
This load is performed efficiently: each checkpoint shard is loaded one by one in RAM and deleted after being loaded in the model.