Models
The base classes PreTrainedModel, TFPreTrainedModel, and FlaxPreTrainedModel implement the common methods for loading/saving a model either from a local file or directory, or from a pretrained model configuration provided by the library (downloaded from HuggingFace’s AWS S3 repository).
PreTrainedModel and TFPreTrainedModel also implement a few methods which are common among all the models to:
- resize the input token embeddings when new tokens are added to the vocabulary
- prune the attention heads of the model.
The other methods that are common to each model are defined in ModuleUtilsMixin
(for the PyTorch models) and TFModuleUtilsMixin (for the TensorFlow models) or
for text generation, GenerationMixin (for the PyTorch models),
TFGenerationMixin (for the TensorFlow models) and
FlaxGenerationMixin (for the Flax/JAX models).
PreTrainedModel
Base class for all models.
PreTrainedModel takes care of storing the configuration of the models and handles methods for loading, downloading and saving models as well as a few methods common to all models to:
- resize the input embeddings,
- prune heads in the self-attention heads.
Class attributes (overridden by derived classes):
config_class (PretrainedConfig) — A subclass of PretrainedConfig to use as configuration class for this model architecture.
load_tf_weights (
Callable) — A python method for loading a TensorFlow checkpoint in a PyTorch model, taking as arguments:- model (PreTrainedModel) — An instance of the model on which to load the TensorFlow checkpoint.
- config (
PreTrainedConfig) — An instance of the configuration associated to the model. - path (
str) — A path to the TensorFlow checkpoint.
base_model_prefix (
str) — A string indicating the attribute associated to the base model in derived classes of the same architecture adding modules on top of the base model.is_parallelizable (
bool) — A flag indicating whether this model supports model parallelization.
(
repo_path_or_name: typing.Optional[str] = None
repo_url: typing.Optional[str] = None
use_temp_dir: bool = False
commit_message: typing.Optional[str] = None
organization: typing.Optional[str] = None
private: typing.Optional[bool] = None
use_auth_token: typing.Union[bool, str, NoneType] = None
**model_card_kwargs
)
→
str
Parameters
-
repo_path_or_name (
str, optional) — Can either be a repository name for your model in the Hub or a path to a local folder (in which case the repository will have the name of that local folder). If not specified, will default to the name given byrepo_urland a local directory with that name will be created. -
repo_url (
str, optional) — Specify this in case you want to push to an existing repository in the hub. If unspecified, a new repository will be created in your namespace (unless you specify anorganization) withrepo_name. -
use_temp_dir (
bool, optional, defaults toFalse) — Whether or not to clone the distant repo in a temporary directory or inrepo_path_or_nameinside the current working directory. This will slow things down if you are making changes in an existing repo since you will need to clone the repo before every push. -
commit_message (
str, optional) — Message to commit while pushing. Will default to"add model". -
organization (
str, optional) — Organization in which you want to push your model (you must be a member of this organization). -
private (
bool, optional) — Whether or not the repository created should be private (requires a paying subscription). -
use_auth_token (
boolorstr, optional) — The token to use as HTTP bearer authorization for remote files. IfTrue, will use the token generated when runningtransformers-cli login(stored in~/.huggingface). Will default toTrueifrepo_urlis not specified.
Returns
str
The url of the commit of your model in the given repository.
Upload the model checkpoint to the 🤗 Model Hub while synchronizing a local clone of the repo in
repo_path_or_name.
Examples:
from transformers import AutoModel
model = AutoModel.from_pretrained("bert-base-cased")
# Push the model to your namespace with the name "my-finetuned-bert" and have a local clone in the
# _my-finetuned-bert_ folder.
model.push_to_hub("my-finetuned-bert")
# Push the model to your namespace with the name "my-finetuned-bert" with no local clone.
model.push_to_hub("my-finetuned-bert", use_temp_dir=True)
# Push the model to an organization with the name "my-finetuned-bert" and have a local clone in the
# _my-finetuned-bert_ folder.
model.push_to_hub("my-finetuned-bert", organization="huggingface")
# Make a change to an existing repo that has been cloned locally in _my-finetuned-bert_.
model.push_to_hub("my-finetuned-bert", repo_url="https://huggingface.co/sgugger/my-finetuned-bert")( pretrained_model_name_or_path: typing.Union[str, os.PathLike, NoneType] *model_args **kwargs )
Parameters
-
pretrained_model_name_or_path (
stroros.PathLike, optional) — Can be either:- A string, the model id of a pretrained model hosted inside a model repo on huggingface.co.
Valid model ids can be located at the root-level, like
bert-base-uncased, or namespaced under a user or organization name, likedbmdz/bert-base-german-cased. - A path to a directory containing model weights saved using
save_pretrained(), e.g.,
./my_model_directory/. - A path or url to a tensorflow index checkpoint file (e.g,
./tf_model/model.ckpt.index). In this case,from_tfshould be set toTrueand a configuration object should be provided asconfigargument. This loading path is slower than converting the TensorFlow checkpoint in a PyTorch model using the provided conversion scripts and loading the PyTorch model afterwards. - A path or url to a model folder containing a flax checkpoint file in .msgpack format (e.g,
./flax_model/containingflax_model.msgpack). In this case,from_flaxshould be set toTrue. Noneif you are both providing the configuration and state dictionary (resp. with keyword argumentsconfigandstate_dict).
- A string, the model id of a pretrained model hosted inside a model repo on huggingface.co.
Valid model ids can be located at the root-level, like
-
model_args (sequence of positional arguments, optional) —
All remaining positional arguments will be passed to the underlying model’s
__init__method. -
config (
Union[PretrainedConfig, str, os.PathLike], optional) — Can be either:- an instance of a class derived from PretrainedConfig,
- a string or path valid as input to from_pretrained().
Instantiate a pretrained pytorch model from a pre-trained model configuration.
The model is set in evaluation mode by default using model.eval() (Dropout modules are deactivated). To
train the model, you should first set it back in training mode with model.train().
The warning Weights from XXX not initialized from pretrained model means that the weights of XXX do not come pretrained with the rest of the model. It is up to you to train those weights with a downstream fine-tuning task.
The warning Weights from XXX not used in YYY means that the layer XXX is not used by YYY, therefore those weights are discarded.
Passing `use_auth_token=True“ is required when you want to use a private model.
Activate the special “offline-mode” to use this method in a firewalled environment.
Examples:
>>> from transformers import BertConfig, BertModel
>>> # Download model and configuration from huggingface.co and cache.
>>> model = BertModel.from_pretrained('bert-base-uncased')
>>> # Model was saved using _save_pretrained('./test/saved_model/')_ (for example purposes, not runnable).
>>> model = BertModel.from_pretrained('./test/saved_model/')
>>> # Update configuration during loading.
>>> model = BertModel.from_pretrained('bert-base-uncased', output_attentions=True)
>>> assert model.config.output_attentions == True
>>> # Loading from a TF checkpoint file instead of a PyTorch model (slower, for example purposes, not runnable).
>>> config = BertConfig.from_json_file('./tf_model/my_tf_model_config.json')
>>> model = BertModel.from_pretrained('./tf_model/my_tf_checkpoint.ckpt.index', from_tf=True, config=config)
>>> # Loading from a Flax checkpoint file instead of a PyTorch model (slower)
>>> model = BertModel.from_pretrained('bert-base-uncased', from_flax=True)(
)
→
nn.Module
Returns
nn.Module
A torch module mapping vocabulary to hidden states.
Returns the model’s input embeddings.
(
)
→
nn.Module
Returns
nn.Module
A torch module mapping hidden states to vocabulary.
Returns the model’s output embeddings.
Deactivates gradient checkpointing for the current model.
Note that in other frameworks this feature can be referred to as “activation checkpointing” or “checkpoint activations”.
Activates gradient checkpointing for the current model.
Note that in other frameworks this feature can be referred to as “activation checkpointing” or “checkpoint activations”.
If needed prunes and maybe initializes weights.
A method executed at the end of each Transformer model initialization, to execute code that needs the model’s modules properly initialized (such as weight initialization).
( heads_to_prune: typing.Dict[int, typing.List[int]] )
Prunes heads of the base model.
(
new_num_tokens: typing.Optional[int] = None
)
→
torch.nn.Embedding
Parameters
-
new_num_tokens (
int, optional) — The number of new tokens in the embedding matrix. Increasing the size will add newly initialized vectors at the end. Reducing the size will remove vectors from the end. If not provided orNone, just returns a pointer to the input tokenstorch.nn.Embeddingmodule of the model without doing anything.
Returns
torch.nn.Embedding
Pointer to the input tokens Embeddings Module of the model.
Resizes input token embeddings matrix of the model if new_num_tokens != config.vocab_size.
Takes care of tying weights embeddings afterwards if the model class has a tie_weights() method.
( save_directory: typing.Union[str, os.PathLike] save_config: bool = True state_dict: typing.Optional[dict] = None save_function: typing.Callable = <function save at 0x7f745f558a60> push_to_hub: bool = False **kwargs )
Parameters
-
save_directory (
stroros.PathLike) — Directory to which to save. Will be created if it doesn’t exist. -
save_config (
bool, optional, defaults toTrue) — Whether or not to save the config of the model. Useful when in distributed training like TPUs and need to call this function on all processes. In this case, setsave_config=Trueonly on the main process to avoid race conditions. -
state_dict (nested dictionary of
torch.Tensor) — The state dictionary of the model to save. Will default toself.state_dict(), but can be used to only save parts of the model or if special precautions need to be taken when recovering the state dictionary of a model (like when using model parallelism). -
save_function (
Callable) — The function to use to save the state dictionary. Useful on distributed training like TPUs when one need to replacetorch.saveby another method. -
push_to_hub (
bool, optional, defaults toFalse) — Whether or not to push your model to the Hugging Face model hub after saving it.
Save a model and its configuration file to a directory, so that it can be re-loaded using the
[from_pretrained()](/docs/transformers/v4.14.1/en/main_classes/model#transformers.PreTrainedModel.from_pretrained) class method.
( value: Module )
Set model’s input embeddings.
Tie the weights between the input embeddings and the output embeddings.
If the torchscript flag is set in the configuration, can’t handle parameter sharing so we are cloning
the weights instead.
Model Instantiation dtype
Under Pytorch a model normally gets instantiated with torch.float32 format. This can be an issue if one tries to
load a model whose weights are in fp16, since it’d require twice as much memory. To overcome this limitation, you can
either explicitly pass the desired dtype using torch_dtype argument:
model = T5ForConditionalGeneration.from_pretrained("t5", torch_dtype=torch.float16)or, if you want the model to always load in the most optimal memory pattern, you can use the special value "auto",
and then dtype will be automatically derived from the model’s weights:
model = T5ForConditionalGeneration.from_pretrained("t5", torch_dtype="auto")Models instantiated from scratch can also be told which dtype to use with:
config = T5Config.from_pretrained("t5")
model = AutoModel.from_config(config)Due to Pytorch design, this functionality is only available for floating dtypes.
ModuleUtilsMixin
A few utilities for torch.nn.Modules, to be used as a mixin.
Add a memory hook before and after each sub-module forward pass to record increase in memory consumption.
Increase in memory consumption is stored in a mem_rss_diff attribute for each module and can be reset to
zero with model.reset_memory_hooks_state().
(
input_dict: typing.Dict[str, typing.Union[torch.Tensor, typing.Any]]
)
→
int
Helper function to estimate the total number of tokens from the model inputs.
(
input_dict: typing.Dict[str, typing.Union[torch.Tensor, typing.Any]]
exclude_embeddings: bool = True
)
→
int
Parameters
-
batch_size (
int) — The batch size for the forward pass. -
sequence_length (
int) — The number of tokens in each line of the batch. -
exclude_embeddings (
bool, optional, defaults toTrue) — Whether or not to count embedding and softmax operations.
Returns
int
The number of floating-point operations.
Get number of (optionally, non-embeddings) floating-point operations for the forward and backward passes of a
batch with this transformer model. Default approximation neglects the quadratic dependency on the number of
tokens (valid if 12 * d_model << sequence_length) as laid out in this paper section 2.1. Should be overridden for transformers with parameter
re-use e.g. Albert or Universal Transformers, or if doing long-range modeling with very high sequence lengths.
( attention_mask: Tensor input_shape: typing.Tuple[int] device: <property object at 0x7f740b173400> )
Parameters
-
attention_mask (
torch.Tensor) — Mask with ones indicating tokens to attend to, zeros for tokens to ignore. -
input_shape (
Tuple[int]) — The shape of the input to the model. device — (torch.device): The device of the input to the model.
Returns
torch.Tensor The extended attention mask, with a the same dtype as attention_mask.dtype.
Makes broadcastable attention and causal masks so that future and masked tokens are ignored.
( head_mask: typing.Optional[torch.Tensor] num_hidden_layers: int is_attention_chunked: bool = False )
Parameters
-
head_mask (
torch.Tensorwith shape[num_heads]or[num_hidden_layers x num_heads], optional) — The mask indicating if we should keep the heads or not (1.0 for keep, 0.0 for discard). -
num_hidden_layers (
int) — The number of hidden layers in the model. isattention_chunked — (bool, _optional, defaults toFalse): Whether or not the attentions scores are computed by chunks or not.
Returns
torch.Tensor with shape [num_hidden_layers x batch x num_heads x seq_length x seq_length] or
list with [None] for each layer.
Prepare the head mask if needed.
(
encoder_attention_mask: Tensor
)
→
torch.Tensor
Invert an attention mask (e.g., switches 0. and 1.).
(
only_trainable: bool = False
exclude_embeddings: bool = False
)
→
int
Get number of (optionally, trainable or non-embeddings) parameters in the module.
Reset the mem_rss_diff attribute of each module (see
add_memory_hooks()).
TFPreTrainedModel
Base class for all TF models.
TFPreTrainedModel takes care of storing the configuration of the models and handles methods for loading, downloading and saving models as well as a few methods common to all models to:
- resize the input embeddings,
- prune heads in the self-attention heads.
Class attributes (overridden by derived classes):
- config_class (PretrainedConfig) — A subclass of PretrainedConfig to use as configuration class for this model architecture.
- base_model_prefix (
str) — A string indicating the attribute associated to the base model in derived classes of the same architecture adding modules on top of the base model.
(
repo_path_or_name: typing.Optional[str] = None
repo_url: typing.Optional[str] = None
use_temp_dir: bool = False
commit_message: typing.Optional[str] = None
organization: typing.Optional[str] = None
private: typing.Optional[bool] = None
use_auth_token: typing.Union[bool, str, NoneType] = None
**model_card_kwargs
)
→
str
Parameters
-
repo_path_or_name (
str, optional) — Can either be a repository name for your model in the Hub or a path to a local folder (in which case the repository will have the name of that local folder). If not specified, will default to the name given byrepo_urland a local directory with that name will be created. -
repo_url (
str, optional) — Specify this in case you want to push to an existing repository in the hub. If unspecified, a new repository will be created in your namespace (unless you specify anorganization) withrepo_name. -
use_temp_dir (
bool, optional, defaults toFalse) — Whether or not to clone the distant repo in a temporary directory or inrepo_path_or_nameinside the current working directory. This will slow things down if you are making changes in an existing repo since you will need to clone the repo before every push. -
commit_message (
str, optional) — Message to commit while pushing. Will default to"add model". -
organization (
str, optional) — Organization in which you want to push your model (you must be a member of this organization). -
private (
bool, optional) — Whether or not the repository created should be private (requires a paying subscription). -
use_auth_token (
boolorstr, optional) — The token to use as HTTP bearer authorization for remote files. IfTrue, will use the token generated when runningtransformers-cli login(stored in~/.huggingface). Will default toTrueifrepo_urlis not specified.
Returns
str
The url of the commit of your model in the given repository.
Upload the model checkpoint to the 🤗 Model Hub while synchronizing a local clone of the repo in
repo_path_or_name.
Examples:
from transformers import TFAutoModel
model = TFAutoModel.from_pretrained("bert-base-cased")
# Push the model to your namespace with the name "my-finetuned-bert" and have a local clone in the
# _my-finetuned-bert_ folder.
model.push_to_hub("my-finetuned-bert")
# Push the model to your namespace with the name "my-finetuned-bert" with no local clone.
model.push_to_hub("my-finetuned-bert", use_temp_dir=True)
# Push the model to an organization with the name "my-finetuned-bert" and have a local clone in the
# _my-finetuned-bert_ folder.
model.push_to_hub("my-finetuned-bert", organization="huggingface")
# Make a change to an existing repo that has been cloned locally in _my-finetuned-bert_.
model.push_to_hub("my-finetuned-bert", repo_url="https://huggingface.co/sgugger/my-finetuned-bert")( optimizer = 'rmsprop' loss = 'passthrough' metrics = None loss_weights = None weighted_metrics = None run_eagerly = None steps_per_execution = None **kwargs )
This is a thin wrapper that sets the model’s loss output head as the loss if the user does not specify a loss function themselves.
( pretrained_model_name_or_path *model_args **kwargs )
Parameters
-
pretrained_model_name_or_path (
str, optional) — Can be either:- A string, the model id of a pretrained model hosted inside a model repo on huggingface.co.
Valid model ids can be located at the root-level, like
bert-base-uncased, or namespaced under a user or organization name, likedbmdz/bert-base-german-cased. - A path to a directory containing model weights saved using
save_pretrained(), e.g.,
./my_model_directory/. - A path or url to a PyTorch state_dict save file (e.g,
./pt_model/pytorch_model.bin). In this case,from_ptshould be set toTrueand a configuration object should be provided asconfigargument. This loading path is slower than converting the PyTorch model in a TensorFlow model using the provided conversion scripts and loading the TensorFlow model afterwards. Noneif you are both providing the configuration and state dictionary (resp. with keyword argumentsconfigandstate_dict).
- A string, the model id of a pretrained model hosted inside a model repo on huggingface.co.
Valid model ids can be located at the root-level, like
-
model_args (sequence of positional arguments, optional) —
All remaining positional arguments will be passed to the underlying model’s
__init__method. -
config (
Union[PretrainedConfig, str], optional) — Can be either:- an instance of a class derived from PretrainedConfig,
- a string valid as input to from_pretrained().
Instantiate a pretrained TF 2.0 model from a pre-trained model configuration.
The warning Weights from XXX not initialized from pretrained model means that the weights of XXX do not come pretrained with the rest of the model. It is up to you to train those weights with a downstream fine-tuning task.
The warning Weights from XXX not used in YYY means that the layer XXX is not used by YYY, therefore those weights are discarded.
Passing use_auth_token=True is required when you want to use a private model.
Examples:
>>> from transformers import BertConfig, TFBertModel
>>> # Download model and configuration from huggingface.co and cache.
>>> model = TFBertModel.from_pretrained('bert-base-uncased')
>>> # Model was saved using _save_pretrained('./test/saved_model/')_ (for example purposes, not runnable).
>>> model = TFBertModel.from_pretrained('./test/saved_model/')
>>> # Update configuration during loading.
>>> model = TFBertModel.from_pretrained('bert-base-uncased', output_attentions=True)
>>> assert model.config.output_attentions == True
>>> # Loading from a Pytorch model file instead of a TensorFlow checkpoint (slower, for example purposes, not runnable).
>>> config = BertConfig.from_json_file('./pt_model/my_pt_model_config.json')
>>> model = TFBertModel.from_pretrained('./pt_model/my_pytorch_model.bin', from_pt=True, config=config)(
)
→
tf.Variable
Returns
tf.Variable
The weights representing the bias, None if not an LM model.
Dict of bias attached to an LM head. The key represents the name of the bias attribute.
(
)
→
tf.Variable
Returns
tf.Variable
The embeddings layer mapping vocabulary to hidden states.
Returns the model’s input embeddings layer.
(
)
→
tf.keras.layers.Layer
Returns
tf.keras.layers.Layer
The LM head layer if the model has one, None if not.
The LM Head layer. This method must be overwritten by all the models that have a lm head.
(
)
→
tf.Variable
Returns
tf.Variable
The new weights mapping vocabulary to hidden states.
Returns the model’s output embeddings
(
)
→
tf.keras.layers.Layer
Returns
tf.keras.layers.Layer
The layer that handles the bias, None if not an LM model.
Get the layer that handles a bias attribute in case the model has an LM head with weights tied to the embeddings
Get the concatenated _prefix name of the bias from the model name to the parent layer
(
repo_path_or_name
)
→
dict
Loads a saved checkpoint (model weights and optimizer state) from a repo. Returns the current epoch count when the checkpoint was made.
( heads_to_prune )
Prunes heads of the base model.
(
new_num_tokens = None
)
→
tf.Variable
Parameters
-
new_num_tokens (
int, optional) — The number of new tokens in the embedding matrix. Increasing the size will add newly initialized vectors at the end. Reducing the size will remove vectors from the end. If not provided orNone, just returns a pointer to the input tokenstf.Variablemodule of the model without doing anything.
Returns
tf.Variable
Pointer to the input tokens Embeddings Module of the model.
Resizes input token embeddings matrix of the model if new_num_tokens != config.vocab_size.
Takes care of tying weights embeddings afterwards if the model class has a tie_weights() method.
( save_directory saved_model = False version = 1 push_to_hub = False **kwargs )
Parameters
-
save_directory (
str) — Directory to which to save. Will be created if it doesn’t exist. -
saved_model (
bool, optional, defaults toFalse) — If the model has to be saved in saved model format as well or not. -
version (
int, optional, defaults to 1) — The version of the saved model. A saved model needs to be versioned in order to be properly loaded by TensorFlow Serving as detailed in the official documentation https://www.tensorflow.org/tfx/serving/serving_basic -
push_to_hub (
bool, optional, defaults toFalse) — Whether or not to push your model to the Hugging Face model hub after saving it.
Save a model and its configuration file to a directory, so that it can be re-loaded using the from_pretrained() class method.
( inputs )
Method used for serving the model.
( output )
Prepare the output of the saved model. Each model must implement this function.
( value )
Set all the bias in the LM head.
( value )
Set model’s input embeddings
( value )
Set model’s output embeddings
A modification of Keras’s default test_step that cleans up the printed metrics when we use a dummy loss.
A modification of Keras’s default train_step that cleans up the printed metrics when we use a dummy loss.
TFModelUtilsMixin
A few utilities for tf.keras.Model, to be used as a mixin.
(
only_trainable: bool = False
)
→
int
Get the number of (optionally, trainable) parameters in the model.
FlaxPreTrainedModel
( config: PretrainedConfig module: Module input_shape: typing.Tuple = (1, 1) seed: int = 0 dtype: dtype = <class 'jax._src.numpy.lax_numpy.float32'> )
Base class for all models.
FlaxPreTrainedModel takes care of storing the configuration of the models and handles methods for loading, downloading and saving models.
Class attributes (overridden by derived classes):
- config_class (PretrainedConfig) — A subclass of PretrainedConfig to use as configuration class for this model architecture.
- base_model_prefix (
str) — A string indicating the attribute associated to the base model in derived classes of the same architecture adding modules on top of the base model.
(
repo_path_or_name: typing.Optional[str] = None
repo_url: typing.Optional[str] = None
use_temp_dir: bool = False
commit_message: typing.Optional[str] = None
organization: typing.Optional[str] = None
private: typing.Optional[bool] = None
use_auth_token: typing.Union[bool, str, NoneType] = None
**model_card_kwargs
)
→
str
Parameters
-
repo_path_or_name (
str, optional) — Can either be a repository name for your model in the Hub or a path to a local folder (in which case the repository will have the name of that local folder). If not specified, will default to the name given byrepo_urland a local directory with that name will be created. -
repo_url (
str, optional) — Specify this in case you want to push to an existing repository in the hub. If unspecified, a new repository will be created in your namespace (unless you specify anorganization) withrepo_name. -
use_temp_dir (
bool, optional, defaults toFalse) — Whether or not to clone the distant repo in a temporary directory or inrepo_path_or_nameinside the current working directory. This will slow things down if you are making changes in an existing repo since you will need to clone the repo before every push. -
commit_message (
str, optional) — Message to commit while pushing. Will default to"add model". -
organization (
str, optional) — Organization in which you want to push your model (you must be a member of this organization). -
private (
bool, optional) — Whether or not the repository created should be private (requires a paying subscription). -
use_auth_token (
boolorstr, optional) — The token to use as HTTP bearer authorization for remote files. IfTrue, will use the token generated when runningtransformers-cli login(stored in~/.huggingface). Will default toTrueifrepo_urlis not specified.
Returns
str
The url of the commit of your model in the given repository.
Upload the model checkpoint to the 🤗 Model Hub while synchronizing a local clone of the repo in
repo_path_or_name.
Examples:
from transformers import FlaxAutoModel
model = FlaxAutoModel.from_pretrained("bert-base-cased")
# Push the model to your namespace with the name "my-finetuned-bert" and have a local clone in the
# _my-finetuned-bert_ folder.
model.push_to_hub("my-finetuned-bert")
# Push the model to your namespace with the name "my-finetuned-bert" with no local clone.
model.push_to_hub("my-finetuned-bert", use_temp_dir=True)
# Push the model to an organization with the name "my-finetuned-bert" and have a local clone in the
# _my-finetuned-bert_ folder.
model.push_to_hub("my-finetuned-bert", organization="huggingface")
# Make a change to an existing repo that has been cloned locally in _my-finetuned-bert_.
model.push_to_hub("my-finetuned-bert", repo_url="https://huggingface.co/sgugger/my-finetuned-bert")( pretrained_model_name_or_path: typing.Union[str, os.PathLike] dtype: dtype = <class 'jax._src.numpy.lax_numpy.float32'> *model_args **kwargs )
Parameters
-
pretrained_model_name_or_path (
stroros.PathLike) — Can be either:- A string, the model id of a pretrained model hosted inside a model repo on huggingface.co.
Valid model ids can be located at the root-level, like
bert-base-uncased, or namespaced under a user or organization name, likedbmdz/bert-base-german-cased. - A path to a directory containing model weights saved using
save_pretrained(), e.g.,
./my_model_directory/. - A path or url to a pt index checkpoint file (e.g,
./tf_model/model.ckpt.index). In this case,from_ptshould be set toTrue.
- A string, the model id of a pretrained model hosted inside a model repo on huggingface.co.
Valid model ids can be located at the root-level, like
-
dtype (
jax.numpy.dtype, optional, defaults tojax.numpy.float32) — The data type of the computation. Can be one ofjax.numpy.float32,jax.numpy.float16(on GPUs) andjax.numpy.bfloat16(on TPUs).This can be used to enable mixed-precision training or half-precision inference on GPUs or TPUs. If specified all the computation will be performed with the given
dtype.Note that this only specifies the dtype of the computation and does not influence the dtype of model parameters.
If you wish to change the dtype of the model parameters, see to_fp16() and to_bf16().
-
model_args (sequence of positional arguments, optional) —
All remaining positional arguments will be passed to the underlying model’s
__init__method. -
config (
Union[PretrainedConfig, str, os.PathLike], optional) — Can be either:- an instance of a class derived from PretrainedConfig,
- a string or path valid as input to from_pretrained().
Instantiate a pretrained flax model from a pre-trained model configuration.
The warning Weights from XXX not initialized from pretrained model means that the weights of XXX do not come pretrained with the rest of the model. It is up to you to train those weights with a downstream fine-tuning task.
The warning Weights from XXX not used in YYY means that the layer XXX is not used by YYY, therefore those weights are discarded.
Examples:
>>> from transformers import BertConfig, FlaxBertModel
>>> # Download model and configuration from huggingface.co and cache.
>>> model = FlaxBertModel.from_pretrained('bert-base-cased')
>>> # Model was saved using _save_pretrained('./test/saved_model/')_ (for example purposes, not runnable).
>>> model = FlaxBertModel.from_pretrained('./test/saved_model/')
>>> # Loading from a PyTorch checkpoint file instead of a PyTorch model (slower, for example purposes, not runnable).
>>> config = BertConfig.from_json_file('./pt_model/config.json')
>>> model = FlaxBertModel.from_pretrained('./pt_model/pytorch_model.bin', from_pt=True, config=config)( save_directory: typing.Union[str, os.PathLike] params = None push_to_hub = False **kwargs )
Save a model and its configuration file to a directory, so that it can be re-loaded using the
[from_pretrained()](/docs/transformers/v4.14.1/en/main_classes/model#transformers.FlaxPreTrainedModel.from_pretrained) class method
( params: typing.Union[typing.Dict, flax.core.frozen_dict.FrozenDict] mask: typing.Any = None )
Cast the floating-point params to jax.numpy.bfloat16. This returns a new params tree and does not
cast the params in place.
This method can be used on TPU to explicitly convert the model parameters to bfloat16 precision to do full half-precision training or to save weights in bfloat16 for inference in order to save memory and improve speed.
Examples:
>>> from transformers import FlaxBertModel
>>> # load model
>>> model = FlaxBertModel.from_pretrained('bert-base-cased')
>>> # By default, the model parameters will be in fp32 precision, to cast these to bfloat16 precision
>>> model.params = model.to_bf16(model.params)
>>> # If you want don't want to cast certain parameters (for example layer norm bias and scale)
>>> # then pass the mask as follows
>>> from flax import traverse_util
>>> model = FlaxBertModel.from_pretrained('bert-base-cased')
>>> flat_params = traverse_util.flatten_dict(model.params)
>>> mask = {path: (path[-2] != ("LayerNorm", "bias") and path[-2:] != ("LayerNorm", "scale")) for path in flat_params}
>>> mask = traverse_util.unflatten_dict(mask)
>>> model.params = model.to_bf16(model.params, mask)( params: typing.Union[typing.Dict, flax.core.frozen_dict.FrozenDict] mask: typing.Any = None )
Cast the floating-point parmas to jax.numpy.float16. This returns a new params tree and does not
cast the params in place.
This method can be used on GPU to explicitly convert the model parameters to float16 precision to do full half-precision training or to save weights in float16 for inference in order to save memory and improve speed.
Examples:
>>> from transformers import FlaxBertModel
>>> # load model
>>> model = FlaxBertModel.from_pretrained('bert-base-cased')
>>> # By default, the model params will be in fp32, to cast these to float16
>>> model.params = model.to_fp16(model.params)
>>> # If you want don't want to cast certain parameters (for example layer norm bias and scale)
>>> # then pass the mask as follows
>>> from flax import traverse_util
>>> model = FlaxBertModel.from_pretrained('bert-base-cased')
>>> flat_params = traverse_util.flatten_dict(model.params)
>>> mask = {path: (path[-2] != ("LayerNorm", "bias") and path[-2:] != ("LayerNorm", "scale")) for path in flat_params}
>>> mask = traverse_util.unflatten_dict(mask)
>>> model.params = model.to_fp16(model.params, mask)( params: typing.Union[typing.Dict, flax.core.frozen_dict.FrozenDict] mask: typing.Any = None )
Cast the floating-point parmas to jax.numpy.float32. This method can be used to explicitly convert the
model parameters to fp32 precision. This returns a new params tree and does not cast the params in
place.
Examples:
>>> from transformers import FlaxBertModel
>>> # Download model and configuration from huggingface.co
>>> model = FlaxBertModel.from_pretrained('bert-base-cased')
>>> # By default, the model params will be in fp32, to illustrate the use of this method,
>>> # we'll first cast to fp16 and back to fp32
>>> model.params = model.to_f16(model.params)
>>> # now cast back to fp32
>>> model.params = model.to_fp32(model.params)Generation
A class containing all of the functions supporting generation, to be used as a mixin in PreTrainedModel.
Implement in subclasses of PreTrainedModel for custom behavior to adjust the logits in the generate method.
( input_ids: LongTensor beam_scorer: BeamScorer logits_processor: typing.Optional[transformers.generation_logits_process.LogitsProcessorList] = None stopping_criteria: typing.Optional[transformers.generation_stopping_criteria.StoppingCriteriaList] = None logits_warper: typing.Optional[transformers.generation_logits_process.LogitsProcessorList] = None max_length: typing.Optional[int] = None pad_token_id: typing.Optional[int] = None eos_token_id: typing.Optional[int] = None output_attentions: typing.Optional[bool] = None output_hidden_states: typing.Optional[bool] = None output_scores: typing.Optional[bool] = None return_dict_in_generate: typing.Optional[bool] = None synced_gpus: typing.Optional[bool] = None **model_kwargs )
Parameters
-
input_ids (
torch.LongTensorof shape(batch_size, sequence_length)) — The sequence used as a prompt for the generation. -
beam_scorer (
BeamScorer) — A derived instance of BeamScorer that defines how beam hypotheses are constructed, stored and sorted during generation. For more information, the documentation of BeamScorer should be read. -
logits_processor (
LogitsProcessorList, optional) — An instance of LogitsProcessorList. List of instances of class derived from LogitsProcessor used to modify the prediction scores of the language modeling head applied at each generation step. -
stopping_criteria (
StoppingCriteriaList, optional) — An instance of StoppingCriteriaList. List of instances of class derived from StoppingCriteria used to tell if the generation loop should stop. -
logits_warper (
LogitsProcessorList, optional) — An instance of LogitsProcessorList. List of instances of class derived from LogitsWarper used to warp the prediction score distribution of the language modeling head applied before multinomial sampling at each generation step. -
max_length (
int, optional, defaults to 20) — DEPRECATED. Uselogits_processororstopping_criteriadirectly to cap the number of generated tokens. The maximum length of the sequence to be generated. -
pad_token_id (
int, optional) — The id of the padding token. -
eos_token_id (
int, optional) — The id of the end-of-sequence token. -
output_attentions (
bool, optional, defaults to False) — Whether or not to return the attentions tensors of all attention layers. Seeattentionsunder returned tensors for more details. -
output_hidden_states (
bool, optional, defaults to False) — Whether or not to return the hidden states of all layers. Seehidden_statesunder returned tensors for more details. -
output_scores (
bool, optional, defaults to False) — Whether or not to return the prediction scores. Seescoresunder returned tensors for more details. -
return_dict_in_generate (
bool, optional, defaults to False) — Whether or not to return a ModelOutput instead of a plain tuple. -
synced_gpus (
bool, optional, defaults toFalse) — Whether to continue running the while loop until max_length (needed for ZeRO stage 3) model_kwargs — Additional model specific kwargs will be forwarded to theforwardfunction of the model. If model is an encoder-decoder model the kwargs should includeencoder_outputs.
Returns
BeamSampleDecoderOnlyOutput,
BeamSampleEncoderDecoderOutput or obj:torch.LongTensor: A
torch.LongTensor containing the generated tokens (default behaviour) or a
BeamSampleDecoderOnlyOutput if
model.config.is_encoder_decoder=False and return_dict_in_generate=True or a
BeamSampleEncoderDecoderOutput if
model.config.is_encoder_decoder=True.
Generates sequences for models with a language modeling head using beam search with multinomial sampling.
Examples:
>>> from transformers import (
... AutoTokenizer,
... AutoModelForSeq2SeqLM,
... LogitsProcessorList,
... MinLengthLogitsProcessor,
... TopKLogitsWarper,
... TemperatureLogitsWarper,
... BeamSearchScorer,
... )
>>> import torch
>>> tokenizer = AutoTokenizer.from_pretrained("t5-base")
>>> model = AutoModelForSeq2SeqLM.from_pretrained("t5-base")
>>> encoder_input_str = "translate English to German: How old are you?"
>>> encoder_input_ids = tokenizer(encoder_input_str, return_tensors="pt").input_ids
>>> # lets run beam search using 3 beams
>>> num_beams = 3
>>> # define decoder start token ids
>>> input_ids = torch.ones((num_beams, 1), device=model.device, dtype=torch.long)
>>> input_ids = input_ids * model.config.decoder_start_token_id
>>> # add encoder_outputs to model keyword arguments
>>> model_kwargs = {
... "encoder_outputs": model.get_encoder()(encoder_input_ids.repeat_interleave(num_beams, dim=0), return_dict=True)
... }
>>> # instantiate beam scorer
>>> beam_scorer = BeamSearchScorer(
... batch_size=1,
... max_length=model.config.max_length,
... num_beams=num_beams,
... device=model.device,
... )
>>> # instantiate logits processors
>>> logits_processor = LogitsProcessorList([
... MinLengthLogitsProcessor(5, eos_token_id=model.config.eos_token_id)
... ])
>>> # instantiate logits processors
>>> logits_warper = LogitsProcessorList([
... TopKLogitsWarper(50),
... TemperatureLogitsWarper(0.7),
... ])
>>> outputs = model.beam_sample(
... input_ids, beam_scorer, logits_processor=logits_processor, logits_warper=logits_warper, **model_kwargs
... )
>>> print("Generated:", tokenizer.batch_decode(outputs, skip_special_tokens=True))( input_ids: LongTensor beam_scorer: BeamScorer logits_processor: typing.Optional[transformers.generation_logits_process.LogitsProcessorList] = None stopping_criteria: typing.Optional[transformers.generation_stopping_criteria.StoppingCriteriaList] = None max_length: typing.Optional[int] = None pad_token_id: typing.Optional[int] = None eos_token_id: typing.Optional[int] = None output_attentions: typing.Optional[bool] = None output_hidden_states: typing.Optional[bool] = None output_scores: typing.Optional[bool] = None return_dict_in_generate: typing.Optional[bool] = None synced_gpus: typing.Optional[bool] = None **model_kwargs )
Parameters
-
input_ids (
torch.LongTensorof shape(batch_size, sequence_length)) — The sequence used as a prompt for the generation. -
beam_scorer (
BeamScorer) — An derived instance of BeamScorer that defines how beam hypotheses are constructed, stored and sorted during generation. For more information, the documentation of BeamScorer should be read. -
logits_processor (
LogitsProcessorList, optional) — An instance of LogitsProcessorList. List of instances of class derived from LogitsProcessor used to modify the prediction scores of the language modeling head applied at each generation step. -
stopping_criteria (
StoppingCriteriaList, optional) — An instance of StoppingCriteriaList. List of instances of class derived from StoppingCriteria used to tell if the generation loop should stop. -
max_length (
int, optional, defaults to 20) — DEPRECATED. Uselogits_processororstopping_criteriadirectly to cap the number of generated tokens. The maximum length of the sequence to be generated. -
pad_token_id (
int, optional) — The id of the padding token. -
eos_token_id (
int, optional) — The id of the end-of-sequence token. -
output_attentions (
bool, optional, defaults to False) — Whether or not to return the attentions tensors of all attention layers. Seeattentionsunder returned tensors for more details. -
output_hidden_states (
bool, optional, defaults to False) — Whether or not to return the hidden states of all layers. Seehidden_statesunder returned tensors for more details. -
output_scores (
bool, optional, defaults to False) — Whether or not to return the prediction scores. Seescoresunder returned tensors for more details. -
return_dict_in_generate (
bool, optional, defaults to False) — Whether or not to return a ModelOutput instead of a plain tuple. -
synced_gpus (
bool, optional, defaults toFalse) — Whether to continue running the while loop until max_length (needed for ZeRO stage 3) model_kwargs — Additional model specific kwargs will be forwarded to theforwardfunction of the model. If model is an encoder-decoder model the kwargs should includeencoder_outputs.
Returns
generation_utilsBeamSearchDecoderOnlyOutput,
BeamSearchEncoderDecoderOutput or obj:torch.LongTensor: A
torch.LongTensor containing the generated tokens (default behaviour) or a
BeamSearchDecoderOnlyOutput if
model.config.is_encoder_decoder=False and return_dict_in_generate=True or a
BeamSearchEncoderDecoderOutput if
model.config.is_encoder_decoder=True.
Generates sequences for models with a language modeling head using beam search decoding.
Examples:
>>> from transformers import (
... AutoTokenizer,
... AutoModelForSeq2SeqLM,
... LogitsProcessorList,
... MinLengthLogitsProcessor,
... BeamSearchScorer,
... )
>>> import torch
>>> tokenizer = AutoTokenizer.from_pretrained("t5-base")
>>> model = AutoModelForSeq2SeqLM.from_pretrained("t5-base")
>>> encoder_input_str = "translate English to German: How old are you?"
>>> encoder_input_ids = tokenizer(encoder_input_str, return_tensors="pt").input_ids
>>> # lets run beam search using 3 beams
>>> num_beams = 3
>>> # define decoder start token ids
>>> input_ids = torch.ones((num_beams, 1), device=model.device, dtype=torch.long)
>>> input_ids = input_ids * model.config.decoder_start_token_id
>>> # add encoder_outputs to model keyword arguments
>>> model_kwargs = {
... "encoder_outputs": model.get_encoder()(encoder_input_ids.repeat_interleave(num_beams, dim=0), return_dict=True)
... }
>>> # instantiate beam scorer
>>> beam_scorer = BeamSearchScorer(
... batch_size=1,
... num_beams=num_beams,
... device=model.device,
... )
>>> # instantiate logits processors
>>> logits_processor = LogitsProcessorList([
... MinLengthLogitsProcessor(5, eos_token_id=model.config.eos_token_id),
... ])
>>> outputs = model.beam_search(input_ids, beam_scorer, logits_processor=logits_processor, **model_kwargs)
>>> print("Generated:", tokenizer.batch_decode(outputs, skip_special_tokens=True))(
input_ids: typing.Optional[torch.LongTensor] = None
max_length: typing.Optional[int] = None
min_length: typing.Optional[int] = None
do_sample: typing.Optional[bool] = None
early_stopping: typing.Optional[bool] = None
num_beams: typing.Optional[int] = None
temperature: typing.Optional[float] = None
top_k: typing.Optional[int] = None
top_p: typing.Optional[float] = None
repetition_penalty: typing.Optional[float] = None
bad_words_ids: typing.Optional[typing.Iterable[int]] = None
bos_token_id: typing.Optional[int] = None
pad_token_id: typing.Optional[int] = None
eos_token_id: typing.Optional[int] = None
length_penalty: typing.Optional[float] = None
no_repeat_ngram_size: typing.Optional[int] = None
encoder_no_repeat_ngram_size: typing.Optional[int] = None
num_return_sequences: typing.Optional[int] = None
max_time: typing.Optional[float] = None
max_new_tokens: typing.Optional[int] = None
decoder_start_token_id: typing.Optional[int] = None
use_cache: typing.Optional[bool] = None
num_beam_groups: typing.Optional[int] = None
diversity_penalty: typing.Optional[float] = None
prefix_allowed_tokens_fn: typing.Union[typing.Callable[[int, torch.Tensor], typing.List[int]], NoneType] = None
output_attentions: typing.Optional[bool] = None
output_hidden_states: typing.Optional[bool] = None
output_scores: typing.Optional[bool] = None
return_dict_in_generate: typing.Optional[bool] = None
forced_bos_token_id: typing.Optional[int] = None
forced_eos_token_id: typing.Optional[int] = None
remove_invalid_values: typing.Optional[bool] = None
synced_gpus: typing.Optional[bool] = None
**model_kwargs
)
→
ModelOutput or torch.LongTensor
Parameters
-
input_ids (
torch.LongTensorof shape(batch_size, sequence_length), optional) — The sequence used as a prompt for the generation. IfNonethe method initializes it withbos_token_idand a batch size of 1. -
max_length (
int, optional, defaults tomodel.config.max_length) — The maximum length of the sequence to be generated. -
max_new_tokens (
int, optional, defaults to None) — The maximum numbers of tokens to generate, ignore the current number of tokens. Use eithermax_new_tokensormax_lengthbut not both, they serve the same purpose. -
min_length (
int, optional, defaults to 10) — The minimum length of the sequence to be generated. -
do_sample (
bool, optional, defaults toFalse) — Whether or not to use sampling ; use greedy decoding otherwise. -
early_stopping (
bool, optional, defaults toFalse) — Whether to stop the beam search when at leastnum_beamssentences are finished per batch or not. -
num_beams (
int, optional, defaults to 1) — Number of beams for beam search. 1 means no beam search. -
temperature (
float, optional, defaults to 1.0) — The value used to module the next token probabilities. -
top_k (
int, optional, defaults to 50) — The number of highest probability vocabulary tokens to keep for top-k-filtering. -
top_p (
float, optional, defaults to 1.0) — If set to float < 1, only the most probable tokens with probabilities that add up totop_por higher are kept for generation. -
repetition_penalty (
float, optional, defaults to 1.0) — The parameter for repetition penalty. 1.0 means no penalty. See this paper for more details. -
pad_token_id (
int, optional) — The id of the padding token. -
bos_token_id (
int, optional) — The id of the beginning-of-sequence token. -
eos_token_id (
int, optional) — The id of the end-of-sequence token. -
length_penalty (
float, optional, defaults to 1.0) — Exponential penalty to the length. 1.0 means no penalty. Set to values < 1.0 in order to encourage the model to generate shorter sequences, to a value > 1.0 in order to encourage the model to produce longer sequences. -
no_repeat_ngram_size (
int, optional, defaults to 0) — If set to int > 0, all ngrams of that size can only occur once. -
encoder_no_repeat_ngram_size (
int, optional, defaults to 0) — If set to int > 0, all ngrams of that size that occur in theencoder_input_idscannot occur in thedecoder_input_ids. -
bad_words_ids(
List[List[int]], optional) — List of token ids that are not allowed to be generated. In order to get the tokens of the words that should not appear in the generated text, usetokenizer(bad_word, add_prefix_space=True).input_ids. -
num_return_sequences(
int, optional, defaults to 1) — The number of independently computed returned sequences for each element in the batch. -
max_time(
float, optional, defaults to None) — The maximum amount of time you allow the computation to run for in seconds. generation will still finish the current pass after allocated time has been passed. -
attention_mask (
torch.LongTensorof shape(batch_size, sequence_length), optional) — Mask to avoid performing attention on padding token indices. Mask values are in[0, 1], 1 for tokens that are not masked, and 0 for masked tokens. If not provided, will default to a tensor the same shape asinput_idsthat masks the pad token. What are attention masks? -
decoder_start_token_id (
int, optional) — If an encoder-decoder model starts decoding with a different token than bos, the id of that token. usecache — (bool, _optional, defaults toTrue): Whether or not the model should use the past last key/values attentions (if applicable to the model) to speed up decoding. -
num_beam_groups (
int, optional, defaults to 1) — Number of groups to dividenum_beamsinto in order to ensure diversity among different groups of beams. this paper for more details. -
diversity_penalty (
float, optional, defaults to 0.0) — This value is subtracted from a beam’s score if it generates a token same as any beam from other group at a particular time. Note thatdiversity_penaltyis only effective ifgroup beam searchis enabled. prefixallowed_tokens_fn — (Callable[[int, torch.Tensor], List[int]], _optional): If provided, this function constraints the beam search to allowed tokens only at each step. If not provided no constraint is applied. This function takes 2 arguments: the batch IDbatch_idandinput_ids. It has to return a list with the allowed tokens for the next generation step conditioned on the batch IDbatch_idand the previously generated tokensinputs_ids. This argument is useful for constrained generation conditioned on the prefix, as described in Autoregressive Entity Retrieval. -
output_attentions (
bool, optional, defaults to False) — Whether or not to return the attentions tensors of all attention layers. Seeattentionsunder returned tensors for more details. -
output_hidden_states (
bool, optional, defaults to False) — Whether or not to return the hidden states of all layers. Seehidden_statesunder returned tensors for more details. -
output_scores (
bool, optional, defaults to False) — Whether or not to return the prediction scores. Seescoresunder returned tensors for more details. -
return_dict_in_generate (
bool, optional, defaults to False) — Whether or not to return a ModelOutput instead of a plain tuple. -
forced_bos_token_id (
int, optional) — The id of the token to force as the first generated token after thedecoder_start_token_id. Useful for multilingual models like mBART where the first generated token needs to be the target language token. -
forced_eos_token_id (
int, optional) — The id of the token to force as the last generated token whenmax_lengthis reached. -
remove_invalid_values (
bool, optional) — Whether to remove possible nan and inf outputs of the model to prevent the generation method to crash. Note that usingremove_invalid_valuescan slow down generation. -
synced_gpus (
bool, optional, defaults toFalse) — Whether to continue running the while loop until max_length (needed for ZeRO stage 3)model_kwargs — Additional model specific kwargs will be forwarded to the
forwardfunction of the model. If the model is an encoder-decoder model, encoder specific kwargs should not be prefixed and decoder specific kwargs should be prefixed with _decoder__.
Returns
ModelOutput or torch.LongTensor
A
ModelOutput (if return_dict_in_generate=True or when
config.return_dict_in_generate=True) or a torch.FloatTensor.
If the model is not an encoder-decoder model (model.config.is_encoder_decoder=False), the
possible ModelOutput types are:
- GreedySearchDecoderOnlyOutput,
- SampleDecoderOnlyOutput,
- BeamSearchDecoderOnlyOutput,
- BeamSampleDecoderOnlyOutput
If the model is an encoder-decoder model (model.config.is_encoder_decoder=True), the possible
ModelOutput types are:
Generates sequences for models with a language modeling head. The method currently supports greedy decoding, multinomial sampling, beam-search decoding, and beam-search multinomial sampling.
Apart from input_ids and attention_mask, all the arguments below will default to the value of the
attribute of the same name inside the PretrainedConfig of the model. The default values
indicated are the default values of those config.
Most of these parameters are explained in more detail in this blog post.
Examples:
>>> from transformers import AutoTokenizer, AutoModelForCausalLM, AutoModelForSeq2SeqLM
>>> tokenizer = AutoTokenizer.from_pretrained("distilgpt2")
>>> model = AutoModelForCausalLM.from_pretrained("distilgpt2")
>>> # do greedy decoding without providing a prompt
>>> outputs = model.generate(max_length=40)
>>> print("Generated:", tokenizer.decode(outputs[0], skip_special_tokens=True))
>>> tokenizer = AutoTokenizer.from_pretrained("t5-base")
>>> model = AutoModelForSeq2SeqLM.from_pretrained("t5-base")
>>> document = (
... "at least two people were killed in a suspected bomb attack on a passenger bus "
... "in the strife-torn southern philippines on monday , the military said."
... )
>>> # encode input context
>>> input_ids = tokenizer(document, return_tensors="pt").input_ids
>>> # generate 3 independent sequences using beam search decoding (5 beams)
>>> # with T5 encoder-decoder model conditioned on short news article.
>>> outputs = model.generate(input_ids=input_ids, num_beams=5, num_return_sequences=3)
>>> print("Generated:", tokenizer.batch_decode(outputs, skip_special_tokens=True))
>>> tokenizer = AutoTokenizer.from_pretrained("distilgpt2")
>>> model = AutoModelForCausalLM.from_pretrained("distilgpt2")
>>> input_context = "The dog"
>>> # encode input context
>>> input_ids = tokenizer(input_context, return_tensors="pt").input_ids
>>> # generate 3 candidates using sampling
>>> outputs = model.generate(input_ids=input_ids, max_length=20, num_return_sequences=3, do_sample=True)
>>> print("Generated:", tokenizer.batch_decode(outputs, skip_special_tokens=True))
>>> tokenizer = AutoTokenizer.from_pretrained("ctrl")
>>> model = AutoModelForCausalLM.from_pretrained("ctrl")
>>> # "Legal" is one of the control codes for ctrl
>>> input_context = "Legal My neighbor is"
>>> # encode input context
>>> input_ids = tokenizer(input_context, return_tensors="pt").input_ids
>>> outputs = model.generate(input_ids=input_ids, max_length=20, repetition_penalty=1.2)
>>> print("Generated:", tokenizer.decode(outputs[0], skip_special_tokens=True))
>>> tokenizer = AutoTokenizer.from_pretrained("gpt2", use_fast=False)
>>> model = AutoModelForCausalLM.from_pretrained("gpt2")
>>> input_context = "My cute dog"
>>> # get tokens of words that should not be generated
>>> bad_words_ids = tokenizer(["idiot", "stupid", "shut up"], add_prefix_space=True).input_ids
>>> # encode input context
>>> input_ids = tokenizer(input_context, return_tensors="pt").input_ids
>>> # generate sequences without allowing bad_words to be generated
>>> outputs = model.generate(input_ids=input_ids, max_length=20, do_sample=True, bad_words_ids=bad_words_ids)
>>> print("Generated:", tokenizer.decode(outputs[0], skip_special_tokens=True))( input_ids: LongTensor logits_processor: typing.Optional[transformers.generation_logits_process.LogitsProcessorList] = None stopping_criteria: typing.Optional[transformers.generation_stopping_criteria.StoppingCriteriaList] = None max_length: typing.Optional[int] = None pad_token_id: typing.Optional[int] = None eos_token_id: typing.Optional[int] = None output_attentions: typing.Optional[bool] = None output_hidden_states: typing.Optional[bool] = None output_scores: typing.Optional[bool] = None return_dict_in_generate: typing.Optional[bool] = None synced_gpus: typing.Optional[bool] = None **model_kwargs )
Parameters
-
input_ids (
torch.LongTensorof shape(batch_size, sequence_length)) — The sequence used as a prompt for the generation. -
logits_processor (
LogitsProcessorList, optional) — An instance of LogitsProcessorList. List of instances of class derived from LogitsProcessor used to modify the prediction scores of the language modeling head applied at each generation step. -
stopping_criteria (
StoppingCriteriaList, optional) — An instance of StoppingCriteriaList. List of instances of class derived from StoppingCriteria used to tell if the generation loop should stop. -
max_length (
int, optional, defaults to 20) — DEPRECATED. Uselogits_processororstopping_criteriadirectly to cap the number of generated tokens. The maximum length of the sequence to be generated. -
pad_token_id (
int, optional) — The id of the padding token. -
eos_token_id (
int, optional) — The id of the end-of-sequence token. -
output_attentions (
bool, optional, defaults to False) — Whether or not to return the attentions tensors of all attention layers. Seeattentionsunder returned tensors for more details. -
output_hidden_states (
bool, optional, defaults to False) — Whether or not to return the hidden states of all layers. Seehidden_statesunder returned tensors for more details. -
output_scores (
bool, optional, defaults to False) — Whether or not to return the prediction scores. Seescoresunder returned tensors for more details. -
return_dict_in_generate (
bool, optional, defaults to False) — Whether or not to return a ModelOutput instead of a plain tuple. -
synced_gpus (
bool, optional, defaults toFalse) — Whether to continue running the while loop until max_length (needed for ZeRO stage 3) model_kwargs — Additional model specific keyword arguments will be forwarded to theforwardfunction of the model. If model is an encoder-decoder model the kwargs should includeencoder_outputs.
Returns
GreedySearchDecoderOnlyOutput,
GreedySearchEncoderDecoderOutput or obj:torch.LongTensor: A
torch.LongTensor containing the generated tokens (default behaviour) or a
GreedySearchDecoderOnlyOutput if
model.config.is_encoder_decoder=False and return_dict_in_generate=True or a
GreedySearchEncoderDecoderOutput if
model.config.is_encoder_decoder=True.
Generates sequences for models with a language modeling head using greedy decoding.
Examples:
>>> from transformers import (
... AutoTokenizer,
... AutoModelForCausalLM,
... LogitsProcessorList,
... MinLengthLogitsProcessor,
... )
>>> tokenizer = AutoTokenizer.from_pretrained("gpt2")
>>> model = AutoModelForCausalLM.from_pretrained("gpt2")
>>> # set pad_token_id to eos_token_id because GPT2 does not have a EOS token
>>> model.config.pad_token_id = model.config.eos_token_id
>>> input_prompt = "Today is a beautiful day, and"
>>> input_ids = tokenizer(input_prompt, return_tensors="pt").input_ids
>>> # instantiate logits processors
>>> logits_processor = LogitsProcessorList([
... MinLengthLogitsProcessor(15, eos_token_id=model.config.eos_token_id),
... ])
>>> outputs = model.greedy_search(input_ids, logits_processor=logits_processor)
>>> print("Generated:", tokenizer.batch_decode(outputs, skip_special_tokens=True))( input_ids: LongTensor beam_scorer: BeamScorer logits_processor: typing.Optional[transformers.generation_logits_process.LogitsProcessorList] = None stopping_criteria: typing.Optional[transformers.generation_stopping_criteria.StoppingCriteriaList] = None max_length: typing.Optional[int] = None pad_token_id: typing.Optional[int] = None eos_token_id: typing.Optional[int] = None output_attentions: typing.Optional[bool] = None output_hidden_states: typing.Optional[bool] = None output_scores: typing.Optional[bool] = None return_dict_in_generate: typing.Optional[bool] = None synced_gpus: typing.Optional[bool] = None **model_kwargs )
Parameters
-
input_ids (
torch.LongTensorof shape(batch_size, sequence_length)) — The sequence used as a prompt for the generation. -
beam_scorer (
BeamScorer) — An derived instance of BeamScorer that defines how beam hypotheses are constructed, stored and sorted during generation. For more information, the documentation of BeamScorer should be read. -
logits_processor (
LogitsProcessorList, optional) — An instance of LogitsProcessorList. List of instances of class derived from LogitsProcessor used to modify the prediction scores of the language modeling head applied at each generation step. -
stopping_criteria (
StoppingCriteriaList, optional) — An instance of StoppingCriteriaList. List of instances of class derived from StoppingCriteria used to tell if the generation loop should stop. -
max_length (
int, optional, defaults to 20) — DEPRECATED. Uselogits_processororstopping_criteriadirectly to cap the number of generated tokens. The maximum length of the sequence to be generated. -
pad_token_id (
int, optional) — The id of the padding token. -
eos_token_id (
int, optional) — The id of the end-of-sequence token. -
output_attentions (
bool, optional, defaults to False) — Whether or not to return the attentions tensors of all attention layers. Seeattentionsunder returned tensors for more details. -
output_hidden_states (
bool, optional, defaults to False) — Whether or not to return the hidden states of all layers. Seehidden_statesunder returned tensors for more details. -
output_scores (
bool, optional, defaults to False) — Whether or not to return the prediction scores. Seescoresunder returned tensors for more details. -
return_dict_in_generate (
bool, optional, defaults to False) — Whether or not to return a ModelOutput instead of a plain tuple. -
synced_gpus (
bool, optional, defaults toFalse) — Whether to continue running the while loop until max_length (needed for ZeRO stage 3)model_kwargs — Additional model specific kwargs that will be forwarded to the
forwardfunction of the model. If model is an encoder-decoder model the kwargs should includeencoder_outputs.
Returns
BeamSearchDecoderOnlyOutput,
BeamSearchEncoderDecoderOutput or obj:torch.LongTensor: A
torch.LongTensor containing the generated tokens (default behaviour) or a
BeamSearchDecoderOnlyOutput if
BeamSearchDecoderOnlyOutput if
model.config.is_encoder_decoder=False and return_dict_in_generate=True or a
BeamSearchEncoderDecoderOutput if
model.config.is_encoder_decoder=True.
Generates sequences for models with a language modeling head using beam search decoding.
Examples:
>>> from transformers import (
... AutoTokenizer,
... AutoModelForSeq2SeqLM,
... LogitsProcessorList,
... MinLengthLogitsProcessor,
... HammingDiversityLogitsProcessor,
... BeamSearchScorer,
... )
>>> import torch
>>> tokenizer = AutoTokenizer.from_pretrained("t5-base")
>>> model = AutoModelForSeq2SeqLM.from_pretrained("t5-base")
>>> encoder_input_str = "translate English to German: How old are you?"
>>> encoder_input_ids = tokenizer(encoder_input_str, return_tensors="pt").input_ids
>>> # lets run diverse beam search using 6 beams
>>> num_beams = 6
>>> # define decoder start token ids
>>> input_ids = torch.ones((num_beams, 1), device=model.device, dtype=torch.long)
>>> input_ids = input_ids * model.config.decoder_start_token_id
>>> # add encoder_outputs to model keyword arguments
>>> model_kwargs = {
... "encoder_outputs": model.get_encoder()(encoder_input_ids.repeat_interleave(num_beams, dim=0), return_dict=True)
... }
>>> # instantiate beam scorer
>>> beam_scorer = BeamSearchScorer(
... batch_size=1,
... max_length=model.config.max_length,
... num_beams=num_beams,
... device=model.device,
... num_beam_groups=3
... )
>>> # instantiate logits processors
>>> logits_processor = LogitsProcessorList([
... HammingDiversityLogitsProcessor(5.5, num_beams=6, num_beam_groups=3),
... MinLengthLogitsProcessor(5, eos_token_id=model.config.eos_token_id),
... ])
>>> outputs = model.group_beam_search(input_ids, beam_scorer, logits_processor=logits_processor, **model_kwargs)
>>> print("Generated:", tokenizer.batch_decode(outputs, skip_special_tokens=True))Implement in subclasses of PreTrainedModel for custom behavior to prepare inputs in the generate method.
( input_ids: LongTensor logits_processor: typing.Optional[transformers.generation_logits_process.LogitsProcessorList] = None stopping_criteria: typing.Optional[transformers.generation_stopping_criteria.StoppingCriteriaList] = None logits_warper: typing.Optional[transformers.generation_logits_process.LogitsProcessorList] = None max_length: typing.Optional[int] = None pad_token_id: typing.Optional[int] = None eos_token_id: typing.Optional[int] = None output_attentions: typing.Optional[bool] = None output_hidden_states: typing.Optional[bool] = None output_scores: typing.Optional[bool] = None return_dict_in_generate: typing.Optional[bool] = None synced_gpus: typing.Optional[bool] = None **model_kwargs )
Parameters
-
input_ids (
torch.LongTensorof shape(batch_size, sequence_length)) — The sequence used as a prompt for the generation. -
logits_processor (
LogitsProcessorList, optional) — An instance of LogitsProcessorList. List of instances of class derived from LogitsProcessor used to modify the prediction scores of the language modeling head applied at each generation step. -
stopping_criteria (
StoppingCriteriaList, optional) — An instance of StoppingCriteriaList. List of instances of class derived from StoppingCriteria used to tell if the generation loop should stop. -
logits_warper (
LogitsProcessorList, optional) — An instance of LogitsProcessorList. List of instances of class derived from LogitsWarper used to warp the prediction score distribution of the language modeling head applied before multinomial sampling at each generation step. -
max_length (
int, optional, defaults to 20) — DEPRECATED. Uselogits_processororstopping_criteriadirectly to cap the number of generated tokens. The maximum length of the sequence to be generated. -
pad_token_id (
int, optional) — The id of the padding token. -
eos_token_id (
int, optional) — The id of the end-of-sequence token. -
output_attentions (
bool, optional, defaults to False) — Whether or not to return the attentions tensors of all attention layers. Seeattentionsunder returned tensors for more details. -
output_hidden_states (
bool, optional, defaults to False) — Whether or not to return the hidden states of all layers. Seehidden_statesunder returned tensors for more details. -
output_scores (
bool, optional, defaults to False) — Whether or not to return the prediction scores. Seescoresunder returned tensors for more details. -
return_dict_in_generate (
bool, optional, defaults to False) — Whether or not to return a ModelOutput instead of a plain tuple. -
synced_gpus (
bool, optional, defaults toFalse) — Whether to continue running the while loop until max_length (needed for ZeRO stage 3) model_kwargs — Additional model specific kwargs will be forwarded to theforwardfunction of the model. If model is an encoder-decoder model the kwargs should includeencoder_outputs.
Returns
SampleDecoderOnlyOutput,
SampleEncoderDecoderOutput or obj:torch.LongTensor: A
torch.LongTensor containing the generated tokens (default behaviour) or a
SampleDecoderOnlyOutput if
model.config.is_encoder_decoder=False and return_dict_in_generate=True or a
SampleEncoderDecoderOutput if
model.config.is_encoder_decoder=True.
Generates sequences for models with a language modeling head using multinomial sampling.
Examples:
>>> from transformers import (
... AutoTokenizer,
... AutoModelForCausalLM,
... LogitsProcessorList,
... MinLengthLogitsProcessor,
... TopKLogitsWarper,
... TemperatureLogitsWarper,
... )
>>> tokenizer = AutoTokenizer.from_pretrained("gpt2")
>>> model = AutoModelForCausalLM.from_pretrained("gpt2")
>>> # set pad_token_id to eos_token_id because GPT2 does not have a EOS token
>>> model.config.pad_token_id = model.config.eos_token_id
>>> input_prompt = "Today is a beautiful day, and"
>>> input_ids = tokenizer(input_prompt, return_tensors="pt").input_ids
>>> # instantiate logits processors
>>> logits_processor = LogitsProcessorList([
... MinLengthLogitsProcessor(15, eos_token_id=model.config.eos_token_id),
... ])
>>> # instantiate logits processors
>>> logits_warper = LogitsProcessorList([
... TopKLogitsWarper(50),
... TemperatureLogitsWarper(0.7),
... ])
>>> outputs = model.sample(input_ids, logits_processor=logits_processor, logits_warper=logits_warper)
>>> print("Generated:", tokenizer.batch_decode(outputs, skip_special_tokens=True))A class containing all of the functions supporting generation, to be used as a mixin in TFPreTrainedModel.
( logits cur_len max_length forced_bos_token_id forced_eos_token_id **kwargs )
Implement in subclasses of PreTrainedModel for custom behavior to adjust the logits in the generate method.
(
input_ids = None
max_length = None
min_length = None
do_sample = None
early_stopping = None
num_beams = None
temperature = None
top_k = None
top_p = None
repetition_penalty = None
bad_words_ids = None
bos_token_id = None
pad_token_id = None
eos_token_id = None
length_penalty = None
no_repeat_ngram_size = None
num_return_sequences = None
attention_mask = None
decoder_start_token_id = None
use_cache = None
output_scores = None
output_attentions = None
output_hidden_states = None
return_dict_in_generate = None
forced_bos_token_id = None
forced_eos_token_id = None
**model_kwargs
)
→
ModelOutput or tf.Tensor
Parameters
-
input_ids (
tf.Tensorofdtype=tf.int32and shape(batch_size, sequence_length), optional) — The sequence used as a prompt for the generation. IfNonethe method initializes it withbos_token_idand a batch size of 1. -
max_length (
int, optional, defaults to 20) — The maximum length of the sequence to be generated. -
min_length (
int, optional, defaults to 10) — The minimum length of the sequence to be generated. -
do_sample (
bool, optional, defaults toFalse) — Whether or not to use sampling ; use greedy decoding otherwise. -
early_stopping (
bool, optional, defaults toFalse) — Whether to stop the beam search when at leastnum_beamssentences are finished per batch or not. -
num_beams (
int, optional, defaults to 1) — Number of beams for beam search. 1 means no beam search. -
temperature (
float, optional, defaults to 1.0) — The value used to module the next token probabilities. -
top_k (
int, optional, defaults to 50) — The number of highest probability vocabulary tokens to keep for top-k-filtering. -
top_p (
float, optional, defaults to 1.0) — If set to float < 1, only the most probable tokens with probabilities that add up totop_por higher are kept for generation. -
repetition_penalty (
float, optional, defaults to 1.0) — The parameter for repetition penalty. 1.0 means no penalty. See this paper for more details. -
pad_token_id (
int, optional) — The id of the padding token. -
bos_token_id (
int, optional) — The id of the beginning-of-sequence token. -
eos_token_id (
int, optional) — The id of the end-of-sequence token. -
length_penalty (
float, optional, defaults to 1.0) — Exponential penalty to the length. 1.0 means no penalty.Set to values < 1.0 in order to encourage the model to generate shorter sequences, to a value > 1.0 in order to encourage the model to produce longer sequences.
-
no_repeat_ngram_size (
int, optional, defaults to 0) — If set to int > 0, all ngrams of that size can only occur once. -
bad_words_ids(
List[int], optional) — List of token ids that are not allowed to be generated. In order to get the tokens of the words that should not appear in the generated text, usetokenizer.encode(bad_word, add_prefix_space=True). -
num_return_sequences(
int, optional, defaults to 1) — The number of independently computed returned sequences for each element in the batch. -
attention_mask (
tf.Tensorofdtype=tf.int32and shape(batch_size, sequence_length), optional) — Mask to avoid performing attention on padding token indices. Mask values are in[0, 1], 1 for tokens that are not masked, and 0 for masked tokens.If not provided, will default to a tensor the same shape as
input_idsthat masks the pad token. -
decoder_start_token_id (
int, optional) — If an encoder-decoder model starts decoding with a different token than bos, the id of that token. usecache — (bool, _optional, defaults toTrue): Whether or not the model should use the past last key/values attentions (if applicable to the model) to speed up decoding. -
output_attentions (
bool, optional, defaults to False) — Whether or not to return the attentions tensors of all attention layers. Seeattentionsunder returned tensors for more details. -
output_hidden_states (
bool, optional, defaults to False) — Whether or not to return the hidden states of all layers. Seehidden_statesunder returned tensors for more details. -
output_scores (
bool, optional, defaults to False) — Whether or not to return the prediction scores. Seescoresunder returned tensors for more details. -
return_dict_in_generate (
bool, optional, defaults to False) — Whether or not to return a ModelOutput instead of a plain tuple. -
forced_bos_token_id (
int, optional) — The id of the token to force as the first generated token after thedecoder_start_token_id. Useful for multilingual models like mBART where the first generated token needs to be the target language token. -
forced_eos_token_id (
int, optional) — The id of the token to force as the last generated token whenmax_lengthis reached. model_specific_kwargs — Additional model specific kwargs will be forwarded to theforwardfunction of the model.
Returns
ModelOutput or tf.Tensor
A
ModelOutput (if return_dict_in_generate=True or when
config.return_dict_in_generate=True) or a tf.Tensor.
If the model is not an encoder-decoder model (model.config.is_encoder_decoder=False), the
possible ModelOutput types are:
TFGreedySearchDecoderOnlyOutput,TFSampleDecoderOnlyOutput,TFBeamSearchDecoderOnlyOutput,TFBeamSampleDecoderOnlyOutput
If the model is an encoder-decoder model (model.config.is_encoder_decoder=True), the possible
ModelOutput types are:
TFGreedySearchEncoderDecoderOutput,TFSampleEncoderDecoderOutput,TFBeamSearchEncoderDecoderOutput,TFBeamSampleEncoderDecoderOutput
Generates sequences for models with a language modeling head. The method currently supports greedy decoding, beam-search decoding, sampling with temperature, sampling with top-k or nucleus sampling.
Adapted in part from Facebook’s XLM beam search code.
Apart from input_ids and attention_mask, all the arguments below will default to the value of the
attribute of the same name inside the PretrainedConfig of the model. The default values
indicated are the default values of those config.
Most of these parameters are explained in more detail in this blog post.
Examples:
tokenizer = AutoTokenizer.from_pretrained('distilgpt2') # Initialize tokenizer
model = TFAutoModelWithLMHead.from_pretrained('distilgpt2') # Download model and configuration from huggingface.co and cache.
outputs = model.generate(max_length=40) # do greedy decoding
print(f'Generated: {tokenizer.decode(outputs[0], skip_special_tokens=True)}')
tokenizer = AutoTokenizer.from_pretrained('openai-gpt') # Initialize tokenizer
model = TFAutoModelWithLMHead.from_pretrained('openai-gpt') # Download model and configuration from huggingface.co and cache.
input_context = 'The dog'
input_ids = tokenizer.encode(input_context, return_tensors='tf') # encode input context
outputs = model.generate(input_ids=input_ids, num_beams=5, num_return_sequences=3, temperature=1.5) # generate 3 independent sequences using beam search decoding (5 beams) with sampling from initial context 'The dog'
for i in range(3): # 3 output sequences were generated
print(f'Generated {i}: {tokenizer.decode(outputs[i], skip_special_tokens=True)}')
tokenizer = AutoTokenizer.from_pretrained('distilgpt2') # Initialize tokenizer
model = TFAutoModelWithLMHead.from_pretrained('distilgpt2') # Download model and configuration from huggingface.co and cache.
input_context = 'The dog'
input_ids = tokenizer.encode(input_context, return_tensors='tf') # encode input context
outputs = model.generate(input_ids=input_ids, max_length=40, temperature=0.7, num_return_sequences=3, do_sample=True) # generate 3 candidates using sampling
for i in range(3): # 3 output sequences were generated
print(f'Generated {i}: {tokenizer.decode(outputs[i], skip_special_tokens=True)}')
tokenizer = AutoTokenizer.from_pretrained('ctrl') # Initialize tokenizer
model = TFAutoModelWithLMHead.from_pretrained('ctrl') # Download model and configuration from huggingface.co and cache.
input_context = 'Legal My neighbor is' # "Legal" is one of the control codes for ctrl
input_ids = tokenizer.encode(input_context, return_tensors='tf') # encode input context
outputs = model.generate(input_ids=input_ids, max_length=50, temperature=0.7, repetition_penalty=1.2) # generate sequences
print(f'Generated: {tokenizer.decode(outputs[0], skip_special_tokens=True)}')
tokenizer = AutoTokenizer.from_pretrained('gpt2') # Initialize tokenizer
model = TFAutoModelWithLMHead.from_pretrained('gpt2') # Download model and configuration from huggingface.co and cache.
input_context = 'My cute dog'
bad_words_ids = [tokenizer.encode(bad_word, add_prefix_space=True) for bad_word in ['idiot', 'stupid', 'shut up']]
input_ids = tokenizer.encode(input_context, return_tensors='tf') # encode input context
outputs = model.generate(input_ids=input_ids, max_length=100, do_sample=True, bad_words_ids=bad_words_ids) # generate sequences without allowing bad_words to be generatedImplement in subclasses of TFPreTrainedModel for custom behavior to prepare inputs in the generate method.
A class containing all of the functions supporting generation, to be used as a mixin in FlaxPreTrainedModel.
( input_ids: ndarray max_length: typing.Optional[int] = None pad_token_id: typing.Optional[int] = None bos_token_id: typing.Optional[int] = None eos_token_id: typing.Optional[int] = None decoder_start_token_id: typing.Optional[int] = None do_sample: typing.Optional[bool] = None prng_key: typing.Optional[jax._src.numpy.lax_numpy.ndarray] = None top_k: typing.Optional[int] = None top_p: typing.Optional[float] = None temperature: typing.Optional[float] = None num_beams: typing.Optional[int] = None no_repeat_ngram_size: typing.Optional[int] = None min_length: typing.Optional[int] = None forced_bos_token_id: typing.Optional[int] = None forced_eos_token_id: typing.Optional[int] = None length_penalty: typing.Optional[float] = None early_stopping: typing.Optional[bool] = None trace: bool = True params: typing.Union[typing.Dict[str, jax._src.numpy.lax_numpy.ndarray], NoneType] = None **model_kwargs )
Parameters
-
input_ids (
jnp.ndarrayof shape(batch_size, sequence_length)) — The sequence used as a prompt for the generation. -
max_length (
int, optional, defaults to 20) — The maximum length of the sequence to be generated. -
do_sample (
bool, optional, defaults toFalse) — Whether or not to use sampling ; use greedy decoding otherwise. -
temperature (
float, optional, defaults to 1.0) — The value used to module the next token probabilities. -
top_k (
int, optional, defaults to 50) — The number of highest probability vocabulary tokens to keep for top-k-filtering. -
top_p (
float, optional, defaults to 1.0) — If set to float < 1, only the most probable tokens with probabilities that add up totop_por higher are kept for generation. -
pad_token_id (
int, optional) — The id of the padding token. -
bos_token_id (
int, optional) — The id of the beginning-of-sequence token. -
eos_token_id (
int, optional) — The id of the end-of-sequence token. -
num_beams (
int, optional, defaults to 1) — Number of beams for beam search. 1 means no beam search. -
decoder_start_token_id (
int, optional) — If an encoder-decoder model starts decoding with a different token than bos, the id of that token. -
trace (
bool, optional, defaults toTrue) — Whether to trace generation. Settingtrace=Falseshould only be used for debugging and will lead to a considerably slower runtime. -
params (
Dict[str, jnp.ndarray], optional) — Optionally the model parameters can be passed. Can be useful for parallelized generation. model_kwargs — Additional model specific kwargs will be forwarded to theforwardfunction of the model.
Returns
Generates sequences for models with a language modeling head. The method currently supports greedy decoding, and, multinomial sampling.
Apart from input_ids, all the arguments below will default to the value of the attribute of the same
name inside the PretrainedConfig of the model. The default values indicated are the
default values of those config.
Most of these parameters are explained in more detail in this blog post.
Examples:
>>> from transformers import AutoTokenizer, FlaxAutoModelForCausalLM
>>> tokenizer = AutoTokenizer.from_pretrained("distilgpt2")
>>> model = FlaxAutoModelForCausalLM.from_pretrained("distilgpt2")
>>> input_context = "The dog"
>>> # encode input context
>>> input_ids = tokenizer(input_context, return_tensors="np").input_ids
>>> # generate candidates using sampling
>>> outputs = model.generate(input_ids=input_ids, max_length=20, top_k=30, do_sample=True)
>>> print("Generated:", tokenizer.batch_decode(outputs, skip_special_tokens=True))Pushing to the Hub
A Mixin containing the functionality to push a model or tokenizer to the hub.
(
repo_path_or_name: typing.Optional[str] = None
repo_url: typing.Optional[str] = None
use_temp_dir: bool = False
commit_message: typing.Optional[str] = None
organization: typing.Optional[str] = None
private: typing.Optional[bool] = None
use_auth_token: typing.Union[bool, str, NoneType] = None
**model_card_kwargs
)
→
str
Parameters
-
repo_path_or_name (
str, optional) — Can either be a repository name for your {object} in the Hub or a path to a local folder (in which case the repository will have the name of that local folder). If not specified, will default to the name given byrepo_urland a local directory with that name will be created. -
repo_url (
str, optional) — Specify this in case you want to push to an existing repository in the hub. If unspecified, a new repository will be created in your namespace (unless you specify anorganization) withrepo_name. -
use_temp_dir (
bool, optional, defaults toFalse) — Whether or not to clone the distant repo in a temporary directory or inrepo_path_or_nameinside the current working directory. This will slow things down if you are making changes in an existing repo since you will need to clone the repo before every push. -
commit_message (
str, optional) — Message to commit while pushing. Will default to"add {object}". -
organization (
str, optional) — Organization in which you want to push your {object} (you must be a member of this organization). -
private (
bool, optional) — Whether or not the repository created should be private (requires a paying subscription). -
use_auth_token (
boolorstr, optional) — The token to use as HTTP bearer authorization for remote files. IfTrue, will use the token generated when runningtransformers-cli login(stored in~/.huggingface). Will default toTrueifrepo_urlis not specified.
Returns
str
The url of the commit of your {object} in the given repository.
Upload the {object_files} to the 🤗 Model Hub while synchronizing a local clone of the repo in
repo_path_or_name.
Examples:
from transformers import {object_class}
{object} = {object_class}.from_pretrained("bert-base-cased")
# Push the {object} to your namespace with the name "my-finetuned-bert" and have a local clone in the
# _my-finetuned-bert_ folder.
{object}.push_to_hub("my-finetuned-bert")
# Push the {object} to your namespace with the name "my-finetuned-bert" with no local clone.
{object}.push_to_hub("my-finetuned-bert", use_temp_dir=True)
# Push the {object} to an organization with the name "my-finetuned-bert" and have a local clone in the
# _my-finetuned-bert_ folder.
{object}.push_to_hub("my-finetuned-bert", organization="huggingface")
# Make a change to an existing repo that has been cloned locally in _my-finetuned-bert_.
{object}.push_to_hub("my-finetuned-bert", repo_url="https://huggingface.co/sgugger/my-finetuned-bert")