Generation
Each framework has a generate method for text generation implemented in their respective GenerationMixin
class:
- PyTorch generate() is implemented in GenerationMixin.
- TensorFlow generate() is implemented in TFGenerationMixin.
- Flax/JAX generate() is implemented in FlaxGenerationMixin.
Regardless of your framework of choice, you can parameterize the generate method with a GenerationConfig class instance. Please refer to this class for the complete list of generation parameters, which control the behavior of the generation method.
To learn how to inspect a model’s generation configuration, what are the defaults, how to change the parameters ad hoc, and how to create and save a customized generation configuration, refer to the text generation strategies guide.
GenerationConfig
class transformers.GenerationConfig
< source >( **kwargs )
Parameters that control the length of the output
-
max_length (
int
, optional, defaults to 20) — The maximum length the generated tokens can have. Corresponds to the length of the input prompt +max_new_tokens
. In general, prefer the use ofmax_new_tokens
, which ignores the number of tokens in the prompt. -
max_new_tokens (
int
, optional) — The maximum numbers of tokens to generate, ignoring the number of tokens in the prompt. -
min_length (
int
, optional, defaults to 0) — The minimum length of the sequence to be generated. Corresponds to the length of the input prompt +min_new_tokens
. In general, prefer the use ofmin_new_tokens
, which ignores the number of tokens in the prompt. -
min_new_tokens (
int
, optional) — The minimum numbers of tokens to generate, ignoring the number of tokens in the prompt. -
early_stopping (
bool
, optional, defaults toFalse
) — Whether to stop the beam search when at leastnum_beams
sentences are finished per batch or not. -
max_time(
float
, optional) — The maximum amount of time you allow the computation to run for in seconds. generation will still finish the current pass after allocated time has been passed.
Parameters that control the generation strategy used
-
do_sample (
bool
, optional, defaults toFalse
) — Whether or not to use sampling ; use greedy decoding otherwise. -
num_beams (
int
, optional, defaults to 1) — Number of beams for beam search. 1 means no beam search. -
num_beam_groups (
int
, optional, defaults to 1) — Number of groups to dividenum_beams
into in order to ensure diversity among different groups of beams. this paper for more details. -
penalty_alpha (
float
, optional) — The values balance the model confidence and the degeneration penalty in contrastive search decoding. -
use_cache (
bool
, optional, defaults toTrue
) — Whether or not the model should use the past last key/values attentions (if applicable to the model) to speed up decoding.
Parameters for manipulation of the model output logits
-
temperature (
float
, optional, defaults to 1.0) — The value used to module the next token probabilities. -
top_k (
int
, optional, defaults to 50) — The number of highest probability vocabulary tokens to keep for top-k-filtering. -
top_p (
float
, optional, defaults to 1.0) — If set to float < 1, only the smallest set of most probable tokens with probabilities that add up totop_p
or higher are kept for generation. -
typical_p (
float
, optional, defaults to 1.0) — Local typicality measures how similar the conditional probability of predicting a target token next is to the expected conditional probability of predicting a random token next, given the partial text already generated. If set to float < 1, the smallest set of the most locally typical tokens with probabilities that add up totypical_p
or higher are kept for generation. See this paper for more details. -
epsilon_cutoff (
float
, optional, defaults to 0.0) — If set to float strictly between 0 and 1, only tokens with a conditional probability greater thanepsilon_cutoff
will be sampled. In the paper, suggested values range from 3e-4 to 9e-4, depending on the size of the model. See Truncation Sampling as Language Model Desmoothing for more details. -
eta_cutoff (
float
, optional, defaults to 0.0) — Eta sampling is a hybrid of locally typical sampling and epsilon sampling. If set to float strictly between 0 and 1, a token is only considered if it is greater than eithereta_cutoff
orsqrt(eta_cutoff) * exp(-entropy(softmax(next_token_logits)))
. The latter term is intuitively the expected next token probability, scaled bysqrt(eta_cutoff)
. In the paper, suggested values range from 3e-4 to 2e-3, depending on the size of the model. See Truncation Sampling as Language Model Desmoothing for more details. -
diversity_penalty (
float
, optional, defaults to 0.0) — This value is subtracted from a beam’s score if it generates a token same as any beam from other group at a particular time. Note thatdiversity_penalty
is only effective ifgroup beam search
is enabled. -
repetition_penalty (
float
, optional, defaults to 1.0) — The parameter for repetition penalty. 1.0 means no penalty. See this paper for more details. -
encoder_repetition_penalty (
float
, optional, defaults to 1.0) — The paramater for encoder_repetition_penalty. An exponential penalty on sequences that are not in the original input. 1.0 means no penalty. -
length_penalty (
float
, optional, defaults to 1.0) — Exponential penalty to the length that is used with beam-based generation. It is applied as an exponent to the sequence length, which in turn is used to divide the score of the sequence. Since the score is the log likelihood of the sequence (i.e. negative),length_penalty
> 0.0 promotes longer sequences, whilelength_penalty
< 0.0 encourages shorter sequences. -
no_repeat_ngram_size (
int
, optional, defaults to 0) — If set to int > 0, all ngrams of that size can only occur once. -
bad_words_ids(
List[List[int]]
, optional) — List of token ids that are not allowed to be generated. In order to get the token ids of the words that should not appear in the generated text, usetokenizer(bad_words, add_prefix_space=True, add_special_tokens=False).input_ids
. -
force_words_ids(
List[List[int]]
orList[List[List[int]]]
, optional) — List of token ids that must be generated. If given aList[List[int]]
, this is treated as a simple list of words that must be included, the opposite tobad_words_ids
. If givenList[List[List[int]]]
, this triggers a disjunctive constraint, where one can allow different forms of each word. -
renormalize_logits (
bool
, optional, defaults toFalse
) — Whether to renormalize the logits after applying all the logits processors or warpers (including the custom ones). It’s highly recommended to set this flag toTrue
as the search algorithms suppose the score logits are normalized but some logit processors or warpers break the normalization. -
constraints (
List[Constraint]
, optional) — Custom constraints that can be added to the generation to ensure that the output will contain the use of certain tokens as defined byConstraint
objects, in the most sensible way possible. -
forced_bos_token_id (
int
, optional, defaults tomodel.config.forced_bos_token_id
) — The id of the token to force as the first generated token after thedecoder_start_token_id
. Useful for multilingual models like mBART where the first generated token needs to be the target language token. -
forced_eos_token_id (
Union[int, List[int]]
, optional, defaults tomodel.config.forced_eos_token_id
) — The id of the token to force as the last generated token whenmax_length
is reached. Optionally, use a list to set multiple end-of-sequence tokens. -
remove_invalid_values (
bool
, optional, defaults tomodel.config.remove_invalid_values
) — Whether to remove possible nan and inf outputs of the model to prevent the generation method to crash. Note that usingremove_invalid_values
can slow down generation. -
exponential_decay_length_penalty (
tuple(int, float)
, optional) — This Tuple adds an exponentially increasing length penalty, after a certain amount of tokens have been generated. The tuple shall consist of:(start_index, decay_factor)
wherestart_index
indicates where penalty starts anddecay_factor
represents the factor of exponential decay -
suppress_tokens (
List[int]
, optional) — A list of tokens that will be suppressed at generation. TheSupressTokens
logit processor will set their log probs to-inf
so that they are not sampled. -
begin_suppress_tokens (
List[int]
, optional) — A list of tokens that will be suppressed at the beginning of the generation. TheSupressBeginTokens
logit processor will set their log probs to-inf
so that they are not sampled. -
forced_decoder_ids (
List[List[int]]
, optional) — A list of pairs of integers which indicates a mapping from generation indices to token indices that will be forced before sampling. For example,[[1, 123]]
means the second generated token will always be a token of index 123.
Parameters that define the output variables of `generate`
-
num_return_sequences(
int
, optional, defaults to 1) — The number of independently computed returned sequences for each element in the batch. -
output_attentions (
bool
, optional, defaults toFalse
) — Whether or not to return the attentions tensors of all attention layers. Seeattentions
under returned tensors for more details. - output_hidden_states (
bool
, optional, defaults toFalse
) — Whether or not to return the hidden states of all layers. Seehidden_states
under returned tensors for more details. -
output_scores (
bool
, optional, defaults toFalse
) — Whether or not to return the prediction scores. Seescores
under returned tensors for more details. -
return_dict_in_generate (
bool
, optional, defaults toFalse
) — Whether or not to return a ModelOutput instead of a plain tuple.
Special tokens that can be used at generation time
-
pad_token_id (
int
, optional) — The id of the padding token. -
bos_token_id (
int
, optional) — The id of the beginning-of-sequence token. -
eos_token_id (
Union[int, List[int]]
, optional) — The id of the end-of-sequence token. Optionally, use a list to set multiple end-of-sequence tokens.
Generation parameters exclusive to encoder-decoder models
-
encoder_no_repeat_ngram_size (
int
, optional, defaults to 0) — If set to int > 0, all ngrams of that size that occur in theencoder_input_ids
cannot occur in thedecoder_input_ids
. -
decoder_start_token_id (
int
, optional) — If an encoder-decoder model starts decoding with a different token than bos, the id of that token.
Wild card
Class that holds a configuration for a generation task. A generate
call supports the following generation methods
for text-decoder, text-to-text, speech-to-text, and vision-to-text models:
- greedy decoding by calling greedy_search() if
num_beams=1
anddo_sample=False
- contrastive search by calling contrastive_search() if
penalty_alpha>0.
andtop_k>1
- multinomial sampling by calling sample() if
num_beams=1
anddo_sample=True
- beam-search decoding by calling beam_search() if
num_beams>1
anddo_sample=False
- beam-search multinomial sampling by calling beam_sample() if
num_beams>1
anddo_sample=True
- diverse beam-search decoding by calling group_beam_search(), if
num_beams>1
andnum_beam_groups>1
- constrained beam-search decoding by calling constrained_beam_search(), if
constraints!=None
orforce_words_ids!=None
You do not need to call any of the above methods directly. Pass custom parameter values to ‘generate’. To learn more about decoding strategies refer to the text generation strategies guide.
from_pretrained
< source >( pretrained_model_name: typing.Union[str, os.PathLike] config_file_name: typing.Union[str, os.PathLike, NoneType] = None **kwargs ) → GenerationConfig
Parameters
-
pretrained_model_name (
str
oros.PathLike
) — This can be either:- a string, the model id of a pretrained model configuration hosted inside a model repo on
huggingface.co. Valid model ids can be located at the root-level, like
bert-base-uncased
, or namespaced under a user or organization name, likedbmdz/bert-base-german-cased
. - a path to a directory containing a configuration file saved using the
save_pretrained() method, e.g.,
./my_model_directory/
.
- a string, the model id of a pretrained model configuration hosted inside a model repo on
huggingface.co. Valid model ids can be located at the root-level, like
-
config_file_name (
str
oros.PathLike
, optional, defaults to"generation_config.json"
) — Name of the generation configuration JSON file to be loaded frompretrained_model_name
. -
cache_dir (
str
oros.PathLike
, optional) — Path to a directory in which a downloaded pretrained model configuration should be cached if the standard cache should not be used. -
force_download (
bool
, optional, defaults toFalse
) — Whether or not to force to (re-)download the configuration files and override the cached versions if they exist. -
resume_download (
bool
, optional, defaults toFalse
) — Whether or not to delete incompletely received file. Attempts to resume the download if such a file exists. -
proxies (
Dict[str, str]
, optional) — A dictionary of proxy servers to use by protocol or endpoint, e.g.,{'http': 'foo.bar:3128', 'http://hostname': 'foo.bar:4012'}.
The proxies are used on each request. -
use_auth_token (
str
orbool
, optional) — The token to use as HTTP bearer authorization for remote files. IfTrue
, or not specified, will use the token generated when runninghuggingface-cli login
(stored in~/.huggingface
). -
revision (
str
, optional, defaults to"main"
) — The specific model version to use. It can be a branch name, a tag name, or a commit id, since we use a git-based system for storing models and other artifacts on huggingface.co, sorevision
can be any identifier allowed by git.To test a pull request you made on the Hub, you can pass `revision=“refs/pr/
“. -
return_unused_kwargs (
bool
, optional, defaults toFalse
) — IfFalse
, then this function returns just the final configuration object.If
True
, then this functions returns aTuple(config, unused_kwargs)
where unused_kwargs is a dictionary consisting of the key/value pairs whose keys are not configuration attributes: i.e., the part ofkwargs
which has not been used to updateconfig
and is otherwise ignored. -
subfolder (
str
, optional, defaults to""
) — In case the relevant files are located inside a subfolder of the model repo on huggingface.co, you can specify the folder name here. -
kwargs (
Dict[str, Any]
, optional) — The values in kwargs of any keys which are configuration attributes will be used to override the loaded values. Behavior concerning key/value pairs whose keys are not configuration attributes is controlled by thereturn_unused_kwargs
keyword parameter.
Returns
The configuration object instantiated from this pretrained model.
Instantiate a GenerationConfig from a generation configuration file.
Examples:
>>> from transformers import GenerationConfig
>>> # Download configuration from huggingface.co and cache.
>>> generation_config = GenerationConfig.from_pretrained("gpt2")
>>> # E.g. config was saved using *save_pretrained('./test/saved_model/')*
>>> generation_config.save_pretrained("./test/saved_model/")
>>> generation_config = GenerationConfig.from_pretrained("./test/saved_model/")
>>> # You can also specify configuration names to your generation configuration file
>>> generation_config.save_pretrained("./test/saved_model/", config_file_name="my_configuration.json")
>>> generation_config = GenerationConfig.from_pretrained("./test/saved_model/", "my_configuration.json")
>>> # If you'd like to try a minor variation to an existing configuration, you can also pass generation
>>> # arguments to `.from_pretrained()`. Be mindful that typos and unused arguments will be ignored
>>> generation_config, unused_kwargs = GenerationConfig.from_pretrained(
... "gpt2", top_k=1, foo=False, return_unused_kwargs=True
... )
>>> generation_config.top_k
1
>>> unused_kwargs
{'foo': False}
from_model_config
< source >( model_config: PretrainedConfig ) → GenerationConfig
Instantiates a GenerationConfig from a PretrainedConfig. This function is useful to convert legacy PretrainedConfig objects, which may contain generation parameters, into a stand-alone GenerationConfig.
save_pretrained
< source >( save_directory: typing.Union[str, os.PathLike] config_file_name: typing.Union[str, os.PathLike, NoneType] = None push_to_hub: bool = False **kwargs )
Parameters
-
save_directory (
str
oros.PathLike
) — Directory where the configuration JSON file will be saved (will be created if it does not exist). -
config_file_name (
str
oros.PathLike
, optional, defaults to"generation_config.json"
) — Name of the generation configuration JSON file to be saved insave_directory
. -
push_to_hub (
bool
, optional, defaults toFalse
) — Whether or not to push your model to the Hugging Face model hub after saving it. You can specify the repository you want to push to withrepo_id
(will default to the name ofsave_directory
in your namespace). kwargs — Additional key word arguments passed along to the push_to_hub() method.
Save a generation configuration object to the directory save_directory
, so that it can be re-loaded using the
from_pretrained() class method.
GenerationMixin
A class containing all functions for auto-regressive text generation, to be used as a mixin in PreTrainedModel.
The class exposes generate(), which can be used for:
- greedy decoding by calling greedy_search() if
num_beams=1
anddo_sample=False
- contrastive search by calling contrastive_search() if
penalty_alpha>0
andtop_k>1
- multinomial sampling by calling sample() if
num_beams=1
anddo_sample=True
- beam-search decoding by calling beam_search() if
num_beams>1
anddo_sample=False
- beam-search multinomial sampling by calling beam_sample() if
num_beams>1
anddo_sample=True
- diverse beam-search decoding by calling group_beam_search(), if
num_beams>1
andnum_beam_groups>1
- constrained beam-search decoding by calling constrained_beam_search(), if
constraints!=None
orforce_words_ids!=None
You do not need to call any of the above methods directly. Pass custom parameter values to ‘generate’ instead. To learn more about decoding strategies refer to the text generation strategies guide.
generate
< source >(
inputs: typing.Optional[torch.Tensor] = None
generation_config: typing.Optional[transformers.generation.configuration_utils.GenerationConfig] = None
logits_processor: typing.Optional[transformers.generation.logits_process.LogitsProcessorList] = None
stopping_criteria: typing.Optional[transformers.generation.stopping_criteria.StoppingCriteriaList] = None
prefix_allowed_tokens_fn: typing.Union[typing.Callable[[int, torch.Tensor], typing.List[int]], NoneType] = None
synced_gpus: typing.Optional[bool] = False
**kwargs
)
→
ModelOutput or torch.LongTensor
Parameters
-
inputs (
torch.Tensor
of varying shape depending on the modality, optional) — The sequence used as a prompt for the generation or as model inputs to the encoder. IfNone
the method initializes it withbos_token_id
and a batch size of 1. For decoder-only modelsinputs
should of in the format ofinput_ids
. For encoder-decoder models inputs can represent any ofinput_ids
,input_values
,input_features
, orpixel_values
. -
generation_config (
~generation.GenerationConfig
, optional) — The generation configuration to be used as base parametrization for the generation call.**kwargs
passed to generate matching the attributes ofgeneration_config
will override them. Ifgeneration_config
is not provided, the default will be used, which had the following loading priority: 1) from thegeneration_config.json
model file, if it exists; 2) from the model configuration. Please note that unspecified parameters will inherit GenerationConfig’s default values, whose documentation should be checked to parameterize generation. -
logits_processor (
LogitsProcessorList
, optional) — Custom logits processors that complement the default logits processors built from arguments and generation config. If a logit processor is passed that is already created with the arguments or a generation config an error is thrown. This feature is intended for advanced users. -
stopping_criteria (
StoppingCriteriaList
, optional) — Custom stopping criteria that complement the default stopping criteria built from arguments and a generation config. If a stopping criteria is passed that is already created with the arguments or a generation config an error is thrown. This feature is intended for advanced users. -
prefix_allowed_tokens_fn (
Callable[[int, torch.Tensor], List[int]]
, optional) — If provided, this function constraints the beam search to allowed tokens only at each step. If not provided no constraint is applied. This function takes 2 arguments: the batch IDbatch_id
andinput_ids
. It has to return a list with the allowed tokens for the next generation step conditioned on the batch IDbatch_id
and the previously generated tokensinputs_ids
. This argument is useful for constrained generation conditioned on the prefix, as described in Autoregressive Entity Retrieval. -
synced_gpus (
bool
, optional, defaults toFalse
) — Whether to continue running the while loop until maxlength (needed for ZeRO stage 3) kwargs — Ad hoc parametrization ofgenerate_config
and/or additional model-specific kwargs that will be forwarded to theforward
function of the model. If the model is an encoder-decoder model, encoder specific kwargs should not be prefixed and decoder specific kwargs should be prefixed with *decoder*.
Returns
ModelOutput or torch.LongTensor
A ModelOutput (if return_dict_in_generate=True
or when config.return_dict_in_generate=True
) or a torch.FloatTensor
.
If the model is not an encoder-decoder model (model.config.is_encoder_decoder=False
), the possible
ModelOutput types are:
- GreedySearchDecoderOnlyOutput,
- SampleDecoderOnlyOutput,
- BeamSearchDecoderOnlyOutput,
- BeamSampleDecoderOnlyOutput
If the model is an encoder-decoder model (model.config.is_encoder_decoder=True
), the possible
ModelOutput types are:
Generates sequences of token ids for models with a language modeling head.
Most generation-controlling parameters are set in generation_config
which, if not passed, will be set to the
model’s default generation configuration. You can override any generation_config
by passing the corresponding
parameters to generate(), e.g. .generate(inputs, num_beams=4, do_sample=True)
.
For an overview of generation strategies and code examples, check out the following guide.
compute_transition_scores
< source >(
sequences: Tensor
scores: typing.Tuple[torch.Tensor]
beam_indices: typing.Optional[torch.Tensor] = None
normalize_logits: bool = False
)
→
torch.Tensor
Parameters
-
sequences (
torch.LongTensor
) — The generated sequences. The second dimension (sequence_length) is either equal tomax_length
or shorter if all batches finished early due to theeos_token_id
. -
scores (
tuple(torch.FloatTensor)
) — Transition scores for each vocabulary token at each generation step. Beam transition scores consisting of log probabilities of tokens conditioned on log softmax of previously generated tokens Tuple oftorch.FloatTensor
with up tomax_new_tokens
elements (one element for each generated token), with each tensor of shape(batch_size*num_beams, config.vocab_size)
. -
beam_indices (
tuple(tuple(torch.LongTensor))
, optional) — Beam indices of generated token id at each generation step.torch.LongTensor
of shape(batch_size*num_return_sequences, input_ids.shape[-1])
. Only required if anum_beams>1
at generate-time. -
normalize_logits (
bool
, optional, defaults toFalse
) — Whether to normalize the logits (which, for legacy reasons, may be unnormalized).
Returns
torch.Tensor
A torch.Tensor
of shape (batch_size*num_return_sequences, sequence_length)
containing
the transition scores (logits)
Computes the transition scores of sequences given the generation scores (and beam indices, if beam search was used). This is a convenient method to quicky obtain the scores of the selected tokens at generation time.
Examples:
>>> from transformers import GPT2Tokenizer, AutoModelForCausalLM
>>> import numpy as np
>>> tokenizer = GPT2Tokenizer.from_pretrained("gpt2")
>>> model = AutoModelForCausalLM.from_pretrained("gpt2")
>>> tokenizer.pad_token_id = tokenizer.eos_token_id
>>> inputs = tokenizer(["Today is"], return_tensors="pt")
>>> # Example 1: Print the scores for each token generated with Greedy Search
>>> outputs = model.generate(**inputs, max_new_tokens=5, return_dict_in_generate=True, output_scores=True)
>>> transition_scores = model.compute_transition_scores(
... outputs.sequences, outputs.scores, normalize_logits=True
... )
>>> input_length = inputs.input_ids.shape[1]
>>> generated_tokens = outputs.sequences[:, input_length:]
>>> for tok, score in zip(generated_tokens[0], transition_scores[0]):
... # | token | token string | logits | probability
... print(f"| {tok:5d} | {tokenizer.decode(tok):8s} | {score.numpy():.3f} | {np.exp(score.numpy()):.2%}")
| 262 | the | -1.414 | 24.33%
| 1110 | day | -2.609 | 7.36%
| 618 | when | -2.010 | 13.40%
| 356 | we | -1.859 | 15.58%
| 460 | can | -2.508 | 8.14%
>>> # Example 2: Reconstruct the sequence scores from Beam Search
>>> outputs = model.generate(
... **inputs,
... max_new_tokens=5,
... num_beams=4,
... num_return_sequences=4,
... return_dict_in_generate=True,
... output_scores=True,
... )
>>> transition_scores = model.compute_transition_scores(
... outputs.sequences, outputs.scores, outputs.beam_indices, normalize_logits=False
... )
>>> # If you sum the generated tokens' scores and apply the length penalty, you'll get the sequence scores.
>>> # Tip: set `normalize_logits=True` to recompute the scores from the normalized logits.
>>> output_length = inputs.input_ids.shape[1] + np.sum(transition_scores.numpy() < 0, axis=1)
>>> length_penalty = model.generation_config.length_penalty
>>> reconstructed_scores = transition_scores.sum(axis=1) / (output_length**length_penalty)
>>> print(np.allclose(outputs.sequences_scores, reconstructed_scores))
True
greedy_search
< source >( input_ids: LongTensor logits_processor: typing.Optional[transformers.generation.logits_process.LogitsProcessorList] = None stopping_criteria: typing.Optional[transformers.generation.stopping_criteria.StoppingCriteriaList] = None max_length: typing.Optional[int] = None pad_token_id: typing.Optional[int] = None eos_token_id: typing.Union[int, typing.List[int], NoneType] = None output_attentions: typing.Optional[bool] = None output_hidden_states: typing.Optional[bool] = None output_scores: typing.Optional[bool] = None return_dict_in_generate: typing.Optional[bool] = None synced_gpus: typing.Optional[bool] = False **model_kwargs )
Parameters
-
input_ids (
torch.LongTensor
of shape(batch_size, sequence_length)
) — The sequence used as a prompt for the generation. -
logits_processor (
LogitsProcessorList
, optional) — An instance of LogitsProcessorList. List of instances of class derived from LogitsProcessor used to modify the prediction scores of the language modeling head applied at each generation step. -
stopping_criteria (
StoppingCriteriaList
, optional) — An instance of StoppingCriteriaList. List of instances of class derived from StoppingCriteria used to tell if the generation loop should stop. -
max_length (
int
, optional, defaults to 20) — DEPRECATED. Uselogits_processor
orstopping_criteria
directly to cap the number of generated tokens. The maximum length of the sequence to be generated. -
pad_token_id (
int
, optional) — The id of the padding token. -
eos_token_id (
int
, optional) — The id of the end-of-sequence token. -
output_attentions (
bool
, optional, defaults toFalse
) — Whether or not to return the attentions tensors of all attention layers. Seeattentions
under returned tensors for more details. - output_hidden_states (
bool
, optional, defaults toFalse
) — Whether or not to return the hidden states of all layers. Seehidden_states
under returned tensors for more details. -
output_scores (
bool
, optional, defaults toFalse
) — Whether or not to return the prediction scores. Seescores
under returned tensors for more details. -
return_dict_in_generate (
bool
, optional, defaults toFalse
) — Whether or not to return a ModelOutput instead of a plain tuple. -
synced_gpus (
bool
, optional, defaults toFalse
) — Whether to continue running the while loop until max_length (needed for ZeRO stage 3) model_kwargs — Additional model specific keyword arguments will be forwarded to theforward
function of the model. If model is an encoder-decoder model the kwargs should includeencoder_outputs
.
Generates sequences of token ids for models with a language modeling head using greedy decoding and can be used for text-decoder, text-to-text, speech-to-text, and vision-to-text models.
In most cases, you do not need to call greedy_search() directly. Use generate() instead. For an overview of generation strategies and code examples, check the following guide.
Examples:
>>> from transformers import (
... AutoTokenizer,
... AutoModelForCausalLM,
... LogitsProcessorList,
... MinLengthLogitsProcessor,
... StoppingCriteriaList,
... MaxLengthCriteria,
... )
>>> tokenizer = AutoTokenizer.from_pretrained("gpt2")
>>> model = AutoModelForCausalLM.from_pretrained("gpt2")
>>> # set pad_token_id to eos_token_id because GPT2 does not have a PAD token
>>> model.generation_config.pad_token_id = model.generation_config.eos_token_id
>>> input_prompt = "It might be possible to"
>>> input_ids = tokenizer(input_prompt, return_tensors="pt").input_ids
>>> # instantiate logits processors
>>> logits_processor = LogitsProcessorList(
... [
... MinLengthLogitsProcessor(10, eos_token_id=model.generation_config.eos_token_id),
... ]
... )
>>> stopping_criteria = StoppingCriteriaList([MaxLengthCriteria(max_length=20)])
>>> outputs = model.greedy_search(
... input_ids, logits_processor=logits_processor, stopping_criteria=stopping_criteria
... )
>>> tokenizer.batch_decode(outputs, skip_special_tokens=True)
["It might be possible to get a better understanding of the nature of the problem, but it's not"]
sample
< source >(
input_ids: LongTensor
logits_processor: typing.Optional[transformers.generation.logits_process.LogitsProcessorList] = None
stopping_criteria: typing.Optional[transformers.generation.stopping_criteria.StoppingCriteriaList] = None
logits_warper: typing.Optional[transformers.generation.logits_process.LogitsProcessorList] = None
max_length: typing.Optional[int] = None
pad_token_id: typing.Optional[int] = None
eos_token_id: typing.Union[int, typing.List[int], NoneType] = None
output_attentions: typing.Optional[bool] = None
output_hidden_states: typing.Optional[bool] = None
output_scores: typing.Optional[bool] = None
return_dict_in_generate: typing.Optional[bool] = None
synced_gpus: typing.Optional[bool] = False
**model_kwargs
)
→
SampleDecoderOnlyOutput, SampleEncoderDecoderOutput or torch.LongTensor
Parameters
-
input_ids (
torch.LongTensor
of shape(batch_size, sequence_length)
) — The sequence used as a prompt for the generation. -
logits_processor (
LogitsProcessorList
, optional) — An instance of LogitsProcessorList. List of instances of class derived from LogitsProcessor used to modify the prediction scores of the language modeling head applied at each generation step. -
stopping_criteria (
StoppingCriteriaList
, optional) — An instance of StoppingCriteriaList. List of instances of class derived from StoppingCriteria used to tell if the generation loop should stop. -
logits_warper (
LogitsProcessorList
, optional) — An instance of LogitsProcessorList. List of instances of class derived from LogitsWarper used to warp the prediction score distribution of the language modeling head applied before multinomial sampling at each generation step. -
max_length (
int
, optional, defaults to 20) — DEPRECATED. Uselogits_processor
orstopping_criteria
directly to cap the number of generated tokens. The maximum length of the sequence to be generated. -
pad_token_id (
int
, optional) — The id of the padding token. -
eos_token_id (
int
, optional) — The id of the end-of-sequence token. -
output_attentions (
bool
, optional, defaults toFalse
) — Whether or not to return the attentions tensors of all attention layers. Seeattentions
under returned tensors for more details. - output_hidden_states (
bool
, optional, defaults toFalse
) — Whether or not to return the hidden states of all layers. Seehidden_states
under returned tensors for more details. -
output_scores (
bool
, optional, defaults toFalse
) — Whether or not to return the prediction scores. Seescores
under returned tensors for more details. -
return_dict_in_generate (
bool
, optional, defaults toFalse
) — Whether or not to return a ModelOutput instead of a plain tuple. -
synced_gpus (
bool
, optional, defaults toFalse
) — Whether to continue running the while loop until max_length (needed for ZeRO stage 3) model_kwargs — Additional model specific kwargs will be forwarded to theforward
function of the model. If model is an encoder-decoder model the kwargs should includeencoder_outputs
.
Returns
SampleDecoderOnlyOutput, SampleEncoderDecoderOutput or torch.LongTensor
A torch.LongTensor
containing the generated tokens (default behaviour) or a
SampleDecoderOnlyOutput if model.config.is_encoder_decoder=False
and
return_dict_in_generate=True
or a SampleEncoderDecoderOutput if
model.config.is_encoder_decoder=True
.
Generates sequences of token ids for models with a language modeling head using multinomial sampling and can be used for text-decoder, text-to-text, speech-to-text, and vision-to-text models.
In most cases, you do not need to call sample() directly. Use generate() instead. For an overview of generation strategies and code examples, check the following guide.
Examples:
>>> from transformers import (
... AutoTokenizer,
... AutoModelForCausalLM,
... LogitsProcessorList,
... MinLengthLogitsProcessor,
... TopKLogitsWarper,
... TemperatureLogitsWarper,
... StoppingCriteriaList,
... MaxLengthCriteria,
... )
>>> import torch
>>> tokenizer = AutoTokenizer.from_pretrained("gpt2")
>>> model = AutoModelForCausalLM.from_pretrained("gpt2")
>>> # set pad_token_id to eos_token_id because GPT2 does not have a EOS token
>>> model.config.pad_token_id = model.config.eos_token_id
>>> model.generation_config.pad_token_id = model.config.eos_token_id
>>> input_prompt = "Today is a beautiful day, and"
>>> input_ids = tokenizer(input_prompt, return_tensors="pt").input_ids
>>> # instantiate logits processors
>>> logits_processor = LogitsProcessorList(
... [
... MinLengthLogitsProcessor(15, eos_token_id=model.generation_config.eos_token_id),
... ]
... )
>>> # instantiate logits processors
>>> logits_warper = LogitsProcessorList(
... [
... TopKLogitsWarper(50),
... TemperatureLogitsWarper(0.7),
... ]
... )
>>> stopping_criteria = StoppingCriteriaList([MaxLengthCriteria(max_length=20)])
>>> torch.manual_seed(0)
>>> outputs = model.sample(
... input_ids,
... logits_processor=logits_processor,
... logits_warper=logits_warper,
... stopping_criteria=stopping_criteria,
... )
>>> tokenizer.batch_decode(outputs, skip_special_tokens=True)
['Today is a beautiful day, and a wonderful day.\n\nI was lucky enough to meet the']
beam_search
< source >( input_ids: LongTensor beam_scorer: BeamScorer logits_processor: typing.Optional[transformers.generation.logits_process.LogitsProcessorList] = None stopping_criteria: typing.Optional[transformers.generation.stopping_criteria.StoppingCriteriaList] = None max_length: typing.Optional[int] = None pad_token_id: typing.Optional[int] = None eos_token_id: typing.Union[int, typing.List[int], NoneType] = None output_attentions: typing.Optional[bool] = None output_hidden_states: typing.Optional[bool] = None output_scores: typing.Optional[bool] = None return_dict_in_generate: typing.Optional[bool] = None synced_gpus: typing.Optional[bool] = False **model_kwargs )
Parameters
-
input_ids (
torch.LongTensor
of shape(batch_size, sequence_length)
) — The sequence used as a prompt for the generation. -
beam_scorer (
BeamScorer
) — An derived instance of BeamScorer that defines how beam hypotheses are constructed, stored and sorted during generation. For more information, the documentation of BeamScorer should be read. -
logits_processor (
LogitsProcessorList
, optional) — An instance of LogitsProcessorList. List of instances of class derived from LogitsProcessor used to modify the prediction scores of the language modeling head applied at each generation step. -
stopping_criteria (
StoppingCriteriaList
, optional) — An instance of StoppingCriteriaList. List of instances of class derived from StoppingCriteria used to tell if the generation loop should stop. -
max_length (
int
, optional, defaults to 20) — DEPRECATED. Uselogits_processor
orstopping_criteria
directly to cap the number of generated tokens. The maximum length of the sequence to be generated. -
pad_token_id (
int
, optional) — The id of the padding token. -
eos_token_id (
int
, optional) — The id of the end-of-sequence token. -
output_attentions (
bool
, optional, defaults toFalse
) — Whether or not to return the attentions tensors of all attention layers. Seeattentions
under returned tensors for more details. - output_hidden_states (
bool
, optional, defaults toFalse
) — Whether or not to return the hidden states of all layers. Seehidden_states
under returned tensors for more details. -
output_scores (
bool
, optional, defaults toFalse
) — Whether or not to return the prediction scores. Seescores
under returned tensors for more details. -
return_dict_in_generate (
bool
, optional, defaults toFalse
) — Whether or not to return a ModelOutput instead of a plain tuple. -
synced_gpus (
bool
, optional, defaults toFalse
) — Whether to continue running the while loop until max_length (needed for ZeRO stage 3) model_kwargs — Additional model specific kwargs will be forwarded to theforward
function of the model. If model is an encoder-decoder model the kwargs should includeencoder_outputs
.
Generates sequences of token ids for models with a language modeling head using beam search decoding and can be used for text-decoder, text-to-text, speech-to-text, and vision-to-text models.
In most cases, you do not need to call beam_search() directly. Use generate() instead. For an overview of generation strategies and code examples, check the following guide.
Examples:
>>> from transformers import (
... AutoTokenizer,
... AutoModelForSeq2SeqLM,
... LogitsProcessorList,
... MinLengthLogitsProcessor,
... BeamSearchScorer,
... )
>>> import torch
>>> tokenizer = AutoTokenizer.from_pretrained("t5-base")
>>> model = AutoModelForSeq2SeqLM.from_pretrained("t5-base")
>>> encoder_input_str = "translate English to German: How old are you?"
>>> encoder_input_ids = tokenizer(encoder_input_str, return_tensors="pt").input_ids
>>> # lets run beam search using 3 beams
>>> num_beams = 3
>>> # define decoder start token ids
>>> input_ids = torch.ones((num_beams, 1), device=model.device, dtype=torch.long)
>>> input_ids = input_ids * model.config.decoder_start_token_id
>>> # add encoder_outputs to model keyword arguments
>>> model_kwargs = {
... "encoder_outputs": model.get_encoder()(
... encoder_input_ids.repeat_interleave(num_beams, dim=0), return_dict=True
... )
... }
>>> # instantiate beam scorer
>>> beam_scorer = BeamSearchScorer(
... batch_size=1,
... num_beams=num_beams,
... device=model.device,
... )
>>> # instantiate logits processors
>>> logits_processor = LogitsProcessorList(
... [
... MinLengthLogitsProcessor(5, eos_token_id=model.config.eos_token_id),
... ]
... )
>>> outputs = model.beam_search(input_ids, beam_scorer, logits_processor=logits_processor, **model_kwargs)
>>> tokenizer.batch_decode(outputs, skip_special_tokens=True)
['Wie alt bist du?']
beam_sample
< source >( input_ids: LongTensor beam_scorer: BeamScorer logits_processor: typing.Optional[transformers.generation.logits_process.LogitsProcessorList] = None stopping_criteria: typing.Optional[transformers.generation.stopping_criteria.StoppingCriteriaList] = None logits_warper: typing.Optional[transformers.generation.logits_process.LogitsProcessorList] = None max_length: typing.Optional[int] = None pad_token_id: typing.Optional[int] = None eos_token_id: typing.Union[int, typing.List[int], NoneType] = None output_attentions: typing.Optional[bool] = None output_hidden_states: typing.Optional[bool] = None output_scores: typing.Optional[bool] = None return_dict_in_generate: typing.Optional[bool] = None synced_gpus: typing.Optional[bool] = False **model_kwargs )
Parameters
-
input_ids (
torch.LongTensor
of shape(batch_size, sequence_length)
) — The sequence used as a prompt for the generation. -
beam_scorer (
BeamScorer
) — A derived instance of BeamScorer that defines how beam hypotheses are constructed, stored and sorted during generation. For more information, the documentation of BeamScorer should be read. -
logits_processor (
LogitsProcessorList
, optional) — An instance of LogitsProcessorList. List of instances of class derived from LogitsProcessor used to modify the prediction scores of the language modeling head applied at each generation step. -
stopping_criteria (
StoppingCriteriaList
, optional) — An instance of StoppingCriteriaList. List of instances of class derived from StoppingCriteria used to tell if the generation loop should stop. -
logits_warper (
LogitsProcessorList
, optional) — An instance of LogitsProcessorList. List of instances of class derived from LogitsWarper used to warp the prediction score distribution of the language modeling head applied before multinomial sampling at each generation step. -
max_length (
int
, optional, defaults to 20) — DEPRECATED. Uselogits_processor
orstopping_criteria
directly to cap the number of generated tokens. The maximum length of the sequence to be generated. -
pad_token_id (
int
, optional) — The id of the padding token. -
eos_token_id (
int
, optional) — The id of the end-of-sequence token. -
output_attentions (
bool
, optional, defaults toFalse
) — Whether or not to return the attentions tensors of all attention layers. Seeattentions
under returned tensors for more details. - output_hidden_states (
bool
, optional, defaults toFalse
) — Whether or not to return the hidden states of all layers. Seehidden_states
under returned tensors for more details. -
output_scores (
bool
, optional, defaults toFalse
) — Whether or not to return the prediction scores. Seescores
under returned tensors for more details. -
return_dict_in_generate (
bool
, optional, defaults toFalse
) — Whether or not to return a ModelOutput instead of a plain tuple. -
synced_gpus (
bool
, optional, defaults toFalse
) — Whether to continue running the while loop until max_length (needed for ZeRO stage 3) model_kwargs — Additional model specific kwargs will be forwarded to theforward
function of the model. If model is an encoder-decoder model the kwargs should includeencoder_outputs
.
Generates sequences of token ids for models with a language modeling head using beam search multinomial sampling and can be used for text-decoder, text-to-text, speech-to-text, and vision-to-text models.
In most cases, you do not need to call beam_sample() directly. Use generate() instead. For an overview of generation strategies and code examples, check the following guide.
Examples:
>>> from transformers import (
... AutoTokenizer,
... AutoModelForSeq2SeqLM,
... LogitsProcessorList,
... MinLengthLogitsProcessor,
... TopKLogitsWarper,
... TemperatureLogitsWarper,
... BeamSearchScorer,
... )
>>> import torch
>>> tokenizer = AutoTokenizer.from_pretrained("t5-base")
>>> model = AutoModelForSeq2SeqLM.from_pretrained("t5-base")
>>> encoder_input_str = "translate English to German: How old are you?"
>>> encoder_input_ids = tokenizer(encoder_input_str, return_tensors="pt").input_ids
>>> # lets run beam search using 3 beams
>>> num_beams = 3
>>> # define decoder start token ids
>>> input_ids = torch.ones((num_beams, 1), device=model.device, dtype=torch.long)
>>> input_ids = input_ids * model.config.decoder_start_token_id
>>> # add encoder_outputs to model keyword arguments
>>> model_kwargs = {
... "encoder_outputs": model.get_encoder()(
... encoder_input_ids.repeat_interleave(num_beams, dim=0), return_dict=True
... )
... }
>>> # instantiate beam scorer
>>> beam_scorer = BeamSearchScorer(
... batch_size=1,
... max_length=model.config.max_length,
... num_beams=num_beams,
... device=model.device,
... )
>>> # instantiate logits processors
>>> logits_processor = LogitsProcessorList(
... [MinLengthLogitsProcessor(5, eos_token_id=model.config.eos_token_id)]
... )
>>> # instantiate logits processors
>>> logits_warper = LogitsProcessorList(
... [
... TopKLogitsWarper(50),
... TemperatureLogitsWarper(0.7),
... ]
... )
>>> outputs = model.beam_sample(
... input_ids, beam_scorer, logits_processor=logits_processor, logits_warper=logits_warper, **model_kwargs
... )
>>> tokenizer.batch_decode(outputs, skip_special_tokens=True)
['Wie alt bist du?']
contrastive_search
< source >( input_ids: LongTensor top_k: typing.Optional[int] = 1 penalty_alpha: typing.Optional[float] = 0 logits_processor: typing.Optional[transformers.generation.logits_process.LogitsProcessorList] = None logits_warper: typing.Optional[transformers.generation.logits_process.LogitsProcessorList] = None stopping_criteria: typing.Optional[transformers.generation.stopping_criteria.StoppingCriteriaList] = None pad_token_id: typing.Optional[int] = None eos_token_id: typing.Union[int, typing.List[int], NoneType] = None output_attentions: typing.Optional[bool] = None output_hidden_states: typing.Optional[bool] = None output_scores: typing.Optional[bool] = None return_dict_in_generate: typing.Optional[bool] = None synced_gpus: typing.Optional[bool] = False **model_kwargs )
Parameters
-
input_ids (
torch.LongTensor
of shape(batch_size, sequence_length)
) — The sequence used as a prompt for the generation. -
top_k (
int
, optional, defaults to 1) — The size of the candidate set that is used to re-rank for contrastive search -
penalty_alpha (
float
, optional, defaults to 0) — The degeneration penalty for contrastive search; activate when it is larger than 0 -
logits_processor (
LogitsProcessorList
, optional) — An instance of LogitsProcessorList. List of instances of class derived from LogitsProcessor used to modify the prediction scores of the language modeling head applied at each generation step. -
logits_warper (
LogitsProcessorList
, optional) — An instance of LogitsProcessorList. List of instances of class derived from LogitsWarper used to warp the prediction score distribution of the language modeling head applied before multinomial sampling at each generation step. -
stopping_criteria (
StoppingCriteriaList
, optional) — An instance of StoppingCriteriaList. List of instances of class derived from StoppingCriteria used to tell if the generation loop should stop. -
pad_token_id (
int
, optional) — The id of the padding token. -
eos_token_id (
int
, optional) — The id of the end-of-sequence token. -
output_attentions (
bool
, optional, defaults toFalse
) — Whether or not to return the attentions tensors of all attention layers. Seeattentions
under returned tensors for more details. - output_hidden_states (
bool
, optional, defaults toFalse
) — Whether or not to return the hidden states of all layers. Seehidden_states
under returned tensors for more details. -
output_scores (
bool
, optional, defaults toFalse
) — Whether or not to return the prediction scores. Seescores
under returned tensors for more details. -
return_dict_in_generate (
bool
, optional, defaults toFalse
) — Whether or not to return a ModelOutput instead of a plain tuple. -
synced_gpus (
bool
, optional, defaults toFalse
) — Whether to continue running the while loop until max_length (needed for ZeRO stage 3) model_kwargs — Additional model specific keyword arguments will be forwarded to theforward
function of the model. If model is an encoder-decoder model the kwargs should includeencoder_outputs
.
Generates sequences of token ids for models with a language modeling head using contrastive search and can be used for text-decoder, text-to-text, speech-to-text, and vision-to-text models.
In most cases, you do not need to call contrastive_search() directly. Use generate() instead. For an overview of generation strategies and code examples, check the following guide.
Examples:
>>> from transformers import (
... AutoTokenizer,
... AutoModelForCausalLM,
... StoppingCriteriaList,
... MaxLengthCriteria,
... )
>>> tokenizer = AutoTokenizer.from_pretrained("facebook/opt-125m")
>>> model = AutoModelForCausalLM.from_pretrained("facebook/opt-125m")
>>> # set pad_token_id to eos_token_id because OPT does not have a PAD token
>>> model.config.pad_token_id = model.config.eos_token_id
>>> input_prompt = "DeepMind Company is"
>>> input_ids = tokenizer(input_prompt, return_tensors="pt")
>>> stopping_criteria = StoppingCriteriaList([MaxLengthCriteria(max_length=64)])
>>> outputs = model.contrastive_search(
... **input_ids, penalty_alpha=0.6, top_k=4, stopping_criteria=stopping_criteria
... )
>>> tokenizer.batch_decode(outputs, skip_special_tokens=True)
['DeepMind Company is a company that focuses on the development and commercialization of artificial intelligence (AI). DeepMind’s mission is to help people understand and solve problems that are difficult to solve in the world today.\n\nIn this post, we talk about the benefits of deep learning in business and how it']
group_beam_search
< source >( input_ids: LongTensor beam_scorer: BeamScorer logits_processor: typing.Optional[transformers.generation.logits_process.LogitsProcessorList] = None stopping_criteria: typing.Optional[transformers.generation.stopping_criteria.StoppingCriteriaList] = None max_length: typing.Optional[int] = None pad_token_id: typing.Optional[int] = None eos_token_id: typing.Union[int, typing.List[int], NoneType] = None output_attentions: typing.Optional[bool] = None output_hidden_states: typing.Optional[bool] = None output_scores: typing.Optional[bool] = None return_dict_in_generate: typing.Optional[bool] = None synced_gpus: typing.Optional[bool] = False **model_kwargs )
Parameters
-
input_ids (
torch.LongTensor
of shape(batch_size, sequence_length)
) — The sequence used as a prompt for the generation. -
beam_scorer (
BeamScorer
) — An derived instance of BeamScorer that defines how beam hypotheses are constructed, stored and sorted during generation. For more information, the documentation of BeamScorer should be read. -
logits_processor (
LogitsProcessorList
, optional) — An instance of LogitsProcessorList. List of instances of class derived from LogitsProcessor used to modify the prediction scores of the language modeling head applied at each generation step. -
stopping_criteria (
StoppingCriteriaList
, optional) — An instance of StoppingCriteriaList. List of instances of class derived from StoppingCriteria used to tell if the generation loop should stop. -
max_length (
int
, optional, defaults to 20) — DEPRECATED. Uselogits_processor
orstopping_criteria
directly to cap the number of generated tokens. The maximum length of the sequence to be generated. -
pad_token_id (
int
, optional) — The id of the padding token. -
eos_token_id (
int
, optional) — The id of the end-of-sequence token. -
output_attentions (
bool
, optional, defaults toFalse
) — Whether or not to return the attentions tensors of all attention layers. Seeattentions
under returned tensors for more details. - output_hidden_states (
bool
, optional, defaults toFalse
) — Whether or not to return the hidden states of all layers. Seehidden_states
under returned tensors for more details. -
output_scores (
bool
, optional, defaults toFalse
) — Whether or not to return the prediction scores. Seescores
under returned tensors for more details. -
return_dict_in_generate (
bool
, optional, defaults toFalse
) — Whether or not to return a ModelOutput instead of a plain tuple. -
synced_gpus (
bool
, optional, defaults toFalse
) — Whether to continue running the while loop until max_length (needed for ZeRO stage 3)model_kwargs — Additional model specific kwargs that will be forwarded to the
forward
function of the model. If model is an encoder-decoder model the kwargs should includeencoder_outputs
.
Generates sequences of token ids for models with a language modeling head using diverse beam search decoding and can be used for text-decoder, text-to-text, speech-to-text, and vision-to-text models.
In most cases, you do not need to call group_beam_search() directly. Use generate() instead. For an overview of generation strategies and code examples, check the following guide.
Examples:
>>> from transformers import (
... AutoTokenizer,
... AutoModelForSeq2SeqLM,
... LogitsProcessorList,
... MinLengthLogitsProcessor,
... HammingDiversityLogitsProcessor,
... BeamSearchScorer,
... )
>>> import torch
>>> tokenizer = AutoTokenizer.from_pretrained("t5-base")
>>> model = AutoModelForSeq2SeqLM.from_pretrained("t5-base")
>>> encoder_input_str = "translate English to German: How old are you?"
>>> encoder_input_ids = tokenizer(encoder_input_str, return_tensors="pt").input_ids
>>> # lets run diverse beam search using 6 beams
>>> num_beams = 6
>>> # define decoder start token ids
>>> input_ids = torch.ones((num_beams, 1), device=model.device, dtype=torch.long)
>>> input_ids = input_ids * model.config.decoder_start_token_id
>>> # add encoder_outputs to model keyword arguments
>>> model_kwargs = {
... "encoder_outputs": model.get_encoder()(
... encoder_input_ids.repeat_interleave(num_beams, dim=0), return_dict=True
... )
... }
>>> # instantiate beam scorer
>>> beam_scorer = BeamSearchScorer(
... batch_size=1,
... max_length=model.config.max_length,
... num_beams=num_beams,
... device=model.device,
... num_beam_groups=3,
... )
>>> # instantiate logits processors
>>> logits_processor = LogitsProcessorList(
... [
... HammingDiversityLogitsProcessor(5.5, num_beams=6, num_beam_groups=3),
... MinLengthLogitsProcessor(5, eos_token_id=model.config.eos_token_id),
... ]
... )
>>> outputs = model.group_beam_search(
... input_ids, beam_scorer, logits_processor=logits_processor, **model_kwargs
... )
>>> tokenizer.batch_decode(outputs, skip_special_tokens=True)
['Wie alt bist du?']
constrained_beam_search
< source >( input_ids: LongTensor constrained_beam_scorer: ConstrainedBeamSearchScorer logits_processor: typing.Optional[transformers.generation.logits_process.LogitsProcessorList] = None stopping_criteria: typing.Optional[transformers.generation.stopping_criteria.StoppingCriteriaList] = None max_length: typing.Optional[int] = None pad_token_id: typing.Optional[int] = None eos_token_id: typing.Union[int, typing.List[int], NoneType] = None output_attentions: typing.Optional[bool] = None output_hidden_states: typing.Optional[bool] = None output_scores: typing.Optional[bool] = None return_dict_in_generate: typing.Optional[bool] = None synced_gpus: typing.Optional[bool] = None **model_kwargs )
Parameters
-
input_ids (
torch.LongTensor
of shape(batch_size, sequence_length)
) — The sequence used as a prompt for the generation. -
constrained_beam_scorer (
ConstrainedBeamSearchScorer
) — A derived instance of BeamScorer that defines how beam hypotheses are constructed, stored and sorted during generation, while satisfying a list of positive constraints. For more information, the documentation of ConstrainedBeamSearchScorer should be read. -
logits_processor (
LogitsProcessorList
, optional) — An instance of LogitsProcessorList. List of instances of class derived from LogitsProcessor used to modify the prediction scores of the language modeling head applied at each generation step. -
stopping_criteria (
StoppingCriteriaList
, optional) — An instance of StoppingCriteriaList. List of instances of class derived from StoppingCriteria used to tell if the generation loop should stop. -
logits_warper (
LogitsProcessorList
, optional) — An instance of LogitsProcessorList. List of instances of class derived from LogitsWarper used to warp the prediction score distribution of the language modeling head applied before multinomial sampling at each generation step. -
max_length (
int
, optional, defaults to 20) — DEPRECATED. Uselogits_processor
orstopping_criteria
directly to cap the number of generated tokens. The maximum length of the sequence to be generated. -
pad_token_id (
int
, optional) — The id of the padding token. -
eos_token_id (
int
, optional) — The id of the end-of-sequence token. -
output_attentions (
bool
, optional, defaults toFalse
) — Whether or not to return the attentions tensors of all attention layers. Seeattentions
under returned tensors for more details. - output_hidden_states (
bool
, optional, defaults toFalse
) — Whether or not to return the hidden states of all layers. Seehidden_states
under returned tensors for more details. -
output_scores (
bool
, optional, defaults toFalse
) — Whether or not to return the prediction scores. Seescores
under returned tensors for more details. -
return_dict_in_generate (
bool
, optional, defaults toFalse
) — Whether or not to return a ModelOutput instead of a plain tuple. -
synced_gpus (
bool
, optional, defaults toFalse
) — Whether to continue running the while loop until max_length (needed for ZeRO stage 3) model_kwargs — Additional model specific kwargs will be forwarded to theforward
function of the model. If model is an encoder-decoder model the kwargs should includeencoder_outputs
.
Generates sequences of token ids for models with a language modeling head using constrained beam search decoding and can be used for text-decoder, text-to-text, speech-to-text, and vision-to-text models.
In most cases, you do not need to call constrained_beam_search() directly. Use generate() instead. For an overview of generation strategies and code examples, check the following guide.
Examples:
>>> from transformers import (
... AutoTokenizer,
... AutoModelForSeq2SeqLM,
... LogitsProcessorList,
... MinLengthLogitsProcessor,
... ConstrainedBeamSearchScorer,
... PhrasalConstraint,
... )
>>> import torch
>>> tokenizer = AutoTokenizer.from_pretrained("t5-base")
>>> model = AutoModelForSeq2SeqLM.from_pretrained("t5-base")
>>> encoder_input_str = "translate English to German: How old are you?"
>>> encoder_input_ids = tokenizer(encoder_input_str, return_tensors="pt").input_ids
>>> # lets run beam search using 3 beams
>>> num_beams = 3
>>> # define decoder start token ids
>>> input_ids = torch.ones((num_beams, 1), device=model.device, dtype=torch.long)
>>> input_ids = input_ids * model.config.decoder_start_token_id
>>> # add encoder_outputs to model keyword arguments
>>> model_kwargs = {
... "encoder_outputs": model.get_encoder()(
... encoder_input_ids.repeat_interleave(num_beams, dim=0), return_dict=True
... )
... }
>>> constraint_str = "Sie"
>>> constraint_token_ids = tokenizer.encode(constraint_str)[:-1] # slice to remove eos token
>>> constraints = [PhrasalConstraint(token_ids=constraint_token_ids)]
>>> # instantiate beam scorer
>>> beam_scorer = ConstrainedBeamSearchScorer(
... batch_size=1, num_beams=num_beams, device=model.device, constraints=constraints
... )
>>> # instantiate logits processors
>>> logits_processor = LogitsProcessorList(
... [
... MinLengthLogitsProcessor(5, eos_token_id=model.config.eos_token_id),
... ]
... )
>>> outputs = model.constrained_beam_search(
... input_ids, beam_scorer, constraints=constraints, logits_processor=logits_processor, **model_kwargs
... )
>>> tokenizer.batch_decode(outputs, skip_special_tokens=True)
['Wie alt sind Sie?']
TFGenerationMixin
A class containing all of the functions supporting generation, to be used as a mixin in TFPreTrainedModel.
The class exposes generate(), which can be used for:
- greedy decoding by calling
greedy_search()
ifnum_beams=1
anddo_sample=False
- contrastive search by calling
contrastive_search()
ifpenalty_alpha>0
andtop_k>1
- multinomial sampling by calling
sample()
ifnum_beams=1
anddo_sample=True
- beam-search decoding by calling
beam_search()
ifnum_beams>1
You do not need to call any of the above methods directly. Pass custom parameter values to ‘generate’ instead. To learn more about decoding strategies refer to the text generation strategies guide.
generate
< source >(
input_ids: typing.Optional[tensorflow.python.framework.ops.Tensor] = None
generation_config: typing.Optional[transformers.generation.configuration_utils.GenerationConfig] = None
seed = None
**kwargs
)
→
ModelOutput or tf.Tensor
Parameters
-
input_ids (
tf.Tensor
ofdtype=tf.int32
and shape(batch_size, sequence_length)
, optional) — The sequence used as a prompt for the generation. IfNone
the method initializes it withbos_token_id
and a batch size of 1. -
generation_config (
~generation.GenerationConfig
, optional) — The generation configuration to be used as base parametrization for the generation call.**kwargs
passed to generate matching the attributes ofgeneration_config
will override them. Ifgeneration_config
is not provided, the default will be used, which had the following loading priority: 1) from thegeneration_config.json
model file, if it exists; 2) from the model configuration. Please note that unspecified parameters will inherit GenerationConfig’s default values, whose documentation should be checked to parameterize generation. -
seed (
List[int]
, optional) — Random seed to control sampling, containing two integers, used whendo_sample
isTrue
. See theseed
argument from stateless functions intf.random
. kwargs — Ad hoc parametrization ofgenerate_config
and/or additional model-specific kwargs that will be forwarded to theforward
function of the model. If the model is an encoder-decoder model, encoder specific kwargs should not be prefixed and decoder specific kwargs should be prefixed with decoder_.
Returns
ModelOutput or tf.Tensor
A ModelOutput (if return_dict_in_generate=True
or when
config.return_dict_in_generate=True
) or a tf.Tensor
.
If the model is not an encoder-decoder model (model.config.is_encoder_decoder=False
), the possible
ModelOutput types are:
TFGreedySearchDecoderOnlyOutput
,TFSampleDecoderOnlyOutput
,TFBeamSearchDecoderOnlyOutput
,TFBeamSampleDecoderOnlyOutput
If the model is an encoder-decoder model (model.config.is_encoder_decoder=True
), the possible
ModelOutput types are:
TFGreedySearchEncoderDecoderOutput
,TFSampleEncoderDecoderOutput
,TFBeamSearchEncoderDecoderOutput
,TFBeamSampleEncoderDecoderOutput
Generates sequences of token ids for models with a language modeling head.
Most generation-controlling parameters are set in generation_config
which, if not passed, will be set to the
model’s default generation configuration. You can override any generation_config
by passing the corresponding
parameters to generate, e.g. .generate(inputs, num_beams=4, do_sample=True)
.
For an overview of generation strategies and code examples, check out the following guide.
FlaxGenerationMixin
A class containing all functions for auto-regressive text generation, to be used as a mixin in FlaxPreTrainedModel.
The class exposes generate(), which can be used for:
- greedy decoding by calling
_greedy_search()
ifnum_beams=1
anddo_sample=False
- multinomial sampling by calling
_sample()
ifnum_beams=1
anddo_sample=True
- beam-search decoding by calling
_beam_search()
ifnum_beams>1
anddo_sample=False
You do not need to call any of the above methods directly. Pass custom parameter values to ‘generate’ instead. To learn more about decoding strategies refer to the text generation strategies guide.
generate
< source >( input_ids: ndarray generation_config: typing.Optional[transformers.generation.configuration_utils.GenerationConfig] = None prng_key: typing.Optional[jax._src.numpy.ndarray.ndarray] = None trace: bool = True params: typing.Union[typing.Dict[str, jax._src.numpy.ndarray.ndarray], NoneType] = None **kwargs )
Parameters
-
input_ids (
jnp.ndarray
of shape(batch_size, sequence_length)
) — The sequence used as a prompt for the generation. -
generation_config (
~generation.GenerationConfig
, optional) — The generation configuration to be used as base parametrization for the generation call.**kwargs
passed to generate matching the attributes ofgeneration_config
will override them. Ifgeneration_config
is not provided, the default will be used, which had the following loading priority: 1) from thegeneration_config.json
model file, if it exists; 2) from the model configuration. Please note that unspecified parameters will inherit GenerationConfig’s default values, whose documentation should be checked to parameterize generation. -
trace (
bool
, optional, defaults toTrue
) — Whether to trace generation. Settingtrace=False
should only be used for debugging and will lead to a considerably slower runtime. -
params (
Dict[str, jnp.ndarray]
, optional) — Optionally the model parameters can be passed. Can be useful for parallelized generation. kwargs — Ad hoc parametrization ofgenerate_config
and/or additional model-specific kwargs that will be forwarded to theforward
function of the model. If the model is an encoder-decoder model, encoder specific kwargs should not be prefixed and decoder specific kwargs should be prefixed with decoder_.
Generates sequences of token ids for models with a language modeling head.